docs(audit): SPIKE NAS verify mechanisms + squash matrix — Q1-Q5 answered, Route A accepted-with-caveats

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-11 08:20:00 +02:00
parent 8ca1272db6
commit b57f6c1509
3 changed files with 328 additions and 52 deletions
@@ -3,6 +3,18 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 - **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no
  blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES
  wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid
  1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats**
  (`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown
  EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash
  is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted
  INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**:
  dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll,
  ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production
  TASK is the operator side's next move.
 - **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo
  ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the
  demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0
@@ -2,59 +2,48 @@
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
-## TASK — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (hub v0.44.0) — 2026-07-10
+## SPIKE — NAS verify mechanisms + NFS squash matrix — 2026-07-11
-**LIVE on k3s (Synced/Healthy, image 0.44.0, startup logs `WG peer-sync enabled` + `PBS DR tenantsync
+**Class:** spike (findings doc is the only artifact; no production code touched).
-enabled`). NOT live-provisioned — no real customer was provisioned from this task; slice 2 (the agent
+**Record:** `documentation/audits/SPIKE-nas-verify-2026-07-11.md`.
-apply-bridge) + the supervised Peti enable are pending.** Builds on SPIKE-pbs-tier-provisioning
+**Baselines (read-only):** felhom-agent `300f06722b` (v0.80.0, `netmount.go` = recipe under test),
-(00afadc). Commits: ce6a566 (code) + 32ca1f6 (live fix + image bump).
+felhom-controller `3db9126121` (v0.112.0, the probe container), felhom.eu `8ca1272db6`.
-### What shipped
+### Verdicts (all five questions answered live, evidence in the record)
 - **ep0 surface (🛑 Viktor-approved, installed + smoked live):** `scripts/felhom-tenantsync.sh` v1.0.0
  as a SECOND forced-command key for `felhom-peersync` (own sudoers drop-in; **peersync script +
  sudoers sha256 asserted byte-identical before/after**). Ops: `provision` (existing token → typed
  `token_exists`; own-ns self-check with one regen retry then rollback), `reissue` (delete→recreate→
  re-grant — delete purges token ACLs, the spike fact), `fingerprint`. Secret hygiene: the token
  secret rides stdout ONLY (all tool stdout → stderr; hub error paths never embed stdout).
  Smoke (throwaway `spike-smoke`, torn down): all ops green; **live fix caught by the smoke** —
  `proxmox-backup-client --output-format json` wraps as `{"data":[…]}` → ns-exists check fixed to
  `(.data // .)` and reinstalled before the hub ever used the surface.
  **Orphan cleanup (spike flag #1):** `root@pam!spike` token + `/datastore/scratch` DatastoreAdmin
  ACL removed; ep0 now holds exactly the real `demo-felhom-01` tenancy.
 - **Hub `internal/tenantsync`:** wgsync-twin SSH client (pinned host key, constrained
  HostKeyAlgorithms, per-op JSON exec, typed `ErrTokenExists`). Contract test
  `TestErrors_NeverEmbedStdout` pins the no-stdout-in-errors rule.
 - **Provisioning flow (`web/pbsdr.go`):** config-form section "PBS DR tier (ep0)" → on enable+save
  (detached-ctx atom): WG peer existence check (the agent self-registers its key; absence
  fail-closes with "has not reported a WG key yet") → tenantsync provision → secret stored
  consume-once HOST-scoped (`host_pbs_secrets`, the `one_time_secrets` twin) → non-secret `pbs_dr`
  descriptor (`enabled, storage_id, pbs_tunnel_ip, datastore, namespace, token_id, fingerprint`)
  merged into the HOST `desired_json` + `SetHostDesired` generation bump. Already-provisioned
  re-save = success-no-op; disable = `enabled:false` descriptor, tenancy kept; **Re-issue PBS
  credentials** button = the F4 precedent (fresh consume-once secret + bump).
 - **Consume endpoint:** `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key self-scoped
  (global = operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the
  secret. *(Deviation: task spec wrote `/host/{id}/…`; implemented under `/hosts/` matching every
  other agent-facing route — slice 2 must use the `/hosts/` form.)*
 - **Deploy:** `TENANTSYNC_SSH_KEY_FILE` env + optional `Secret/tenantsync` mount (key generated on
  180, moved into the Secret, local copy shredded); same endpoint addr + pinned host key as peersync.
-### Tests + red-proofs (all run, mutations reverted, gate green)
+- **Q1 — in-guest automount trigger: PASS (make-or-break).** A `pct exec` ls AND a docker
- Happy path: descriptor lands in desired_json + gen bump exactly 1; secret consumable once and
+  bind-mount access from inside guest 9201 both wake the host systemd automount (~1 s); content
-  absent from ConfigJSON/desired-state/captured logs (grep-asserts).
+  correct, nfs4 propagates host→guest; idle-unmount + re-trigger proven. Q1c open item: whether a
- **Red-proof consume-once:** dropping the `consumed_at` UPDATE → store test re-served `"secret-1"`,
+  trigger installed BEFORE guest start carries through at boot needs a restart-window test (no
-  API test returned 200 with the secret again → both FAIL as designed.
+  restart permitted this spike).
- **Red-proof fail-closed:** swallowing the applyPBSDR error in handleConfigUpdate → all four
+- **Q2 — controller uid-1000 probe: PASS.** `docker exec -u 1000:1000` chain proof + `setpriv
-  fail-closed subtests FAIL with 303 (half-enabled save).
+  --reuid 1000` mechanism proof (same syscalls as `syscall.Credential`); CapEff `a80425fb` has
- **Red-proof idempotency:** dropping the already-provisioned short-circuit → re-save re-keyed the
+  SETUID/SETGID; server-side files land `101000:101000`.
-  token (2 provision calls), staged a fresh secret, bumped spuriously → FAIL.
+- **Q3 — Route A: ACCEPTED-with-caveats.** `all_squash` to an arbitrary server uid (1050) gives the
- Also covered: foreign-host 403 non-burning, reissue rotation (old secret dead), disable no-op on
+  guest-uid-1000 app full read/create/rename/**hardlink**/delete. Caveats: guest sees
-  second save, storage-id edit updates descriptor without re-provision, tenantsync client
+  `nobody:nogroup`; `chown` fails immediate-clean EPERM; `chmod` SUCCEEDS and persists server-side.
-  round-trip against an in-process SSH server incl. wrong-host-key refusal.
+  q3c pinned the June WRONG case: `anonuid` without `all_squash` never applies to non-root traffic —
- UI leg: render tests (`TestPBSDR_FormRendersState` + suite) — the hub UI is password-gated, so
+  wire uid 101000 fails on others-perms. Consumer guidance must require the "map/squash ALL users"
-  render tests are the verification method per repo convention.
+  mode, not just an anon-uid field.
 - **Q4 — error taxonomy + timing:** full table in the record. Headlines: NFSv4 returns the IDENTICAL
  `No such file or directory` for nonexistent vs not-permitted exports (merge those error
  categories); dead server = systemd `timeout` at **90 s** (systemd kills mount.nfs before its 2-min
  retry — no diagnostic string); **`retry=0` recommended for the PRODUCTION unit** (dead-server
  verify 90 s→3.1 s with clean `No route to host`; guest on-demand access 91 s→3.8 s ENODEV; happy
  path untouched; autofs retries fresh per access). SMB splits cleanly: `error(13)`=credentials,
  `error(2)`=share name. **Budget (Q4-vii):** verify cannot ride the agentapi global 15 s timeout
  (`client.go:87`) — worst case is systemd's 90 s; give AddNetStorage its own longer-lived path or
  async status-poll, ~120 s end-to-end, agent-side `systemctl start` bounded at ~95 s.
 - **Q5 — plain-user SMB: PASS.** Stock user + rw share, zero server-side uid config, all 8 probe
  steps pass INCLUDING hardlink (on Debian Samba/ext4 — re-verify on a real appliance); server-side
  ownership = the authenticated account; guest sees forced `1000:1000/0664`.
-### Operator follow-ups
+### Safety rails held
- Slice 2 (agent apply-bridge) then the supervised Peti enable (floor + agent update first).
+Isolated scratch root `/srv/nas-spike2/` on 180 (backups `.bak-nasspike2`); no iptables/network
- The demo host's missing §4b step-3 storage ACL grants (spike flag #2) — re-assert before slice 2
+manipulation (unreachable case = verified-unused 192.168.0.199); no guest restart; spike mounts only
-  codifies the self-grant.
+under `/mnt/felhom-drives/spike2-*`. **Teardown verified:** exports back to pre-count 2, smb.conf
 sections 6, spike users/scratch/units/creds/mountpoints all gone, 0 spike2 mounts on host and in
 guest, controller 0.112.0 healthy, spike-pulled alpine image removed.
 **Next:** the production "NAS verify-before-commit + UI redesign" TASK is written by the operator
 side on these findings (STOP honored — no production work started).
@@ -0,0 +1,275 @@
 # SPIKE — NAS verify mechanisms + NFS squash matrix, simulated on DooPlex
 **Date:** 2026-07-11
 **Class:** Spike (empirical validation; output = this doc only). Throwaway probes; no production code
 written or modified.
 **Goal:** validate the four unproven mechanisms the upcoming "NAS verify-before-commit + UI redesign"
 production TASK depends on, and pin down the June spike's `anonuid=1000` WRONG case. Q1 in-guest
 automount trigger; Q2 controller uid-1000 probe; Q3 NFS squash matrix (Route A decision); Q4 verify
 error taxonomy + timing; Q5 plain-user SMB.
 **Simulated NAS:** real nfs-kernel-server + Samba on **DooPlex 192.168.0.180**, isolated scratch root
 `/srv/nas-spike2/` (all exports/shares spike-named; live Longhorn exports + the six original Samba
 sections untouched, verified by count at teardown). Mount side: **felhom-pve** (192.168.0.162), all
 mounts under `/mnt/felhom-drives/spike2-*` via hand-written throwaway units carrying EXACTLY
 `netmount.go`'s option strings. Probe side: live demo guest **9201**, probes as guest uid 1000 via
 `docker exec -u 1000:1000 felhom-controller` (the production probe's exact vantage point).
 > **Sim-vs-real-NAS caveat (restated from June):** this validates the Linux *mechanism* against a
 > Debian NFS/Samba server. Synology/QNAP specifics — their `squash`/"Map all users" UIs, SMB dialect
 > ceilings, per-share ACL models, whether their SMB stack permits hardlinks — remain unvalidated until
 > the virtual-dsm confirmation pass.
 **Baselines (read-only anchors, verified live 2026-07-11):**
 | Repo | `main` @ commit | Version | Role |
 |------|-----------------|---------|------|
 | felhom-agent | `300f06722b` | v0.80.0 | `internal/storage/netmount.go` = the locked recipe under test |
 | felhom-controller | `3db9126121` | v0.112.0 | the container the Q2/Q3/Q5 probes ran in |
 | felhom.eu | `8ca1272db6` | — | this doc's home |
 **This doc's commit:** `PENDING` (inserted by the follow-up CHANGELOG commit).
 ---
 ## Verdict: **ALL FIVE QUESTIONS ANSWERED — no blocker for the production TASK**
 - **Q1 (make-or-break) PASS** — an in-guest access (bare `pct exec ls` AND a docker-container
  bind-mount access) wakes the **host** systemd automount; content appears correctly, never an empty
  dir or ENOENT.
 - **Q2 PASS** — the controller container (root, in-guest) can drop to uid/gid 1000 (`setpriv`, the
  same setresuid/setresgid syscalls `syscall.Credential` uses) and write through the share path;
  SETUID/SETGID capability bits confirmed present.
 - **Q3 — Route A ACCEPTED-with-caveats** — `all_squash` to an arbitrary server-local uid gives the
  guest-uid-1000 app full read+write+rename+**hardlink**+delete. Caveats: guest-visible ownership is
  `nobody:nogroup` (65534), `chown` fails (immediate clean EPERM, no hang), `chmod` **succeeds** (and
  persists server-side). The June WRONG case is pinned: `anonuid` **without** `all_squash` is a no-op
  for non-root traffic — the wire uid 101000 hits the export unmapped and fails on others-perms.
 - **Q4 taxonomy captured** — table below. Two headline facts: NFSv4 makes "export doesn't exist" and
  "export not permitted for this client" **indistinguishable** (identical error string), and the real
  ceiling on a dead-server mount is **systemd's 90 s start timeout**, not `mount.nfs`'s 2-min retry.
  `retry=0` turns the dead-server case into a **3.1 s clean "No route to host"** and an on-demand
  guest access from 91 s ENODEV into **3.8 s** — recommended for the PRODUCTION unit (evidence in
  Q4-vi).
 - **Q5 PASS** — a plain Samba user + rw share (zero server-side `force user`/uid config) works through
  the client-forced-uid chain, **including hardlinks** (on this Debian/ext4 Samba).
 ---
 ## Q1 — In-guest automount trigger: **PASS (both variants)**
 Setup: export `/srv/nas-spike2/q1` (`rw,all_squash,anonuid=101000,anongid=101000,no_subtree_check`,
 dir `101000:101000`), throwaway `.mount`+`.automount` pair at `/mnt/felhom-drives/spike2-q1` with the
 locked NFS option string (`vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev`), `TimeoutIdleSec=30`.
 Idle state confirmed by grep of `/proc/mounts` only (autofs, no nfs4) — the path was never
 stat'ed host-side.
 **Q1a — first access from inside the running guest:**
 ```
 # host /proc/mounts before: autofs trigger ONLY
 $ time pct exec 9201 -- ls -la /mnt/felhom-drives/spike2-q1
 -rw-r--r-- 1   1000    1000      30 ... MARKER.txt        ← marker listed, owned 1000:1000
 -rw-r--r-- 1   1000    1000 5242880 ... sample.bin
 real  0m1.053s   rc=0
 # host /proc/mounts after:  192.168.0.180:/srv/nas-spike2/q1 ... nfs4 ...soft,timeo=50,retrans=2...
 # guest /proc/mounts after: the SAME nfs4 entry (propagated) — plus the autofs trigger itself
 ```
 The in-guest access woke the host automount in ~1 s (including `pct exec` overhead). None of the
 WRONG outcomes occurred (no hang, no empty dir, no host-only mount).
 **Q1b — idle-unmount + container re-trigger:** after 45 s untouched, host was back to autofs-only
 (idle-unmount works). Then a container bind-mount access —
 `docker run --rm -v /mnt/felhom-drives/spike2-q1:/m alpine ls -la /m` — listed the marker and
 remounted the host nfs4 (3.2 s total, dominated by the one-time alpine image pull). So the
 docker `-v` bind of an idle automount path also triggers correctly — the exact media-container
 pattern.
 **Q1c — ordering observation (read-only, no restart performed):** `mp8: /mnt/felhom-drives` is a
 plain bind; host side is propagation `shared`, guest side `shared,slave`. The **autofs trigger mount
 itself is visible in the guest's `/proc/mounts`** (it propagated in live when the automount was
 enabled, guest running). What this spike could NOT prove without a guest restart: whether a trigger
 that exists BEFORE guest start is carried into the guest by the mp8 bind at boot (the June premise
 covered real mounts — felhom-flash/felhom-usb — not autofs triggers). **Open item: needs a
 restart-window test.** Until then the production task should treat "automount installed while guest
 runs" as the proven path (it is also the actual add-storage flow).
 ---
 ## Q2 — Controller drops to uid 1000 and writes: **PASS (both proofs)**
 **Q2a — chain proof (docker-level):**
 ```
 $ docker exec -u 1000:1000 felhom-controller /bin/sh -c 'id && echo probe > .../q2a && cat .../q2a && rm .../q2a'
 uid=1000 gid=1000 groups=1000
 probe
 Q2A_WRITE_READ_DELETE_OK
 # second write left in place; on 180: stat → 101000:101000 644  ← the +100000 recipe end-to-end
 ```
 **Q2b — mechanism proof (the `syscall.Credential` path):**
 ```
 CapEff: 00000000a80425fb    CapBnd: 00000000a80425fb   ← bits 6 (SETGID) + 7 (SETUID) present
 $ command -v setpriv → /usr/bin/setpriv               ← in the image (util-linux, debian bookworm-slim)
 $ docker exec felhom-controller setpriv --reuid 1000 --regid 1000 --clear-groups /bin/sh -c 'id && echo x > .../q2b && ...'
 uid=1000 gid=1000 groups=1000
 Q2B_SETPRIV_OK
 ```
 `setpriv` performs the same setresuid/setresgid syscalls `os/exec` + `SysProcAttr.Credential` will
 use — no EPERM anywhere, so the production probe shape (root controller spawning a uid-1000
 subprocess) is viable as designed. Bonus: `docker exec -u 1000:1000` (Q2a) also works, so a
 future probe could even avoid setuid entirely by exec-ing into itself via the docker socket —
 not needed, just noted.
 ---
 ## Q3 — NFS squash matrix: **Route A ACCEPTED-with-caveats; June WRONG case pinned**
 Three exports, each probed from the controller container as guest uid 1000
 (`docker exec -u 1000:1000 felhom-controller`), 8-step probe (read / create / mkdir+rename /
 **hardlink** / delete / stat / chown+chmod / server-side stat):
 | # | Export | Dir owner | S1 read | S2 create | S3 mkdir+mv | S4 hardlink | S5 delete | S6 stat (guest view) | S7 chown / chmod | server-side new file |
 |---|--------|-----------|---------|-----------|-------------|-------------|-----------|----------------------|------------------|----------------------|
 | a | `rw,all_squash,anonuid=101000,anongid=101000` | 101000:101000 | PASS | PASS | PASS | PASS (links=2) | PASS | `1000:1000` | OK / OK | `101000:101000 644` |
 | b | `rw,all_squash,anonuid=1050,anongid=1050` (throwaway user `spike2a`) | 1050:1050 | PASS | PASS | PASS | **PASS (links=2)** | PASS | `65534:65534` (nobody) | **EPERM (clean, ~7 ms total)** / **SUCCEEDS** | `1050:1050 644` |
 | c | `rw,anonuid=1000,anongid=1000` (NO all_squash) | 1000:1000 (755) | PASS (world-readable) | **FAIL** `Permission denied` | **FAIL** | — | — | `65534:65534` | EPERM / EPERM | none created |
 **Route A verdict (one-line decision input): ACCEPTED-with-caveats — `all_squash` to ANY
 server-local uid gives the guest-uid-1000 app full read+write+rename+hardlink+delete; no 101000
 needed anywhere on the NAS.** Exact caveat list:
 1. **Guest-visible ownership is `nobody:nogroup` (65534:65534)** — cosmetic for apps that just
   read/write, but apps that *check* ownership may complain.
 2. **`chown` fails** — immediate clean `Operation not permitted`, no hang (measured ~7 ms for the
   chown+chmod pair).
 3. **`chmod` SUCCEEDS** (refinement over the expected caveat): with `all_squash` every request runs
   as the anon identity, which owns every file on the share — so mode changes are permitted and
   **persist server-side** (the q3b marker really became `600` on the NAS). Apps that chmod won't
   error; operators should know modes are app-controlled.
 4. **Hardlink works** — the *arr atomic-move pattern is safe on Route A.
 **The June WRONG case, pinned (q3c):** `anonuid=1000` without `all_squash` squashes **root only**;
 the guest-uid-1000 app arrives on the wire as its host-mapped uid **101000, unmapped**, and is
 evaluated against the `1000:1000 755` dir's **others**-perms → read OK, every write
 `Permission denied`. So the June finding ("anonuid=1000 → not writable") was real but its cause is
 "anonuid without all_squash is a no-op for non-root traffic", NOT "uid 1000 is inherently wrong".
 With `all_squash` added, even `anonuid=1000` would behave like row (b). The production guidance can
 state this truthfully: **the consumer recipe requires the "map/squash ALL users" mode, not just an
 anonymous-uid field.**
 ---
 ## Q4 — Verify error taxonomy + timing (production verify inputs)
 Method: throwaway `.mount` unit per case, `time systemctl start <unit>` (the production verify
 mechanism), journal captured, unit stopped/reset between cases. NFS cases use the locked NFS option
 string; SMB cases the locked SMB string (creds file 0600 on the host).
 | # | Case | Duration | start rc | Unit result | Distinguishing journal/dmesg string (verbatim) |
 |---|------|----------|----------|-------------|-----------------------------------------------|
 | i | server unreachable (192.168.0.199, verified unused) | **90.1 s** | 1 | `timeout` | `Mounting timed out. Terminating.` + `Mount process exited, code=killed, status=15/TERM` (NO mount.nfs line — systemd kills it first) |
 | i′ | same, options + `retry=0` | **3.1 s** | 1 | `exit-code` (32) | `mount.nfs4: No route to host for 192.168.0.199:/srv/nope on /mnt/...` |
 | ii | reachable, nonexistent export | 0.081 s | 1 | `exit-code` (32) | `mount.nfs4: mounting 192.168.0.180:/srv/nas-spike2/nope failed, reason given by server: No such file or directory` |
 | iii | export exists, NOT permitted for this client (scoped to 192.168.0.1) | 0.080 s | 1 | `exit-code` (32) | **IDENTICAL to (ii)** — `...failed, reason given by server: No such file or directory` |
 | iv | SMB wrong password | 0.175 s | 1 | `exit-code` (32) | `mount error(13): Permission denied`; dmesg: `Status code returned 0xc000006d STATUS_LOGON_FAILURE` |
 | v | SMB wrong share name (valid creds) | 0.458 s | 1 | `exit-code` (32) | `mount error(2): No such file or directory`; dmesg: `CIFS: VFS: BAD_NETWORK_NAME: \\192.168.0.180\nope2` |
 Happy-path references measured the same day: NFS automount trigger <0.1 s (mount side), SMB plain
 mount 0.085 s.
 **Taxonomy consequences for the production verify's Hungarian error map:**
 - Cases (ii) and (iii) MUST be one merged category over NFSv4 ("a megosztás nem található, vagy a
  szerver nem engedélyezi ennek a gépnek") — the pseudo-fs hides non-permitted exports; do not promise
  the user a distinction the protocol cannot give.
 - The unreachable case is cleanly classifiable only with `retry=0` (`No route to host` / connection
  errors); with the default retry the verify sees a generic systemd `timeout` with no mount.nfs
  diagnostic at all.
 - SMB splits cleanly: `error(13)` = credentials, `error(2)` = share name, plus dmesg NT-status lines
  if ever needed.
 - All failures share `exit status 32` — classify on the journal string, not the exit code.
 **Q4-vi — the `retry=0` stance: put it in the PRODUCTION unit** (not only a verify-time transient
 unit). Evidence — a guest process accessing the **idle automount path while the NAS is down**:
 - without `retry=0`: blocks **91 s**, then `ls: cannot open directory ...: No such device` (ENODEV;
  systemd's 90 s start timeout is what ends it);
 - with `retry=0`: same clean ENODEV in **3.8 s**; an immediate second access re-attempts and fails
  in the same ~3.8 s (autofs retries fresh per access — no stuck state).
 `retry` only governs retrying a FAILED first attempt, so the happy path is untouched, and a briefly
 down NAS costs an app ~4 s per access instead of a 90 s wedge. There is no scenario in this model
 where the 2-min foreground retry helps: the automount re-triggers a fresh attempt on every access
 anyway. Caveat, stated honestly: 3.1–3.8 s is the same-LAN ARP-failure case; a black-holed-but-routed
 server (SYN drops) would sit in TCP connect until systemd's 90 s cap — `retry=0` narrows, but does
 not eliminate, the worst case.
 **Q4-vii — verify timeout budget recommendation:** with `retry=0` in the unit, the measured cases
 bound a LAN verify at **≤ ~5 s** (mount trigger + probe write), but the true worst case remains
 **systemd's 90 s** (black-holed server). The agentapi client's global 15 s `http.Client` Timeout
 (`client.go:87`) therefore **cannot carry the verify call**: a synchronous verify inside
 `POST /api/storage/netstorage/add` would time out client-side exactly in the cases verify exists to
 catch. Recommendation: the production task gives AddNetStorage its own longer-lived path — either a
 dedicated client/context with a **~120 s** end-to-end budget, or (better, matching the existing
 detached-format pattern) an async start + status-poll. Agent-side, bound the `systemctl start` step
 at ~95 s (just above systemd's cap) so the agent always returns a classified result rather than its
 own timeout.
 ---
 ## Q5 — SMB without server-side force user: **PASS (hardlink included)**
 Server: Samba share `[spike2-plain]`, `read only = no`, `valid users = spike2smb` (a plain throwaway
 user, uid 1051), **no `force user`/`force group`**, dir owned `spike2smb`. Client: the locked SMB
 option string verbatim (`vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid,
 file_mode=0664,dir_mode=0775,_netdev`). Mounted in 0.085 s; probe as guest uid 1000:
 | Step | Result |
 |------|--------|
 | S1 read | PASS — marker listed `1000:1000` (client-forced), content read |
 | S2 create + read-back | PASS |
 | S3 mkdir + rename | PASS |
 | S4 **hardlink** | **PASS** (`link-count=2`) — on THIS Samba/ext4; a NAS SMB stack may still refuse it, re-verify on the appliance |
 | S5 delete | PASS |
 | S6 stat | `1000:1000, mode 664` (uid/gid + file_mode client overrides, as designed) |
 | S7 chown/chmod | both "succeed" (client accepts them; guest view stays 1000:1000/0664; server mode of a touched file ended `644`/`744` — i.e. cosmetic client-side, not faithfully persisted) |
 | S8 server-side stat | `spike2smb:spike2smb (1051:1051)` — **writes land as the authenticated user, zero server-side uid config** |
 Consumer-SMB consequence: a stock NAS account with rw on the share is sufficient; ownership fidelity
 on the NAS is "everything belongs to the connecting account" — same shape as Route A on NFS.
 ---
 ## Teardown verification — DONE (all counts restored)
 - **180:** `/etc/exports` + `/etc/samba/smb.conf` restored from the `.bak-nasspike2` copies, baks
  deleted. Verified: **2** non-comment export lines (pre-spike count 2), **6** smb.conf `[...]`
  sections (pre-spike 6), `exportfs -v` shows **0** spike2 entries, users `spike2a`/`spike2smb`
  removed (`id` → no such user), `/srv/nas-spike2` gone. Longhorn exports, PBS, k3s, Gitea, the six
  Samba sections untouched.
 - **felhom-pve:** all spike2 units stopped (automounts first, then mounts), unit files removed,
  `daemon-reload` + `reset-failed`; `grep -c spike2 /proc/mounts` → **0**; `systemctl list-units
  '*spike2*' --all` → empty; `/mnt/felhom-drives` holds only `felhom-usb`; both SMB creds files
  removed; `/mnt/felhom-drives/spike2-*` mountpoint dirs removed.
 - **Guest 9201:** `grep -c spike2 /proc/mounts` → **0**; `felhom-controller:0.112.0` Up (healthy);
  the alpine image pulled during Q1b removed.
 - SMB passwords were throwaway, never committed; the creds files lived 0600 on felhom-pve for the
  spike's duration only.
 ---
 ## Observations (out of scope — documented, NOT acted on)
 1. **Guest `/proc/mounts` duplicates propagated entries** (each spike mount appeared twice in the
   guest) — an artifact of the mp8 bind topology; harmless, but a naive "grep -c" health check would
   double-count.
 2. The live NFS4 mount line shows **`fatal_neterrors=none`** — a newer kernel NFS option present on
   the PVE host's kernel; not in the locked recipe string, purely informational.
 3. `docker exec -u 1000:1000` into the production controller container works today (Q2a) — an
   alternative probe transport via the docker socket if the Credential path ever grew a problem.
 4. `useradd --system --uid 1050` warns (`uid greater than SYS_UID_MAX`) but works — irrelevant to
   findings.
 5. The Q4 error strings were produced by `mount.nfs4`/`mount.cifs` on PVE (util-linux/nfs-utils of
   Proxmox 8) — string-matching in the production verify should tolerate minor version drift
   (match on the stable substrings: `No such file or directory`, `No route to host`,
   `error(13)`, `error(2)`, `timed out`).
 6. Samba on 180 reloaded config via `smbcontrol all reload-config` without disturbing the six live
   shares — useful for the future virtual-dsm pass too.
 No secrets in this doc. All spike credentials were throwaway and are now deleted.