# AUDIT — NAS-backed CWA deploy: storage mismatch + "not a separate drive" RCA (2026-07-11) **Scope:** diagnostic only — no fixes applied, demo left as-found. Demo box felhom-pve (192.168.0.162), guest 9201, controller v0.116.1, agent v0.83.0, hub v0.46.0. NFS export `192.168.0.180:/mnt/5_hdd/felhom-data` (DooPlex). All times CEST unless marked UTC. --- ## TL;DR The re-add of `nas-media` at 14:16 **worked end-to-end and verified against the real NFS** — the new host automount propagated live into the running guest (shared→slave carries *new* mounts). CWA was deployed correctly against the NAS (`app.yaml` `HDD_PATH: /mnt/felhom-drives/nas-media`) and its library physically lives on the DooPlex export. What broke everything was a **second guest reboot at 15:39:11** (`pct reboot 9201`, root@pam): the fresh guest namespace received the `felhom-usb` ext4 submount but **not** the autofs trigger — since then the guest path is an empty, root-owned local stub. CWA has been **down since 15:39** (Docker could not recreate the bind source: `mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` — which, silver lining, also prevented a single byte from landing on the system drive). The UI showing "felhom-usb" is a pure display bug: the deployed-app storage select marks `selected` by `IsDefault` only, never by the stored `HDD_PATH`. The hub WARN was correct and was the **only** signal that fired. **Blast radius: zero data loss, zero data misplaced.** Library (434 KB metadata.db + 3 authors, 5.4 MB total) safe on the export. CWA down ~4 h, silently. --- ## 1. Timeline (reconstructed from host journal, docker inspect, file mtimes) | Time (CEST) | Event | Evidence | |---|---|---| | Jun 15 / Jul 6 | Old CWA cycle leftovers on felhom-usb (`userdata/import/calibre`, `backups/primary/calibre-web`) — unrelated to today | mtimes on felhom-usb | | Jul 11 morning | Q1c supervised test: guest reboot #1, NAS trigger lost, old share later removed | Q1c findings (prior session) | | 14:16:35 | `nas-media` re-added: agent `mkdir` + automount unit installed, agent verify OK ("netverify: mount verified") | host journal, agent log | | 14:16:37 | Registered in controller settings (`kind: network`, mapped uid/gid 1000) | settings.json | | 14:16:50 | **CWA deployed with `HDD_PATH: /mnt/felhom-drives/nas-media`** | app.yaml `deployed_at` | | 14:18–15:36 | In-guest controller (comm `felhom-controll`, cgroup `/lxc/9201/...docker...`) triggers the **host** automount every ~5 min — proof the trigger was live *inside the guest* and the periodic checks saw the real NFS | host journal ("Got automount request … triggered by … (felhom-controll)") | | 14:20–14:24 | Calibre library ingested (Pogátsa Zoltán, Robin Cook, Stephen King) onto the NFS | export file mtimes | | 15:05:47 | CWA container (re)start — the last running instance | docker inspect StartedAt | | **15:39:11** | **`pct reboot 9201` (root@pam) — reboot #2.** Automount requests from the guest stop dead at 15:36:54 and never resume | host journal UPID `vzreboot` | | 15:39:15 | CWA exits (128); Docker restart (`unless-stopped`) fails permanently: `error while creating mount source path '/mnt/felhom-drives/nas-media/userdata/media/books': mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` | docker inspect State | | 17:13:25 | Controller container recreated (cause unrecorded; RestartCount=0 → not a crash loop) | docker inspect | | ~19:40 | Audit evidence captured (below); marker test + host-retrigger probe run and cleaned up | this session | ## 2. Evidence ### 2.1 Mount reality (H1) — guest first, then host Guest 9201 (`pct exec`, before any host-side access this session): ``` /dev/mapper/pve-root /mnt/felhom-drives ext4 rw,relatime,errors=remount-ro 0 0 /dev/sdd1 /mnt/felhom-drives/felhom-usb ext4 rw,relatime 0 0 (no nfs4, no autofs anywhere) /mnt/felhom-drives/nas-media: empty dir, drwxr-xr-x nobody:nogroup, Device: 252,1 (pve-root), Birth == Modify == 2026-07-11 12:16:35 UTC ← never written since creation df: /dev/mapper/pve-root 98497780 blocks → the SYSTEM drive ``` Controller container (`docker exec felhom-controller`): identical — plain dir on pve-root. This is the namespace the add-time probe and the 5-min health checks run in. Host: ``` systemd-1 /mnt/felhom-drives/nas-media autofs rw,relatime,...,timeout=60,direct ... ← armed trigger mnt-felhom\x2ddrives-nas\x2dmedia.automount loaded active waiting mnt-felhom\x2ddrives-nas\x2dmedia.mount loaded inactive dead ← normal 60 s idle unmount ``` Propagation (mountinfo): host `/mnt/felhom-drives` = `shared:179`, autofs child = `shared:221`; guest bind = `shared:262 master:179` (a proper slave!), and the guest **did** receive the felhom-usb ext4 submount (`master:51`) — but not the autofs mount. Observed rule (matches Q1c): the guest-start bind replicates ext4 submounts but **not autofs direct mounts**; new host mounts created **while the guest runs** do propagate (proven at 14:16). Marker test: `RCA-host-marker` created in the export on DooPlex → **NOT visible in the guest**. Guest-side `touch` into the stub → `permission denied` (root-owned dir, unmapped in the userns) — the guest path is not just misdirected, it is completely unwritable. Markers cleaned up. ### 2.2 Where CWA's data actually is (H3 vs H4) - `app.yaml` (`/opt/docker/stacks/calibre-web/app.yaml`): **`HDD_PATH: /mnt/felhom-drives/nas-media`**, locked field, `deployed_at 2026-07-11T12:16:50Z`. - `docker inspect calibre-web` binds: `/mnt/felhom-drives/nas-media/userdata/media/books → /calibre-library` and `.../userdata/import/calibre → /cwa-book-ingest`. Config in a named volume. - DooPlex export: full CWA skeleton + **library** (`userdata/media/books/metadata.db` 434 KB, 3 author dirs, `.calnotes`), plus `backups/primary/calibre-web` manifest+compose. 5.4 MB total. `metadata.db-shm/-wal` left behind by the reboot kill (routine SQLite recovery on next open). - Guest stub: empty (mtime == birth). felhom-usb: only Jun 15 / Jul 6 leftovers from an earlier CWA cycle; `userdata/media/books` on felhom-usb is empty. ### 2.3 Observability shakedown (§3.3) The controller/agent capture rings could **not** answer H3 after the fact: - The controller container was recreated at 17:13 (in-memory ring + docker logs of the old container gone — the bootstrap deploy mechanism `docker run`s a fresh container). - Even live, the ring wraps in **~10 minutes** on an idle demo box: earliest entry 17:39:30 UTC at a 17:47 UTC pull; the buffer is dominated by 10-second `[stacks] Status refresh` + periodic `[agentapi] GET /disks` DEBUG spam. - Hub-side bundles (72 h custody) requested by the operator during the observability validation predate this incident's window; only a bundle pulled between 14:16 and 15:39 would hold the add and deploy flows. Worth a look on hub.felhom.eu, not counted on. H3 was answered from durable artifacts instead (app.yaml + docker inspect + export contents). The add-flow agent-side lines *were* recoverable — from the host **journal**, not the ring. ### 2.4 Code paths (all cited against v0.116.1 tree) - **Probe (H2):** `controller/internal/web/netprobe.go:32-47` (`NetProbeChild`) — write + readback + delete only; `netprobe_linux.go:21-43` re-execs as uid/gid 1000 **in the controller container's namespace**. There is **no fstype / statfs-magic / device-identity check** anywhere in the probe. Orchestrated from `netstorage_job.go:129-138`. - **Deploy persist (H3):** the dropdown submits `HDD_PATH` as the option `value` (the real path), saved to app.yaml as a locked field — verified correct on the box; no substitution point found or needed (live evidence already refutes H3). - **Deploy view render (H4):** `internal/web/templates/deploy.html:562-571` — the select renders `{{if .IsDefault}}selected{{end}}` per option and, for a deployed app, just adds `disabled` (line 563). **The stored `HDD_PATH` is never compared.** Handler side (`internal/web/handlers.go:330-342`) passes only the schedulable-path list; the app's current value isn't passed into the select at all. Live page confirms: *neither* option carries `selected` (no path currently has `is_default` in settings.json), so the browser displays the first option — felhom-usb. - **Hub warning (H5):** `internal/monitor/healthcheck.go:299-302` → `system.IsMountPoint` (`internal/system/mounts_linux.go:19-29`): `Stat_t.Dev` of path vs parent, evaluated in the controller container. With the stub both are 252:1 → WARN. **Correct positive.** - **The masking (why nothing else fired):** `internal/web/handlers.go:1383-1436` (`networkStorageWarnings`) decides the dashboard "NAS unreachable" badge by asking the **agent** (`ListNetStorage`) — and the agent's *host-namespace* view is genuinely healthy (`idle`). Nothing evaluates the share from the namespace the apps actually consume it in. ### 2.5 Propagation-repair probe (§3.5) Host-side access re-triggered the autofs fine (nfs4 mounted on the host) → guest still saw the stub. The NFS mounts as a **child of the host's autofs mount** (peer group `shared:221`), which has no counterpart in the guest — so re-triggering can never heal a guest that lost the trigger. What must be re-established is the **autofs trigger mount itself** in the guest namespace. Two facts from today constrain the fix: 1. A **newly created** host mount under `/mnt/felhom-drives` (`shared:179`) propagates into running guests' slave binds — proven by the 14:16 re-add healing the then-running guest. 2. Therefore **restarting the automount unit while guests run** (cheap: the NFS mount is idle-unmounted anyway) re-plants the trigger in every running guest in one shot. This is the natural `ReassertNetworkMounts` mechanism; it must fire after every guest start (and prudently verify guest-visibility afterwards rather than assume). ## 3. Hypothesis verdicts | # | Hypothesis | Verdict | |---|---|---| | H1 | Guest-side stub | **TRUE, refined.** The re-add itself propagated into the live guest and verified against the real NFS. The stub was created by the *second* reboot (15:39) — a fresh guest namespace does not inherit a pre-existing autofs trigger (Q1c mechanism re-confirmed), while it *does* inherit ext4 submounts. | | H2 | Probe hole (no fstype check) | **TRUE in code, did not fire here.** The probe verified the real NFS at add time. Today's stub would fail the probe anyway (`not_writable`, root-owned dir) — the hole only bites when a stub happens to be uid-1000-writable. The *operative* gap is broader: no guest/controller-namespace re-verification ever runs after add-time, and the periodic NAS health signal is measured host-side (see H5/masking). | | H3 | Deploy persisted wrong path | **FALSE.** app.yaml, docker binds and physical data all say nas-media. | | H4 | Display-resolution bug | **TRUE.** `deploy.html:567` selects by `IsDefault` only; stored `HDD_PATH` never consulted; with no default flagged, the browser shows the first option (felhom-usb). Display-only. | | H5 | Hub warning correct | **TRUE.** Device-ID check (healthcheck.go:299) correctly detected path-on-system-device from the controller namespace — the only guard that caught the state. Promote its mechanism to add/deploy-time refusal and to the NAS-unreachable badge. | | S-D | Garbled "TÁ*rhely" label | **REAL, data-level.** settings.json stores the felhom-usb label as `Tárhely (felhom-usb)` ("Tárhely" UTF-8 bytes re-decoded as CP1250) since its registration on **2026-07-08**; nas-media's label (2026-07-11) is clean. Current label-generation code (`settings.go:891-893`) is clean UTF-8 — one-time corruption in whatever wrote the 07-08 registration. Needs data repair + writer RCA. | ## 4. The four RCA questions 1. **Guest-side reality / did the re-add restore propagation?** The guest path is an empty, root-owned local directory on the system device (252:1) with no NFS or autofs anywhere in the guest. The re-add **did** restore propagation into the then-running guest (14:16→15:36 proven healthy); the 15:39 `pct reboot` severed it again. Fresh adds heal running guests; nothing heals a rebooted guest today. 2. **Where is CWA's data / H3 xor H4?** Physically on the DooPlex export (5.4 MB, intact incl. metadata.db); `app.yaml` says nas-media. **Display bug (H4)**, not a persistence bug. CWA itself has been down since 15:39 — the operator's "runs and looked like working" was the pre-reboot state. 3. **Why did the verify pipeline pass?** Legitimately — at add time the mount was real in the probing namespace. The design gaps: (a) the probe asserts writability, never network-filesystem identity (statfs magic / device ≠ rootfs-device); (b) verification runs once, at add time, in a state that a guest reboot silently invalidates; (c) ongoing NAS health is asked of the agent (host namespace), so the guest-side stub is invisible. The minimal add-time check that would refuse today's state: statfs the path in the controller namespace and require the expected network fstype — cheap, and it also converts the post-reboot stub into a hard, attributable failure whenever any re-verification runs. 4. **Hub warning correct or false positive?** Correct behavior, right namespace, only working guard. Promote it: a path that fails the device-ID / fstype test must not be deployable-to, and should flip the NAS badge, not just a generic system WARN. **Blast radius:** no data on the system drive (docker's own `mkdir` was blocked by the root-owned stub — accidental but effective backstop); no data lost; CWA down ~4 h with no alert beyond the hub storage WARN. **Customer-plausible: fully.** Any NAS-backed app + any guest reboot (agent update rollout, host reboot, power blip) reproduces this on a real customer box: app down or — worse, if the stub is ever writable (e.g. a future uid-1000-owned mountpoint dir) — silent writes to the system drive. This ordering drives the fix list. ## 5. Prioritized fix tasks (each a separate follow-up; none implemented here) 1. **HIGH — agent `ReassertNetworkMounts` on guest start** (existing backlog `FOLLOWUP-nas-automount-guest-reboot-reassert.md`, now enriched): mechanism = restart/re-create the automount unit after a guest starts (new-mount propagation into running guests is proven); must cover all net shares, fire on every guest start (not only reboot-the-verb), and post-verify guest visibility instead of assuming. Host-side re-triggering alone provably cannot heal a guest. 2. **HIGH — verify in the consuming namespace, continuously:** (a) add-time probe gains a statfs fstype/device check (H2 fix — a stub must never verify); (b) deploy-time refusal: a network `HDD_PATH` whose in-namespace fstype is not the expected network fs is not selectable/deployable (promotes the H5 check); (c) `networkStorageWarnings` must combine the agent's host view with a controller-namespace check so a guest-side stub raises the NAS badge (today's masking). 3. **MEDIUM — boot-time app-start failure must alert:** CWA sat in `Exited(128)`/restart-failed for hours with no issue event. The restart error string ("error while creating mount source path") is exactly the signal to surface (app-issue event → hub), likely reusing the shipped app-log issue pipeline. 4. **MEDIUM — H4 display fix:** deployed-app storage select must mark `selected` by the stored `HDD_PATH` (fallback label when the path is no longer in the schedulable list), not by `IsDefault`. Also check why no storage path currently carries `is_default`. 5. **LOW — S-D label repair + writer RCA:** fix the stored felhom-usb label (one relabel call), then find/red-proof the 2026-07-08 write path that CP1250-mangled it (drive attach/registration flow). 6. **LOW — ring usefulness:** exclude/rate-limit the 10-second status-refresh DEBUG spam from the capture ring (10-minute effective window defeats after-the-fact RCA), and note that bootstrap container recreation discards both ring and docker logs — the host journal and hub bundles are the only durable channels. ## 6. Remediation — PERFORMED (operator GO given in-session, executed ~20:14–20:20 CEST) The proposed recipe ran exactly as written and **doubles as a successful live pre-test of the fix-1 mechanism**: 1. `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'` on the host (guest running) → the autofs trigger appeared in the guest namespace **immediately** (propagation of a fresh host mount into a running guest's slave bind, re-proven). 2. In-guest `ls` woke the automount; guest AND controller container now show the `nfs4` mount with the export content (`backups/`, `userdata/`, correct uid-1000 mapping). 3. `docker start calibre-web` → `running`, healthcheck `healthy`; `/calibre-library` shows `metadata.db` + all three author dirs; SQLite WAL recovered cleanly (0-byte wal, shm re-touched). 4. uid-1000 write probe from inside the CWA container through the NFS: **WRITE-OK**. 5. App answers over traefik (`Host: books.demo-felhom.eu`): 301→302 login redirect chain, alive. No rsync was needed (nothing was misplaced). The controller health WARN (S-B) is expected to clear on the next 5-minute monitor cycle now that `IsMountPoint` sees nfs4 ≠ pve-root. **Fix-1 design note validated live:** restarting the `.automount` unit is a sufficient, cheap, data-safe reassert mechanism for running guests; the agent follow-up only needs to invoke it after guest starts and post-verify guest visibility. ## 7. Follow-up — fixes 1+2+4 SHIPPED (2026-07-11 late) Agent **v0.84.0** (`felhom-agent` commit `474b858`: `ReassertNetworkMounts` — guest-hook post-start leg + agent-startup sweep, §8 decision table, stop+enable-now on the granted verbs) + controller **v0.117.0** (`felhom-controller` commits `c0f3e12`+`4a9c54a`: statfs namespace classifier, probe `not_network_fs` assertion, deploy-time stub refusal, dashboard stub badge, deploy-view selected-by-stored-HDD_PATH rider). Q1c re-run live (`pct reboot 9201`, operator-GO): the hook re-armed and verified the share DURING the start task; CWA auto-started healthy; health ok; no badge. **Q1c is green — reboot survival is automatic.** Fixes 3 (boot-failure alerting), 5 (label repair), 6 (ring revision) remain queued. Feature doc updated: `documentation/controller/network-storage-nas.md` §Reboot survival. --- *Evidence gathered non-destructively 2026-07-11 ~19:35–19:50 CEST; only artifacts created were the two `RCA-` markers (removed). Sources: host journal, pct exec, docker inspect, settings.json, app.yaml, DooPlex export listing, live page fetch, v0.116.1 source.*