felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md

AUDIT — NAS-backed CWA deploy: storage mismatch + "not a separate drive" RCA (2026-07-11)

Scope: diagnostic only — no fixes applied, demo left as-found. Demo box felhom-pve (192.168.0.162), guest 9201, controller v0.116.1, agent v0.83.0, hub v0.46.0. NFS export 192.168.0.180:/mnt/5_hdd/felhom-data (DooPlex). All times CEST unless marked UTC.

---

TL;DR

The re-add of nas-media at 14:16 worked end-to-end and verified against the real NFS — the new host automount propagated live into the running guest (shared→slave carries new mounts). CWA was deployed correctly against the NAS (app.yaml HDD_PATH: /mnt/felhom-drives/nas-media) and its library physically lives on the DooPlex export. What broke everything was a second guest reboot at 15:39:11 (pct reboot 9201, root@pam): the fresh guest namespace received the felhom-usb ext4 submount but not the autofs trigger — since then the guest path is an empty, root-owned local stub. CWA has been down since 15:39 (Docker could not recreate the bind source: mkdir /mnt/felhom-drives/nas-media/userdata: permission denied — which, silver lining, also prevented a single byte from landing on the system drive). The UI showing "felhom-usb" is a pure display bug: the deployed-app storage select marks selected by IsDefault only, never by the stored HDD_PATH. The hub WARN was correct and was the only signal that fired.

Blast radius: zero data loss, zero data misplaced. Library (434 KB metadata.db + 3 authors, 5.4 MB total) safe on the export. CWA down ~4 h, silently.

---

1. Timeline (reconstructed from host journal, docker inspect, file mtimes)

Time (CEST)	Event	Evidence
Jun 15 / Jul 6	Old CWA cycle leftovers on felhom-usb (userdata/import/calibre, backups/primary/calibre-web) — unrelated to today	mtimes on felhom-usb
Jul 11 morning	Q1c supervised test: guest reboot #1, NAS trigger lost, old share later removed	Q1c findings (prior session)
14:16:35	nas-media re-added: agent mkdir + automount unit installed, agent verify OK ("netverify: mount verified")	host journal, agent log
14:16:37	Registered in controller settings (kind: network, mapped uid/gid 1000)	settings.json
14:16:50	CWA deployed with HDD_PATH: /mnt/felhom-drives/nas-media	app.yaml deployed_at
14:18–15:36	In-guest controller (comm felhom-controll, cgroup /lxc/9201/...docker...) triggers the host automount every ~5 min — proof the trigger was live inside the guest and the periodic checks saw the real NFS	host journal ("Got automount request … triggered by … (felhom-controll)")
14:20–14:24	Calibre library ingested (Pogátsa Zoltán, Robin Cook, Stephen King) onto the NFS	export file mtimes
15:05:47	CWA container (re)start — the last running instance	docker inspect StartedAt
15:39:11	pct reboot 9201 (root@pam) — reboot #2. Automount requests from the guest stop dead at 15:36:54 and never resume	host journal UPID vzreboot
15:39:15	CWA exits (128); Docker restart (unless-stopped) fails permanently: error while creating mount source path '/mnt/felhom-drives/nas-media/userdata/media/books': mkdir /mnt/felhom-drives/nas-media/userdata: permission denied	docker inspect State
17:13:25	Controller container recreated (cause unrecorded; RestartCount=0 → not a crash loop)	docker inspect
~19:40	Audit evidence captured (below); marker test + host-retrigger probe run and cleaned up	this session

2. Evidence

2.1 Mount reality (H1) — guest first, then host

Guest 9201 (pct exec, before any host-side access this session):

/dev/mapper/pve-root /mnt/felhom-drives ext4 rw,relatime,errors=remount-ro 0 0
/dev/sdd1 /mnt/felhom-drives/felhom-usb ext4 rw,relatime 0 0
(no nfs4, no autofs anywhere)

/mnt/felhom-drives/nas-media: empty dir, drwxr-xr-x nobody:nogroup,
Device: 252,1 (pve-root), Birth == Modify == 2026-07-11 12:16:35 UTC  ← never written since creation
df: /dev/mapper/pve-root 98497780 blocks → the SYSTEM drive

Controller container (docker exec felhom-controller): identical — plain dir on pve-root. This is the namespace the add-time probe and the 5-min health checks run in.

Host:

systemd-1 /mnt/felhom-drives/nas-media autofs rw,relatime,...,timeout=60,direct ...   ← armed trigger
mnt-felhom\x2ddrives-nas\x2dmedia.automount  loaded active waiting
mnt-felhom\x2ddrives-nas\x2dmedia.mount      loaded inactive dead      ← normal 60 s idle unmount

Propagation (mountinfo): host /mnt/felhom-drives = shared:179, autofs child = shared:221; guest bind = shared:262 master:179 (a proper slave!), and the guest did receive the felhom-usb ext4 submount (master:51) — but not the autofs mount. Observed rule (matches Q1c): the guest-start bind replicates ext4 submounts but not autofs direct mounts; new host mounts created while the guest runs do propagate (proven at 14:16).

Marker test: RCA-host-marker created in the export on DooPlex → NOT visible in the guest. Guest-side touch into the stub → permission denied (root-owned dir, unmapped in the userns) — the guest path is not just misdirected, it is completely unwritable. Markers cleaned up.

2.2 Where CWA's data actually is (H3 vs H4)

• app.yaml (/opt/docker/stacks/calibre-web/app.yaml): HDD_PATH: /mnt/felhom-drives/nas-media, locked field, deployed_at 2026-07-11T12:16:50Z.
• docker inspect calibre-web binds: /mnt/felhom-drives/nas-media/userdata/media/books → /calibre-library and .../userdata/import/calibre → /cwa-book-ingest. Config in a named volume.
• DooPlex export: full CWA skeleton + library (userdata/media/books/metadata.db 434 KB, 3 author dirs, .calnotes), plus backups/primary/calibre-web manifest+compose. 5.4 MB total. metadata.db-shm/-wal left behind by the reboot kill (routine SQLite recovery on next open).
• Guest stub: empty (mtime == birth). felhom-usb: only Jun 15 / Jul 6 leftovers from an earlier CWA cycle; userdata/media/books on felhom-usb is empty.

2.3 Observability shakedown (§3.3)

The controller/agent capture rings could not answer H3 after the fact:

• The controller container was recreated at 17:13 (in-memory ring + docker logs of the old container gone — the bootstrap deploy mechanism docker runs a fresh container).
• Even live, the ring wraps in ~10 minutes on an idle demo box: earliest entry 17:39:30 UTC at a 17:47 UTC pull; the buffer is dominated by 10-second [stacks] Status refresh + periodic [agentapi] GET /disks DEBUG spam.
• Hub-side bundles (72 h custody) requested by the operator during the observability validation predate this incident's window; only a bundle pulled between 14:16 and 15:39 would hold the add and deploy flows. Worth a look on hub.felhom.eu, not counted on.

H3 was answered from durable artifacts instead (app.yaml + docker inspect + export contents). The add-flow agent-side lines were recoverable — from the host journal, not the ring.

2.4 Code paths (all cited against v0.116.1 tree)

• Probe (H2): controller/internal/web/netprobe.go:32-47 (NetProbeChild) — write + readback
  • delete only; netprobe_linux.go:21-43 re-execs as uid/gid 1000 in the controller container's namespace. There is no fstype / statfs-magic / device-identity check anywhere in the probe. Orchestrated from netstorage_job.go:129-138.
• Deploy persist (H3): the dropdown submits HDD_PATH as the option value (the real path), saved to app.yaml as a locked field — verified correct on the box; no substitution point found or needed (live evidence already refutes H3).
• Deploy view render (H4): internal/web/templates/deploy.html:562-571 — the select renders {{if .IsDefault}}selected{{end}} per option and, for a deployed app, just adds disabled (line 563). The stored HDD_PATH is never compared. Handler side (internal/web/handlers.go:330-342) passes only the schedulable-path list; the app's current value isn't passed into the select at all. Live page confirms: neither option carries selected (no path currently has is_default in settings.json), so the browser displays the first option — felhom-usb.
• Hub warning (H5): internal/monitor/healthcheck.go:299-302 → system.IsMountPoint (internal/system/mounts_linux.go:19-29): Stat_t.Dev of path vs parent, evaluated in the controller container. With the stub both are 252:1 → WARN. Correct positive.
• The masking (why nothing else fired): internal/web/handlers.go:1383-1436 (networkStorageWarnings) decides the dashboard "NAS unreachable" badge by asking the agent (ListNetStorage) — and the agent's host-namespace view is genuinely healthy (idle). Nothing evaluates the share from the namespace the apps actually consume it in.

2.5 Propagation-repair probe (§3.5)

Host-side access re-triggered the autofs fine (nfs4 mounted on the host) → guest still saw the stub. The NFS mounts as a child of the host's autofs mount (peer group shared:221), which has no counterpart in the guest — so re-triggering can never heal a guest that lost the trigger. What must be re-established is the autofs trigger mount itself in the guest namespace. Two facts from today constrain the fix:

1. A newly created host mount under /mnt/felhom-drives (shared:179) propagates into running guests' slave binds — proven by the 14:16 re-add healing the then-running guest.
2. Therefore restarting the automount unit while guests run (cheap: the NFS mount is idle-unmounted anyway) re-plants the trigger in every running guest in one shot. This is the natural ReassertNetworkMounts mechanism; it must fire after every guest start (and prudently verify guest-visibility afterwards rather than assume).

3. Hypothesis verdicts

#	Hypothesis	Verdict
H1	Guest-side stub	TRUE, refined. The re-add itself propagated into the live guest and verified against the real NFS. The stub was created by the second reboot (15:39) — a fresh guest namespace does not inherit a pre-existing autofs trigger (Q1c mechanism re-confirmed), while it does inherit ext4 submounts.
H2	Probe hole (no fstype check)	TRUE in code, did not fire here. The probe verified the real NFS at add time. Today's stub would fail the probe anyway (not_writable, root-owned dir) — the hole only bites when a stub happens to be uid-1000-writable. The operative gap is broader: no guest/controller-namespace re-verification ever runs after add-time, and the periodic NAS health signal is measured host-side (see H5/masking).
H3	Deploy persisted wrong path	FALSE. app.yaml, docker binds and physical data all say nas-media.
H4	Display-resolution bug	TRUE. deploy.html:567 selects by IsDefault only; stored HDD_PATH never consulted; with no default flagged, the browser shows the first option (felhom-usb). Display-only.
H5	Hub warning correct	TRUE. Device-ID check (healthcheck.go:299) correctly detected path-on-system-device from the controller namespace — the only guard that caught the state. Promote its mechanism to add/deploy-time refusal and to the NAS-unreachable badge.
S-D	Garbled "TÁ*rhely" label	REAL, data-level. settings.json stores the felhom-usb label as Tárhely (felhom-usb) ("Tárhely" UTF-8 bytes re-decoded as CP1250) since its registration on 2026-07-08; nas-media's label (2026-07-11) is clean. Current label-generation code (settings.go:891-893) is clean UTF-8 — one-time corruption in whatever wrote the 07-08 registration. Needs data repair + writer RCA.

4. The four RCA questions

1. Guest-side reality / did the re-add restore propagation? The guest path is an empty, root-owned local directory on the system device (252:1) with no NFS or autofs anywhere in the guest. The re-add did restore propagation into the then-running guest (14:16→15:36 proven healthy); the 15:39 pct reboot severed it again. Fresh adds heal running guests; nothing heals a rebooted guest today.
2. Where is CWA's data / H3 xor H4? Physically on the DooPlex export (5.4 MB, intact incl. metadata.db); app.yaml says nas-media. Display bug (H4), not a persistence bug. CWA itself has been down since 15:39 — the operator's "runs and looked like working" was the pre-reboot state.
3. Why did the verify pipeline pass? Legitimately — at add time the mount was real in the probing namespace. The design gaps: (a) the probe asserts writability, never network-filesystem identity (statfs magic / device ≠ rootfs-device); (b) verification runs once, at add time, in a state that a guest reboot silently invalidates; (c) ongoing NAS health is asked of the agent (host namespace), so the guest-side stub is invisible. The minimal add-time check that would refuse today's state: statfs the path in the controller namespace and require the expected network fstype — cheap, and it also converts the post-reboot stub into a hard, attributable failure whenever any re-verification runs.
4. Hub warning correct or false positive? Correct behavior, right namespace, only working guard. Promote it: a path that fails the device-ID / fstype test must not be deployable-to, and should flip the NAS badge, not just a generic system WARN.

Blast radius: no data on the system drive (docker's own mkdir was blocked by the root-owned stub — accidental but effective backstop); no data lost; CWA down ~4 h with no alert beyond the hub storage WARN. Customer-plausible: fully. Any NAS-backed app + any guest reboot (agent update rollout, host reboot, power blip) reproduces this on a real customer box: app down or — worse, if the stub is ever writable (e.g. a future uid-1000-owned mountpoint dir) — silent writes to the system drive. This ordering drives the fix list.

5. Prioritized fix tasks (each a separate follow-up; none implemented here)

1. HIGH — agent ReassertNetworkMounts on guest start (existing backlog FOLLOWUP-nas-automount-guest-reboot-reassert.md, now enriched): mechanism = restart/re-create the automount unit after a guest starts (new-mount propagation into running guests is proven); must cover all net shares, fire on every guest start (not only reboot-the-verb), and post-verify guest visibility instead of assuming. Host-side re-triggering alone provably cannot heal a guest.
2. HIGH — verify in the consuming namespace, continuously: (a) add-time probe gains a statfs fstype/device check (H2 fix — a stub must never verify); (b) deploy-time refusal: a network HDD_PATH whose in-namespace fstype is not the expected network fs is not selectable/deployable (promotes the H5 check); (c) networkStorageWarnings must combine the agent's host view with a controller-namespace check so a guest-side stub raises the NAS badge (today's masking).
3. MEDIUM — boot-time app-start failure must alert: CWA sat in Exited(128)/restart-failed for hours with no issue event. The restart error string ("error while creating mount source path") is exactly the signal to surface (app-issue event → hub), likely reusing the shipped app-log issue pipeline.
4. MEDIUM — H4 display fix: deployed-app storage select must mark selected by the stored HDD_PATH (fallback label when the path is no longer in the schedulable list), not by IsDefault. Also check why no storage path currently carries is_default.
5. LOW — S-D label repair + writer RCA: fix the stored felhom-usb label (one relabel call), then find/red-proof the 2026-07-08 write path that CP1250-mangled it (drive attach/registration flow).
6. LOW — ring usefulness: exclude/rate-limit the 10-second status-refresh DEBUG spam from the capture ring (10-minute effective window defeats after-the-fact RCA), and note that bootstrap container recreation discards both ring and docker logs — the host journal and hub bundles are the only durable channels.

6. Remediation — PERFORMED (operator GO given in-session, executed ~20:14–20:20 CEST)

The proposed recipe ran exactly as written and doubles as a successful live pre-test of the fix-1 mechanism:

1. systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount' on the host (guest running) → the autofs trigger appeared in the guest namespace immediately (propagation of a fresh host mount into a running guest's slave bind, re-proven).
2. In-guest ls woke the automount; guest AND controller container now show the nfs4 mount with the export content (backups/, userdata/, correct uid-1000 mapping).
3. docker start calibre-web → running, healthcheck healthy; /calibre-library shows metadata.db + all three author dirs; SQLite WAL recovered cleanly (0-byte wal, shm re-touched).
4. uid-1000 write probe from inside the CWA container through the NFS: WRITE-OK.
5. App answers over traefik (Host: books.demo-felhom.eu): 301→302 login redirect chain, alive.

No rsync was needed (nothing was misplaced). The controller health WARN (S-B) is expected to clear on the next 5-minute monitor cycle now that IsMountPoint sees nfs4 ≠ pve-root.

Fix-1 design note validated live: restarting the .automount unit is a sufficient, cheap, data-safe reassert mechanism for running guests; the agent follow-up only needs to invoke it after guest starts and post-verify guest visibility.

7. Follow-up — fixes 1+2+4 SHIPPED (2026-07-11 late)

Agent v0.84.0 (felhom-agent commit 474b858: ReassertNetworkMounts — guest-hook post-start leg + agent-startup sweep, §8 decision table, stop+enable-now on the granted verbs) + controller v0.117.0 (felhom-controller commits c0f3e12+4a9c54a: statfs namespace classifier, probe not_network_fs assertion, deploy-time stub refusal, dashboard stub badge, deploy-view selected-by-stored-HDD_PATH rider). Q1c re-run live (pct reboot 9201, operator-GO): the hook re-armed and verified the share DURING the start task; CWA auto-started healthy; health ok; no badge. Q1c is green — reboot survival is automatic. Fixes 3 (boot-failure alerting), 5 (label repair), 6 (ring revision) remain queued. Feature doc updated: documentation/controller/network-storage-nas.md §Reboot survival.

---

Evidence gathered non-destructively 2026-07-11 ~19:35–19:50 CEST; only artifacts created were the two RCA- markers (removed). Sources: host journal, pct exec, docker inspect, settings.json, app.yaml, DooPlex export listing, live page fetch, v0.116.1 source.