felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md

# AUDIT — NAS-backed CWA deploy: storage mismatch + "not a separate drive" RCA (2026-07-11)

**Scope:** diagnostic only — no fixes applied, demo left as-found. Demo box felhom-pve
(192.168.0.162), guest 9201, controller v0.116.1, agent v0.83.0, hub v0.46.0. NFS export
`192.168.0.180:/mnt/5_hdd/felhom-data` (DooPlex). All times CEST unless marked UTC.

---

## TL;DR

The re-add of `nas-media` at 14:16 **worked end-to-end and verified against the real NFS** — the
new host automount propagated live into the running guest (shared→slave carries *new* mounts). CWA
was deployed correctly against the NAS (`app.yaml` `HDD_PATH: /mnt/felhom-drives/nas-media`) and
its library physically lives on the DooPlex export. What broke everything was a **second guest
reboot at 15:39:11** (`pct reboot 9201`, root@pam): the fresh guest namespace received the
`felhom-usb` ext4 submount but **not** the autofs trigger — since then the guest path is an empty,
root-owned local stub. CWA has been **down since 15:39** (Docker could not recreate the bind
source: `mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` — which, silver lining,
also prevented a single byte from landing on the system drive). The UI showing "felhom-usb" is a
pure display bug: the deployed-app storage select marks `selected` by `IsDefault` only, never by
the stored `HDD_PATH`. The hub WARN was correct and was the **only** signal that fired.

**Blast radius: zero data loss, zero data misplaced.** Library (434 KB metadata.db + 3 authors,
5.4 MB total) safe on the export. CWA down ~4 h, silently.

---

## 1. Timeline (reconstructed from host journal, docker inspect, file mtimes)

| Time (CEST) | Event | Evidence |
|---|---|---|
| Jun 15 / Jul 6 | Old CWA cycle leftovers on felhom-usb (`userdata/import/calibre`, `backups/primary/calibre-web`) — unrelated to today | mtimes on felhom-usb |
| Jul 11 morning | Q1c supervised test: guest reboot #1, NAS trigger lost, old share later removed | Q1c findings (prior session) |
| 14:16:35 | `nas-media` re-added: agent `mkdir` + automount unit installed, agent verify OK ("netverify: mount verified") | host journal, agent log |
| 14:16:37 | Registered in controller settings (`kind: network`, mapped uid/gid 1000) | settings.json |
| 14:16:50 | **CWA deployed with `HDD_PATH: /mnt/felhom-drives/nas-media`** | app.yaml `deployed_at` |
| 14:18–15:36 | In-guest controller (comm `felhom-controll`, cgroup `/lxc/9201/...docker...`) triggers the **host** automount every ~5 min — proof the trigger was live *inside the guest* and the periodic checks saw the real NFS | host journal ("Got automount request … triggered by … (felhom-controll)") |
| 14:20–14:24 | Calibre library ingested (Pogátsa Zoltán, Robin Cook, Stephen King) onto the NFS | export file mtimes |
| 15:05:47 | CWA container (re)start — the last running instance | docker inspect StartedAt |
| **15:39:11** | **`pct reboot 9201` (root@pam) — reboot #2.** Automount requests from the guest stop dead at 15:36:54 and never resume | host journal UPID `vzreboot` |
| 15:39:15 | CWA exits (128); Docker restart (`unless-stopped`) fails permanently: `error while creating mount source path '/mnt/felhom-drives/nas-media/userdata/media/books': mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` | docker inspect State |
| 17:13:25 | Controller container recreated (cause unrecorded; RestartCount=0 → not a crash loop) | docker inspect |
| ~19:40 | Audit evidence captured (below); marker test + host-retrigger probe run and cleaned up | this session |

## 2. Evidence

### 2.1 Mount reality (H1) — guest first, then host

Guest 9201 (`pct exec`, before any host-side access this session):

```
/dev/mapper/pve-root /mnt/felhom-drives ext4 rw,relatime,errors=remount-ro 0 0
/dev/sdd1 /mnt/felhom-drives/felhom-usb ext4 rw,relatime 0 0
(no nfs4, no autofs anywhere)

/mnt/felhom-drives/nas-media: empty dir, drwxr-xr-x nobody:nogroup,
Device: 252,1 (pve-root), Birth == Modify == 2026-07-11 12:16:35 UTC  ← never written since creation
df: /dev/mapper/pve-root 98497780 blocks → the SYSTEM drive
```

Controller container (`docker exec felhom-controller`): identical — plain dir on pve-root.
This is the namespace the add-time probe and the 5-min health checks run in.

Host:

```
systemd-1 /mnt/felhom-drives/nas-media autofs rw,relatime,...,timeout=60,direct ...   ← armed trigger
mnt-felhom\x2ddrives-nas\x2dmedia.automount  loaded active waiting
mnt-felhom\x2ddrives-nas\x2dmedia.mount      loaded inactive dead      ← normal 60 s idle unmount
```

Propagation (mountinfo): host `/mnt/felhom-drives` = `shared:179`, autofs child = `shared:221`;
guest bind = `shared:262 master:179` (a proper slave!), and the guest **did** receive the
felhom-usb ext4 submount (`master:51`) — but not the autofs mount. Observed rule (matches Q1c):
the guest-start bind replicates ext4 submounts but **not autofs direct mounts**; new host mounts
created **while the guest runs** do propagate (proven at 14:16).

Marker test: `RCA-host-marker` created in the export on DooPlex → **NOT visible in the guest**.
Guest-side `touch` into the stub → `permission denied` (root-owned dir, unmapped in the userns) —
the guest path is not just misdirected, it is completely unwritable. Markers cleaned up.

### 2.2 Where CWA's data actually is (H3 vs H4)

- `app.yaml` (`/opt/docker/stacks/calibre-web/app.yaml`): **`HDD_PATH: /mnt/felhom-drives/nas-media`**,
  locked field, `deployed_at 2026-07-11T12:16:50Z`.
- `docker inspect calibre-web` binds: `/mnt/felhom-drives/nas-media/userdata/media/books
  → /calibre-library` and `.../userdata/import/calibre → /cwa-book-ingest`. Config in a named volume.
- DooPlex export: full CWA skeleton + **library** (`userdata/media/books/metadata.db` 434 KB,
  3 author dirs, `.calnotes`), plus `backups/primary/calibre-web` manifest+compose. 5.4 MB total.
  `metadata.db-shm/-wal` left behind by the reboot kill (routine SQLite recovery on next open).
- Guest stub: empty (mtime == birth). felhom-usb: only Jun 15 / Jul 6 leftovers from an earlier
  CWA cycle; `userdata/media/books` on felhom-usb is empty.

### 2.3 Observability shakedown (§3.3)

The controller/agent capture rings could **not** answer H3 after the fact:
- The controller container was recreated at 17:13 (in-memory ring + docker logs of the old
  container gone — the bootstrap deploy mechanism `docker run`s a fresh container).
- Even live, the ring wraps in **~10 minutes** on an idle demo box: earliest entry 17:39:30 UTC at
  a 17:47 UTC pull; the buffer is dominated by 10-second `[stacks] Status refresh` + periodic
  `[agentapi] GET /disks` DEBUG spam.
- Hub-side bundles (72 h custody) requested by the operator during the observability validation
  predate this incident's window; only a bundle pulled between 14:16 and 15:39 would hold the add
  and deploy flows. Worth a look on hub.felhom.eu, not counted on.

H3 was answered from durable artifacts instead (app.yaml + docker inspect + export contents).
The add-flow agent-side lines *were* recoverable — from the host **journal**, not the ring.

### 2.4 Code paths (all cited against v0.116.1 tree)

- **Probe (H2):** `controller/internal/web/netprobe.go:32-47` (`NetProbeChild`) — write + readback
  + delete only; `netprobe_linux.go:21-43` re-execs as uid/gid 1000 **in the controller
  container's namespace**. There is **no fstype / statfs-magic / device-identity check** anywhere
  in the probe. Orchestrated from `netstorage_job.go:129-138`.
- **Deploy persist (H3):** the dropdown submits `HDD_PATH` as the option `value` (the real path),
  saved to app.yaml as a locked field — verified correct on the box; no substitution point found or
  needed (live evidence already refutes H3).
- **Deploy view render (H4):** `internal/web/templates/deploy.html:562-571` — the select renders
  `{{if .IsDefault}}selected{{end}}` per option and, for a deployed app, just adds `disabled`
  (line 563). **The stored `HDD_PATH` is never compared.** Handler side
  (`internal/web/handlers.go:330-342`) passes only the schedulable-path list; the app's current
  value isn't passed into the select at all. Live page confirms: *neither* option carries
  `selected` (no path currently has `is_default` in settings.json), so the browser displays the
  first option — felhom-usb.
- **Hub warning (H5):** `internal/monitor/healthcheck.go:299-302` → `system.IsMountPoint`
  (`internal/system/mounts_linux.go:19-29`): `Stat_t.Dev` of path vs parent, evaluated in the
  controller container. With the stub both are 252:1 → WARN. **Correct positive.**
- **The masking (why nothing else fired):** `internal/web/handlers.go:1383-1436`
  (`networkStorageWarnings`) decides the dashboard "NAS unreachable" badge by asking the **agent**
  (`ListNetStorage`) — and the agent's *host-namespace* view is genuinely healthy (`idle`).
  Nothing evaluates the share from the namespace the apps actually consume it in.

### 2.5 Propagation-repair probe (§3.5)

Host-side access re-triggered the autofs fine (nfs4 mounted on the host) → guest still saw the
stub. The NFS mounts as a **child of the host's autofs mount** (peer group `shared:221`), which has
no counterpart in the guest — so re-triggering can never heal a guest that lost the trigger.
What must be re-established is the **autofs trigger mount itself** in the guest namespace. Two
facts from today constrain the fix:
1. A **newly created** host mount under `/mnt/felhom-drives` (`shared:179`) propagates into running
   guests' slave binds — proven by the 14:16 re-add healing the then-running guest.
2. Therefore **restarting the automount unit while guests run** (cheap: the NFS mount is
   idle-unmounted anyway) re-plants the trigger in every running guest in one shot. This is the
   natural `ReassertNetworkMounts` mechanism; it must fire after every guest start (and prudently
   verify guest-visibility afterwards rather than assume).

## 3. Hypothesis verdicts

| # | Hypothesis | Verdict |
|---|---|---|
| H1 | Guest-side stub | **TRUE, refined.** The re-add itself propagated into the live guest and verified against the real NFS. The stub was created by the *second* reboot (15:39) — a fresh guest namespace does not inherit a pre-existing autofs trigger (Q1c mechanism re-confirmed), while it *does* inherit ext4 submounts. |
| H2 | Probe hole (no fstype check) | **TRUE in code, did not fire here.** The probe verified the real NFS at add time. Today's stub would fail the probe anyway (`not_writable`, root-owned dir) — the hole only bites when a stub happens to be uid-1000-writable. The *operative* gap is broader: no guest/controller-namespace re-verification ever runs after add-time, and the periodic NAS health signal is measured host-side (see H5/masking). |
| H3 | Deploy persisted wrong path | **FALSE.** app.yaml, docker binds and physical data all say nas-media. |
| H4 | Display-resolution bug | **TRUE.** `deploy.html:567` selects by `IsDefault` only; stored `HDD_PATH` never consulted; with no default flagged, the browser shows the first option (felhom-usb). Display-only. |
| H5 | Hub warning correct | **TRUE.** Device-ID check (healthcheck.go:299) correctly detected path-on-system-device from the controller namespace — the only guard that caught the state. Promote its mechanism to add/deploy-time refusal and to the NAS-unreachable badge. |
| S-D | Garbled "TÁ*rhely" label | **REAL, data-level.** settings.json stores the felhom-usb label as `Tárhely (felhom-usb)` ("Tárhely" UTF-8 bytes re-decoded as CP1250) since its registration on **2026-07-08**; nas-media's label (2026-07-11) is clean. Current label-generation code (`settings.go:891-893`) is clean UTF-8 — one-time corruption in whatever wrote the 07-08 registration. Needs data repair + writer RCA. |

## 4. The four RCA questions

1. **Guest-side reality / did the re-add restore propagation?** The guest path is an empty,
   root-owned local directory on the system device (252:1) with no NFS or autofs anywhere in the
   guest. The re-add **did** restore propagation into the then-running guest (14:16→15:36 proven
   healthy); the 15:39 `pct reboot` severed it again. Fresh adds heal running guests; nothing
   heals a rebooted guest today.
2. **Where is CWA's data / H3 xor H4?** Physically on the DooPlex export (5.4 MB, intact incl.
   metadata.db); `app.yaml` says nas-media. **Display bug (H4)**, not a persistence bug. CWA
   itself has been down since 15:39 — the operator's "runs and looked like working" was the
   pre-reboot state.
3. **Why did the verify pipeline pass?** Legitimately — at add time the mount was real in the
   probing namespace. The design gaps: (a) the probe asserts writability, never
   network-filesystem identity (statfs magic / device ≠ rootfs-device); (b) verification runs
   once, at add time, in a state that a guest reboot silently invalidates; (c) ongoing NAS health
   is asked of the agent (host namespace), so the guest-side stub is invisible. The minimal
   add-time check that would refuse today's state: statfs the path in the controller namespace and
   require the expected network fstype — cheap, and it also converts the post-reboot stub into a
   hard, attributable failure whenever any re-verification runs.
4. **Hub warning correct or false positive?** Correct behavior, right namespace, only working
   guard. Promote it: a path that fails the device-ID / fstype test must not be deployable-to, and
   should flip the NAS badge, not just a generic system WARN.

**Blast radius:** no data on the system drive (docker's own `mkdir` was blocked by the root-owned
stub — accidental but effective backstop); no data lost; CWA down ~4 h with no alert beyond the hub
storage WARN. **Customer-plausible: fully.** Any NAS-backed app + any guest reboot (agent update
rollout, host reboot, power blip) reproduces this on a real customer box: app down or — worse, if
the stub is ever writable (e.g. a future uid-1000-owned mountpoint dir) — silent writes to the
system drive. This ordering drives the fix list.

## 5. Prioritized fix tasks (each a separate follow-up; none implemented here)

1. **HIGH — agent `ReassertNetworkMounts` on guest start** (existing backlog
   `FOLLOWUP-nas-automount-guest-reboot-reassert.md`, now enriched): mechanism = restart/re-create
   the automount unit after a guest starts (new-mount propagation into running guests is proven);
   must cover all net shares, fire on every guest start (not only reboot-the-verb), and
   post-verify guest visibility instead of assuming. Host-side re-triggering alone provably
   cannot heal a guest.
2. **HIGH — verify in the consuming namespace, continuously:** (a) add-time probe gains a statfs
   fstype/device check (H2 fix — a stub must never verify); (b) deploy-time refusal: a network
   `HDD_PATH` whose in-namespace fstype is not the expected network fs is not selectable/deployable
   (promotes the H5 check); (c) `networkStorageWarnings` must combine the agent's host view with a
   controller-namespace check so a guest-side stub raises the NAS badge (today's masking).
3. **MEDIUM — boot-time app-start failure must alert:** CWA sat in `Exited(128)`/restart-failed
   for hours with no issue event. The restart error string ("error while creating mount source
   path") is exactly the signal to surface (app-issue event → hub), likely reusing the shipped
   app-log issue pipeline.
4. **MEDIUM — H4 display fix:** deployed-app storage select must mark `selected` by the stored
   `HDD_PATH` (fallback label when the path is no longer in the schedulable list), not by
   `IsDefault`. Also check why no storage path currently carries `is_default`.
5. **LOW — S-D label repair + writer RCA:** fix the stored felhom-usb label (one relabel call),
   then find/red-proof the 2026-07-08 write path that CP1250-mangled it (drive attach/registration
   flow).
6. **LOW — ring usefulness:** exclude/rate-limit the 10-second status-refresh DEBUG spam from the
   capture ring (10-minute effective window defeats after-the-fact RCA), and note that bootstrap
   container recreation discards both ring and docker logs — the host journal and hub bundles are
   the only durable channels.

## 6. Remediation — PERFORMED (operator GO given in-session, executed ~20:14–20:20 CEST)

The proposed recipe ran exactly as written and **doubles as a successful live pre-test of the
fix-1 mechanism**:

1. `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'` on the host (guest running)
   → the autofs trigger appeared in the guest namespace **immediately** (propagation of a fresh
   host mount into a running guest's slave bind, re-proven).
2. In-guest `ls` woke the automount; guest AND controller container now show the `nfs4` mount with
   the export content (`backups/`, `userdata/`, correct uid-1000 mapping).
3. `docker start calibre-web` → `running`, healthcheck `healthy`; `/calibre-library` shows
   `metadata.db` + all three author dirs; SQLite WAL recovered cleanly (0-byte wal, shm re-touched).
4. uid-1000 write probe from inside the CWA container through the NFS: **WRITE-OK**.
5. App answers over traefik (`Host: books.demo-felhom.eu`): 301→302 login redirect chain, alive.

No rsync was needed (nothing was misplaced). The controller health WARN (S-B) is expected to clear
on the next 5-minute monitor cycle now that `IsMountPoint` sees nfs4 ≠ pve-root.

**Fix-1 design note validated live:** restarting the `.automount` unit is a sufficient, cheap,
data-safe reassert mechanism for running guests; the agent follow-up only needs to invoke it after
guest starts and post-verify guest visibility.

## 7. Follow-up — fixes 1+2+4 SHIPPED (2026-07-11 late)

Agent **v0.84.0** (`felhom-agent` commit `474b858`: `ReassertNetworkMounts` — guest-hook post-start
leg + agent-startup sweep, §8 decision table, stop+enable-now on the granted verbs) + controller
**v0.117.0** (`felhom-controller` commits `c0f3e12`+`4a9c54a`: statfs namespace classifier, probe
`not_network_fs` assertion, deploy-time stub refusal, dashboard stub badge, deploy-view
selected-by-stored-HDD_PATH rider). Q1c re-run live (`pct reboot 9201`, operator-GO): the hook
re-armed and verified the share DURING the start task; CWA auto-started healthy; health ok; no
badge. **Q1c is green — reboot survival is automatic.** Fixes 3 (boot-failure alerting), 5 (label
repair), 6 (ring revision) remain queued. Feature doc updated:
`documentation/controller/network-storage-nas.md` §Reboot survival.

---
*Evidence gathered non-destructively 2026-07-11 ~19:35–19:50 CEST; only artifacts created were the
two `RCA-` markers (removed). Sources: host journal, pct exec, docker inspect, settings.json,
app.yaml, DooPlex export listing, live page fetch, v0.116.1 source.*