AUDIT: NAS/CWA RCA 2026-07-11 — stub after 2nd reboot; deploy correct; display bug; hub WARN correct
Diagnostic only (no fixes). Full evidence + 6 prioritized fix tasks in documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md; remediation deferred (operator-gated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,232 @@
|
||||
# AUDIT — NAS-backed CWA deploy: storage mismatch + "not a separate drive" RCA (2026-07-11)
|
||||
|
||||
**Scope:** diagnostic only — no fixes applied, demo left as-found. Demo box felhom-pve
|
||||
(192.168.0.162), guest 9201, controller v0.116.1, agent v0.83.0, hub v0.46.0. NFS export
|
||||
`192.168.0.180:/mnt/5_hdd/felhom-data` (DooPlex). All times CEST unless marked UTC.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
The re-add of `nas-media` at 14:16 **worked end-to-end and verified against the real NFS** — the
|
||||
new host automount propagated live into the running guest (shared→slave carries *new* mounts). CWA
|
||||
was deployed correctly against the NAS (`app.yaml` `HDD_PATH: /mnt/felhom-drives/nas-media`) and
|
||||
its library physically lives on the DooPlex export. What broke everything was a **second guest
|
||||
reboot at 15:39:11** (`pct reboot 9201`, root@pam): the fresh guest namespace received the
|
||||
`felhom-usb` ext4 submount but **not** the autofs trigger — since then the guest path is an empty,
|
||||
root-owned local stub. CWA has been **down since 15:39** (Docker could not recreate the bind
|
||||
source: `mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` — which, silver lining,
|
||||
also prevented a single byte from landing on the system drive). The UI showing "felhom-usb" is a
|
||||
pure display bug: the deployed-app storage select marks `selected` by `IsDefault` only, never by
|
||||
the stored `HDD_PATH`. The hub WARN was correct and was the **only** signal that fired.
|
||||
|
||||
**Blast radius: zero data loss, zero data misplaced.** Library (434 KB metadata.db + 3 authors,
|
||||
5.4 MB total) safe on the export. CWA down ~4 h, silently.
|
||||
|
||||
---
|
||||
|
||||
## 1. Timeline (reconstructed from host journal, docker inspect, file mtimes)
|
||||
|
||||
| Time (CEST) | Event | Evidence |
|
||||
|---|---|---|
|
||||
| Jun 15 / Jul 6 | Old CWA cycle leftovers on felhom-usb (`userdata/import/calibre`, `backups/primary/calibre-web`) — unrelated to today | mtimes on felhom-usb |
|
||||
| Jul 11 morning | Q1c supervised test: guest reboot #1, NAS trigger lost, old share later removed | Q1c findings (prior session) |
|
||||
| 14:16:35 | `nas-media` re-added: agent `mkdir` + automount unit installed, agent verify OK ("netverify: mount verified") | host journal, agent log |
|
||||
| 14:16:37 | Registered in controller settings (`kind: network`, mapped uid/gid 1000) | settings.json |
|
||||
| 14:16:50 | **CWA deployed with `HDD_PATH: /mnt/felhom-drives/nas-media`** | app.yaml `deployed_at` |
|
||||
| 14:18–15:36 | In-guest controller (comm `felhom-controll`, cgroup `/lxc/9201/...docker...`) triggers the **host** automount every ~5 min — proof the trigger was live *inside the guest* and the periodic checks saw the real NFS | host journal ("Got automount request … triggered by … (felhom-controll)") |
|
||||
| 14:20–14:24 | Calibre library ingested (Pogátsa Zoltán, Robin Cook, Stephen King) onto the NFS | export file mtimes |
|
||||
| 15:05:47 | CWA container (re)start — the last running instance | docker inspect StartedAt |
|
||||
| **15:39:11** | **`pct reboot 9201` (root@pam) — reboot #2.** Automount requests from the guest stop dead at 15:36:54 and never resume | host journal UPID `vzreboot` |
|
||||
| 15:39:15 | CWA exits (128); Docker restart (`unless-stopped`) fails permanently: `error while creating mount source path '/mnt/felhom-drives/nas-media/userdata/media/books': mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` | docker inspect State |
|
||||
| 17:13:25 | Controller container recreated (cause unrecorded; RestartCount=0 → not a crash loop) | docker inspect |
|
||||
| ~19:40 | Audit evidence captured (below); marker test + host-retrigger probe run and cleaned up | this session |
|
||||
|
||||
## 2. Evidence
|
||||
|
||||
### 2.1 Mount reality (H1) — guest first, then host
|
||||
|
||||
Guest 9201 (`pct exec`, before any host-side access this session):
|
||||
|
||||
```
|
||||
/dev/mapper/pve-root /mnt/felhom-drives ext4 rw,relatime,errors=remount-ro 0 0
|
||||
/dev/sdd1 /mnt/felhom-drives/felhom-usb ext4 rw,relatime 0 0
|
||||
(no nfs4, no autofs anywhere)
|
||||
|
||||
/mnt/felhom-drives/nas-media: empty dir, drwxr-xr-x nobody:nogroup,
|
||||
Device: 252,1 (pve-root), Birth == Modify == 2026-07-11 12:16:35 UTC ← never written since creation
|
||||
df: /dev/mapper/pve-root 98497780 blocks → the SYSTEM drive
|
||||
```
|
||||
|
||||
Controller container (`docker exec felhom-controller`): identical — plain dir on pve-root.
|
||||
This is the namespace the add-time probe and the 5-min health checks run in.
|
||||
|
||||
Host:
|
||||
|
||||
```
|
||||
systemd-1 /mnt/felhom-drives/nas-media autofs rw,relatime,...,timeout=60,direct ... ← armed trigger
|
||||
mnt-felhom\x2ddrives-nas\x2dmedia.automount loaded active waiting
|
||||
mnt-felhom\x2ddrives-nas\x2dmedia.mount loaded inactive dead ← normal 60 s idle unmount
|
||||
```
|
||||
|
||||
Propagation (mountinfo): host `/mnt/felhom-drives` = `shared:179`, autofs child = `shared:221`;
|
||||
guest bind = `shared:262 master:179` (a proper slave!), and the guest **did** receive the
|
||||
felhom-usb ext4 submount (`master:51`) — but not the autofs mount. Observed rule (matches Q1c):
|
||||
the guest-start bind replicates ext4 submounts but **not autofs direct mounts**; new host mounts
|
||||
created **while the guest runs** do propagate (proven at 14:16).
|
||||
|
||||
Marker test: `RCA-host-marker` created in the export on DooPlex → **NOT visible in the guest**.
|
||||
Guest-side `touch` into the stub → `permission denied` (root-owned dir, unmapped in the userns) —
|
||||
the guest path is not just misdirected, it is completely unwritable. Markers cleaned up.
|
||||
|
||||
### 2.2 Where CWA's data actually is (H3 vs H4)
|
||||
|
||||
- `app.yaml` (`/opt/docker/stacks/calibre-web/app.yaml`): **`HDD_PATH: /mnt/felhom-drives/nas-media`**,
|
||||
locked field, `deployed_at 2026-07-11T12:16:50Z`.
|
||||
- `docker inspect calibre-web` binds: `/mnt/felhom-drives/nas-media/userdata/media/books
|
||||
→ /calibre-library` and `.../userdata/import/calibre → /cwa-book-ingest`. Config in a named volume.
|
||||
- DooPlex export: full CWA skeleton + **library** (`userdata/media/books/metadata.db` 434 KB,
|
||||
3 author dirs, `.calnotes`), plus `backups/primary/calibre-web` manifest+compose. 5.4 MB total.
|
||||
`metadata.db-shm/-wal` left behind by the reboot kill (routine SQLite recovery on next open).
|
||||
- Guest stub: empty (mtime == birth). felhom-usb: only Jun 15 / Jul 6 leftovers from an earlier
|
||||
CWA cycle; `userdata/media/books` on felhom-usb is empty.
|
||||
|
||||
### 2.3 Observability shakedown (§3.3)
|
||||
|
||||
The controller/agent capture rings could **not** answer H3 after the fact:
|
||||
- The controller container was recreated at 17:13 (in-memory ring + docker logs of the old
|
||||
container gone — the bootstrap deploy mechanism `docker run`s a fresh container).
|
||||
- Even live, the ring wraps in **~10 minutes** on an idle demo box: earliest entry 17:39:30 UTC at
|
||||
a 17:47 UTC pull; the buffer is dominated by 10-second `[stacks] Status refresh` + periodic
|
||||
`[agentapi] GET /disks` DEBUG spam.
|
||||
- Hub-side bundles (72 h custody) requested by the operator during the observability validation
|
||||
predate this incident's window; only a bundle pulled between 14:16 and 15:39 would hold the add
|
||||
and deploy flows. Worth a look on hub.felhom.eu, not counted on.
|
||||
|
||||
H3 was answered from durable artifacts instead (app.yaml + docker inspect + export contents).
|
||||
The add-flow agent-side lines *were* recoverable — from the host **journal**, not the ring.
|
||||
|
||||
### 2.4 Code paths (all cited against v0.116.1 tree)
|
||||
|
||||
- **Probe (H2):** `controller/internal/web/netprobe.go:32-47` (`NetProbeChild`) — write + readback
|
||||
+ delete only; `netprobe_linux.go:21-43` re-execs as uid/gid 1000 **in the controller
|
||||
container's namespace**. There is **no fstype / statfs-magic / device-identity check** anywhere
|
||||
in the probe. Orchestrated from `netstorage_job.go:129-138`.
|
||||
- **Deploy persist (H3):** the dropdown submits `HDD_PATH` as the option `value` (the real path),
|
||||
saved to app.yaml as a locked field — verified correct on the box; no substitution point found or
|
||||
needed (live evidence already refutes H3).
|
||||
- **Deploy view render (H4):** `internal/web/templates/deploy.html:562-571` — the select renders
|
||||
`{{if .IsDefault}}selected{{end}}` per option and, for a deployed app, just adds `disabled`
|
||||
(line 563). **The stored `HDD_PATH` is never compared.** Handler side
|
||||
(`internal/web/handlers.go:330-342`) passes only the schedulable-path list; the app's current
|
||||
value isn't passed into the select at all. Live page confirms: *neither* option carries
|
||||
`selected` (no path currently has `is_default` in settings.json), so the browser displays the
|
||||
first option — felhom-usb.
|
||||
- **Hub warning (H5):** `internal/monitor/healthcheck.go:299-302` → `system.IsMountPoint`
|
||||
(`internal/system/mounts_linux.go:19-29`): `Stat_t.Dev` of path vs parent, evaluated in the
|
||||
controller container. With the stub both are 252:1 → WARN. **Correct positive.**
|
||||
- **The masking (why nothing else fired):** `internal/web/handlers.go:1383-1436`
|
||||
(`networkStorageWarnings`) decides the dashboard "NAS unreachable" badge by asking the **agent**
|
||||
(`ListNetStorage`) — and the agent's *host-namespace* view is genuinely healthy (`idle`).
|
||||
Nothing evaluates the share from the namespace the apps actually consume it in.
|
||||
|
||||
### 2.5 Propagation-repair probe (§3.5)
|
||||
|
||||
Host-side access re-triggered the autofs fine (nfs4 mounted on the host) → guest still saw the
|
||||
stub. The NFS mounts as a **child of the host's autofs mount** (peer group `shared:221`), which has
|
||||
no counterpart in the guest — so re-triggering can never heal a guest that lost the trigger.
|
||||
What must be re-established is the **autofs trigger mount itself** in the guest namespace. Two
|
||||
facts from today constrain the fix:
|
||||
1. A **newly created** host mount under `/mnt/felhom-drives` (`shared:179`) propagates into running
|
||||
guests' slave binds — proven by the 14:16 re-add healing the then-running guest.
|
||||
2. Therefore **restarting the automount unit while guests run** (cheap: the NFS mount is
|
||||
idle-unmounted anyway) re-plants the trigger in every running guest in one shot. This is the
|
||||
natural `ReassertNetworkMounts` mechanism; it must fire after every guest start (and prudently
|
||||
verify guest-visibility afterwards rather than assume).
|
||||
|
||||
## 3. Hypothesis verdicts
|
||||
|
||||
| # | Hypothesis | Verdict |
|
||||
|---|---|---|
|
||||
| H1 | Guest-side stub | **TRUE, refined.** The re-add itself propagated into the live guest and verified against the real NFS. The stub was created by the *second* reboot (15:39) — a fresh guest namespace does not inherit a pre-existing autofs trigger (Q1c mechanism re-confirmed), while it *does* inherit ext4 submounts. |
|
||||
| H2 | Probe hole (no fstype check) | **TRUE in code, did not fire here.** The probe verified the real NFS at add time. Today's stub would fail the probe anyway (`not_writable`, root-owned dir) — the hole only bites when a stub happens to be uid-1000-writable. The *operative* gap is broader: no guest/controller-namespace re-verification ever runs after add-time, and the periodic NAS health signal is measured host-side (see H5/masking). |
|
||||
| H3 | Deploy persisted wrong path | **FALSE.** app.yaml, docker binds and physical data all say nas-media. |
|
||||
| H4 | Display-resolution bug | **TRUE.** `deploy.html:567` selects by `IsDefault` only; stored `HDD_PATH` never consulted; with no default flagged, the browser shows the first option (felhom-usb). Display-only. |
|
||||
| H5 | Hub warning correct | **TRUE.** Device-ID check (healthcheck.go:299) correctly detected path-on-system-device from the controller namespace — the only guard that caught the state. Promote its mechanism to add/deploy-time refusal and to the NAS-unreachable badge. |
|
||||
| S-D | Garbled "TÁ*rhely" label | **REAL, data-level.** settings.json stores the felhom-usb label as `Tárhely (felhom-usb)` ("Tárhely" UTF-8 bytes re-decoded as CP1250) since its registration on **2026-07-08**; nas-media's label (2026-07-11) is clean. Current label-generation code (`settings.go:891-893`) is clean UTF-8 — one-time corruption in whatever wrote the 07-08 registration. Needs data repair + writer RCA. |
|
||||
|
||||
## 4. The four RCA questions
|
||||
|
||||
1. **Guest-side reality / did the re-add restore propagation?** The guest path is an empty,
|
||||
root-owned local directory on the system device (252:1) with no NFS or autofs anywhere in the
|
||||
guest. The re-add **did** restore propagation into the then-running guest (14:16→15:36 proven
|
||||
healthy); the 15:39 `pct reboot` severed it again. Fresh adds heal running guests; nothing
|
||||
heals a rebooted guest today.
|
||||
2. **Where is CWA's data / H3 xor H4?** Physically on the DooPlex export (5.4 MB, intact incl.
|
||||
metadata.db); `app.yaml` says nas-media. **Display bug (H4)**, not a persistence bug. CWA
|
||||
itself has been down since 15:39 — the operator's "runs and looked like working" was the
|
||||
pre-reboot state.
|
||||
3. **Why did the verify pipeline pass?** Legitimately — at add time the mount was real in the
|
||||
probing namespace. The design gaps: (a) the probe asserts writability, never
|
||||
network-filesystem identity (statfs magic / device ≠ rootfs-device); (b) verification runs
|
||||
once, at add time, in a state that a guest reboot silently invalidates; (c) ongoing NAS health
|
||||
is asked of the agent (host namespace), so the guest-side stub is invisible. The minimal
|
||||
add-time check that would refuse today's state: statfs the path in the controller namespace and
|
||||
require the expected network fstype — cheap, and it also converts the post-reboot stub into a
|
||||
hard, attributable failure whenever any re-verification runs.
|
||||
4. **Hub warning correct or false positive?** Correct behavior, right namespace, only working
|
||||
guard. Promote it: a path that fails the device-ID / fstype test must not be deployable-to, and
|
||||
should flip the NAS badge, not just a generic system WARN.
|
||||
|
||||
**Blast radius:** no data on the system drive (docker's own `mkdir` was blocked by the root-owned
|
||||
stub — accidental but effective backstop); no data lost; CWA down ~4 h with no alert beyond the hub
|
||||
storage WARN. **Customer-plausible: fully.** Any NAS-backed app + any guest reboot (agent update
|
||||
rollout, host reboot, power blip) reproduces this on a real customer box: app down or — worse, if
|
||||
the stub is ever writable (e.g. a future uid-1000-owned mountpoint dir) — silent writes to the
|
||||
system drive. This ordering drives the fix list.
|
||||
|
||||
## 5. Prioritized fix tasks (each a separate follow-up; none implemented here)
|
||||
|
||||
1. **HIGH — agent `ReassertNetworkMounts` on guest start** (existing backlog
|
||||
`FOLLOWUP-nas-automount-guest-reboot-reassert.md`, now enriched): mechanism = restart/re-create
|
||||
the automount unit after a guest starts (new-mount propagation into running guests is proven);
|
||||
must cover all net shares, fire on every guest start (not only reboot-the-verb), and
|
||||
post-verify guest visibility instead of assuming. Host-side re-triggering alone provably
|
||||
cannot heal a guest.
|
||||
2. **HIGH — verify in the consuming namespace, continuously:** (a) add-time probe gains a statfs
|
||||
fstype/device check (H2 fix — a stub must never verify); (b) deploy-time refusal: a network
|
||||
`HDD_PATH` whose in-namespace fstype is not the expected network fs is not selectable/deployable
|
||||
(promotes the H5 check); (c) `networkStorageWarnings` must combine the agent's host view with a
|
||||
controller-namespace check so a guest-side stub raises the NAS badge (today's masking).
|
||||
3. **MEDIUM — boot-time app-start failure must alert:** CWA sat in `Exited(128)`/restart-failed
|
||||
for hours with no issue event. The restart error string ("error while creating mount source
|
||||
path") is exactly the signal to surface (app-issue event → hub), likely reusing the shipped
|
||||
app-log issue pipeline.
|
||||
4. **MEDIUM — H4 display fix:** deployed-app storage select must mark `selected` by the stored
|
||||
`HDD_PATH` (fallback label when the path is no longer in the schedulable list), not by
|
||||
`IsDefault`. Also check why no storage path currently carries `is_default`.
|
||||
5. **LOW — S-D label repair + writer RCA:** fix the stored felhom-usb label (one relabel call),
|
||||
then find/red-proof the 2026-07-08 write path that CP1250-mangled it (drive attach/registration
|
||||
flow).
|
||||
6. **LOW — ring usefulness:** exclude/rate-limit the 10-second status-refresh DEBUG spam from the
|
||||
capture ring (10-minute effective window defeats after-the-fact RCA), and note that bootstrap
|
||||
container recreation discards both ring and docker logs — the host journal and hub bundles are
|
||||
the only durable channels.
|
||||
|
||||
## 6. Remediation — DEFERRED (operator-gated, operator not in session)
|
||||
|
||||
Demo left as-found. Proposed recipe when GO is given (no rsync needed — nothing is misplaced):
|
||||
|
||||
1. `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'` on the host (guest running).
|
||||
2. Verify in-guest: `pct exec 9201 -- grep nas-media /proc/mounts` after an in-guest `ls` of the
|
||||
path (expect autofs, then nfs4), and confirm `RCA`-free export content is visible.
|
||||
3. `pct exec 9201 -- docker start calibre-web`; confirm the library loads (SQLite recovers the
|
||||
-wal/-shm on open).
|
||||
4. Optionally re-run the uid-1000 write path from inside the CWA container.
|
||||
|
||||
This doubles as a live pre-test of the fix-1 mechanism.
|
||||
|
||||
---
|
||||
*Evidence gathered non-destructively 2026-07-11 ~19:35–19:50 CEST; only artifacts created were the
|
||||
two `RCA-` markers (removed). Sources: host journal, pct exec, docker inspect, settings.json,
|
||||
app.yaml, DooPlex export listing, live page fetch, v0.116.1 source.*
|
||||
Reference in New Issue
Block a user