AUDIT: NAS/CWA RCA 2026-07-11 — stub after 2nd reboot; deploy correct; display bug; hub WARN correct
Diagnostic only (no fixes). Full evidence + 6 prioritized fix tasks in documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md; remediation deferred (operator-gated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -2,55 +2,32 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## Hub v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate — 2026-07-11
|
||||
## NAS-backed CWA deploy RCA — diagnostic audit (no fixes) — 2026-07-11
|
||||
|
||||
felhom.eu's third of the cross-repo observability task (agent v0.83.0 + controller v0.116.1 shipped
|
||||
in their repos). Commits: hub `e35b1ae`, manifest bump + conventions runbook `7c038df`.
|
||||
**Deployed:** `felhom-hub:0.46.0`, ArgoCD `Synced/Healthy`, rollout complete (17:03 CEST).
|
||||
Deliverable: `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md` + CONTEXT.md note. No code
|
||||
changes anywhere; demo left as-found (remediation §6 operator-gated, deferred).
|
||||
|
||||
### What shipped
|
||||
**Root cause chain (all five hypotheses resolved):**
|
||||
- The 14:16 nas-media re-add verified **legitimately** — a newly created host automount propagates
|
||||
into *running* guests (proven: the in-guest controller triggered the host autofs every 5 min,
|
||||
14:18–15:36). H1's real trigger was a **second** `pct reboot 9201` at 15:39:11: a fresh guest
|
||||
namespace inherits ext4 submounts but not the autofs trigger → empty root-owned stub since.
|
||||
- CWA was deployed **correctly** to the NAS (app.yaml `HDD_PATH`, docker binds, and the physical
|
||||
library — 5.4 MB incl. metadata.db — all on the DooPlex export). H3 FALSE.
|
||||
- The UI showing "felhom-usb" is a pure display bug (H4): `deploy.html:567` marks `selected` by
|
||||
`IsDefault` only, never the stored HDD_PATH; with no default flagged, the browser shows the
|
||||
first option.
|
||||
- The hub "nem külön meghajtón" WARN was **correct** (H5) — the device-ID check in the controller
|
||||
namespace was the only guard that fired. The dashboard NAS badge masked the state because it
|
||||
asks the **agent** (host namespace, genuinely healthy).
|
||||
- CWA has been DOWN since 15:39 (docker restart fails: `mkdir .../userdata: permission denied` on
|
||||
the stub) — which also blocked any writes to the system drive. **Zero data loss, zero misplaced.**
|
||||
- Probe hole (H2) confirmed in code (no fstype/device check, `netprobe.go`) but did not fire here.
|
||||
- S-D confirmed real: settings.json stores the felhom-usb label CP1250-mangled since 2026-07-08.
|
||||
- Observability shakedown: capture ring wraps in ~10 min (status-refresh spam) and dies with
|
||||
container recreation — rings could not answer H3; host journal + durable artifacts did.
|
||||
|
||||
- **Store** (`hub/internal/store/logbundle.go` + schema): `log_bundle_requests` (pending intent per
|
||||
scope+component — controller scope = customer_id / report ACK; agent scope = host_id / heartbeat
|
||||
envelope) + `log_bundles` (gzip, newest-3, **72 h TTL** purged on the existing 60 s sweep).
|
||||
`SaveLogBundle` runs the **token-pattern secret gate BEFORE storing**: a hit stores a
|
||||
`blocked: possible secret` flag row and NO payload (fail-closed, WARN hub-side); `[REDACTED]`
|
||||
shapes + public checksums pass by design.
|
||||
- **Channels** (additive both directions): report ACK `controller_log_requested` + ingest of
|
||||
`controller_log_tail`; heartbeat envelope `log_tail_requested` + ingest of `log_tail`.
|
||||
Consume-once on arrival. A pre-0.83 agent never fulfills → the request stays visibly `pending`
|
||||
(S6-tested, harmless).
|
||||
- **UI** (host detail, English per the hub operator surface): Diagnostics section — "Request
|
||||
controller logs" / "Request agent logs" (CSRF forms), state rows (`pending` with the honest
|
||||
latency hint: controller ≤ ~15 min report interval, agent ≈ heartbeat cadence; `available` with
|
||||
View/Download; `blocked`), 72 h custody note. The hosts "read-only" invariant is amended by test:
|
||||
these two forms are the ONLY actions.
|
||||
- **Conventions codified:** `documentation/runbooks/logging-conventions.md` (levels, logs-are-
|
||||
English, keys-never-values, capture layers, pull posture) + a one-liner in each repo CLAUDE.md.
|
||||
|
||||
### Red-proofs (demonstrated, restored)
|
||||
|
||||
| Scenario | Reverted shape | Failure seen |
|
||||
|---|---|---|
|
||||
| S4 secret gate | gate disabled | `secret-shaped token not blocked` |
|
||||
| S2/consume-once | clear-on-arrival removed | `request survived fulfillment` (store + API tests) |
|
||||
|
||||
S5 TTL: injectable clock — kept at +71 h, purged at +73 h. Keep-newest-3 + byte-cap belts tested.
|
||||
|
||||
### Live evidence / honest latencies
|
||||
|
||||
- Hub 0.46.0 live (ArgoCD Synced/Healthy, image verified, clean startup log).
|
||||
- Box halves live-proven from 9201: controller ring + agent ring both readable at
|
||||
`logging.level=info` through the new Debug tabs (see felhom-controller/REPORT.md timestamps).
|
||||
- **Operator follow-up (HUMAN — hub UI is password-gated, CC cannot log in):** on
|
||||
`hub.felhom.eu/hosts/<demo host>` click both request buttons; expect the agent bundle within one
|
||||
heartbeat (~15 min) and the controller bundle on the next report (≤ ~15 min); download both and
|
||||
confirm the controller bundle carries the NAS-add phase lines + the `operator log pull served`
|
||||
INFO line. The full round-trip is already proven by `internal/api/logbundle_ack_test.go`.
|
||||
- Blocked-secret branch: test-level only by design (no real secret planted live).
|
||||
|
||||
### Open / observations
|
||||
|
||||
- Hub bearer key rotation still DUE (pre-existing; manifests/hub.yaml + screenshot-exposed).
|
||||
- `log_bundles` UI states `requested → pending` are merged (a request IS pending from the box's
|
||||
view); `expired` renders as absence after the purge — states documented in the template hint.
|
||||
**Fix tasks (audit §5, priority order):** 1) agent ReassertNetworkMounts on guest start
|
||||
(automount-unit restart mechanism now proven); 2) verify in the consuming namespace (probe fstype
|
||||
check + deploy-time refusal + NAS badge combines controller-ns view); 3) boot-time app-start
|
||||
failure alerting; 4) H4 select fix; 5) label repair + writer RCA; 6) ring spam exclusion.
|
||||
|
||||
Reference in New Issue
Block a user