diff --git a/REPORT.md b/REPORT.md index 59a3c64..4a4c2fb 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,45 +1,98 @@ -# REPORT — F2: alert on born/persistent-down channel, v0.91.0 +# REPORT — NAS network storage Part A2 (controller registry/UI/health), v0.92.0 -**Task:** close F2 (the transition-only alerting gap the full-stack testrun surfaced) — a channel -failure present at **startup/reseed** was dashboard-only, no operator email, forever. Part of the -F2 (Part A) + prevention (Part B) slice; this repo is **Part A (controller)**. +**Date:** 2026-06-30 +**Task:** `TASK.md` — NAS Part A2 (+ hub dispatcher critical-severity fix, "Part 0"). Implementation class. -**Baseline:** controller `main` @ `77bccf1` (live 0.90.0) → **v0.91.0** @ `b2ad871` (image -`…/felhom-controller:0.91.0`). Companions: hub **v0.21.0** (F2 hub checkers), agent **v0.46.0** (B.1). +## Baselines -## The gap + fix (`internal/channelhealth/checker.go`) +| Repo | before → after | +|------|----------------| +| felhom-controller | v0.91.0 → **v0.92.0** (commit `364dc50`) | +| felhom.eu (hub, Part 0) | v0.23.0 → **v0.24.0** (code `0ff1d3c`, manifest `87f0794`) | +| felhom-agent | v0.50.0 (A1, unchanged) | -Pre-fix `Check()`: a confirmed (past-debounce) down with `prev==""` hit `// first observation seeds, -no alert`, then `prev==newState` every cycle → **never alerted**. A controller that *boots* into a -broken channel (e.g. right after a leaf regen + controller restart) was dashboard-only. +## What was built -**Fix — an `alerted` flag** (have we emitted for the CURRENT down-spell?): -- **UP:** `alerted=false` (re-arm); if `prev` was `down:*` → `NotifyRecovered`. -- **DOWN, transient + `consecutiveDown < N`:** unchanged debounce (suppress; unseeded → assume up). -- **DOWN, confirmed:** dashboard down. If `prev` was up/unseeded **or** the reason changed → new spell - → `alerted=false`. Then `if !alerted → NotifyDown; alerted=true`. Removed the `prev==""` - silent-seed-for-down branch. -- Net: non-transient **born-down alerts on cycle 1**; transient born-down still needs N≥2 (cold-boot - agent-not-yet-up race); **healthy** first-obs still silent; reason-change re-alerts; recovery re-arms. +The controller side of NAS network storage: an operator adds a customer's NAS share and points a media app +at it, all via the UI — proxying to the validated agent foundation (A1 `/netstorage/*`). A NAS is a +**distinct "network" storage kind**, NOT a drive: no enroll/eject/decommission/migrate/wipe/SMART. -## Tests (all green: `go build/vet/test ./...`) -F2 **born-down red-proof** (`TestF2_BornDownNonTransient_AlertsOnce`: one alert on cycle 1) + companion -(`…OldSeedSilentLogicWouldNotAlert`: the old `prev==""` path would not have alerted); born-down -transient still debounced; recovery re-arms the spell; healthy first-obs silent. The existing -debounce/transition/classify tests still pass. +### Part 0 — hub dispatcher (felhom.eu, v0.24.0, commit `0ff1d3c`) +- `internal/notify/dispatcher.go` `ProcessEvent`: routed only `warning`/`error` → a `critical`-severity + event was **silently dropped**. Now routes `warning`/`error`/`critical` (`severityNotifies`); an + unrecognized severity is **logged**, not dropped; `info` stays a deliberate non-notify. +- `host_disk_critical` now emits its **natural `critical` severity** (`FormatOperatorEmail` styles it 🔴). +- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil,nil)` + for a customer with no notification row — an event for such a customer would have **panicked/crashed the + hub**. Now guards `prefs == nil`. +- Verified safe first (§9 rule 5): **no** controller event emits `critical` (all info/warning/error), so no + surprise alert volume. Tests: `severityNotifies` + companion (pre-fix predicate drops `critical`), + ProcessEvent routes `critical`, unknown logged, info silent. -## Live validation (felhom-pve guest 9201) — PASS -Induced a **born-down non-transient**: regenerated the agent leaf (channel breaks) then restarted the -controller so its FIRST channel observation was `pin_mismatch`: -``` -[channel] agent channel DOWN (unseeded->down:pin_mismatch): … TLS pin mismatch … -Event pushed: agent_channel_pin_mismatch (error) — … (HTTP 200) -``` -**`unseeded->down`** = the born-down case, and it **ALERTED** (~60 s, one cycle) — vs the pre-fix -silent seed. Recovered via R1 leaf restore: `[channel] agent channel recovered` + -`agent_channel_recovered`, leaf==pin MATCH, dashboard banner cleared. This is the capstone's missing -half — **detection at startup, not only on a live transition.** +### A2 — controller (v0.92.0, commit `364dc50`) +- `internal/agentapi/client.go`: `AddNetStorage`/`ListNetStorage`/`RemoveNetStorage` + `NetworkMountStatus` + mirror. **SMB credential passes straight through to the agent (0600 file); never persisted by the + controller.** +- `internal/settings/settings.go`: `StoragePath.Kind` discriminator + network descriptors (no password); + `IsNetwork()`/`IsNetworkStoragePath()`; `NetworkMountRoot`. +- `internal/web/netstorage_handlers.go` (NEW): `POST /api/storage/netstorage/{add,remove}` + `GET`; registers + a Kind=network `StoragePath`; merges the agent's live per-share health. +- **Kind-gating (safety centerpiece):** `refuseNetworkLifecycle` blocks the drive ops (eject/decommission/ + migrate/wipe) on a network path; the drive-absent gate (`planDriveGates`) and the missing-storage surface + **skip network paths** → an `unreachable` NAS is a recoverable warning, never the drive stop-cascade. + `networkStorageWarnings` drives a distinct app-card badge. +- UI (`settings.html`): a "Hálózati tárhely (NAS)" section (add form NFS/SMB, health badges, remove); shares + auto-selectable as a media app's `HDD_PATH`. -## NOT changed -The swap, capability, or drive-gate logic; the debounce threshold; the agentapi pin/transport. Alerting -behaviour only. +## Tests — both repos `go build ./... && go vet ./... && go test ./...` GREEN +- agentapi: round-trip (creds forwarded, mapping/protocol forwarded, health states; `idle`/`ok` not + degraded, `unreachable` degraded). +- registry **Kind-gate companion**: the drive-lifecycle handlers refuse a network path (400 before any + agent/stackMgr use); a DRIVE path is NOT gate-refused (proves the gate keys on Kind, not the path). +- **`unreachable`≠`missing` companion**: a network path is never "missing"; the drive gate produces NO Stop + action for an absent network path, while it DOES for an absent drive path under the same parent. +- Part 0 dispatcher tests (above). + +## Live e2e validation (against the isolated sim NAS) + +**Method:** the demo controller has no password set, so CSRF is skipped — I drove the **exact UI endpoints** +(`/api/storage/netstorage/*`) via curl from inside guest 9201's controller container (the full controller +server pipeline runs; an acceptable proxy per CLAUDE.md). The storage-page badge was verified from the +rendered HTML. **Sim NAS** = a throwaway privileged LXC (NFS, `anonuid/anongid=101000`, own config) on +felhom-pve — never touched production storage; destroyed after. + +- **A — add via controller:** `POST /api/storage/netstorage/add` (nfs) → `registered:true`, registered as + `/mnt/felhom-drives/media kind=network sched=True proto=nfs`; the agent mounted it; `GET` showed health + `idle` then `ok` after first access. The share is absent from `/api/disks` (distinct class). +- **B — media app reads from NAS:** jellyfin (uid 1000, healthy) bound to the controller-registered NAS path + **read the media library** (`Movies/SampleMovie (2024).mkv` owned `1000:1000` via the +100000 idmap, plus + `readme.txt`). (jellyfin isn't in the app catalog — run as a raw uid-1000 container to exercise the real + A1 chain; see Observations.) +- **C — NAS disappears (no cascade):** black-holed the sim → `GET /api/storage/netstorage` health + `unreachable` in ~2s; the settings page rendered the "Nem elérhető" badge in the "Hálózati tárhely (NAS)" + section; **guest + controller + jellyfin stayed healthy**; over a drive-gate tick the network path was + **NOT** marked disconnected and **no app was stopped** (cascade prevented). On NAS return → health `ok`, + no remount. +- **D — remove via controller:** `POST /api/storage/netstorage/remove` → `removed:true`, list empty, host + units gone + unmounted. +- **E — Part 0:** unit-validated (critical routes; companion proves the pre-fix drop); hub v0.24.0 deployed + via ArgoCD (Synced/Healthy), `host_disk` now emits natural `critical`. (Not re-fired live — the demo host + root is at 24% after the prior task's prune, below threshold.) + +Deployed: hub v0.24.0 (ArgoCD sync, Synced/Healthy); controller v0.92.0 on guest 9201 +(`docker ps … :0.92.0 Up (healthy)`, clean logs). Sim NAS + test jellyfin torn down; `/mnt/felhom-drives` +back to only the real drives. + +## Observations / limitations +- **Part B (restic-over-SFTP NAS backup target)** still pending — separate task. +- **No media app in the catalog** (jellyfin/immich/…): A2's HDD_PATH selectability is wired (the NAS path is + Schedulable and appears in the dropdown), but the full "deploy a catalog media app onto the NAS via the + deploy flow" couldn't run; jellyfin was a raw uid-1000 container for the read proof. Adding jellyfin to + `app-catalog-felhom.eu` is a small follow-up that would make the app-card badge live-exercisable too. +- **Mixed-uid share sharing** is a documented v1 limit (a share's +100000 uid mapping is fixed at add-time → + one app or same-uid apps). +- **Write apps on a soft NFS NAS:** a mid-disappearance in-flight file can truncate (spike) → prefer + atomic-write apps or SSD-staging; v1 test used a READ app (jellyfin). +- Per-storage worst-fill alerting (hub host_disk follow-on) still pending. + +No secrets in any committed file. SMB/NFS test credentials were throwaway and out-of-band only. diff --git a/controller/README.md b/controller/README.md index fcf9960..f1e9f78 100644 --- a/controller/README.md +++ b/controller/README.md @@ -635,6 +635,22 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an ### 4. Storage Management +> **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB) +> is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**. +> The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no +> mount authority** and **never persists the SMB password** (it passes the credential to the agent's add +> request, which writes the 0600 file). Endpoints `POST /api/storage/netstorage/{add,remove}` + +> `GET /api/storage/netstorage` (merges the agent's per-share liveness: `ok` | `idle` (benign idle-unmount) +> | `unreachable` | `unknown`). A network share registers as `/mnt/felhom-drives/`, is **selectable as +> a media app's `HDD_PATH`** (Schedulable), and shows in the "Hálózati tárhely (NAS)" settings section. +> - **NOT a drive:** the drive lifecycle (eject/decommission/migrate/wipe/SMART) is **refused** on a network +> path server-side (`refuseNetworkLifecycle`). The **drive-absent gate** (`planDriveGates`) and the +> **missing-storage** surface **skip network paths**, so an `unreachable` NAS is a **recoverable warning** +> (`networkStorageWarnings` → a distinct app-card badge), **never** the drive "missing → stopped" cascade. +> - **Limits (v1):** a share's `+100000` uid mapping is fixed at add-time (one app / same-uid apps); for +> write apps on a soft NFS mount, an in-flight file can truncate if the NAS vanishes mid-write (prefer +> atomic-write apps / SSD-staging). + > **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now > visible in the guest at a STABLE path **`/mnt/felhom-drives/`**, NOT the raw `/mnt/`. The > agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's