docs: REPORT + README for controller v0.92.0 (NAS Part A2, live-validated A-D)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
@@ -1,45 +1,98 @@
|
||||
# REPORT — F2: alert on born/persistent-down channel, v0.91.0
|
||||
# REPORT — NAS network storage Part A2 (controller registry/UI/health), v0.92.0
|
||||
|
||||
**Task:** close F2 (the transition-only alerting gap the full-stack testrun surfaced) — a channel
|
||||
failure present at **startup/reseed** was dashboard-only, no operator email, forever. Part of the
|
||||
F2 (Part A) + prevention (Part B) slice; this repo is **Part A (controller)**.
|
||||
**Date:** 2026-06-30
|
||||
**Task:** `TASK.md` — NAS Part A2 (+ hub dispatcher critical-severity fix, "Part 0"). Implementation class.
|
||||
|
||||
**Baseline:** controller `main` @ `77bccf1` (live 0.90.0) → **v0.91.0** @ `b2ad871` (image
|
||||
`…/felhom-controller:0.91.0`). Companions: hub **v0.21.0** (F2 hub checkers), agent **v0.46.0** (B.1).
|
||||
## Baselines
|
||||
|
||||
## The gap + fix (`internal/channelhealth/checker.go`)
|
||||
| Repo | before → after |
|
||||
|------|----------------|
|
||||
| felhom-controller | v0.91.0 → **v0.92.0** (commit `364dc50`) |
|
||||
| felhom.eu (hub, Part 0) | v0.23.0 → **v0.24.0** (code `0ff1d3c`, manifest `87f0794`) |
|
||||
| felhom-agent | v0.50.0 (A1, unchanged) |
|
||||
|
||||
Pre-fix `Check()`: a confirmed (past-debounce) down with `prev==""` hit `// first observation seeds,
|
||||
no alert`, then `prev==newState` every cycle → **never alerted**. A controller that *boots* into a
|
||||
broken channel (e.g. right after a leaf regen + controller restart) was dashboard-only.
|
||||
## What was built
|
||||
|
||||
**Fix — an `alerted` flag** (have we emitted for the CURRENT down-spell?):
|
||||
- **UP:** `alerted=false` (re-arm); if `prev` was `down:*` → `NotifyRecovered`.
|
||||
- **DOWN, transient + `consecutiveDown < N`:** unchanged debounce (suppress; unseeded → assume up).
|
||||
- **DOWN, confirmed:** dashboard down. If `prev` was up/unseeded **or** the reason changed → new spell
|
||||
→ `alerted=false`. Then `if !alerted → NotifyDown; alerted=true`. Removed the `prev==""`
|
||||
silent-seed-for-down branch.
|
||||
- Net: non-transient **born-down alerts on cycle 1**; transient born-down still needs N≥2 (cold-boot
|
||||
agent-not-yet-up race); **healthy** first-obs still silent; reason-change re-alerts; recovery re-arms.
|
||||
The controller side of NAS network storage: an operator adds a customer's NAS share and points a media app
|
||||
at it, all via the UI — proxying to the validated agent foundation (A1 `/netstorage/*`). A NAS is a
|
||||
**distinct "network" storage kind**, NOT a drive: no enroll/eject/decommission/migrate/wipe/SMART.
|
||||
|
||||
## Tests (all green: `go build/vet/test ./...`)
|
||||
F2 **born-down red-proof** (`TestF2_BornDownNonTransient_AlertsOnce`: one alert on cycle 1) + companion
|
||||
(`…OldSeedSilentLogicWouldNotAlert`: the old `prev==""` path would not have alerted); born-down
|
||||
transient still debounced; recovery re-arms the spell; healthy first-obs silent. The existing
|
||||
debounce/transition/classify tests still pass.
|
||||
### Part 0 — hub dispatcher (felhom.eu, v0.24.0, commit `0ff1d3c`)
|
||||
- `internal/notify/dispatcher.go` `ProcessEvent`: routed only `warning`/`error` → a `critical`-severity
|
||||
event was **silently dropped**. Now routes `warning`/`error`/`critical` (`severityNotifies`); an
|
||||
unrecognized severity is **logged**, not dropped; `info` stays a deliberate non-notify.
|
||||
- `host_disk_critical` now emits its **natural `critical` severity** (`FormatOperatorEmail` styles it 🔴).
|
||||
- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil,nil)`
|
||||
for a customer with no notification row — an event for such a customer would have **panicked/crashed the
|
||||
hub**. Now guards `prefs == nil`.
|
||||
- Verified safe first (§9 rule 5): **no** controller event emits `critical` (all info/warning/error), so no
|
||||
surprise alert volume. Tests: `severityNotifies` + companion (pre-fix predicate drops `critical`),
|
||||
ProcessEvent routes `critical`, unknown logged, info silent.
|
||||
|
||||
## Live validation (felhom-pve guest 9201) — PASS
|
||||
Induced a **born-down non-transient**: regenerated the agent leaf (channel breaks) then restarted the
|
||||
controller so its FIRST channel observation was `pin_mismatch`:
|
||||
```
|
||||
[channel] agent channel DOWN (unseeded->down:pin_mismatch): … TLS pin mismatch …
|
||||
Event pushed: agent_channel_pin_mismatch (error) — … (HTTP 200)
|
||||
```
|
||||
**`unseeded->down`** = the born-down case, and it **ALERTED** (~60 s, one cycle) — vs the pre-fix
|
||||
silent seed. Recovered via R1 leaf restore: `[channel] agent channel recovered` +
|
||||
`agent_channel_recovered`, leaf==pin MATCH, dashboard banner cleared. This is the capstone's missing
|
||||
half — **detection at startup, not only on a live transition.**
|
||||
### A2 — controller (v0.92.0, commit `364dc50`)
|
||||
- `internal/agentapi/client.go`: `AddNetStorage`/`ListNetStorage`/`RemoveNetStorage` + `NetworkMountStatus`
|
||||
mirror. **SMB credential passes straight through to the agent (0600 file); never persisted by the
|
||||
controller.**
|
||||
- `internal/settings/settings.go`: `StoragePath.Kind` discriminator + network descriptors (no password);
|
||||
`IsNetwork()`/`IsNetworkStoragePath()`; `NetworkMountRoot`.
|
||||
- `internal/web/netstorage_handlers.go` (NEW): `POST /api/storage/netstorage/{add,remove}` + `GET`; registers
|
||||
a Kind=network `StoragePath`; merges the agent's live per-share health.
|
||||
- **Kind-gating (safety centerpiece):** `refuseNetworkLifecycle` blocks the drive ops (eject/decommission/
|
||||
migrate/wipe) on a network path; the drive-absent gate (`planDriveGates`) and the missing-storage surface
|
||||
**skip network paths** → an `unreachable` NAS is a recoverable warning, never the drive stop-cascade.
|
||||
`networkStorageWarnings` drives a distinct app-card badge.
|
||||
- UI (`settings.html`): a "Hálózati tárhely (NAS)" section (add form NFS/SMB, health badges, remove); shares
|
||||
auto-selectable as a media app's `HDD_PATH`.
|
||||
|
||||
## NOT changed
|
||||
The swap, capability, or drive-gate logic; the debounce threshold; the agentapi pin/transport. Alerting
|
||||
behaviour only.
|
||||
## Tests — both repos `go build ./... && go vet ./... && go test ./...` GREEN
|
||||
- agentapi: round-trip (creds forwarded, mapping/protocol forwarded, health states; `idle`/`ok` not
|
||||
degraded, `unreachable` degraded).
|
||||
- registry **Kind-gate companion**: the drive-lifecycle handlers refuse a network path (400 before any
|
||||
agent/stackMgr use); a DRIVE path is NOT gate-refused (proves the gate keys on Kind, not the path).
|
||||
- **`unreachable`≠`missing` companion**: a network path is never "missing"; the drive gate produces NO Stop
|
||||
action for an absent network path, while it DOES for an absent drive path under the same parent.
|
||||
- Part 0 dispatcher tests (above).
|
||||
|
||||
## Live e2e validation (against the isolated sim NAS)
|
||||
|
||||
**Method:** the demo controller has no password set, so CSRF is skipped — I drove the **exact UI endpoints**
|
||||
(`/api/storage/netstorage/*`) via curl from inside guest 9201's controller container (the full controller
|
||||
server pipeline runs; an acceptable proxy per CLAUDE.md). The storage-page badge was verified from the
|
||||
rendered HTML. **Sim NAS** = a throwaway privileged LXC (NFS, `anonuid/anongid=101000`, own config) on
|
||||
felhom-pve — never touched production storage; destroyed after.
|
||||
|
||||
- **A — add via controller:** `POST /api/storage/netstorage/add` (nfs) → `registered:true`, registered as
|
||||
`/mnt/felhom-drives/media kind=network sched=True proto=nfs`; the agent mounted it; `GET` showed health
|
||||
`idle` then `ok` after first access. The share is absent from `/api/disks` (distinct class).
|
||||
- **B — media app reads from NAS:** jellyfin (uid 1000, healthy) bound to the controller-registered NAS path
|
||||
**read the media library** (`Movies/SampleMovie (2024).mkv` owned `1000:1000` via the +100000 idmap, plus
|
||||
`readme.txt`). (jellyfin isn't in the app catalog — run as a raw uid-1000 container to exercise the real
|
||||
A1 chain; see Observations.)
|
||||
- **C — NAS disappears (no cascade):** black-holed the sim → `GET /api/storage/netstorage` health
|
||||
`unreachable` in ~2s; the settings page rendered the "Nem elérhető" badge in the "Hálózati tárhely (NAS)"
|
||||
section; **guest + controller + jellyfin stayed healthy**; over a drive-gate tick the network path was
|
||||
**NOT** marked disconnected and **no app was stopped** (cascade prevented). On NAS return → health `ok`,
|
||||
no remount.
|
||||
- **D — remove via controller:** `POST /api/storage/netstorage/remove` → `removed:true`, list empty, host
|
||||
units gone + unmounted.
|
||||
- **E — Part 0:** unit-validated (critical routes; companion proves the pre-fix drop); hub v0.24.0 deployed
|
||||
via ArgoCD (Synced/Healthy), `host_disk` now emits natural `critical`. (Not re-fired live — the demo host
|
||||
root is at 24% after the prior task's prune, below threshold.)
|
||||
|
||||
Deployed: hub v0.24.0 (ArgoCD sync, Synced/Healthy); controller v0.92.0 on guest 9201
|
||||
(`docker ps … :0.92.0 Up (healthy)`, clean logs). Sim NAS + test jellyfin torn down; `/mnt/felhom-drives`
|
||||
back to only the real drives.
|
||||
|
||||
## Observations / limitations
|
||||
- **Part B (restic-over-SFTP NAS backup target)** still pending — separate task.
|
||||
- **No media app in the catalog** (jellyfin/immich/…): A2's HDD_PATH selectability is wired (the NAS path is
|
||||
Schedulable and appears in the dropdown), but the full "deploy a catalog media app onto the NAS via the
|
||||
deploy flow" couldn't run; jellyfin was a raw uid-1000 container for the read proof. Adding jellyfin to
|
||||
`app-catalog-felhom.eu` is a small follow-up that would make the app-card badge live-exercisable too.
|
||||
- **Mixed-uid share sharing** is a documented v1 limit (a share's +100000 uid mapping is fixed at add-time →
|
||||
one app or same-uid apps).
|
||||
- **Write apps on a soft NFS NAS:** a mid-disappearance in-flight file can truncate (spike) → prefer
|
||||
atomic-write apps or SSD-staging; v1 test used a READ app (jellyfin).
|
||||
- Per-storage worst-fill alerting (hub host_disk follow-on) still pending.
|
||||
|
||||
No secrets in any committed file. SMB/NFS test credentials were throwaway and out-of-band only.
|
||||
|
||||
@@ -635,6 +635,22 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
|
||||
|
||||
### 4. Storage Management
|
||||
|
||||
> **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB)
|
||||
> is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**.
|
||||
> The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no
|
||||
> mount authority** and **never persists the SMB password** (it passes the credential to the agent's add
|
||||
> request, which writes the 0600 file). Endpoints `POST /api/storage/netstorage/{add,remove}` +
|
||||
> `GET /api/storage/netstorage` (merges the agent's per-share liveness: `ok` | `idle` (benign idle-unmount)
|
||||
> | `unreachable` | `unknown`). A network share registers as `/mnt/felhom-drives/<name>`, is **selectable as
|
||||
> a media app's `HDD_PATH`** (Schedulable), and shows in the "Hálózati tárhely (NAS)" settings section.
|
||||
> - **NOT a drive:** the drive lifecycle (eject/decommission/migrate/wipe/SMART) is **refused** on a network
|
||||
> path server-side (`refuseNetworkLifecycle`). The **drive-absent gate** (`planDriveGates`) and the
|
||||
> **missing-storage** surface **skip network paths**, so an `unreachable` NAS is a **recoverable warning**
|
||||
> (`networkStorageWarnings` → a distinct app-card badge), **never** the drive "missing → stopped" cascade.
|
||||
> - **Limits (v1):** a share's `+100000` uid mapping is fixed at add-time (one app / same-uid apps); for
|
||||
> write apps on a soft NFS mount, an in-flight file can truncate if the NAS vanishes mid-write (prefer
|
||||
> atomic-write apps / SSD-staging).
|
||||
|
||||
> **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now
|
||||
> visible in the guest at a STABLE path **`/mnt/felhom-drives/<name>`**, NOT the raw `/mnt/<name>`. The
|
||||
> agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's
|
||||
|
||||
Reference in New Issue
Block a user