From cbba20aa732b43e939bee1e64804270beb592859 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 29 Jun 2026 22:01:57 +0200 Subject: [PATCH] docs: felhom.eu REPORT (hub F2 + install-script preservation) + TESTRUN F2-closed Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ --- REPORT.md | 73 ++++++++++--------- .../audits/TESTRUN-fullstack-2026-06-29.md | 15 ++++ 2 files changed, 53 insertions(+), 35 deletions(-) diff --git a/REPORT.md b/REPORT.md index 3b4415d..e644636 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,43 +2,46 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md). -## Agent capability-degraded operator alert — hub leg (HostCapabilityChecker), hub v0.19.0 +## F2 (hub checkers) + prevention (host-install leaf preservation) — hub v0.21.0 + install script -**Date:** 2026-06-29 · **Baseline:** hub `main` @ `1958204` (v0.18.0) → **v0.19.0** @ `b7b165b`; -manifest synced @ `2312572`. Companion: felhom-agent **v0.44.0**. +**Date:** 2026-06-29. Companions: felhom-controller **v0.91.0** (Part A controller), felhom-agent +**v0.46.0** (Part B.1). This repo carries **Part A (hub checkers, F2)** + **Part B.2 (install script)**. -### What & why -felhom-agent v0.44.0 now rides a `capabilities` snapshot on its host report (each required `sudo -n` -grant: ok/degraded). This is the hub leg: detect the ok→degraded transition and alert the **operator** -— closing the loop that let the 2026-06-28 non-root-cutover regressions go undetected until breakage. +### Part A — hub F2 (`internal/monitor/host_capability.go`, `host_staleness.go`), hub v0.21.0 +A host already **degraded** (capability) or **stale/down** (staleness) when the hub (re)started was +seeded silently and **never alerted**. Fix: the constructors now seed only HEALTHY hosts; an unhealthy +host is left **unseeded** so the first `Check()` emits once (`emitTransition(…, "unknown", newState, +…)`). The dispatcher's 1 h operator cooldown dedups the re-emit across a hub bounce. Healthy first-obs +still silent. +- Tests: born-degraded **red-proof** (degraded at construction → unseeded → one + `agent_capability_degraded` on first Check, no dup next); the staleness test updated to F2 (born-stale + → unseeded → one `host_stale`). Build/vet/test green. +- **Live:** deployed v0.21.0; startup logs confirm the change — + `Host capability checker initialized: 1 ok, 0 degraded (degraded left unseeded → first Check emits)` + and the staleness sibling likewise. -### Implementation -- **`monitor.HostCapabilityChecker` (NEW):** a deliberate sibling of `HostStalenessChecker` — same - per-host state map, seed-without-event, emit-only-on-transition. A host is `degraded` iff its latest - report has any **Critical** capability `status:"degraded"`; non-critical degradations never alert. - Runs on the existing 60s sweep alongside the staleness checkers. -- **Events:** `agent_capability_degraded` (warning, ok→degraded; names the degraded caps + gated - features in message + details JSON), `agent_capability_recovered` (info, degraded→ok). Routed via - the existing `Dispatcher.ProcessEvent` — **operator-only** (not a customer notification toggle, same - mechanism as `host_stale`) with the standard 1 h operator cooldown. -- **`store.GetHostCapabilities` (NEW):** reads the snapshot from the latest host-report's `report_json` - per host, keyed on `MAX(id)` (not `received_at` — within-second ties would return multiple rows). - **No schema migration.** A pre-v0.44.0 agent (no `capabilities`) reads as `ok` → no false alert. -- Cross-repo `host-report.golden.json` mirrors `capabilities: []` (byte-identical with the agent copy). +### Part B.2 — `scripts/felhom-host-install.sh` (prevention) +- **`--preserve-state-from DIR`:** before the first agent start, copy `local-api.{crt,key}` + + `local-tokens.log` into `/var/lib/felhom-agent`, chown `felhom-agent`, modes 644/600/600, and + **validate the leaf parses** (`openssl x509 -noout`, else refuse). Keeps the pinned fingerprint + **stable** across a reinstall — no controller re-bootstrap. Distinct from `--preserve-from` (config + merge only). +- **Populated-host guard:** if the state dir has **no** leaf AND the host already runs guests AND + neither `--preserve-state-from` nor `--allow-new-leaf` was given → **refuse** with a clear message + (converts the silent 2026-06-28 footgun into a hard stop). A plain re-run on an intact host (leaf + present) is unaffected. `bash -n` clean. +- **Live logic validation (isolated, non-destructive — full installer not run):** on felhom-pve with + real openssl/pct — preserve copies with correct modes and the **fingerprint is preserved** across the + copy (`60b5974d…` source == dest); an unparseable leaf is rejected; the guard returns REFUSE for + leaf-absent + guests-present; the live agent state dir was untouched. -### Tests (all green: `go build/vet/test ./...`) -`TestHostCapabilityChecker` (seed no-event → ok→degraded ONE event → steady degraded none → -degraded→ok recovered), `…_NonCriticalIgnored`, `…_OldAgentNoCaps`. The golden contract test + -key-set tests pass with the mirrored field. +### NOT yet run — Part B live reinstall (B.4) — **awaiting supervised** +A full `felhom-host-install.sh` reinstall on felhom-pve (with `--preserve-state-from` → preserved fp, +channel stays up; without it on the populated host → guard refuses) is the operator's supervised call. +The agent B.1 loud-regenerate and the preserve/guard logic are proven; only the end-to-end reinstall +is deferred. -### Live validation (k3s felhom-system) — deployed + verified -- Built + pushed `felhom-hub:0.19.0` on 180; bumped `manifests/hub.yaml`; ArgoCD hard-refresh → sync → - rollout. **Synced / Healthy**, running image `…felhom-hub:0.19.0`. -- Startup: `felhom-hub 0.19.0 starting` + `Host capability checker initialized: 1 ok, 0 degraded` — - the checker is live and reading the demo host's snapshot (agent reports 40/40 ok). - -### NOT yet live-validated (deferred) -The full live agent→hub→operator-email degraded transition: the hub alert fires only on a **Critical** -capability degraded, which on the live host would require removing a Critical grant (forbidden — would -re-break drives) or a separate test host, plus a real operator email. **Fully covered by the unit -tests above.** Deferred to a scratch-host exercise. +### Testrun residual fixed +The full-stack testrun's guest reboot had left guest 9201's `bootstrap.json` a placeholder (redeploy +blocked); restored the real file from the R1 aside copy (customer.id=demo-felhom) → the bootstrap +redeploy path works again (controller 0.91.0 deployed through it). diff --git a/documentation/audits/TESTRUN-fullstack-2026-06-29.md b/documentation/audits/TESTRUN-fullstack-2026-06-29.md index 6df1301..3127925 100644 --- a/documentation/audits/TESTRUN-fullstack-2026-06-29.md +++ b/documentation/audits/TESTRUN-fullstack-2026-06-29.md @@ -156,6 +156,21 @@ next fresh boot). --- +## F2 CLOSED (2026-06-29) + prevention shipped + +The testrun surfaced **F2**: a channel/host failure present at startup/reseed alerted only on a live +transition, so a born-down (e.g. a controller booting into pin_mismatch) was dashboard-only, no +operator email. **Closed** — controller **v0.91.0** (`alerted`-flag rework) + hub **v0.21.0** (checkers +seed only healthy; born-degraded/stale emits on first Check, cooldown dedups). Live-validated: a +controller restarted into a regenerated-leaf channel logged `[channel] agent channel DOWN +(unseeded->down:pin_mismatch)` and **pushed `agent_channel_pin_mismatch`** (~60 s) — the capstone's +missing half (detection at startup). **Prevention** for the original incident also shipped: agent +**v0.46.0** loud-WARNs a regenerated leaf (`leaf REGENERATED — pins now INVALID`), and +`felhom-host-install.sh` gained `--preserve-state-from` (keeps the leaf/pin stable across a reinstall) ++ a populated-host guard (refuse to regenerate the leaf on a host with guests). The §K prevention +recommendation is now implemented; the full live reinstall is the only piece awaiting a supervised run. +(F1 swap-rollback `RestartCount` hardening remains backlog.) + ## Final state — known-good agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45