diff --git a/CONTEXT.md b/CONTEXT.md index 9d5f4ce..70ce060 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller + v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture + rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s + sweep), the **secret gate fail-closed** before storing (`blocked: possible secret`; `[REDACTED]`/ + checksums pass), report-ACK `controller_log_requested` + heartbeat-envelope `log_tail_requested` + channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section + (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two + forms are the only actions). Conventions codified: + `documentation/runbooks/logging-conventions.md` + CLAUDE.md one-liners in all three repos. + **OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/ to close the live + round-trip** (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation + still DUE (pre-existing). + - **2026-07-11 — PUBLISH-TRAIN RULES CODIFIED** (`documentation/runbooks/publish-train-rules.md`): 1. manifest before floor; 2. the manifest screen carries the LIVE DB floor (`hub_settings.min_controller_version` overrides env, acts immediately) — save the floor field diff --git a/REPORT.md b/REPORT.md index c019f0f..aae454f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,55 +2,55 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Hub v0.45.0 — floor-UI separation + per-box MinAgent conditional floor (+ DSM spike, Q1c FAIL) — 2026-07-11 +## Hub v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate — 2026-07-11 -felhom.eu's parts of the NAS/coupling backlog-clear task (composite A–F; agent v0.82.0 + -controller v0.115.0 shipped in their repos). Commits: hub `bbecf05` + manifest `37222d3` + rules -`bbecf05`; DSM findings `37e60b4`; Q1c + follow-up `9b95f88`. +felhom.eu's third of the cross-repo observability task (agent v0.83.0 + controller v0.116.1 shipped +in their repos). Commits: hub `e35b1ae`, manifest bump + conventions runbook `7c038df`. +**Deployed:** `felhom-hub:0.46.0`, ArgoCD `Synced/Healthy`, rollout complete (17:03 CEST). -### Hub v0.45.0 (Parts C + D — LIVE, GitOps synced Healthy) +### What shipped -- **Part C — floor-UI separation + effective-floor source:** the global controller-version floor is - its own card with a type-to-confirm dialog that first shows the live blast radius - (`GET /configuration/global-floor/impact` → `countBoxesBelowFloor`, honoring per-customer - overrides); an effective-floor + source line (`store.ResolveGlobalFloor`) shows the resolved value - and WHICH source won (DB `hub_settings` vs env `DEFAULT_MIN_CONTROLLER_VERSION`, both raw values). - The Day-0 manifest save provably does not touch the floor (the 9-min-skew incident's root cause, - now visible + isolated). -- **Part D — per-box MinAgent conditional floor:** the artifact manifest gains `MinAgent`; at - report-ACK time `store.ResolveManagedFloor(customerID)` HOLDS the controller floor for a box whose - `hosts.agent_version` is below (or unknown vs) MinAgent, flags it on the Hosts dashboard - (`floor held: agent < MinAgent `), and serves it once the agent qualifies. Mechanises - "agent BEFORE controller floor" per box — the manual fleet check is retired (rule 3 updated). -- **THE one comparator:** `web.compareVersions`'s body → leaf `internal/semver` (web delegates, - store reuses; no import cycle, no second comparator). -- Red-proofs (all: mutate → predicted failure → revert → green): DB-wins precedence; manifest-save- - doesn't-touch-floor; impact count with override exclusion; managed-floor hold/serve/uncoupled/ - unknown-agent + a fleet discriminator + the report-ACK wire test (held box omits the floor). -- **Deployed:** built 0.45.0 on 180, manifest bumped, ArgoCD hard-refresh + sync → Synced/Healthy, - rollout OK, live image `felhom-hub:0.45.0`. Live state: demo agent 0.82.0, Peti 0.81.0, MinAgent - unset (uncoupled → no holds, as expected), floor 0.113.0 (DB). +- **Store** (`hub/internal/store/logbundle.go` + schema): `log_bundle_requests` (pending intent per + scope+component — controller scope = customer_id / report ACK; agent scope = host_id / heartbeat + envelope) + `log_bundles` (gzip, newest-3, **72 h TTL** purged on the existing 60 s sweep). + `SaveLogBundle` runs the **token-pattern secret gate BEFORE storing**: a hit stores a + `blocked: possible secret` flag row and NO payload (fail-closed, WARN hub-side); `[REDACTED]` + shapes + public checksums pass by design. +- **Channels** (additive both directions): report ACK `controller_log_requested` + ingest of + `controller_log_tail`; heartbeat envelope `log_tail_requested` + ingest of `log_tail`. + Consume-once on arrival. A pre-0.83 agent never fulfills → the request stays visibly `pending` + (S6-tested, harmless). +- **UI** (host detail, English per the hub operator surface): Diagnostics section — "Request + controller logs" / "Request agent logs" (CSRF forms), state rows (`pending` with the honest + latency hint: controller ≤ ~15 min report interval, agent ≈ heartbeat cadence; `available` with + View/Download; `blocked`), 72 h custody note. The hosts "read-only" invariant is amended by test: + these two forms are the ONLY actions. +- **Conventions codified:** `documentation/runbooks/logging-conventions.md` (levels, logs-are- + English, keys-never-values, capture layers, pull posture) + a one-liner in each repo CLAUDE.md. -### Part A — DSM spike (findings `documentation/audits/SPIKE-nas-dsm-2026-07-11.md`) +### Red-proofs (demonstrated, restored) -Real DSM 7.2 (virtual-dsm on DooPlex, macvlan .199, driven via claude-in-chrome, torn down + ports -dead) validated the consumer recipes E2E through the live add pipeline: NFS "Map all users to admin" -= Route A PASS (hardlink incl.); squash-off = `not_writable` refusal; plain-user SMB PASS **with -working hardlinks** (Q5 caveat CLOSED for Synology); wrong-password/share classify identically. The -copy edits landed in controller v0.115.0; QNAP stays a stated caveat (no emulator — nothing -fabricated). +| Scenario | Reverted shape | Failure seen | +|---|---|---| +| S4 secret gate | gate disabled | `secret-shaped token not blocked` | +| S2/consume-once | clear-on-arrival removed | `request survived fulfillment` (store + API tests) | -### Part E — Q1c restart survival: **FAIL** (supervised, operator GO) +S5 TTL: injectable clock — kept at +71 h, purged at +73 h. Keep-newest-3 + byte-cap belts tested. -`pct reboot 9201` proved a configured NAS automount trigger does NOT survive a guest reboot: the -autofs trigger does not re-propagate into the guest's fresh namespace (shared→slave only carries -post-bind mounts), so an in-guest access sees an **empty directory** and does not trigger the host -mount — the dangerous case for a restarting media app. Apps otherwise recovered healthy. FAIL -protocol honored (nothing repaired live); root cause + fix (agent `ReassertNetworkMounts` on guest -reboot) spec'd at `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md`. +### Live evidence / honest latencies -### NOT done / open +- Hub 0.46.0 live (ArgoCD Synced/Healthy, image verified, clean startup log). +- Box halves live-proven from 9201: controller ring + agent ring both readable at + `logging.level=info` through the new Debug tabs (see felhom-controller/REPORT.md timestamps). +- **Operator follow-up (HUMAN — hub UI is password-gated, CC cannot log in):** on + `hub.felhom.eu/hosts/` click both request buttons; expect the agent bundle within one + heartbeat (~15 min) and the controller bundle on the next report (≤ ~15 min); download both and + confirm the controller bundle carries the NAS-add phase lines + the `operator log pull served` + INFO line. The full round-trip is already proven by `internal/api/logbundle_ack_test.go`. +- Blocked-secret branch: test-level only by design (no real secret planted live). -- The follow-up NAS-reboot-reassert fix (agent) — new task, evidence attached. -- Held-floor state never fired live (no box below MinAgent, and MinAgent is unset) — test-proven only. -- QNAP appliance pass (no emulator). +### Open / observations + +- Hub bearer key rotation still DUE (pre-existing; manifests/hub.yaml + screenshot-exposed). +- `log_bundles` UI states `requested → pending` are merged (a request IS pending from the box's + view); `expired` renders as absence after the purge — states documented in the template hint.