diff --git a/documentation/audits/night-2026-10-25/redproof-hub-r1287-r1289.txt b/documentation/audits/night-2026-10-25/redproof-hub-r1287-r1289.txt new file mode 100644 index 00000000..186c5697 --- /dev/null +++ b/documentation/audits/night-2026-10-25/redproof-hub-r1287-r1289.txt @@ -0,0 +1,11 @@ +# R-1287: the new tests against the old hub (dispatcher.go + templates.go stashed), 2026-10-25 +=== RUN TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure + r1287_setaside_test.go:47: a set-aside is dressed as a red backup failure: "[Felhom] 🔴 c1: 1 of 1 apps failed to back up (backup run)" +--- FAIL: TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure (0.04s) +=== RUN TestR1287_ARunWithARealFailureKeepsItsRedSubject +--- PASS: TestR1287_ARunWithARealFailureKeepsItsRedSubject (0.04s) + +# R-1289: the Valid check removed +=== RUN TestR1289_ATestBuildIsNeverTheFleetsNewest + floor_ui_test.go:264: NewestReportedControllerVersion() = "0.344.0-rc.1", want 0.343.0 — a test build is not a release +--- FAIL: TestR1289_ATestBuildIsNeverTheFleetsNewest (0.04s) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 21dd06b5..79a9e33a 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -311,7 +311,7 @@ stopping line that lies. | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC | | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC | | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC | -| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | OPEN | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC | +| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed ()"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC | ## Hub & operator — 13 rows (P3 5, P4 8) @@ -330,7 +330,7 @@ stopping line that lies. | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | -| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | OPEN | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC | +| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): suffixed versions are skipped (`TestR1289_ATestBuildIsNeverTheFleetsNewest`, red-proved) | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 29ab9037..be6c4ffe 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,16 @@ +## v0.171.0 (unreleased, 2026-10-25 night) — a set-aside mails as a set-aside (R-1287); a test build is never „the fleet's newest" (R-1289) — committed, NOT deployed + +- **R-1287** (`internal/notify`): a `backup_run_failures` digest whose every entry is leg `unit-set-aside` (a D47 + set-aside, controller ≥ 0.343.0) is sent as a **warning** with its own subject „⚠️ : a backup of an earlier + install was set aside — nothing failed ()" and a body that says where the copy is and when it is expected. A run + with any real failure keeps its red „N of M apps failed to back up". Works for boxes already on 0.343.0 (no + controller change). `r1287_setaside_test.go` on the real dispatch path; seen failing on the old code with the exact + subject measured on Tester 1 („🔴 … 1 of 1 apps failed to back up"). +- **R-1289** (`store.NewestReportedControllerVersion`): versions with a suffix (`-rc.N`, `-dev…`) are skipped — the + hub's semver compare cannot read them and called them equal to everything, so a dev build no box runs stayed „the + fleet's newest" in the R-120 golden gate. `TestR1289_ATestBuildIsNeverTheFleetsNewest` (red first). +- Red-proofs: `documentation/audits/night-2026-10-25/redproof-hub-r1287-r1289.txt`. + ## v0.170.0 (2026-10-25) — the escrow package in one transaction (R-1254) — DEPLOYED on the operator's yes 2026-10-25 15:50 UTC (image `sha256:6478498d…`; ArgoCD sync of the hub Deployment only); Tester 1's escrow ceremony stored its package in one transaction; no alarm mail in the watch after - **The new escrow key, its sealed-password hash and the identity blob are stored in ONE transaction** diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index ac8d78ad..2820a090 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -206,6 +206,14 @@ func (d *Dispatcher) processEvent(customerID, eventType, severity, message, mess return } + // R-1287: a D47 set-aside arrives as backup_run_failures (leg `unit-set-aside`, controller ≥ 0.343.0, error) — but + // nothing failed: a reinstall's earlier backup was kept whole. A run of only set-asides is a WARNING here, whatever + // the box sent, so it is never dressed as a red backup failure (renderBackupRunFailures words it). + if eventType == "backup_run_failures" && (severity == "error" || severity == "critical") && setAsideOnly(detailsJSON) { + d.logger.Printf("[INFO] %s backup_run_failures for %s holds only set-aside legs — sent as warning (R-1287)", severity, customerID) + severity = "warning" + } + // "test" bypass — send directly to customer email, skip prefs/cooldown if eventType == "test" { d.sendTestEmail(customerID) diff --git a/hub/internal/notify/r1287_setaside_test.go b/hub/internal/notify/r1287_setaside_test.go new file mode 100644 index 00000000..b30b03af --- /dev/null +++ b/hub/internal/notify/r1287_setaside_test.go @@ -0,0 +1,65 @@ +package notify + +import ( + "strings" + "testing" +) + +// R-1287: a D47 set-aside reaches the hub as backup_run_failures, leg `unit-set-aside`, severity error, Failed 1 — +// the payload controller 0.343.0 sends (cmd/controller/main.go SetUnitSetAsideNotify). Measured 2026-10-25 on Tester 1: +// two mails „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" while nothing failed. The consequence asserted: +// the operator's mail says set aside / nothing failed, as a warning; a run with a real failure keeps its red subject. +// RED-PROOF: drop the setAsideApps branch in renderBackupRunFailures and the severity override in processEvent → the +// subject is the red „failed to back up" and the record reads error. +const setAsidePayload = `{"run_id":"unit-set-aside-tandoor-20261025T173155Z","run_kind":"backup","failed":1,"attempted":1,` + + `"apps":[{"app":"tandoor","leg":"unit-set-aside","reason":"the backup of an earlier install was set aside whole (D47)"}]}` + +func r1287Dispatch(t *testing.T, details string) (subjects []string, severity string) { + t.Helper() + st := opOnlyStore(t) + d := opOnlyDispatcher(t, st, &sentTo{}) + d.sendEmailFn = func(to, subject, body string, headers map[string]string) error { + if to == "operator@felhom.eu" { + subjects = append(subjects, subject) + } + return nil + } + d.ProcessEvent("c1", "backup_run_failures", "error", `App "tandoor": its backup belonged to an earlier install`, details, "controller") + logs, err := st.GetRecentNotifications("c1", 10) + if err != nil { + t.Fatal(err) + } + for _, l := range logs { + if l.EventType == "backup_run_failures" { + severity = l.Severity + } + } + return subjects, severity +} + +func TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure(t *testing.T) { + subjects, sev := r1287Dispatch(t, setAsidePayload) + if len(subjects) != 1 { + t.Fatalf("the operator got %d mail(s), want 1: %v", len(subjects), subjects) + } + s := subjects[0] + if strings.Contains(s, "failed to back up") || strings.Contains(s, "🔴") { + t.Fatalf("a set-aside is dressed as a red backup failure: %q", s) + } + if !strings.Contains(s, "set aside — nothing failed") || !strings.Contains(s, "tandoor") || !strings.Contains(s, "⚠️") { + t.Fatalf("the subject must say set aside / nothing failed and name the app: %q", s) + } + if sev != "warning" { + t.Fatalf("the record reads %q, want warning", sev) + } +} + +func TestR1287_ARunWithARealFailureKeepsItsRedSubject(t *testing.T) { + mixed := `{"run_kind":"backup","failed":2,"attempted":5,"apps":[` + + `{"app":"tandoor","leg":"unit-set-aside","reason":"set aside (D47)"},` + + `{"app":"immich","leg":"recovery-unit","reason":"disk full"}]}` + subjects, sev := r1287Dispatch(t, mixed) + if len(subjects) != 1 || !strings.Contains(subjects[0], "2 of 5 apps failed to back up") || sev != "error" { + t.Fatalf("a run with a real failure must stay red and error: %v %q", subjects, sev) + } +} diff --git a/hub/internal/notify/templates.go b/hub/internal/notify/templates.go index 8ee48411..ef07c30b 100644 --- a/hub/internal/notify/templates.go +++ b/hub/internal/notify/templates.go @@ -350,6 +350,9 @@ func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bo if kind == "" { kind = "backup" } + if setAsideApps(d) != nil { + return renderSetAside(customerID, d) + } subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)", customerID, d.Failed, d.Attempted, kind) @@ -392,6 +395,50 @@ func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bo return b.String(), subject, true } +// legUnitSetAside is the controller's leg for a D47 set-aside (controller `cmd/controller/main.go`, ≥ 0.343.0). +const legUnitSetAside = "unit-set-aside" + +// setAsideApps returns the app names when EVERY entry of the digest is a set-aside, else nil (a mixed run is a failure +// run and keeps its red subject; the set-aside lines then read as what they are in the list). +func setAsideApps(d backupRunDetails) []string { + if len(d.Apps) == 0 { + return nil + } + var apps []string + for _, a := range d.Apps { + if a.Leg != legUnitSetAside { + return nil + } + apps = append(apps, a.App) + } + return apps +} + +// setAsideOnly is setAsideApps for the raw payload (the dispatcher's severity decision). +func setAsideOnly(detailsJSON string) bool { + var d backupRunDetails + if detailsJSON == "" || json.Unmarshal([]byte(detailsJSON), &d) != nil { + return false + } + return setAsideApps(d) != nil +} + +// renderSetAside words a set-aside-only digest (R-1287): nothing failed; an earlier install's backup was kept whole and +// is restorable from the restore page. The subject says so without being opened. +func renderSetAside(customerID string, d backupRunDetails) (string, string, bool) { + apps := setAsideApps(d) + subject := fmt.Sprintf("[Felhom] ⚠️ %s: a backup of an earlier install was set aside — nothing failed (%s)", + customerID, strings.Join(apps, ", ")) + var b strings.Builder + b.WriteString("\n\nSET ASIDE — nothing failed. A backup that belonged to an earlier install of the app was moved aside\n") + b.WriteString("whole (D47) and is listed on the box's restore page; the new install backs up into its own unit.\n") + b.WriteString("Expected after a reinstall. Not expected on an update — look if no one reinstalled the app.\n\n") + for _, a := range d.Apps { + fmt.Fprintf(&b, " %s %s\n", a.App, a.Reason) + } + return b.String(), subject, true +} + // trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app // reason, because the digest prints those figures ONCE for the whole run on its own line. // diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index 8ccc55bf..fe189503 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -2220,7 +2220,9 @@ func (s *Store) NewestReportedControllerVersion() string { continue } v = strings.TrimPrefix(strings.TrimSpace(v), "v") - if v == "" { + // R-1289: a test build (`0.343.0-rc.7`, `0.343.0-dev.r1251a`) is never „the fleet's newest": Compare cannot + // read the suffix and calls it EQUAL to everything, so the first such row read used to win for good. + if v == "" || !semver.Valid(v) { continue } if newest == "" || semver.Compare(v, newest) > 0 { diff --git a/hub/internal/web/floor_ui_test.go b/hub/internal/web/floor_ui_test.go index cb62bc57..59660049 100644 --- a/hub/internal/web/floor_ui_test.go +++ b/hub/internal/web/floor_ui_test.go @@ -250,3 +250,17 @@ func TestNewestReportedControllerIsSemverOrdered(t *testing.T) { t.Errorf("NewestReportedControllerVersion() = %q, want 0.186.0 — a lexical MAX() would return 0.99.0", got) } } + +// R-1289: a dev and an rc build reported AHEAD of the release (in row order and in number) never count — measured +// 2026-10-25: the vouch line read „behind the fleet's 0.343.0-dev.r1251a" while every box ran 0.343.0 or 0.342.0. +// RED-PROOF: drop the Valid check → the first suffixed row is returned. +func TestR1289_ATestBuildIsNeverTheFleetsNewest(t *testing.T) { + _, st := newTestServer(t) + seedReport(t, st, "a", "0.343.0-dev.r1251a") + seedReport(t, st, "b", "0.344.0-rc.1") + seedReport(t, st, "c", "0.342.0") + seedReport(t, st, "d", "0.343.0") + if got := st.NewestReportedControllerVersion(); got != "0.343.0" { + t.Errorf("NewestReportedControllerVersion() = %q, want 0.343.0 — a test build is not a release", got) + } +}