hub v0.171.0 (unreleased): a set-aside-only backup digest mails as a warning that says nothing failed (R-1287); a test build is never the fleet's newest (R-1289). Red-proved. Commit only.
gates / gates (push) Successful in 13m21s
gates / gates (push) Successful in 13m21s
This commit is contained in:
admin committed
1 parent
68ee667a83
commit
f63284861e
8 files changed
+163
-3
No files matched your search
@@ -0,0 +1,11 @@
|
||||
# R-1287: the new tests against the old hub (dispatcher.go + templates.go stashed), 2026-10-25
|
||||
=== RUN TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure
|
||||
r1287_setaside_test.go:47: a set-aside is dressed as a red backup failure: "[Felhom] 🔴 c1: 1 of 1 apps failed to back up (backup run)"
|
||||
--- FAIL: TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure (0.04s)
|
||||
=== RUN TestR1287_ARunWithARealFailureKeepsItsRedSubject
|
||||
--- PASS: TestR1287_ARunWithARealFailureKeepsItsRedSubject (0.04s)
|
||||
|
||||
# R-1289: the Valid check removed
|
||||
=== RUN TestR1289_ATestBuildIsNeverTheFleetsNewest
|
||||
floor_ui_test.go:264: NewestReportedControllerVersion() = "0.344.0-rc.1", want 0.343.0 — a test build is not a release
|
||||
--- FAIL: TestR1289_ATestBuildIsNeverTheFleetsNewest (0.04s)
|
||||
@@ -311,7 +311,7 @@ stopping line that lies.
|
||||
| **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
|
||||
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | OPEN | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
|
||||
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed (<apps>)"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
|
||||
|
||||
|
||||
## Hub & operator — 13 rows (P3 5, P4 8)
|
||||
@@ -330,7 +330,7 @@ stopping line that lies.
|
||||
| **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
|
||||
| **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
|
||||
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
|
||||
| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | OPEN | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC |
|
||||
| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): suffixed versions are skipped (`TestR1289_ATestBuildIsNeverTheFleetsNewest`, red-proved) | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC |
|
||||
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
@@ -1,3 +1,16 @@
|
||||
## v0.171.0 (unreleased, 2026-10-25 night) — a set-aside mails as a set-aside (R-1287); a test build is never „the fleet's newest" (R-1289) — committed, NOT deployed
|
||||
|
||||
- **R-1287** (`internal/notify`): a `backup_run_failures` digest whose every entry is leg `unit-set-aside` (a D47
|
||||
set-aside, controller ≥ 0.343.0) is sent as a **warning** with its own subject „⚠️ <customer>: a backup of an earlier
|
||||
install was set aside — nothing failed (<apps>)" and a body that says where the copy is and when it is expected. A run
|
||||
with any real failure keeps its red „N of M apps failed to back up". Works for boxes already on 0.343.0 (no
|
||||
controller change). `r1287_setaside_test.go` on the real dispatch path; seen failing on the old code with the exact
|
||||
subject measured on Tester 1 („🔴 … 1 of 1 apps failed to back up").
|
||||
- **R-1289** (`store.NewestReportedControllerVersion`): versions with a suffix (`-rc.N`, `-dev…`) are skipped — the
|
||||
hub's semver compare cannot read them and called them equal to everything, so a dev build no box runs stayed „the
|
||||
fleet's newest" in the R-120 golden gate. `TestR1289_ATestBuildIsNeverTheFleetsNewest` (red first).
|
||||
- Red-proofs: `documentation/audits/night-2026-10-25/redproof-hub-r1287-r1289.txt`.
|
||||
|
||||
## v0.170.0 (2026-10-25) — the escrow package in one transaction (R-1254) — DEPLOYED on the operator's yes 2026-10-25 15:50 UTC (image `sha256:6478498d…`; ArgoCD sync of the hub Deployment only); Tester 1's escrow ceremony stored its package in one transaction; no alarm mail in the watch after
|
||||
|
||||
- **The new escrow key, its sealed-password hash and the identity blob are stored in ONE transaction**
|
||||
|
||||
@@ -206,6 +206,14 @@ func (d *Dispatcher) processEvent(customerID, eventType, severity, message, mess
|
||||
return
|
||||
}
|
||||
|
||||
// R-1287: a D47 set-aside arrives as backup_run_failures (leg `unit-set-aside`, controller ≥ 0.343.0, error) — but
|
||||
// nothing failed: a reinstall's earlier backup was kept whole. A run of only set-asides is a WARNING here, whatever
|
||||
// the box sent, so it is never dressed as a red backup failure (renderBackupRunFailures words it).
|
||||
if eventType == "backup_run_failures" && (severity == "error" || severity == "critical") && setAsideOnly(detailsJSON) {
|
||||
d.logger.Printf("[INFO] %s backup_run_failures for %s holds only set-aside legs — sent as warning (R-1287)", severity, customerID)
|
||||
severity = "warning"
|
||||
}
|
||||
|
||||
// "test" bypass — send directly to customer email, skip prefs/cooldown
|
||||
if eventType == "test" {
|
||||
d.sendTestEmail(customerID)
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1287: a D47 set-aside reaches the hub as backup_run_failures, leg `unit-set-aside`, severity error, Failed 1 —
|
||||
// the payload controller 0.343.0 sends (cmd/controller/main.go SetUnitSetAsideNotify). Measured 2026-10-25 on Tester 1:
|
||||
// two mails „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" while nothing failed. The consequence asserted:
|
||||
// the operator's mail says set aside / nothing failed, as a warning; a run with a real failure keeps its red subject.
|
||||
// RED-PROOF: drop the setAsideApps branch in renderBackupRunFailures and the severity override in processEvent → the
|
||||
// subject is the red „failed to back up" and the record reads error.
|
||||
const setAsidePayload = `{"run_id":"unit-set-aside-tandoor-20261025T173155Z","run_kind":"backup","failed":1,"attempted":1,` +
|
||||
`"apps":[{"app":"tandoor","leg":"unit-set-aside","reason":"the backup of an earlier install was set aside whole (D47)"}]}`
|
||||
|
||||
func r1287Dispatch(t *testing.T, details string) (subjects []string, severity string) {
|
||||
t.Helper()
|
||||
st := opOnlyStore(t)
|
||||
d := opOnlyDispatcher(t, st, &sentTo{})
|
||||
d.sendEmailFn = func(to, subject, body string, headers map[string]string) error {
|
||||
if to == "operator@felhom.eu" {
|
||||
subjects = append(subjects, subject)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
d.ProcessEvent("c1", "backup_run_failures", "error", `App "tandoor": its backup belonged to an earlier install`, details, "controller")
|
||||
logs, err := st.GetRecentNotifications("c1", 10)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, l := range logs {
|
||||
if l.EventType == "backup_run_failures" {
|
||||
severity = l.Severity
|
||||
}
|
||||
}
|
||||
return subjects, severity
|
||||
}
|
||||
|
||||
func TestR1287_ASetAsideMailsAsSetAsideNotAsAFailure(t *testing.T) {
|
||||
subjects, sev := r1287Dispatch(t, setAsidePayload)
|
||||
if len(subjects) != 1 {
|
||||
t.Fatalf("the operator got %d mail(s), want 1: %v", len(subjects), subjects)
|
||||
}
|
||||
s := subjects[0]
|
||||
if strings.Contains(s, "failed to back up") || strings.Contains(s, "🔴") {
|
||||
t.Fatalf("a set-aside is dressed as a red backup failure: %q", s)
|
||||
}
|
||||
if !strings.Contains(s, "set aside — nothing failed") || !strings.Contains(s, "tandoor") || !strings.Contains(s, "⚠️") {
|
||||
t.Fatalf("the subject must say set aside / nothing failed and name the app: %q", s)
|
||||
}
|
||||
if sev != "warning" {
|
||||
t.Fatalf("the record reads %q, want warning", sev)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1287_ARunWithARealFailureKeepsItsRedSubject(t *testing.T) {
|
||||
mixed := `{"run_kind":"backup","failed":2,"attempted":5,"apps":[` +
|
||||
`{"app":"tandoor","leg":"unit-set-aside","reason":"set aside (D47)"},` +
|
||||
`{"app":"immich","leg":"recovery-unit","reason":"disk full"}]}`
|
||||
subjects, sev := r1287Dispatch(t, mixed)
|
||||
if len(subjects) != 1 || !strings.Contains(subjects[0], "2 of 5 apps failed to back up") || sev != "error" {
|
||||
t.Fatalf("a run with a real failure must stay red and error: %v %q", subjects, sev)
|
||||
}
|
||||
}
|
||||
@@ -350,6 +350,9 @@ func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bo
|
||||
if kind == "" {
|
||||
kind = "backup"
|
||||
}
|
||||
if setAsideApps(d) != nil {
|
||||
return renderSetAside(customerID, d)
|
||||
}
|
||||
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
|
||||
customerID, d.Failed, d.Attempted, kind)
|
||||
|
||||
@@ -392,6 +395,50 @@ func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bo
|
||||
return b.String(), subject, true
|
||||
}
|
||||
|
||||
// legUnitSetAside is the controller's leg for a D47 set-aside (controller `cmd/controller/main.go`, ≥ 0.343.0).
|
||||
const legUnitSetAside = "unit-set-aside"
|
||||
|
||||
// setAsideApps returns the app names when EVERY entry of the digest is a set-aside, else nil (a mixed run is a failure
|
||||
// run and keeps its red subject; the set-aside lines then read as what they are in the list).
|
||||
func setAsideApps(d backupRunDetails) []string {
|
||||
if len(d.Apps) == 0 {
|
||||
return nil
|
||||
}
|
||||
var apps []string
|
||||
for _, a := range d.Apps {
|
||||
if a.Leg != legUnitSetAside {
|
||||
return nil
|
||||
}
|
||||
apps = append(apps, a.App)
|
||||
}
|
||||
return apps
|
||||
}
|
||||
|
||||
// setAsideOnly is setAsideApps for the raw payload (the dispatcher's severity decision).
|
||||
func setAsideOnly(detailsJSON string) bool {
|
||||
var d backupRunDetails
|
||||
if detailsJSON == "" || json.Unmarshal([]byte(detailsJSON), &d) != nil {
|
||||
return false
|
||||
}
|
||||
return setAsideApps(d) != nil
|
||||
}
|
||||
|
||||
// renderSetAside words a set-aside-only digest (R-1287): nothing failed; an earlier install's backup was kept whole and
|
||||
// is restorable from the restore page. The subject says so without being opened.
|
||||
func renderSetAside(customerID string, d backupRunDetails) (string, string, bool) {
|
||||
apps := setAsideApps(d)
|
||||
subject := fmt.Sprintf("[Felhom] ⚠️ %s: a backup of an earlier install was set aside — nothing failed (%s)",
|
||||
customerID, strings.Join(apps, ", "))
|
||||
var b strings.Builder
|
||||
b.WriteString("\n\nSET ASIDE — nothing failed. A backup that belonged to an earlier install of the app was moved aside\n")
|
||||
b.WriteString("whole (D47) and is listed on the box's restore page; the new install backs up into its own unit.\n")
|
||||
b.WriteString("Expected after a reinstall. Not expected on an update — look if no one reinstalled the app.\n\n")
|
||||
for _, a := range d.Apps {
|
||||
fmt.Fprintf(&b, " %s %s\n", a.App, a.Reason)
|
||||
}
|
||||
return b.String(), subject, true
|
||||
}
|
||||
|
||||
// trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
|
||||
// reason, because the digest prints those figures ONCE for the whole run on its own line.
|
||||
//
|
||||
|
||||
@@ -2220,7 +2220,9 @@ func (s *Store) NewestReportedControllerVersion() string {
|
||||
continue
|
||||
}
|
||||
v = strings.TrimPrefix(strings.TrimSpace(v), "v")
|
||||
if v == "" {
|
||||
// R-1289: a test build (`0.343.0-rc.7`, `0.343.0-dev.r1251a`) is never „the fleet's newest": Compare cannot
|
||||
// read the suffix and calls it EQUAL to everything, so the first such row read used to win for good.
|
||||
if v == "" || !semver.Valid(v) {
|
||||
continue
|
||||
}
|
||||
if newest == "" || semver.Compare(v, newest) > 0 {
|
||||
|
||||
@@ -250,3 +250,17 @@ func TestNewestReportedControllerIsSemverOrdered(t *testing.T) {
|
||||
t.Errorf("NewestReportedControllerVersion() = %q, want 0.186.0 — a lexical MAX() would return 0.99.0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1289: a dev and an rc build reported AHEAD of the release (in row order and in number) never count — measured
|
||||
// 2026-10-25: the vouch line read „behind the fleet's 0.343.0-dev.r1251a" while every box ran 0.343.0 or 0.342.0.
|
||||
// RED-PROOF: drop the Valid check → the first suffixed row is returned.
|
||||
func TestR1289_ATestBuildIsNeverTheFleetsNewest(t *testing.T) {
|
||||
_, st := newTestServer(t)
|
||||
seedReport(t, st, "a", "0.343.0-dev.r1251a")
|
||||
seedReport(t, st, "b", "0.344.0-rc.1")
|
||||
seedReport(t, st, "c", "0.342.0")
|
||||
seedReport(t, st, "d", "0.343.0")
|
||||
if got := st.NewestReportedControllerVersion(); got != "0.343.0" {
|
||||
t.Errorf("NewestReportedControllerVersion() = %q, want 0.343.0 — a test build is not a release", got)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user