hub v0.171.0 (unreleased): a set-aside-only backup digest mails as a warning that says nothing failed (R-1287); a test build is never the fleet's newest (R-1289). Red-proved. Commit only.
gates / gates (push) Successful in 13m21s
gates / gates (push) Successful in 13m21s
This commit is contained in:
1 parent
68ee667a83
commit
f63284861e
8 files changed
+163
-3
No files matched your search
@@ -311,7 +311,7 @@ stopping line that lies.
|
||||
| **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
|
||||
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | OPEN | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
|
||||
| **R-1287** | Monitoring & notifications | P3 | **A D47 set-aside mails the operator as „🔴 tester-1: 1 of 1 apps failed to back up (backup run)" — no backup failed; the body says the right thing („its backup belonged to an earlier install … SET ASIDE whole").** Seen 2026-10-25 on Tester 1 with controller 0.343.0 (attended release run): two such mails at 17:31:56Z and 17:36:55Z. The controller sends the set-aside as `backup_run_failures` leg `unit-set-aside`, `Failed: 1` (so the hub never drops a line that claims zero); the hub's subject for that event is fixed („N of M apps failed to back up", `hub/internal/notify/templates.go:353`). The CLAUDE.md corollary: an alarm's text must say what it counts. After the floor, every household reinstall mails the operator a red „failed to back up". | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): a set-aside-only digest is a warning with „… set aside — nothing failed (<apps>)"; a run with a real failure stays red (`hub/internal/notify/r1287_setaside_test.go`, red-proved) | — | two-repo: the hub words a `unit-set-aside` leg on its own („… a backup of an earlier install was set aside — nothing failed", warning, not error) with a test on the rendered subject; or a new event type through the allow-list (`hub-event-allowlist-gotcha`); one hub release | CC |
|
||||
|
||||
|
||||
## Hub & operator — 13 rows (P3 5, P4 8)
|
||||
@@ -330,7 +330,7 @@ stopping line that lies.
|
||||
| **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
|
||||
| **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
|
||||
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
|
||||
| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | OPEN | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC |
|
||||
| **R-1289** | Hub & operator | P4 | **The hub's „newest controller the fleet reported" can be a dev build that no box runs: `semver.Compare` returns 0 (equal) for any version with a suffix, so „0.343.0-dev.r1251a" — once read first — beats nothing and is never beaten.** Seen 2026-10-25 in the vouch line (attended release run): „golden 0.331.0 unchanged (behind the fleet's 0.343.0-dev.r1251a …)" while the four hosts run 0.343.0 / 0.342.0. `store.NewestReportedControllerVersion` scans every report ever stored (`hub/internal/store/store.go:2210`), and `hub/internal/semver/semver.go:14` gives up on `0-dev…` / `0-rc.N`. Used by the R-120 golden gate (`hub/internal/web/configs.go:1548,1554`): with a suffixed „newest" the gate compares against whatever row came first — today it only words a log line and fails open. | **VERIFY** — fixed on main 2026-10-25 night, waits for the hub release (v0.171.0): suffixed versions are skipped (`TestR1289_ATestBuildIsNeverTheFleetsNewest`, red-proved) | — | skip suffixed versions in `NewestReportedControllerVersion` (or compare with the pre-release rule the controller uses, `util.ParseVersionPre`) + a test with a dev and an rc row ahead of a release row (red first); one hub release | CC |
|
||||
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
Reference in new issue
Block a user