hub v0.162.0 (built, not deployed): no repeat mail after a restart for a known-down host or an already-reported stale restore test (R-1001); the Day-0 agent list leaves out packages without the felhom-agent file, a vouch of one is refused (R-1011); an absent tunnel reads 'none set up' (R-1070)
gates / gates (push) Successful in 7m20s
gates / gates (push) Successful in 7m20s
This commit is contained in:
@@ -1,3 +1,32 @@
|
||||
## v0.162.0 — a hub restart no longer re-mails a known-down box (R-1001); the Day-0 agent list offers only real agent packages (R-1011); a box with no tunnel reads „none set up" (R-1070) (2026-10-19)
|
||||
|
||||
Built, NOT deployed (the night brief of 2026-10-19: the hub is never deployed at night; the deploy is the operator's
|
||||
word in the morning).
|
||||
|
||||
- **No repeat mail after a restart (R-1001).** Measured 2026-10-13: every hub restart re-mailed `host_down` and
|
||||
`restore_test_stale` for Tester 2, silent for days — the staleness checker left a down host unseeded so the first
|
||||
sweep emits, and its comment said the dispatcher's 1-hour cooldown would swallow the repeat (host liveness events
|
||||
bypass it); the restore-test edge lived in memory. Now: at start a host whose newest `host_stale`/`host_down` event
|
||||
is newer than its last report is seeded with that state silently (`store.LatestHostLivenessEvent`, the „Host <id>:"
|
||||
messages); a seeded „stale" still escalates to `host_down`; a host that reports again recovers once and its next
|
||||
silence alarms. `restore_test_stale` reads its edge from the DB on the first sweep after a start: a stale signal for
|
||||
the tier inside the read window and newer than the tier's last proof means the run was reported. A failed lookup, or
|
||||
a signal older than the window, lets the alarm through. Tests `TestHostStaleness_R1001_RestartDoesNotReMail`,
|
||||
`TestHostStaleness_R1001_NewSilenceStillMails`, `TestRestoreTestStale_R1001_RestartDoesNotReMail` — each restart is
|
||||
a NEW checker over the SAME database; red-proved (seeding removed → `the restarted hub re-sent [host_down]`; the DB
|
||||
check removed → `re-sent 2 restore_test_stale signal(s)`).
|
||||
- **The Day-0 agent list (R-1011).** `gitea.FileSHA256` with a file named now answers exactly that file: a version
|
||||
without it is `ErrNoFile` (it wraps `ErrNoSHA`), never the first file's sha. The agent dropdown leaves out the step
|
||||
bundles (`0.156.1-step1` … hold only `felhom-config-bundle.json`; read from the registry 2026-10-19: 6 such
|
||||
versions, every release version holds `felhom-agent`, every golden `golden.tar.zst`), one INFO line per resolve; a
|
||||
vouch of one is refused with the new flash `artifact_file_missing`. Tests `TestFileSHA256_R1011_NeverAnotherFilesSHA`,
|
||||
`TestArtifactChoices_R1011_StepBundleIsNotAnAgentVersion`, a row in `TestArtifactSave_FlashNamesTheCause`;
|
||||
red-proved (the old fallback → `the list offers 0.156.1-step1 (sha "bundle-…")`). Two test fakes named the golden's
|
||||
file after its package; they now serve `golden.tar.zst`.
|
||||
- **The agent's new tunnel state `absent`** (R-1070, agent Unreleased): the System page shows „none set up" (plain, no
|
||||
amber) for a box with no cloudflared container; `not_running` stays red, other states amber. The tunnel alarm counts
|
||||
`not_running` only, unchanged. Test `TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.
|
||||
|
||||
## v0.161.0 — the passphrase gives keys only until the box uses its own (R-1159, operator decision D19); an alarm for an ep0 household with no prune job (R-1165) (2026-10-19)
|
||||
|
||||
- **The retrieval passphrase's keys** (R-1159, D19 = „box key first, then lock"; `05` §8 [DESIGN R-1159]). Measured the
|
||||
|
||||
@@ -771,7 +771,7 @@ type hostReportPayload struct {
|
||||
Tiers []hostForeignKeyArchives `json:"tiers"`
|
||||
} `json:"foreign_key_archives"`
|
||||
Cloudflared struct {
|
||||
Status string `json:"status"` // agent >= 0.141.0: running | not_running | unknown (older: active | inactive | …)
|
||||
Status string `json:"status"` // agent >= 0.141.0: running | not_running | unknown; absent since R-1070 (never alarms) (older: active | inactive | …)
|
||||
Detail string `json:"detail,omitempty"`
|
||||
} `json:"cloudflared"`
|
||||
// DR recipe — the agent's storage/guest/PBS half (secret-free). RawMessage = stored verbatim,
|
||||
|
||||
@@ -70,6 +70,11 @@ func IsNotFound(err error) bool {
|
||||
// ErrNoSHA marks a version whose metadata answered but carried no usable sha256 (R-292).
|
||||
var ErrNoSHA = errors.New("registry listed no sha256")
|
||||
|
||||
// ErrNoFile marks a version that has files but not the one asked for (R-1011, hub 0.162.0): an agent step bundle
|
||||
// (`0.156.1-step1`) holds only felhom-config-bundle.json. It wraps ErrNoSHA, so a caller that only knows ErrNoSHA
|
||||
// still refuses.
|
||||
var ErrNoFile = fmt.Errorf("the package version holds no such file: %w", ErrNoSHA)
|
||||
|
||||
type pkgEntry struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
@@ -100,9 +105,11 @@ type pkgFile struct {
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
// FileSHA256 returns the sha256 of a file inside a generic package version. When the version holds
|
||||
// multiple files it prefers an exact match to preferredFile (e.g. "golden.tar.zst"); otherwise it uses
|
||||
// the first file. This is a cheap metadata call — the artifact bytes are never downloaded.
|
||||
// FileSHA256 returns the sha256 of a file inside a generic package version. With preferredFile set it is EXACTLY that
|
||||
// file: a version without it is ErrNoFile, never another file's sha (R-1011 — the Day-0 list offered agent step
|
||||
// bundles, whose only file is the config bundle, and showed that file's sha as the agent's). Only an empty
|
||||
// preferredFile takes the first file. This is a cheap metadata call — the artifact bytes are never downloaded.
|
||||
// Pinned by TestFileSHA256_R1011_NeverAnotherFilesSHA.
|
||||
func (c *Client) FileSHA256(ctx context.Context, pkgName, version, preferredFile string) (string, error) {
|
||||
u := fmt.Sprintf("%s/api/v1/packages/%s/generic/%s/%s/files",
|
||||
c.baseURL, c.owner, pkgName, version)
|
||||
@@ -114,11 +121,17 @@ func (c *Client) FileSHA256(ctx context.Context, pkgName, version, preferredFile
|
||||
return "", fmt.Errorf("no files listed for %s/%s: %w", pkgName, version, ErrNoSHA)
|
||||
}
|
||||
if preferredFile != "" {
|
||||
names := make([]string, 0, len(files))
|
||||
for _, f := range files {
|
||||
if f.Name == preferredFile && f.SHA256 != "" {
|
||||
if f.Name == preferredFile {
|
||||
if f.SHA256 == "" {
|
||||
return "", fmt.Errorf("no sha256 for %s/%s file %q: %w", pkgName, version, f.Name, ErrNoSHA)
|
||||
}
|
||||
return f.SHA256, nil
|
||||
}
|
||||
names = append(names, f.Name)
|
||||
}
|
||||
return "", fmt.Errorf("%s/%s has no file %q (it holds %s): %w", pkgName, version, preferredFile, strings.Join(names, ", "), ErrNoFile)
|
||||
}
|
||||
if files[0].SHA256 == "" {
|
||||
return "", fmt.Errorf("no sha256 for %s/%s file %q: %w", pkgName, version, files[0].Name, ErrNoSHA)
|
||||
|
||||
@@ -2,6 +2,7 @@ package gitea
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
@@ -86,3 +87,44 @@ func TestGetJSON_ErrorOnNon200(t *testing.T) {
|
||||
t.Fatal("expected an error on HTTP 500")
|
||||
}
|
||||
}
|
||||
|
||||
// R-1011: with a file named, a version without it is ErrNoFile (which is also ErrNoSHA) — never the first file's sha.
|
||||
// Seen 2026-10-14: the Day-0 list showed agent `0.156.1-step1` (only felhom-config-bundle.json) with the bundle's sha.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix fallback to files[0], it fails with
|
||||
//
|
||||
// gitea_test.go: step bundle: sha "bbbb" err <nil>, want ErrNoFile and no sha
|
||||
//
|
||||
// Restored.
|
||||
func TestFileSHA256_R1011_NeverAnotherFilesSHA(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case strings.Contains(r.URL.Path, "/felhom-agent/0.156.1-step1/files"):
|
||||
w.Write([]byte(`[{"name":"felhom-config-bundle.json","sha256":"bbbb"}]`))
|
||||
case strings.Contains(r.URL.Path, "/felhom-agent/0.156.1/files"):
|
||||
w.Write([]byte(`[{"name":"felhom-config-bundle.json","sha256":"bbbb"},{"name":"felhom-agent","sha256":"aaaa"}]`))
|
||||
case strings.Contains(r.URL.Path, "/felhom-agent/0.1.0/files"):
|
||||
w.Write([]byte(`[{"name":"felhom-agent","sha256":""}]`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
c := New(srv.URL, "admin", "u", "t")
|
||||
ctx := context.Background()
|
||||
|
||||
sha, err := c.FileSHA256(ctx, "felhom-agent", "0.156.1-step1", "felhom-agent")
|
||||
if sha != "" || !errors.Is(err, ErrNoFile) || !errors.Is(err, ErrNoSHA) {
|
||||
t.Fatalf("step bundle: sha %q err %v, want ErrNoFile and no sha", sha, err)
|
||||
}
|
||||
if sha, err = c.FileSHA256(ctx, "felhom-agent", "0.156.1", "felhom-agent"); err != nil || sha != "aaaa" {
|
||||
t.Fatalf("the release beside its bundle: sha %q err %v, want the binary's aaaa", sha, err)
|
||||
}
|
||||
if sha, err = c.FileSHA256(ctx, "felhom-agent", "0.1.0", "felhom-agent"); sha != "" || !errors.Is(err, ErrNoSHA) || errors.Is(err, ErrNoFile) {
|
||||
t.Fatalf("the file without a sha: sha %q err %v, want ErrNoSHA (not ErrNoFile)", sha, err)
|
||||
}
|
||||
if sha, err = c.FileSHA256(ctx, "felhom-agent", "0.156.1-step1", ""); err != nil || sha != "bbbb" {
|
||||
t.Fatalf("no file named: the first file stays the answer, got %q %v", sha, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -49,16 +49,31 @@ func NewHostStalenessChecker(s *store.Store, threshold time.Duration, onEvent Ev
|
||||
logger.Printf("[WARN] Host staleness checker: failed to seed states: %v", err)
|
||||
return sc
|
||||
}
|
||||
var okCount, staleCount, downCount int
|
||||
var okCount, staleCount, downCount, knownCount int
|
||||
for _, row := range rows {
|
||||
if s.IsCustomerBlocked(row.CustomerID) {
|
||||
continue
|
||||
}
|
||||
sc.customerOf[row.HostID] = row.CustomerID
|
||||
age := time.Since(row.LastReportAt)
|
||||
// F2: seed only HEALTHY hosts. A host already stale/down at (re)start is left UNSEEDED so the
|
||||
// first Check() observes oldState=="" and emits once (a box that went silent while the hub was
|
||||
// down would otherwise stay silent). The dispatcher's 1h cooldown dedups the re-emit.
|
||||
// F2: a host already stale/down at (re)start is left UNSEEDED so the first Check() observes oldState==""
|
||||
// and emits once (a box that went silent while the hub was down would otherwise stay silent).
|
||||
//
|
||||
// R-1001 (hub 0.162.0): UNLESS the alarm was already raised for this silence — the newest host_stale /
|
||||
// host_down event for the host is newer than its last report. Then the host is seeded with the state that
|
||||
// event recorded, silently. The old comment said „the dispatcher's 1h cooldown dedups the re-emit"; it did
|
||||
// not (host liveness events bypass that cooldown, `dispatcher.go` nodeLivenessEvents), so every hub restart
|
||||
// re-mailed the operator about Tester 2, silent for days (measured 2026-10-13). A host seeded „stale" still
|
||||
// emits host_down when it crosses downAfter — a new transition. A failed lookup leaves it unseeded (the
|
||||
// alarm, never the silence). Pinned by TestHostStaleness_R1001_RestartDoesNotReMail.
|
||||
if age > sc.threshold {
|
||||
if known := sc.alreadyAlarmed(row); known != "" {
|
||||
sc.states[row.HostID] = known
|
||||
sc.downtimeStart[row.HostID] = row.LastReportAt
|
||||
knownCount++
|
||||
continue
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case age > sc.downAfter:
|
||||
downCount++
|
||||
@@ -69,10 +84,30 @@ func NewHostStalenessChecker(s *store.Store, threshold time.Duration, onEvent Ev
|
||||
okCount++
|
||||
}
|
||||
}
|
||||
logger.Printf("[INFO] Host staleness checker initialized: %d ok, %d stale, %d down (stale/down left unseeded → first Check emits; host_stale after %s, host_down after %s)", okCount, staleCount, downCount, threshold, 2*threshold)
|
||||
logger.Printf("[INFO] Host staleness checker initialized: %d ok, %d already alarmed (seeded silently, R-1001), %d stale, %d down (left unseeded → first Check emits; host_stale after %s, host_down after %s)", okCount, knownCount, staleCount, downCount, threshold, 2*threshold)
|
||||
return sc
|
||||
}
|
||||
|
||||
// alreadyAlarmed returns "stale" or "down" when the newest liveness event for the host is that alarm and is not older
|
||||
// than the host's last report (the operator was told about THIS silence), else "".
|
||||
func (sc *HostStalenessChecker) alreadyAlarmed(row store.HostStaleRow) string {
|
||||
ev, err := sc.store.LatestHostLivenessEvent(row.CustomerID, row.HostID)
|
||||
if err != nil {
|
||||
sc.logger.Printf("[WARN] Host staleness: could not read the last alarm for %s (%v) — the first Check alarms (R-1001)", row.HostID, err)
|
||||
return ""
|
||||
}
|
||||
if ev == nil || ev.CreatedAt.Before(row.LastReportAt) {
|
||||
return ""
|
||||
}
|
||||
switch ev.EventType {
|
||||
case "host_down":
|
||||
return "down"
|
||||
case "host_stale":
|
||||
return "stale"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Check evaluates all hosts and emits events on state transitions. Call every 60s.
|
||||
func (sc *HostStalenessChecker) Check() {
|
||||
rows, err := sc.store.GetHostStaleness()
|
||||
@@ -125,7 +160,7 @@ func (sc *HostStalenessChecker) Check() {
|
||||
if oldState == "" {
|
||||
// F2: first observation (incl. a hub restart). Healthy seeds silently; a host already
|
||||
// stale/down at first sight emits once (else a box that went silent while the hub was
|
||||
// down would never alert). The dispatcher's 1h cooldown dedups a re-emit on hub restart.
|
||||
// down would never alert). A host whose alarm was already raised was seeded at start (R-1001).
|
||||
sc.states[row.HostID] = newState
|
||||
if newState != "ok" {
|
||||
sc.emitTransition(row.HostID, row.CustomerID, "unknown", newState, age)
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// R-1001 (measured 2026-10-13 18:33 UTC, hub v0.149.0 rollout): Tester 2, silent 216 h, got a fresh host_down and
|
||||
// restore_test_stale mail on a hub restart. A restart is a NEW checker over the SAME database — that is what these
|
||||
// tests build: alarm with checker 1, then a second checker on the same DB (the restarted hub) must not mail again,
|
||||
// and a host that goes silent after the start must still mail once.
|
||||
|
||||
func r1001Store(t *testing.T) (*store.Store, *sql.DB) {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "r1001.db")
|
||||
st, err := store.New(path, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
db, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { db.Close() })
|
||||
return st, db
|
||||
}
|
||||
|
||||
func r1001Host(t *testing.T, st *store.Store, db *sql.DB, cid, hid string, minutesAgo int) {
|
||||
t.Helper()
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: cid, APIKey: "k-" + cid, RetrievalPassword: "p"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.UpsertHost(&store.Host{HostID: hid, CustomerID: cid, APIKey: "hk-" + hid}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostReport(hid, cid, []byte(`{}`), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
backdate(t, db, hid, minutesAgo)
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with the R-1001 seeding removed from NewHostStalenessChecker (the pre-fix
|
||||
// F2 shape), the restart case fails with
|
||||
//
|
||||
// r1001_restart_test.go: the restarted hub re-sent [host_down] for a host whose silence was already alarmed
|
||||
//
|
||||
// Restored.
|
||||
func TestHostStaleness_R1001_RestartDoesNotReMail(t *testing.T) {
|
||||
st, db := r1001Store(t)
|
||||
r1001Host(t, st, db, "tester-2", "Tester-2-be8404", 216*60) // down for 9 days
|
||||
var got []string
|
||||
on := func(_, et, _, _, _, _ string) { got = append(got, et) }
|
||||
quiet := log.New(io.Discard, "", 0)
|
||||
|
||||
first := NewHostStalenessChecker(st, 30*time.Minute, on, quiet)
|
||||
first.Check()
|
||||
if len(got) != 1 || got[0] != "host_down" {
|
||||
t.Fatalf("the first hub ever to see the silence must alarm once: %v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
restarted := NewHostStalenessChecker(st, 30*time.Minute, on, quiet)
|
||||
restarted.Check()
|
||||
restarted.Check()
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("the restarted hub re-sent %v for a host whose silence was already alarmed", got)
|
||||
}
|
||||
if s := restarted.GetState("Tester-2-be8404"); s != "down" {
|
||||
t.Fatalf("the already-alarmed host must be seeded with its alarm's state, got %q", s)
|
||||
}
|
||||
|
||||
// The box reports again → host_recovered once, and its next silence alarms again (a new run).
|
||||
backdate(t, db, "Tester-2-be8404", 0)
|
||||
restarted.Check()
|
||||
if len(got) != 1 || got[0] != "host_recovered" {
|
||||
t.Fatalf("a seeded host that reports again must recover once: %v", got)
|
||||
}
|
||||
got = nil
|
||||
backdate(t, db, "Tester-2-be8404", 70)
|
||||
again := NewHostStalenessChecker(st, 30*time.Minute, on, quiet) // a restart in the middle of a NEW silence
|
||||
again.Check()
|
||||
if len(got) != 1 || got[0] != "host_down" {
|
||||
t.Fatalf("a new silence after a report must alarm (the old alarm is older than the report): %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The other half of the rule: a host that goes down AFTER the start mails once; an alarm seeded „stale" still
|
||||
// escalates to host_down when it crosses the down threshold (a new transition, not a repeat).
|
||||
func TestHostStaleness_R1001_NewSilenceStillMails(t *testing.T) {
|
||||
st, db := r1001Host2(t)
|
||||
var got []string
|
||||
on := func(_, et, _, _, _, _ string) { got = append(got, et) }
|
||||
quiet := log.New(io.Discard, "", 0)
|
||||
|
||||
sc := NewHostStalenessChecker(st, 30*time.Minute, on, quiet)
|
||||
sc.Check()
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a healthy host at start: no mail, got %v", got)
|
||||
}
|
||||
backdate(t, db, "h-new", 70) // down after the start
|
||||
sc.Check()
|
||||
if len(got) != 1 || got[0] != "host_down" {
|
||||
t.Fatalf("a host down after the start must mail once: %v", got)
|
||||
}
|
||||
|
||||
// stale alarmed, then a restart, then down → host_down (new transition), once.
|
||||
got = nil
|
||||
backdate(t, db, "h-new", 0)
|
||||
sc.Check() // recovered
|
||||
backdate(t, db, "h-new", 40)
|
||||
sc.Check() // host_stale
|
||||
got = nil
|
||||
restarted := NewHostStalenessChecker(st, 30*time.Minute, on, quiet)
|
||||
restarted.Check()
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("restart during an alarmed stale: no mail, got %v", got)
|
||||
}
|
||||
backdate(t, db, "h-new", 70)
|
||||
restarted.Check()
|
||||
if len(got) != 1 || got[0] != "host_down" {
|
||||
t.Fatalf("stale → down after the restart must mail host_down once: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func r1001Host2(t *testing.T) (*store.Store, *sql.DB) {
|
||||
st, db := r1001Store(t)
|
||||
r1001Host(t, st, db, "c-new", "h-new", 0)
|
||||
return st, db
|
||||
}
|
||||
|
||||
// restore_test_stale across a restart: the same DB, a second checker. Both tiers unproven for 20 days.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with the staleAlreadyReported case removed from checkStaleness, it fails
|
||||
// with
|
||||
//
|
||||
// r1001_restart_test.go: the restarted hub re-sent 2 restore_test_stale signal(s)
|
||||
//
|
||||
// Restored.
|
||||
func TestRestoreTestStale_R1001_RestartDoesNotReMail(t *testing.T) {
|
||||
st, db := r1001Store(t)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "k", RetrievalPassword: "p"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(rtReport(t, "", false, now, "")), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := db.Exec(`UPDATE host_reports SET received_at = datetime('now', '-20 days')`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []capturedEvent
|
||||
first := rtChecker(st, &got, nil)
|
||||
first.Check()
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("two tiers unproven for 20 days: want 2 stale signals, got %v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
restarted := rtChecker(st, &got, nil)
|
||||
restarted.Check()
|
||||
restarted.Check()
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("the restarted hub re-sent %d restore_test_stale signal(s)", len(got))
|
||||
}
|
||||
|
||||
// An earlier signal outside the read window does not hold the next one back.
|
||||
if _, err := db.Exec(`UPDATE events SET created_at = datetime('now', '-15 days') WHERE event_type = ?`, EventRestoreTestStale); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got = nil
|
||||
later := rtChecker(st, &got, nil)
|
||||
later.Check()
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("a signal older than the read window must not silence the tier: got %v", got)
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -277,6 +278,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
c.mu.Unlock()
|
||||
|
||||
switch {
|
||||
case state == "stale" && prev == "" && c.staleAlreadyReported(customerID, tier, proven[tier], now):
|
||||
// R-1001: the first sweep after a hub restart — this tier's stale run was already reported.
|
||||
c.logger.Printf("[INFO] restore-test staleness: %s %s tier still unproven, already reported before the restart — not re-sent (R-1001)", customerID, tier)
|
||||
case state == "stale" && prev != "stale":
|
||||
c.emit(customerID, EventRestoreTestStale, "warning", v.reason)
|
||||
case v.verdict == verdictUnknown && prev == "":
|
||||
@@ -287,6 +291,26 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// staleAlreadyReported (R-1001, hub 0.162.0): the edge state lives in memory, so a hub restart forgot that a tier's
|
||||
// stale signal was already sent and the first sweep re-sent it (measured 2026-10-13, Tester 2). The DB holds the
|
||||
// answer: a restore_test_stale event for this tier („<tier> tier: …") inside the read window and newer than the
|
||||
// tier's last proof means this run of staleness was reported. The window bound keeps it honest — a pass between that
|
||||
// event and now would be inside the window and newer than it. An event older than the window, or a failed read, lets
|
||||
// the signal through (the alarm, never the silence). Pinned by TestRestoreTestStale_R1001_RestartDoesNotReMail.
|
||||
func (c *RestoreTestChecker) staleAlreadyReported(customerID, tier string, provenAt, now time.Time) bool {
|
||||
evs, err := c.store.GetEventsByType(customerID, EventRestoreTestStale, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test staleness: could not read the past signals for %s (%v) — sending (R-1001)", customerID, err)
|
||||
return false
|
||||
}
|
||||
for _, e := range evs {
|
||||
if strings.HasPrefix(e.Message, tier+" tier:") && e.CreatedAt.After(provenAt) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
|
||||
@@ -2704,6 +2704,30 @@ func (s *Store) GetLatestEventByType(customerID, eventType string) (*Event, erro
|
||||
return &e, nil
|
||||
}
|
||||
|
||||
// LatestHostLivenessEvent returns the newest host_stale / host_down / host_recovered event for hostID, or nil (R-1001).
|
||||
// The host staleness checker writes every one of them as „Host <id>: …" under the host's customer, so the prefix names
|
||||
// the host (substr, not LIKE: a host id may hold `_`).
|
||||
func (s *Store) LatestHostLivenessEvent(customerID, hostID string) (*Event, error) {
|
||||
prefix := "Host " + hostID + ":"
|
||||
var e Event
|
||||
var createdAt string
|
||||
err := s.db.QueryRow(`
|
||||
SELECT id, customer_id, event_type, severity, message, details_json, source, created_at
|
||||
FROM events
|
||||
WHERE customer_id = ? AND event_type IN ('host_stale', 'host_down', 'host_recovered') AND substr(message, 1, ?) = ?
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1`, customerID, len(prefix), prefix,
|
||||
).Scan(&e.ID, &e.CustomerID, &e.EventType, &e.Severity, &e.Message, &e.DetailsJSON, &e.Source, &createdAt)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
e.CreatedAt = parseSQLiteTime(createdAt)
|
||||
return &e, nil
|
||||
}
|
||||
|
||||
// GetAllRecentEvents returns the most recent events across all customers.
|
||||
func (s *Store) GetAllRecentEvents(limit int) ([]Event, error) {
|
||||
rows, err := s.db.Query(`
|
||||
|
||||
@@ -23,9 +23,10 @@ import (
|
||||
// fakeGitea serves the two endpoints the client uses, with a controllable per-request delay so
|
||||
// "concurrent" is measurable rather than asserted.
|
||||
type fakeGitea struct {
|
||||
versions []string
|
||||
delay time.Duration
|
||||
failVer string // this version's /files call 500s
|
||||
versions []string
|
||||
delay time.Duration
|
||||
failVer string // this version's /files call 500s
|
||||
bundleOnly string // R-1011: this version's package holds only the config bundle (an agent step bundle)
|
||||
|
||||
inFlight int32
|
||||
maxSeen int32
|
||||
@@ -58,6 +59,10 @@ func (f *fakeGitea) start(t *testing.T) *gitea.Client {
|
||||
}
|
||||
}
|
||||
ver := versionFromFilesPath(r.URL.Path)
|
||||
if f.bundleOnly != "" && ver == f.bundleOnly {
|
||||
fmt.Fprintf(w, `[{"name":"felhom-config-bundle.json","sha256":"bundle-%s"}]`, ver)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(w, `[{"name":"felhom-agent","sha256":"sha-%s"}]`, ver)
|
||||
return
|
||||
}
|
||||
@@ -246,3 +251,25 @@ func TestArtifactChoices_CacheIsPerPackage(t *testing.T) {
|
||||
t.Error("precondition")
|
||||
}
|
||||
}
|
||||
|
||||
// R-1011: the Day-0 agent list shows only versions whose package holds the felhom-agent file. A step bundle
|
||||
// (`0.156.1-step1`, only felhom-config-bundle.json) is left out — never listed with the bundle's sha.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-19): with FileSHA256's pre-fix fallback to the first file, it fails with
|
||||
//
|
||||
// artifact_choices_test.go: the list offers 0.156.1-step1 (sha "bundle-0.156.1-step1"), a package with no agent binary
|
||||
//
|
||||
// Restored.
|
||||
func TestArtifactChoices_R1011_StepBundleIsNotAnAgentVersion(t *testing.T) {
|
||||
f := &fakeGitea{versions: []string{"0.157.0", "0.156.1-step1", "0.156.1"}, bundleOnly: "0.156.1-step1"}
|
||||
s := newTestWebServer(t, f.start(t))
|
||||
got := s.artifactChoices(context.Background(), "felhom-agent", "felhom-agent")
|
||||
for _, c := range got {
|
||||
if c.Version == "0.156.1-step1" {
|
||||
t.Fatalf("the list offers %s (sha %q), a package with no agent binary", c.Version, c.SHA256)
|
||||
}
|
||||
}
|
||||
if len(got) != 2 || got[0].SHA256 != "sha-0.157.0" || got[1].Version != "0.156.1" || got[1].SHA256 != "sha-0.156.1" {
|
||||
t.Fatalf("the release versions must stay, each with its own binary's sha: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -51,8 +51,12 @@ func (f *fakeRegistry) start(t *testing.T) *gitea.Client {
|
||||
if f.shaless[seg[5]+"/"+seg[6]] {
|
||||
sha = ""
|
||||
}
|
||||
name := seg[5] // the agent's file is named after its package …
|
||||
if name == pkgGolden {
|
||||
name = fileGolden // … the golden's is not (R-1011: the hub now asks for exactly that file)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`[{"name":"` + seg[5] + `","sha256":"` + sha + `"}]`))
|
||||
_, _ = w.Write([]byte(`[{"name":"` + name + `","sha256":"` + sha + `"}]`))
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
|
||||
@@ -1589,6 +1589,7 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
|
||||
//
|
||||
// artifact_version_missing — the registry answered 404 for that version
|
||||
// artifact_sha_missing — the version exists but the registry listed no sha256
|
||||
// artifact_file_missing — the version exists but its package holds no such file (an agent step bundle, R-1011)
|
||||
// artifact_unverifiable — the registry did not answer usefully (network error, 5xx)
|
||||
// artifact_sha_invalid — a hand-typed sha is not 64 hex characters (manual path only)
|
||||
func (s *Server) resolveArtifactSHA(ctx context.Context, pkg, file, version, submittedSHA string) (string, string) {
|
||||
@@ -1602,6 +1603,8 @@ func (s *Server) resolveArtifactSHA(ctx context.Context, pkg, file, version, sub
|
||||
switch {
|
||||
case gitea.IsNotFound(err):
|
||||
flash = "artifact_version_missing"
|
||||
case errors.Is(err, gitea.ErrNoFile):
|
||||
flash = "artifact_file_missing"
|
||||
case errors.Is(err, gitea.ErrNoSHA):
|
||||
flash = "artifact_sha_missing"
|
||||
}
|
||||
|
||||
@@ -47,8 +47,10 @@ func TestArtifactSave_FlashNamesTheCause(t *testing.T) {
|
||||
}{
|
||||
{"version not found", http.StatusNotFound, "", "artifact_version_missing"},
|
||||
{"registry failing", http.StatusBadGateway, "", "artifact_unverifiable"},
|
||||
{"no sha listed", http.StatusOK, `[{"name":"x","sha256":""}]`, "artifact_sha_missing"},
|
||||
{"healthy", http.StatusOK, `[{"name":"x","sha256":"` + sha + `"}]`, "artifacts_set"},
|
||||
{"no sha listed", http.StatusOK, `[{"name":"felhom-agent","sha256":""},{"name":"golden.tar.zst","sha256":""}]`, "artifact_sha_missing"},
|
||||
// R-1011: a step bundle — files, but not the agent binary — is refused by name, never another file's sha.
|
||||
{"no agent binary (a step bundle)", http.StatusOK, `[{"name":"felhom-config-bundle.json","sha256":"` + sha + `"}]`, "artifact_file_missing"},
|
||||
{"healthy", http.StatusOK, `[{"name":"felhom-agent","sha256":"` + sha + `"},{"name":"golden.tar.zst","sha256":"` + sha + `"}]`, "artifacts_set"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, _ := newTestServer(t)
|
||||
@@ -81,6 +83,7 @@ func TestArtifactFlashes_EachRendersItsOwnText(t *testing.T) {
|
||||
for flash, want := range map[string]string{
|
||||
"artifact_version_missing": "the registry has no such version",
|
||||
"artifact_sha_missing": "no checksum for a chosen version",
|
||||
"artifact_file_missing": "holds no artifact",
|
||||
"artifact_unverifiable": "could not verify",
|
||||
"artifact_sha_invalid": "a checksum you typed is not valid",
|
||||
} {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"log"
|
||||
@@ -385,7 +386,14 @@ func (s *Server) artifactChoices(ctx context.Context, pkg, file string) []artifa
|
||||
wg.Wait()
|
||||
|
||||
out := make([]artifactChoice, 0, len(vers))
|
||||
var noFile []string
|
||||
for i, v := range vers {
|
||||
if errors.Is(errs[i], gitea.ErrNoFile) {
|
||||
// R-1011: a version whose package does not hold the artifact (an agent step bundle) is not an agent
|
||||
// version — left out of the dropdown, one INFO line per resolve rather than a WARN per version.
|
||||
noFile = append(noFile, v)
|
||||
continue
|
||||
}
|
||||
if errs[i] != nil {
|
||||
// unchanged behaviour: a failed lookup drops THAT version, never the whole list
|
||||
s.logger.Printf("[WARN] artifact sha (%s/%s): %v", pkg, v, errs[i])
|
||||
@@ -394,6 +402,10 @@ func (s *Server) artifactChoices(ctx context.Context, pkg, file string) []artifa
|
||||
out = append(out, artifactChoice{Version: v, SHA256: shas[i]})
|
||||
}
|
||||
|
||||
if len(noFile) > 0 {
|
||||
s.logger.Printf("[INFO] artifact choices (%s): %d version(s) without %s left out (R-1011): %s", pkg, len(noFile), file, strings.Join(noFile, ", "))
|
||||
}
|
||||
|
||||
// Only a SUCCESSFUL resolve is cached. A Gitea blip must not pin an empty list in front of the
|
||||
// operator for a minute — the earlier returns leave the cache untouched, and so does this one
|
||||
// when every version's sha lookup failed.
|
||||
|
||||
@@ -298,6 +298,10 @@ func buildSystemRows(lines []osupdates.FleetLine, facts map[string]sysfacts.Syst
|
||||
switch l.Tunnel {
|
||||
case "running":
|
||||
r.Tunnel = plain("running")
|
||||
case "absent":
|
||||
// R-1070 (agent with the `absent` state): no cloudflared container — a box with no tunnel set up (a test
|
||||
// or LAN-only box). Not a fault; the hub's tunnel alarm counts not_running only.
|
||||
r.Tunnel = cell{Text: "none set up", Title: "no cloudflared container in the guest (agent state: absent)"}
|
||||
case "not_running", "inactive":
|
||||
r.Tunnel = cell{Text: l.Tunnel, Class: "bad"}
|
||||
default:
|
||||
|
||||
@@ -272,3 +272,19 @@ func TestSystemPage_KernelCells(t *testing.T) {
|
||||
t.Fatalf("an older agent: %+v %+v", d, step)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1070: the agent's new `absent` tunnel state (no cloudflared container — no tunnel set up) reads as a plain fact,
|
||||
// not as a warning; not_running stays red, an unknown state stays amber.
|
||||
func TestSystemRows_R1070_AbsentTunnelIsNotAWarning(t *testing.T) {
|
||||
lines := []osupdates.FleetLine{{HostID: "a", Tunnel: "absent"}, {HostID: "b", Tunnel: "not_running"}, {HostID: "c", Tunnel: "unknown"}}
|
||||
rows := buildSystemRows(lines, nil, nil, time.Hour, time.Hour, time.Hour, time.Now())
|
||||
if len(rows) != 3 {
|
||||
t.Fatalf("rows: %d", len(rows))
|
||||
}
|
||||
if rows[0].Tunnel.Class != "" || rows[0].Tunnel.Text != "none set up" {
|
||||
t.Errorf("absent: %+v, want a plain „none set up\"", rows[0].Tunnel)
|
||||
}
|
||||
if rows[1].Tunnel.Class != "bad" || rows[2].Tunnel.Class != "warn" {
|
||||
t.Errorf("not_running %+v must stay bad, unknown %+v warn", rows[1].Tunnel, rows[2].Tunnel)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,6 +89,9 @@
|
||||
{{if eq .Flash "artifact_unverifiable"}}
|
||||
<div class="flash flash-error"><strong>Refused: could not verify — this does not mean anything is missing.</strong> The registry did not answer, so the hub cannot tell whether that version is installable. It refuses rather than saving with a warning, because a warning beside a success reads as a success. Manifest unchanged. Check that Gitea is up, then try again. There is deliberately no override: the registry is on your own server, so if it is unreachable the vouch can wait.</div>
|
||||
{{end}}
|
||||
{{if eq .Flash "artifact_file_missing"}}
|
||||
<div class="flash flash-error"><strong>Refused: that version's package holds no artifact.</strong> It has files, but not the agent binary (<code>felhom-agent</code>) or the golden (<code>golden.tar.zst</code>) — an agent step bundle such as <code>0.156.1-step1</code> carries only the config bundle. Vouching it would point fresh installs at a package with no binary. Manifest unchanged. Choose the release version.</div>
|
||||
{{end}}
|
||||
{{if eq .Flash "artifact_sha_missing"}}
|
||||
<div class="flash flash-error"><strong>Refused: no checksum for a chosen version.</strong> The hub records the sha256 it reads from Gitea, and that lookup came back empty — so saving would vouch bytes nobody verified. Manifest unchanged.</div>
|
||||
{{end}}
|
||||
|
||||
Reference in New Issue
Block a user