hub v0.133.0 (R-859 test approvals end with the test; R-840 bundle on the System page, manifest, alarm); installer 1.31.0 (root files from the config bundle); bundle bootstrap script; golden 0.293.0 evidence; rulings 96–99; drill-r50 removed (evidence)
gates / gates (push) Successful in 29s
gates / gates (push) Successful in 29s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1,3 +1,25 @@
|
||||
## v0.133.0 — test approvals end with the test (R-859); the root-file bundle on the System page, in the install manifest, and its alarm (R-840) (2026-10-04)
|
||||
|
||||
**Agent v0.143.0** reports the bundle; an older agent shows `unknown` in the new column (never a guess).
|
||||
|
||||
- **R-859 — test approvals end with the test (`11` §5.3.1).** Every OS approval made while a TEST override
|
||||
(`OS_APPROVE_AFTER`, `OS_APPROVE_NIGHTS`, `OS_DOCKER_APPROVE_NIGHTS`) is active is stored with a `test` mark. At every
|
||||
start WITHOUT an override, each test approval no real approval has superseded is CANCELLED: never served again (no
|
||||
ring-1 box gets a plan from it), ring-1 boxes are bumped, one operator event `os_release_cancelled` each. What boxes
|
||||
already installed stays. A one-time backfill marks the AUTOMATIC approvals made earlier than 24 h after their set was first
|
||||
seen (the 2026-10-04 guest and host sets — approved 1.5 h after first seen). The operator's Docker button approval of
|
||||
that day stays in force (a person approved it; decided by CC unattended — operator may reverse). The System page shows a test
|
||||
approval in amber and lists the cancellations of the last 7 days. The same set is approved again by the ruled wait.
|
||||
- **R-840 — the config bundle.** The Configuration vouch resolves the vouched agent's `felhom-config-bundle.json` sha
|
||||
from the registry by EXACT name (`gitea.ExactFileSHA256` — never the binary's sha as a fallback); the install manifest
|
||||
(`/api/v1/artifacts/<customer>`) serves it as `bundle`; the installer 1.31.0 installs exactly it. The System page has a
|
||||
"Root files" column (the box's bundle; amber when behind the vouched one or changed by hand; red from 7 days); the
|
||||
alarm `os_config_bundle_behind` (operator, warning) fires after `OS_ALARM_BUNDLE_BEHIND_AFTER` (default 7 d; decided by
|
||||
CC unattended — operator may reverse). A box saying `none` counts as behind; `unknown` never does.
|
||||
- `GetLatestHostReportJSON` breaks a same-second tie by id (two reports in one second picked an arbitrary one).
|
||||
- Tests: `testapproval_test.go`, `bundle_alarm_test.go`, `system_bundle_test.go`, `gitea/exact_test.go`. Red-proofs:
|
||||
`documentation/audits/r840-config-bundle-2026-10-04/partD/d-redproof.txt`, `.../partB/hub-bundle-redproof.txt`.
|
||||
|
||||
## v0.132.0 — the System page, the Docker engine release, the crash events (R-852, `09` decisions 87–89) (2026-10-04)
|
||||
|
||||
**Needs agent v0.142.0** for the versions, the Docker step and the crash guard; an older agent shows "no versions
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -423,6 +424,20 @@ func main() {
|
||||
logger.Printf("[ERROR] OS_DOCKER_APPROVE_NIGHTS=%q invalid — keeping 2", v)
|
||||
}
|
||||
}
|
||||
// `11` §5.3.1 (hub v0.133.0): an approval made under ANY of the three TEST overrides carries the test mark, and a start
|
||||
// without them cancels every test approval no real one superseded (no ring-1 box installs it from then on).
|
||||
var overrides []string
|
||||
for _, k := range []string{"OS_APPROVE_AFTER", "OS_APPROVE_NIGHTS", "OS_DOCKER_APPROVE_NIGHTS"} {
|
||||
if v := os.Getenv(k); v != "" {
|
||||
overrides = append(overrides, k+"="+v)
|
||||
}
|
||||
}
|
||||
osSvc.TestOverride = strings.Join(overrides, " ")
|
||||
if cancelled, cerr := osSvc.CancelTestReleases(); cerr != nil {
|
||||
logger.Printf("[ERROR] osupdates: cancelling test approvals at start: %v (cancelled so far: %v)", cerr, cancelled)
|
||||
} else if len(cancelled) > 0 {
|
||||
logger.Printf("[WARN] osupdates: %d TEST approval(s) cancelled at start: %s", len(cancelled), strings.Join(cancelled, ", "))
|
||||
}
|
||||
logger.Printf("[INFO] osupdates: approval rule = every ring-0 box healthy for %s and %d night run(s)", osSvc.ApproveAfter, osSvc.NightsRequired)
|
||||
logger.Printf("[INFO] osupdates: the Docker engine set is approved only by the operator, after %d healthy ring-0 night(s)", osSvc.DockerNights)
|
||||
apiHandler.SetOSUpdateService(osSvc)
|
||||
@@ -437,6 +452,7 @@ func main() {
|
||||
{"OS_ALARM_REBOOT_AFTER", &osSvc.RebootAfter, 14 * 24 * time.Hour},
|
||||
{"OS_ALARM_RING0_STALL_AFTER", &osSvc.Ring0StallAfter, 7 * 24 * time.Hour},
|
||||
{"OS_ALARM_NOT_COVERED_AFTER", &osSvc.NotCoveredAfter, 14 * 24 * time.Hour},
|
||||
{"OS_ALARM_BUNDLE_BEHIND_AFTER", &osSvc.BundleBehindAfter, 7 * 24 * time.Hour},
|
||||
} {
|
||||
*a.dst = a.def
|
||||
if v := os.Getenv(a.env); v != "" {
|
||||
|
||||
@@ -2675,6 +2675,8 @@ func (h *Handler) handleConfigRetrieve(w http.ResponseWriter, r *http.Request, c
|
||||
type artifactManifestResponse struct {
|
||||
Agent artifactEntry `json:"agent"`
|
||||
Golden artifactEntry `json:"golden"`
|
||||
// Bundle is the vouched agent's config bundle (R-840); absent when the vouched agent carries none.
|
||||
Bundle *artifactEntry `json:"bundle,omitempty"`
|
||||
}
|
||||
|
||||
type artifactEntry struct {
|
||||
@@ -2720,6 +2722,9 @@ func (h *Handler) handleArtifactManifest(w http.ResponseWriter, r *http.Request,
|
||||
Agent: artifactEntry{Version: m.AgentVersion, SHA256: m.AgentSHA256},
|
||||
Golden: artifactEntry{Version: m.GoldenVersion, SHA256: m.GoldenSHA256},
|
||||
}
|
||||
if m.BundleSHA256 != "" {
|
||||
resp.Bundle = &artifactEntry{Version: m.AgentVersion, SHA256: m.BundleSHA256}
|
||||
}
|
||||
h.logger.Printf("[INFO] Artifact manifest served for customer %s (agent=%s golden=%s)", customerID, m.AgentVersion, m.GoldenVersion)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package gitea
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-840: a version WITHOUT a config bundle must answer "" — never the binary's sha (FileSHA256's fallback would).
|
||||
func TestExactFileSHA256_NeverFallsBack(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/api/v1/packages/admin/generic/felhom-agent/0.142.1/files":
|
||||
w.Write([]byte(`[{"name":"felhom-agent","sha256":"aaaa"}]`))
|
||||
case "/api/v1/packages/admin/generic/felhom-agent/0.143.0/files":
|
||||
w.Write([]byte(`[{"name":"felhom-agent","sha256":"aaaa"},{"name":"felhom-config-bundle.json","sha256":"bbbb"}]`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
c := New(srv.URL, "admin", "u", "t")
|
||||
if got, err := c.ExactFileSHA256(context.Background(), "felhom-agent", "0.142.1", "felhom-config-bundle.json"); err != nil || got != "" {
|
||||
t.Fatalf("no bundle: got %q %v, want \"\"", got, err)
|
||||
}
|
||||
if got, err := c.ExactFileSHA256(context.Background(), "felhom-agent", "0.143.0", "felhom-config-bundle.json"); err != nil || got != "bbbb" {
|
||||
t.Fatalf("bundle: got %q %v", got, err)
|
||||
}
|
||||
if _, err := c.ExactFileSHA256(context.Background(), "felhom-agent", "9.9.9", "felhom-config-bundle.json"); err == nil {
|
||||
t.Fatal("an unreadable listing must be an error, not \"no bundle\"")
|
||||
}
|
||||
}
|
||||
@@ -107,6 +107,26 @@ func (c *Client) FileSHA256(ctx context.Context, pkgName, version, preferredFile
|
||||
return files[0].SHA256, nil
|
||||
}
|
||||
|
||||
// ExactFileSHA256 returns the sha256 of exactly `file` in the package version, or ("", nil) when the version has no
|
||||
// such file. Unlike FileSHA256 it NEVER falls back to another file: the config bundle (R-840) must not inherit the
|
||||
// binary's sha on a version that carries no bundle.
|
||||
func (c *Client) ExactFileSHA256(ctx context.Context, pkgName, version, file string) (string, error) {
|
||||
u := fmt.Sprintf("%s/api/v1/packages/%s/generic/%s/%s/files", c.baseURL, c.owner, pkgName, version)
|
||||
var files []pkgFile
|
||||
if err := c.getJSON(ctx, u, &files); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, f := range files {
|
||||
if f.Name == file {
|
||||
if f.SHA256 == "" {
|
||||
return "", fmt.Errorf("no sha256 for %s/%s file %q", pkgName, version, file)
|
||||
}
|
||||
return f.SHA256, nil
|
||||
}
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (c *Client) getJSON(ctx context.Context, url string, out interface{}) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
|
||||
@@ -682,10 +682,14 @@ var operatorOnlyEvents = map[string]bool{
|
||||
"offsite_window_large_grant": true,
|
||||
// OS updates (hub v0.130.0, `11` §8 step 2): run failures, rings, switches and approvals are operator facts.
|
||||
// os_update_applied is deliberately NOT here — it is the household's one line (info: recorded, never mailed).
|
||||
"os_update_failed": true,
|
||||
"os_update_health_failed": true,
|
||||
"os_release_approved": true,
|
||||
"os_release_approved_now": true,
|
||||
"os_update_failed": true,
|
||||
"os_update_health_failed": true,
|
||||
"os_release_approved": true,
|
||||
"os_release_approved_now": true,
|
||||
// hub v0.133.0 (`11` §5.3.1): a TEST approval cancelled at a start without the override.
|
||||
"os_release_cancelled": true,
|
||||
// R-840 (hub v0.133.0): a box's root-owned config bundle behind the vouched one for 7 days.
|
||||
"os_config_bundle_behind": true,
|
||||
"os_update_settings_changed": true,
|
||||
// R-841 (hub v0.131.0): the tunnel alarm — a box fact the household can do nothing about from inside.
|
||||
"tunnel_down": true,
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
package osupdates
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
func bundleReport(t *testing.T, f *fix, host, version, sha string) {
|
||||
t.Helper()
|
||||
body := `{"system":{"pve_version":"pve-manager/9.2.2/x","config_bundle":{"version":"` + version + `","bundle_sha256":"` + sha + `"}}}`
|
||||
h, err := f.s.Store.GetHost(host)
|
||||
if err != nil || h == nil {
|
||||
t.Fatalf("no host %s", host)
|
||||
}
|
||||
if err := f.s.Store.SaveHostReport(host, h.CustomerID, []byte(body), store.HostReportDenorm{AgentVersion: "0.143.0"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func vouch(t *testing.T, f *fix, sha string) {
|
||||
t.Helper()
|
||||
if err := f.s.Store.SetArtifactManifest(store.ArtifactManifest{AgentVersion: "0.143.0", AgentSHA256: "x", BundleSHA256: sha}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func count(sent []string, typ string) int {
|
||||
n := 0
|
||||
for _, s := range sent {
|
||||
if s == typ {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// R-840: a box behind the vouched bundle is told to the operator after 7 days — not before, once, and the clock
|
||||
// restarts when the box catches up. "none" (no bundle ever) counts; "unknown" never does.
|
||||
func TestBundleAlarm_AfterSevenDaysBehind(t *testing.T) {
|
||||
f := newFix(t)
|
||||
vouch(t, f, "new")
|
||||
bundleReport(t, f, "cust1", "none", "")
|
||||
sent, _ := f.s.Alarms()
|
||||
if count(sent, EventBundleBehind) != 0 {
|
||||
t.Fatal("alarm on the first sight")
|
||||
}
|
||||
f.now = f.now.Add(6 * 24 * time.Hour)
|
||||
if sent, _ = f.s.Alarms(); count(sent, EventBundleBehind) != 0 {
|
||||
t.Fatal("alarm before 7 days")
|
||||
}
|
||||
f.now = f.now.Add(25 * time.Hour)
|
||||
if sent, _ = f.s.Alarms(); count(sent, EventBundleBehind) != 1 {
|
||||
t.Fatalf("no alarm after 7 days: %v", sent)
|
||||
}
|
||||
if sent, _ = f.s.Alarms(); count(sent, EventBundleBehind) != 0 {
|
||||
t.Fatal("the alarm must not repeat at once")
|
||||
}
|
||||
bundleReport(t, f, "cust1", "0.143.0", "new")
|
||||
f.s.Alarms()
|
||||
if !f.s.Store.BundleBehindSince("cust1").IsZero() {
|
||||
t.Fatal("caught up: the clock must clear")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBundleAlarm_UnknownAndNoVouchedBundleSayNothing(t *testing.T) {
|
||||
f := newFix(t)
|
||||
vouch(t, f, "new")
|
||||
bundleReport(t, f, "cust1", "unknown", "")
|
||||
f.s.Alarms()
|
||||
f.now = f.now.Add(30 * 24 * time.Hour)
|
||||
if sent, _ := f.s.Alarms(); count(sent, EventBundleBehind) != 0 || !f.s.Store.BundleBehindSince("cust1").IsZero() {
|
||||
t.Fatalf("unknown is not a fact: %v", sent)
|
||||
}
|
||||
// control: the same box saying "none" IS behind (proves the report above was read at all)
|
||||
bundleReport(t, f, "cust1", "none", "")
|
||||
f.s.Alarms()
|
||||
if f.s.Store.BundleBehindSince("cust1").IsZero() {
|
||||
t.Fatal("control: a box saying none must start the clock")
|
||||
}
|
||||
g := newFix(t)
|
||||
vouch(t, g, "")
|
||||
bundleReport(t, g, "cust1", "0.143.0", "some-bundle")
|
||||
g.s.Alarms()
|
||||
g.now = g.now.Add(30 * 24 * time.Hour)
|
||||
if sent, _ := g.s.Alarms(); count(sent, EventBundleBehind) != 0 {
|
||||
t.Fatalf("nothing vouched, nothing behind: %v", sent)
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import (
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/semver"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts"
|
||||
)
|
||||
|
||||
// Layers.
|
||||
@@ -68,6 +69,10 @@ const (
|
||||
EventRebootNeeded = "os_reboot_needed" // warning, operator: reboot needed for RebootAfter
|
||||
EventRing0Stalled = "os_ring0_stalled" // error, operator: ring 0 approved nothing for Ring0StallAfter
|
||||
EventNotCovered = "os_not_covered" // warning, operator: fast-lane packages no release covers
|
||||
// EventCancelled: a TEST approval was cancelled because the hub started without the TEST override (`11` §5.3.1).
|
||||
EventCancelled = "os_release_cancelled" // warning, operator
|
||||
// EventBundleBehind: a box's root-owned config bundle has differed from the vouched one for BundleBehindAfter (R-840).
|
||||
EventBundleBehind = "os_config_bundle_behind" // warning, operator
|
||||
)
|
||||
|
||||
// Package is one name=version with its origin ("Debian" | "Debian-Security").
|
||||
@@ -165,9 +170,15 @@ type Service struct {
|
||||
RebootAfter time.Duration // 14 d
|
||||
Ring0StallAfter time.Duration // 7 d
|
||||
NotCoveredAfter time.Duration // 14 d
|
||||
Logger *log.Logger
|
||||
Now func() time.Time
|
||||
Bump func(hostID string)
|
||||
// BundleBehindAfter: a box's config bundle differs from the vouched one this long → an operator alarm (R-840;
|
||||
// decided by CC unattended — operator may reverse). Zero = 7 d.
|
||||
BundleBehindAfter time.Duration
|
||||
Logger *log.Logger
|
||||
Now func() time.Time
|
||||
Bump func(hostID string)
|
||||
// TestOverride names the TEST overrides active at start ("" = none, the ruled waits). Every approval made while it
|
||||
// is set carries the `test` mark; CancelTestReleases cancels them at a start without it (`11` §5.3.1).
|
||||
TestOverride string
|
||||
}
|
||||
|
||||
func (s *Service) now() time.Time {
|
||||
@@ -427,6 +438,8 @@ type ReleaseInfo struct {
|
||||
ApprovedAt time.Time
|
||||
ApprovedBy string
|
||||
Packages int
|
||||
Test bool // a TEST approval still in force: amber on the System page
|
||||
Cancelled string // set on a cancelled one (the page lists the last 7 days')
|
||||
}
|
||||
|
||||
// Releases lists the newest release of every layer (guest, host, Docker); a layer with none is absent.
|
||||
@@ -439,7 +452,20 @@ func (s *Service) Releases() []ReleaseInfo {
|
||||
}
|
||||
var list []Package
|
||||
_ = json.Unmarshal([]byte(rel.PackagesJSON), &list)
|
||||
out = append(out, ReleaseInfo{Layer: layer, ID: rel.ID, ApprovedAt: rel.ApprovedAt, ApprovedBy: rel.ApprovedBy, Packages: len(list)})
|
||||
out = append(out, ReleaseInfo{Layer: layer, ID: rel.ID, ApprovedAt: rel.ApprovedAt, ApprovedBy: rel.ApprovedBy, Packages: len(list), Test: rel.Test})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// CancelledReleases lists the approvals cancelled in the last 7 days (the System page says what stopped being served).
|
||||
func (s *Service) CancelledReleases() []ReleaseInfo {
|
||||
rels, _ := s.Store.CancelledOSReleasesSince(s.now().Add(-7 * 24 * time.Hour))
|
||||
var out []ReleaseInfo
|
||||
for _, r := range rels {
|
||||
var list []Package
|
||||
_ = json.Unmarshal([]byte(r.PackagesJSON), &list)
|
||||
out = append(out, ReleaseInfo{Layer: r.Layer, ID: r.ID, ApprovedAt: r.ApprovedAt, ApprovedBy: r.ApprovedBy, Packages: len(list),
|
||||
Test: r.Test, Cancelled: r.CancelledAt})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -455,6 +481,9 @@ func (s *Service) Candidates() []Status {
|
||||
return append(out, d)
|
||||
}
|
||||
|
||||
// BundleThreshold is the config-bundle alarm's wait (the System page turns the cell red at it).
|
||||
func (s *Service) BundleThreshold() time.Duration { return dflt(s.BundleBehindAfter, 7*24*time.Hour) }
|
||||
|
||||
// Thresholds are the alarm numbers the System page colours by (the same values the alarms use).
|
||||
func (s *Service) Thresholds() (stale, reboot, notCovered time.Duration) {
|
||||
return dflt(s.StaleAfter, 7*24*time.Hour), dflt(s.RebootAfter, 14*24*time.Hour), dflt(s.NotCoveredAfter, 14*24*time.Hour)
|
||||
@@ -577,12 +606,18 @@ func (s *Service) approve(layer, fp string, list []Package, by string) error {
|
||||
at := s.now().UTC().Truncate(time.Second)
|
||||
id := "os-" + layer + "-" + at.Format("20060102-150405")
|
||||
pj, _ := json.Marshal(list)
|
||||
if err := s.Store.SaveOSRelease(store.OSRelease{ID: id, Layer: layer, Fingerprint: fp, ApprovedAt: at, ApprovedBy: by, PackagesJSON: string(pj)}); err != nil {
|
||||
test := s.TestOverride != ""
|
||||
if err := s.Store.SaveOSRelease(store.OSRelease{ID: id, Layer: layer, Fingerprint: fp, ApprovedAt: at, ApprovedBy: by,
|
||||
PackagesJSON: string(pj), Test: test}); err != nil {
|
||||
return err
|
||||
}
|
||||
s.logf("[WARN] osupdates: OS release %s (%s) APPROVED by %s (%d packages, fingerprint %s)", id, layer, by, len(list), fp)
|
||||
s.event("", EventReleaseApprove, "info", fmt.Sprintf("OS release %s (%s) approved (%s, %d packages).", id, layer, by, len(list)),
|
||||
map[string]any{"release_id": id, "layer": layer, "approved_by": by, "packages": len(list), "fingerprint": fp})
|
||||
mark := ""
|
||||
if test {
|
||||
mark = " — TEST approval (" + s.TestOverride + "); cancelled when the hub starts without the override"
|
||||
}
|
||||
s.logf("[WARN] osupdates: OS release %s (%s) APPROVED by %s (%d packages, fingerprint %s)%s", id, layer, by, len(list), fp, mark)
|
||||
s.event("", EventReleaseApprove, "info", fmt.Sprintf("OS release %s (%s) approved (%s, %d packages)%s.", id, layer, by, len(list), mark),
|
||||
map[string]any{"release_id": id, "layer": layer, "approved_by": by, "packages": len(list), "fingerprint": fp, "test": test})
|
||||
if s.Bump != nil && layer != LayerDocker { // a Docker set reaches ring 1 only by a signed job, not the desired state
|
||||
hosts, _ := s.Store.ListHosts()
|
||||
for _, h := range hosts {
|
||||
@@ -594,6 +629,45 @@ func (s *Service) approve(layer, fp string, list []Package, by string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// CancelTestReleases runs at every hub start. Without a TEST override it cancels every test-marked approval that no
|
||||
// real (non-test) approval has superseded: no ring-1 box installs it from then on; what boxes already installed stays.
|
||||
// Each cancellation is an operator event; ring-1 boxes are bumped so their next plan has no cancelled release. With the
|
||||
// override still active it does nothing (the test is still running). Returns the cancelled ids.
|
||||
func (s *Service) CancelTestReleases() ([]string, error) {
|
||||
if s.TestOverride != "" {
|
||||
s.logf("[INFO] osupdates: TEST override active (%s) — test approvals stay in force", s.TestOverride)
|
||||
return nil, nil
|
||||
}
|
||||
var ids []string
|
||||
for _, layer := range AllLayers {
|
||||
rels, err := s.Store.UnsupersededTestReleases(layer)
|
||||
if err != nil {
|
||||
return ids, err
|
||||
}
|
||||
for _, r := range rels {
|
||||
reason := "approved under a TEST override; the hub started without it"
|
||||
if err := s.Store.CancelOSRelease(r.ID, reason, s.now()); err != nil {
|
||||
return ids, err
|
||||
}
|
||||
ids = append(ids, r.ID)
|
||||
s.logf("[WARN] osupdates: TEST approval %s (%s, approved %s by %s) CANCELLED — no ring-1 box installs it from now on",
|
||||
r.ID, layer, r.ApprovedAt.UTC().Format(time.RFC3339), r.ApprovedBy)
|
||||
s.event("", EventCancelled, "warning", fmt.Sprintf("OS release %s (%s) was a TEST approval and is cancelled: "+
|
||||
"no further box installs it. Boxes that already installed it keep it.", r.ID, layer),
|
||||
map[string]any{"release_id": r.ID, "layer": layer, "approved_at": r.ApprovedAt.UTC().Format(time.RFC3339), "approved_by": r.ApprovedBy})
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 && s.Bump != nil {
|
||||
hosts, _ := s.Store.ListHosts()
|
||||
for _, h := range hosts {
|
||||
if st := s.Store.GetOSHostSettings(h.HostID); st.Ring == 1 && st.Enabled {
|
||||
s.Bump(h.HostID)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
func (s *Service) releaseBlock(layer string) *ReleaseBlock {
|
||||
rel, err := s.Store.LatestOSRelease(layer)
|
||||
if err != nil || rel == nil {
|
||||
@@ -860,6 +934,38 @@ func (s *Service) Alarms() ([]string, error) {
|
||||
}
|
||||
}
|
||||
}
|
||||
// 5. R-840: the box's ROOT-OWNED config bundle (sudoers, wrappers, units) differs from the vouched agent's for
|
||||
// BundleBehindAfter. "none" (no bundle ever reached the box) counts as behind; "unknown" (the box could not say)
|
||||
// counts as nothing — never a guess. Only when the vouched agent carries a bundle at all.
|
||||
man := s.Store.GetArtifactManifest()
|
||||
for _, h := range hosts {
|
||||
if man.BundleSHA256 == "" {
|
||||
break
|
||||
}
|
||||
rj, _ := s.Store.GetLatestHostReportJSON(h.CustomerID)
|
||||
sys := sysfacts.Parse(rj)
|
||||
if !sys.Present || sys.Bundle.Version == sysfacts.Unknown {
|
||||
continue
|
||||
}
|
||||
behind := sys.Bundle.BundleSHA256 != man.BundleSHA256
|
||||
since := s.Store.BundleBehindSince(h.HostID)
|
||||
switch {
|
||||
case !behind && !since.IsZero():
|
||||
_ = s.Store.SetBundleBehindSince(h.HostID, time.Time{})
|
||||
since = time.Time{}
|
||||
case behind && since.IsZero():
|
||||
since = now
|
||||
_ = s.Store.SetBundleBehindSince(h.HostID, since)
|
||||
}
|
||||
after := dflt(s.BundleBehindAfter, 7*24*time.Hour)
|
||||
if s.raise("bundle:"+h.HostID, behind && now.Sub(since) >= after, h.CustomerID, EventBundleBehind, "warning",
|
||||
fmt.Sprintf("Root files: %s still runs config bundle %s; the vouched agent %s carries a newer one (since %s). "+
|
||||
"Send it with a signed agent_config_update (`11` §5.4.2).", h.HostID, sys.Bundle.Version, man.AgentVersion,
|
||||
since.UTC().Format("2006-01-02")),
|
||||
map[string]any{"host_id": h.HostID, "box_bundle": sys.Bundle.Version, "vouched_agent": man.AgentVersion, "since": since}) {
|
||||
sent = append(sent, EventBundleBehind)
|
||||
}
|
||||
}
|
||||
// 3. Ring 0 approved nothing for `stall` while ring 0 has pending fast-lane updates: the whole fleet stopped
|
||||
// getting fixes.
|
||||
ring0, _ := s.ring0Hosts()
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
package osupdates
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// `11` §5.3.1 (hub v0.133.0): test approvals end with the test.
|
||||
|
||||
// approveUnderTest makes ring 0 run a set and approves it at once under a TEST override (OS_APPROVE_AFTER=0 shape).
|
||||
func approveUnderTest(t *testing.T, f *fix, override string, set ...Package) string {
|
||||
t.Helper()
|
||||
f.s.TestOverride = override
|
||||
f.s.ApproveAfter, f.s.NightsRequired = 0, 0
|
||||
f.report(t, "hp", "debug", true, set...)
|
||||
f.report(t, "n100", "debug", true, set...)
|
||||
if _, err := f.s.Evaluate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rel, _ := f.s.Store.LatestOSRelease(LayerGuest)
|
||||
if rel == nil {
|
||||
t.Fatal("not approved")
|
||||
}
|
||||
return rel.ID
|
||||
}
|
||||
|
||||
// An approval made under the override carries the mark; one made without it does not.
|
||||
func TestTestApproval_IsMarked(t *testing.T) {
|
||||
f := newFix(t)
|
||||
approveUnderTest(t, f, "OS_APPROVE_AFTER=0s", pk("libc6", "u4"))
|
||||
rel, _ := f.s.Store.LatestOSRelease(LayerGuest)
|
||||
if !rel.Test {
|
||||
t.Fatalf("an approval under a TEST override must be marked: %+v", rel)
|
||||
}
|
||||
if info := f.s.Releases(); len(info) != 1 || !info[0].Test {
|
||||
t.Fatalf("the System page must see the mark: %+v", info)
|
||||
}
|
||||
f.s.TestOverride = ""
|
||||
f.now = f.now.Add(time.Hour)
|
||||
f.report(t, "hp", "debug", true, pk("libc6", "u5"))
|
||||
f.report(t, "n100", "debug", true, pk("libc6", "u5"))
|
||||
f.s.Evaluate()
|
||||
rel, _ = f.s.Store.LatestOSRelease(LayerGuest)
|
||||
if rel.Test {
|
||||
t.Fatalf("an approval without the override must not be marked: %+v", rel)
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence: after a restart without the override, a ring-1 box gets NO plan from the test approval; the
|
||||
// cancellation is an operator event and ring-1 boxes are bumped.
|
||||
func TestTestApproval_CancelledAtAStartWithoutTheOverride(t *testing.T) {
|
||||
f := newFix(t)
|
||||
id := approveUnderTest(t, f, "OS_APPROVE_AFTER=0s", pk("libc6", "u4"))
|
||||
if b := f.s.DesiredBlock("cust1"); b.Release == nil || b.Release.ID != id {
|
||||
t.Fatalf("before the restart the ring-1 box is served the release: %+v", b)
|
||||
}
|
||||
f.events, f.bumps = nil, nil
|
||||
f.s.TestOverride = "" // the hub restarts without the override
|
||||
ids, err := f.s.CancelTestReleases()
|
||||
if err != nil || len(ids) != 1 || ids[0] != id {
|
||||
t.Fatalf("cancelled %v, %v", ids, err)
|
||||
}
|
||||
if b := f.s.DesiredBlock("cust1"); b.Release != nil {
|
||||
t.Fatalf("a ring-1 box must get no plan from a cancelled test approval: %+v", b.Release)
|
||||
}
|
||||
if len(f.events) != 1 || f.events[0] != EventCancelled {
|
||||
t.Fatalf("events = %v", f.events)
|
||||
}
|
||||
if len(f.bumps) != 1 || f.bumps[0] != "cust1" {
|
||||
t.Fatalf("ring-1 boxes must be bumped: %v", f.bumps)
|
||||
}
|
||||
if c := f.s.CancelledReleases(); len(c) != 1 || c[0].ID != id {
|
||||
t.Fatalf("the page must list the cancellation: %+v", c)
|
||||
}
|
||||
// once is enough: a second start cancels nothing more
|
||||
if again, _ := f.s.CancelTestReleases(); len(again) != 0 {
|
||||
t.Fatalf("second start cancelled %v", again)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestApproval_StaysWhileTheOverrideIsStillOn(t *testing.T) {
|
||||
f := newFix(t)
|
||||
approveUnderTest(t, f, "OS_APPROVE_AFTER=0s", pk("libc6", "u4"))
|
||||
if ids, _ := f.s.CancelTestReleases(); len(ids) != 0 {
|
||||
t.Fatalf("the test is still running; nothing to cancel: %v", ids)
|
||||
}
|
||||
if b := f.s.DesiredBlock("cust1"); b.Release == nil {
|
||||
t.Fatal("still served while the override is on")
|
||||
}
|
||||
}
|
||||
|
||||
// A test approval that a REAL approval has superseded is history, not cancelled; the real one stays served.
|
||||
func TestTestApproval_SupersededIsLeftAlone(t *testing.T) {
|
||||
f := newFix(t)
|
||||
old := approveUnderTest(t, f, "OS_APPROVE_AFTER=0s", pk("libc6", "u4"))
|
||||
f.s.TestOverride = ""
|
||||
f.now = f.now.Add(time.Hour)
|
||||
f.report(t, "hp", "debug", true, pk("libc6", "u5"))
|
||||
f.report(t, "n100", "debug", true, pk("libc6", "u5"))
|
||||
f.s.Evaluate()
|
||||
real, _ := f.s.Store.LatestOSRelease(LayerGuest)
|
||||
if real.ID == old || real.Test {
|
||||
t.Fatalf("setup: %+v", real)
|
||||
}
|
||||
if ids, _ := f.s.CancelTestReleases(); len(ids) != 0 {
|
||||
t.Fatalf("a superseded test approval must not be cancelled: %v", ids)
|
||||
}
|
||||
if b := f.s.DesiredBlock("cust1"); b.Release == nil || b.Release.ID != real.ID {
|
||||
t.Fatalf("the real release stays served: %+v", b.Release)
|
||||
}
|
||||
}
|
||||
|
||||
// After a cancellation the SAME set is approved again by the ruled wait (a real release) — the cancel is not a ban.
|
||||
func TestTestApproval_TheSetIsApprovedAgainByTheRuledWait(t *testing.T) {
|
||||
f := newFix(t)
|
||||
set := []Package{pk("libc6", "u4")}
|
||||
approveUnderTest(t, f, "OS_APPROVE_AFTER=0s", set...)
|
||||
f.s.TestOverride = ""
|
||||
f.s.ApproveAfter, f.s.NightsRequired = 24*time.Hour, 1
|
||||
f.s.CancelTestReleases()
|
||||
f.now = f.now.Add(25 * time.Hour)
|
||||
f.report(t, "hp", "night", true, set...)
|
||||
f.report(t, "n100", "night", true, set...)
|
||||
f.s.Evaluate()
|
||||
rel, _ := f.s.Store.LatestOSRelease(LayerGuest)
|
||||
if rel == nil || rel.Test {
|
||||
t.Fatalf("the ruled wait must approve the set again, unmarked: %+v", rel)
|
||||
}
|
||||
}
|
||||
|
||||
// The one-time backfill: on a database from before the mark, an approval earlier than 24 h after its set was first
|
||||
// seen (the 2026-10-04 shape: 1.5 h) is marked test; one after the ruled wait is not.
|
||||
func TestTestApproval_BackfillMarksTheEarlyApprovals(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "hub.db")
|
||||
db, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, q := range []string{
|
||||
`CREATE TABLE os_candidates (fingerprint TEXT PRIMARY KEY, first_seen DATETIME NOT NULL, packages_json TEXT NOT NULL)`,
|
||||
`CREATE TABLE os_releases (id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, approved_at DATETIME NOT NULL, approved_by TEXT NOT NULL, packages_json TEXT NOT NULL, layer TEXT NOT NULL DEFAULT 'guest')`,
|
||||
`INSERT INTO os_candidates VALUES ('fpA', '2026-10-04 11:07:00', '[]'), ('fpB', '2026-10-02 08:00:00', '[]')`,
|
||||
`INSERT INTO os_candidates VALUES ('fpD', '2026-10-04 14:28:00', '[]')`,
|
||||
`INSERT INTO os_releases VALUES ('os-guest-early', 'fpA', '2026-10-04 12:39:33', 'auto', '[]', 'guest'),
|
||||
('os-guest-ruled', 'fpB', '2026-10-03 09:00:00', 'auto', '[]', 'guest'),
|
||||
('os-docker-button', 'fpD', '2026-10-04 14:28:42', 'operator', '[]', 'docker')`,
|
||||
} {
|
||||
if _, err := db.Exec(q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
db.Close()
|
||||
st, err := store.New(path, log.New(os.Stderr, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
rels, _ := st.UnsupersededTestReleases(LayerGuest)
|
||||
if len(rels) != 1 || rels[0].ID != "os-guest-early" {
|
||||
t.Fatalf("backfill: unsuperseded test releases = %+v", rels)
|
||||
}
|
||||
// the operator's own button approval is not backfilled (a person approved it)
|
||||
if d, _ := st.UnsupersededTestReleases(LayerDocker); len(d) != 0 {
|
||||
t.Fatalf("backfill marked the operator's Docker approval: %+v", d)
|
||||
}
|
||||
}
|
||||
@@ -53,9 +53,42 @@ func (s *Store) migrateOSUpdates() error {
|
||||
// Host fast lane (hub v0.131.0): every report and release belongs to a LAYER; old rows are the guest's.
|
||||
s.db.Exec(`ALTER TABLE os_reports ADD COLUMN layer TEXT NOT NULL DEFAULT 'guest'`)
|
||||
s.db.Exec(`ALTER TABLE os_releases ADD COLUMN layer TEXT NOT NULL DEFAULT 'guest'`)
|
||||
// Test approvals end with the test (hub v0.133.0, `11` §5.3.1): an approval made while a TEST override is active
|
||||
// carries `test`; a start without the override CANCELS every test approval no real one has superseded.
|
||||
if !s.hasColumn("os_releases", "test") {
|
||||
if _, err := s.db.Exec(`ALTER TABLE os_releases ADD COLUMN test INTEGER NOT NULL DEFAULT 0`); err != nil {
|
||||
return err
|
||||
}
|
||||
// One-time backfill, from the data: an AUTOMATIC approval earlier than 24 h after its set was first seen
|
||||
// cannot have passed the ruled wait (24 h + one night) — it was a TEST approval (2026-10-04: the guest and host
|
||||
// sets, approved 1.5 h after first seen under OS_APPROVE_*). The operator's own Docker button approval of that day
|
||||
// is NOT backfilled: a person approved it; the override only shortened its precondition (decided by CC
|
||||
// unattended — operator may reverse). From hub v0.133.0 on, every approval under an override is marked.
|
||||
s.db.Exec(`UPDATE os_releases SET test = 1 WHERE approved_by = 'auto' AND EXISTS (SELECT 1 FROM os_candidates c
|
||||
WHERE c.fingerprint = os_releases.fingerprint
|
||||
AND (julianday(os_releases.approved_at) - julianday(c.first_seen)) * 24 < 24)`)
|
||||
}
|
||||
s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancelled_at TEXT NOT NULL DEFAULT ''`)
|
||||
s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancel_reason TEXT NOT NULL DEFAULT ''`)
|
||||
return nil
|
||||
}
|
||||
|
||||
// hasColumn reports whether table has the column (PRAGMA table_info).
|
||||
func (s *Store) hasColumn(table, col string) bool {
|
||||
rows, err := s.db.Query(`SELECT name FROM pragma_table_info(?)`, table)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var n string
|
||||
if rows.Scan(&n) == nil && n == col {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// OSHostSettings is one box's ring and switch.
|
||||
type OSHostSettings struct {
|
||||
HostID string
|
||||
@@ -191,6 +224,22 @@ type OSRelease struct {
|
||||
ApprovedAt time.Time
|
||||
ApprovedBy string
|
||||
PackagesJSON string
|
||||
Test bool // approved while a TEST override was active (`11` §5.3.1)
|
||||
CancelledAt string // "" = in force; a cancelled release is never served (LatestOSRelease skips it)
|
||||
CancelReason string
|
||||
}
|
||||
|
||||
const osReleaseCols = `id, layer, fingerprint, approved_at, approved_by, packages_json, test, cancelled_at, cancel_reason`
|
||||
|
||||
func scanOSRelease(sc interface{ Scan(...any) error }) (*OSRelease, error) {
|
||||
var r OSRelease
|
||||
var at string
|
||||
var test int
|
||||
if err := sc.Scan(&r.ID, &r.Layer, &r.Fingerprint, &at, &r.ApprovedBy, &r.PackagesJSON, &test, &r.CancelledAt, &r.CancelReason); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.ApprovedAt, r.Test = parseSQLiteTime(at), test == 1
|
||||
return &r, nil
|
||||
}
|
||||
|
||||
// SaveOSRelease stores an approved release.
|
||||
@@ -199,25 +248,70 @@ func (s *Store) SaveOSRelease(r OSRelease) error {
|
||||
if layer == "" {
|
||||
layer = "guest"
|
||||
}
|
||||
_, err := s.db.Exec(`INSERT INTO os_releases (id, layer, fingerprint, approved_at, approved_by, packages_json) VALUES (?, ?, ?, ?, ?, ?)`,
|
||||
r.ID, layer, r.Fingerprint, r.ApprovedAt.UTC().Format("2006-01-02 15:04:05"), r.ApprovedBy, r.PackagesJSON)
|
||||
test := 0
|
||||
if r.Test {
|
||||
test = 1
|
||||
}
|
||||
_, err := s.db.Exec(`INSERT INTO os_releases (id, layer, fingerprint, approved_at, approved_by, packages_json, test) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
r.ID, layer, r.Fingerprint, r.ApprovedAt.UTC().Format("2006-01-02 15:04:05"), r.ApprovedBy, r.PackagesJSON, test)
|
||||
return err
|
||||
}
|
||||
|
||||
// LatestOSRelease returns the newest approved release of a layer, or nil.
|
||||
// LatestOSRelease returns the newest approved release of a layer that is IN FORCE (not cancelled), or nil.
|
||||
func (s *Store) LatestOSRelease(layer string) (*OSRelease, error) {
|
||||
var r OSRelease
|
||||
var at string
|
||||
err := s.db.QueryRow(`SELECT id, layer, fingerprint, approved_at, approved_by, packages_json FROM os_releases WHERE layer = ? ORDER BY approved_at DESC, id DESC LIMIT 1`, layer).
|
||||
Scan(&r.ID, &r.Layer, &r.Fingerprint, &at, &r.ApprovedBy, &r.PackagesJSON)
|
||||
r, err := scanOSRelease(s.db.QueryRow(`SELECT `+osReleaseCols+` FROM os_releases WHERE layer = ? AND cancelled_at = ''
|
||||
ORDER BY approved_at DESC, id DESC LIMIT 1`, layer))
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
return r, err
|
||||
}
|
||||
|
||||
// UnsupersededTestReleases returns a layer's test releases still in force and newer than its newest REAL (non-test,
|
||||
// in-force) release — the ones a start without the TEST override must cancel. Newest first.
|
||||
func (s *Store) UnsupersededTestReleases(layer string) ([]OSRelease, error) {
|
||||
rows, err := s.db.Query(`SELECT `+osReleaseCols+` FROM os_releases r WHERE r.layer = ? AND r.test = 1 AND r.cancelled_at = ''
|
||||
AND r.approved_at >= COALESCE((SELECT MAX(approved_at) FROM os_releases WHERE layer = r.layer AND test = 0 AND cancelled_at = ''), '')
|
||||
ORDER BY r.approved_at DESC, r.id DESC`, layer)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.ApprovedAt = parseSQLiteTime(at)
|
||||
return &r, nil
|
||||
defer rows.Close()
|
||||
var out []OSRelease
|
||||
for rows.Next() {
|
||||
r, err := scanOSRelease(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, *r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// CancelOSRelease marks one release cancelled (it is never served again; its row stays as the record).
|
||||
func (s *Store) CancelOSRelease(id, reason string, at time.Time) error {
|
||||
_, err := s.db.Exec(`UPDATE os_releases SET cancelled_at = ?, cancel_reason = ? WHERE id = ? AND cancelled_at = ''`,
|
||||
at.UTC().Format("2006-01-02 15:04:05"), reason, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// CancelledOSReleasesSince lists releases cancelled at or after t (the System page shows them), newest first.
|
||||
func (s *Store) CancelledOSReleasesSince(t time.Time) ([]OSRelease, error) {
|
||||
rows, err := s.db.Query(`SELECT `+osReleaseCols+` FROM os_releases WHERE cancelled_at != '' AND cancelled_at >= ?
|
||||
ORDER BY cancelled_at DESC, id DESC`, t.UTC().Format("2006-01-02 15:04:05"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []OSRelease
|
||||
for rows.Next() {
|
||||
r, err := scanOSRelease(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, *r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// BackdateOSCandidateForTest moves a candidate's first_seen into the past. TEST-ONLY.
|
||||
|
||||
@@ -2118,6 +2118,10 @@ type ArtifactManifest struct {
|
||||
// Recording the hash here does not fix the delivery channel (that is R-50b(b)/(c)) — it makes
|
||||
// DRIFT VISIBLE, which is the cheap honest first step.
|
||||
WrapperSHA256 string `json:"wrapper_sha256"`
|
||||
// BundleSHA256 is the vouched agent version's CONFIG BUNDLE (R-840): every root-owned file the installer writes,
|
||||
// published beside the binary as felhom-agent/<AgentVersion>/felhom-config-bundle.json. "" = the vouched agent
|
||||
// carries none (older than v0.143.0). The installer installs exactly this; boxes behind it raise an alarm.
|
||||
BundleSHA256 string `json:"bundle_sha256"`
|
||||
}
|
||||
|
||||
// hub_settings keys for the artifact manifest (BUNDLE slice). Stored as discrete key/value rows in
|
||||
@@ -2130,6 +2134,7 @@ const (
|
||||
settingArtifactGoldenSHA256 = "artifact_golden_sha256"
|
||||
settingArtifactMinAgent = "artifact_min_agent"
|
||||
settingArtifactWrapperSHA256 = "artifact_wrapper_sha256" // R-50b(a): the vouched felhom-pbs-apply hash
|
||||
settingArtifactBundleSHA256 = "artifact_bundle_sha256" // R-840: the vouched agent's config bundle
|
||||
)
|
||||
|
||||
// settingOperatorPasswordHash is the hub_settings key for the operator login password bcrypt hash,
|
||||
@@ -2178,6 +2183,7 @@ func (s *Store) GetArtifactManifest() ArtifactManifest {
|
||||
GoldenSHA256: s.getSetting(settingArtifactGoldenSHA256),
|
||||
MinAgent: s.getSetting(settingArtifactMinAgent),
|
||||
WrapperSHA256: s.getSetting(settingArtifactWrapperSHA256),
|
||||
BundleSHA256: s.getSetting(settingArtifactBundleSHA256),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2199,7 +2205,30 @@ func (s *Store) SetArtifactManifest(m ArtifactManifest) error {
|
||||
if err := s.setSetting(settingArtifactMinAgent, m.MinAgent); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.setSetting(settingArtifactWrapperSHA256, m.WrapperSHA256)
|
||||
if err := s.setSetting(settingArtifactWrapperSHA256, m.WrapperSHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.setSetting(settingArtifactBundleSHA256, m.BundleSHA256)
|
||||
}
|
||||
|
||||
// BundleBehindSince returns since when a box's config bundle has differed from the vouched one (zero = it matches, or
|
||||
// was never seen behind). R-840's 7-day alarm counts from here.
|
||||
func (s *Store) BundleBehindSince(hostID string) time.Time {
|
||||
v := s.getSetting("bundle_behind_since:" + hostID)
|
||||
if v == "" {
|
||||
return time.Time{}
|
||||
}
|
||||
t, _ := time.Parse(time.RFC3339, v)
|
||||
return t
|
||||
}
|
||||
|
||||
// SetBundleBehindSince records (or, with a zero time, clears) the first moment a box was seen behind.
|
||||
func (s *Store) SetBundleBehindSince(hostID string, t time.Time) error {
|
||||
v := ""
|
||||
if !t.IsZero() {
|
||||
v = t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
return s.setSetting("bundle_behind_since:"+hostID, v)
|
||||
}
|
||||
|
||||
// EffectiveMinControllerVersion resolves the floor that actually applies to a customer: the
|
||||
@@ -3487,7 +3516,7 @@ func (s *Store) SaveHostReport(hostID, customerID string, reportJSON []byte, d H
|
||||
func (s *Store) GetLatestHostReportJSON(customerID string) (string, error) {
|
||||
var j string
|
||||
err := s.db.QueryRow(
|
||||
`SELECT report_json FROM host_reports WHERE customer_id = ? ORDER BY received_at DESC LIMIT 1`,
|
||||
`SELECT report_json FROM host_reports WHERE customer_id = ? ORDER BY received_at DESC, id DESC LIMIT 1`,
|
||||
customerID,
|
||||
).Scan(&j)
|
||||
if err == sql.ErrNoRows {
|
||||
|
||||
@@ -50,6 +50,15 @@ type Guest struct {
|
||||
UnknownReason string `json:"unknown_reason"`
|
||||
}
|
||||
|
||||
// ConfigBundle is the box's root-owned config bundle (R-840, agent v0.143.0): the agent's own read of the record
|
||||
// (version "none" = no bundle ever reached the box), with the drift the wrapper's facts add (files changed by hand).
|
||||
type ConfigBundle struct {
|
||||
Version string `json:"version"` // agent version of the bundle | none | unknown
|
||||
BundleSHA256 string `json:"bundle_sha256"`
|
||||
InstalledAt string `json:"installed_at"`
|
||||
Drift []string `json:"drift"`
|
||||
}
|
||||
|
||||
// System is the whole stanza. Present is false for a report from an agent older than v0.142.0.
|
||||
type System struct {
|
||||
Present bool
|
||||
@@ -60,6 +69,7 @@ type System struct {
|
||||
ReadAt string
|
||||
Host Host
|
||||
Guest Guest
|
||||
Bundle ConfigBundle
|
||||
}
|
||||
|
||||
type wire struct {
|
||||
@@ -70,6 +80,7 @@ type wire struct {
|
||||
Facts json.RawMessage `json:"facts"`
|
||||
FactsError string `json:"facts_error"`
|
||||
ReadAt string `json:"read_at"`
|
||||
ConfigBundle *ConfigBundle `json:"config_bundle"`
|
||||
} `json:"system"`
|
||||
}
|
||||
|
||||
@@ -93,11 +104,25 @@ func Parse(reportJSON string) System {
|
||||
out.PVEVersion, out.KernelVersion = orUnknown(w.System.PVEVersion), orUnknown(w.System.KernelVersion)
|
||||
out.VMID, out.FactsError, out.ReadAt = w.System.VMID, w.System.FactsError, w.System.ReadAt
|
||||
var f struct {
|
||||
Host Host `json:"host"`
|
||||
Host struct {
|
||||
Host
|
||||
ConfigBundle *ConfigBundle `json:"config_bundle"`
|
||||
} `json:"host"`
|
||||
Guest Guest `json:"guest"`
|
||||
}
|
||||
out.Bundle = ConfigBundle{Version: Unknown}
|
||||
if w.System.ConfigBundle != nil && w.System.ConfigBundle.Version != "" {
|
||||
out.Bundle = *w.System.ConfigBundle
|
||||
}
|
||||
if len(w.System.Facts) > 0 && json.Unmarshal(w.System.Facts, &f) == nil {
|
||||
out.Host, out.Guest = f.Host, f.Guest
|
||||
out.Host, out.Guest = f.Host.Host, f.Guest
|
||||
// The wrapper's view adds the drift; its record is the same file the agent read.
|
||||
if fb := f.Host.ConfigBundle; fb != nil && fb.Version != "" && fb.Version != Unknown {
|
||||
if out.Bundle.Version == Unknown {
|
||||
out.Bundle = *fb
|
||||
}
|
||||
out.Bundle.Drift = fb.Drift
|
||||
}
|
||||
}
|
||||
out.Host.Debian, out.Host.KernelRunning = orUnknown(out.Host.Debian), orUnknown(out.Host.KernelRunning)
|
||||
out.Host.KernelNextBoot = orUnknown(out.Host.KernelNextBoot)
|
||||
|
||||
@@ -1320,6 +1320,18 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
|
||||
http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
// R-840: the vouched agent's CONFIG BUNDLE is resolved from the registry by exact name, like the binary. A version
|
||||
// without one (older than v0.143.0) vouches none — the installer then falls back to its per-file fetches.
|
||||
bundleSHA := ""
|
||||
if agentVer != "" && s.gitea != nil {
|
||||
b, berr := s.gitea.ExactFileSHA256(r.Context(), pkgAgent, agentVer, fileBundle)
|
||||
if berr != nil {
|
||||
s.logger.Printf("[WARN] artifact vouch REFUSED: could not read agent %s's config bundle sha: %v", agentVer, berr)
|
||||
http.Redirect(w, r, "/configuration?flash=artifact_unverifiable", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
bundleSHA = b
|
||||
}
|
||||
// R-50b(a): the PBS-DR wrapper hash is operator-typed, not resolved from the package registry —
|
||||
// unlike the agent binary and the golden, this artifact is not published there at all. It is
|
||||
// installed from raw/branch/main, which is exactly the drift this field makes visible.
|
||||
@@ -1366,12 +1378,13 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
|
||||
GoldenSHA256: goldenSHA,
|
||||
MinAgent: minAgent,
|
||||
WrapperSHA256: wrapperSHA,
|
||||
BundleSHA256: bundleSHA,
|
||||
}); err != nil {
|
||||
s.logger.Printf("[ERROR] Failed to set artifact manifest: %v", err)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q wrapper_sha=%t", agentVer, goldenVer, minAgent, wrapperSHA != "")
|
||||
s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q wrapper_sha=%t bundle_sha=%q", agentVer, goldenVer, minAgent, wrapperSHA != "", bundleSHA)
|
||||
// Agent-plane immediate-sync (Direction-2a, v0.59.0): a MinAgent-floor / vouched-agent change is
|
||||
// a fleet-wide agent-plane intent shift. Fire-and-forget nudge every box so it re-reports at
|
||||
// once (the self-update train's signed op / floor re-evaluation lands in seconds, not ≤15 min).
|
||||
|
||||
@@ -33,6 +33,7 @@ import (
|
||||
const (
|
||||
pkgAgent = "felhom-agent"
|
||||
fileAgent = "felhom-agent"
|
||||
fileBundle = "felhom-config-bundle.json" // R-840: the agent's config bundle, beside the binary
|
||||
pkgGolden = "felhom-golden"
|
||||
fileGolden = "golden.tar.zst"
|
||||
)
|
||||
|
||||
@@ -31,9 +31,10 @@ type systemRow struct {
|
||||
FactsNote string
|
||||
// host
|
||||
PVE, KernelRunning, KernelNextBoot, HostDebian cell
|
||||
HostRelease, HostPending, HostNotCovered cell
|
||||
Held, RebootSince, KernelPanic, Oops cell
|
||||
CrashRestarts24h, Guard cell
|
||||
HostRelease, HostPending, HostNotCovered cell
|
||||
Held, RebootSince, KernelPanic, Oops cell
|
||||
CrashRestarts24h, Guard cell
|
||||
Bundle cell // R-840: the root-owned config bundle
|
||||
// guest
|
||||
GuestDebian, GuestRelease, GuestPending, GuestRestart cell
|
||||
// docker
|
||||
@@ -46,13 +47,43 @@ type systemRow struct {
|
||||
type OSSystemView interface {
|
||||
Fleet() ([]osupdates.FleetLine, error)
|
||||
Releases() []osupdates.ReleaseInfo
|
||||
CancelledReleases() []osupdates.ReleaseInfo
|
||||
Candidates() []osupdates.Status
|
||||
Thresholds() (stale, reboot, notCovered time.Duration)
|
||||
BundleThreshold() time.Duration
|
||||
ApproveDocker() (string, error)
|
||||
}
|
||||
|
||||
func plain(s string) cell { return cell{Text: s} }
|
||||
|
||||
// bundleCell is the "Root files" cell (R-840): the box's config bundle against the vouched agent's. Amber while behind,
|
||||
// red from the alarm's wait on, amber when a file was changed by hand (drift); "unknown" is never coloured as a fact.
|
||||
func bundleCell(f sysfacts.System, vouchedAgent, vouchedSHA string, since time.Time, after time.Duration, now time.Time) cell {
|
||||
b := f.Bundle
|
||||
if !f.Present || b.Version == "" || b.Version == sysfacts.Unknown {
|
||||
return unknownCell("")
|
||||
}
|
||||
c := cell{Text: b.Version}
|
||||
switch {
|
||||
case vouchedSHA == "":
|
||||
c.Title = "no vouched bundle to compare with (the vouched agent carries none)"
|
||||
case b.BundleSHA256 != vouchedSHA:
|
||||
c.Class, c.Title = "warn", "behind the vouched agent "+vouchedAgent+"'s bundle — send it with a signed agent_config_update"
|
||||
if !since.IsZero() {
|
||||
c.Title += " (behind since " + since.UTC().Format("2006-01-02 15:04") + " UTC)"
|
||||
if now.Sub(since) >= after {
|
||||
c.Class = "bad"
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(b.Drift) > 0 {
|
||||
c.Text += " (changed by hand)"
|
||||
c.Class = "warn"
|
||||
c.Title = "files differ from the installed bundle: " + strings.Join(b.Drift, ", ")
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func unknownCell(s string) cell {
|
||||
if s == "" || s == sysfacts.Unknown {
|
||||
return cell{Text: "unknown", Class: "warn", Title: "the box could not read it (agent older than v0.142.0, or the guest is down)"}
|
||||
@@ -207,9 +238,16 @@ func (s *Server) handleSystem(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
stale, reboot, notCov := view.Thresholds()
|
||||
rows := buildSystemRows(lines, facts, names, stale, reboot, notCov, time.Now())
|
||||
man := s.store.GetArtifactManifest()
|
||||
for i := range rows {
|
||||
rows[i].Bundle = bundleCell(facts[rows[i].HostID], man.AgentVersion, man.BundleSHA256,
|
||||
s.store.BundleBehindSince(rows[i].HostID), view.BundleThreshold(), time.Now())
|
||||
}
|
||||
data := map[string]interface{}{
|
||||
"Rows": buildSystemRows(lines, facts, names, stale, reboot, notCov, time.Now()),
|
||||
"Rows": rows,
|
||||
"Releases": view.Releases(),
|
||||
"Cancelled": view.CancelledReleases(),
|
||||
"Candidates": view.Candidates(),
|
||||
"Flash": r.URL.Query().Get("flash"),
|
||||
"FlashErr": r.URL.Query().Get("err"),
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts"
|
||||
)
|
||||
|
||||
// R-840: the "Root files" cell — amber behind, red from the alarm's wait, amber on drift, unknown never coloured as a fact.
|
||||
func TestBundleCell(t *testing.T) {
|
||||
now := time.Date(2026, 10, 4, 20, 0, 0, 0, time.UTC)
|
||||
sys := func(rep string) sysfacts.System { return sysfacts.Parse(rep) }
|
||||
cur := sys(`{"system":{"config_bundle":{"version":"0.143.0","bundle_sha256":"new"}}}`)
|
||||
old := sys(`{"system":{"config_bundle":{"version":"none"}}}`)
|
||||
drift := sys(`{"system":{"config_bundle":{"version":"0.143.0","bundle_sha256":"new"},"facts":{"host":{"config_bundle":{"version":"0.143.0","drift":["/usr/local/sbin/felhom-pbs-apply"]}}}}}`)
|
||||
if c := bundleCell(cur, "0.143.0", "new", time.Time{}, 7*24*time.Hour, now); c.Class != "" || c.Text != "0.143.0" {
|
||||
t.Fatalf("current: %+v", c)
|
||||
}
|
||||
if c := bundleCell(old, "0.143.0", "new", now.Add(-time.Hour), 7*24*time.Hour, now); c.Class != "warn" || c.Text != "none" {
|
||||
t.Fatalf("behind 1 h: %+v", c)
|
||||
}
|
||||
if c := bundleCell(old, "0.143.0", "new", now.Add(-8*24*time.Hour), 7*24*time.Hour, now); c.Class != "bad" {
|
||||
t.Fatalf("behind 8 days: %+v", c)
|
||||
}
|
||||
if c := bundleCell(drift, "0.143.0", "new", time.Time{}, 7*24*time.Hour, now); c.Class != "warn" || c.Title == "" {
|
||||
t.Fatalf("drift: %+v", c)
|
||||
}
|
||||
if c := bundleCell(sys(`{}`), "0.143.0", "new", time.Time{}, 7*24*time.Hour, now); c.Text != "unknown" {
|
||||
t.Fatalf("no stanza: %+v", c)
|
||||
}
|
||||
}
|
||||
@@ -41,9 +41,20 @@
|
||||
<div class="rel-grid">
|
||||
{{range .Releases}}
|
||||
<div><strong>{{.Layer}}</strong>: <code>{{.ID}}</code><br>
|
||||
<span class="text-muted">{{.Packages}} packages · {{.ApprovedAt.UTC.Format "2006-01-02 15:04"}} UTC · by {{.ApprovedBy}}</span></div>
|
||||
<span class="text-muted">{{.Packages}} packages · {{.ApprovedAt.UTC.Format "2006-01-02 15:04"}} UTC · by {{.ApprovedBy}}</span>
|
||||
{{if .Test}}<br><span class="c-warn" title="Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).">TEST approval</span>{{end}}</div>
|
||||
{{else}}<div class="text-muted">No release approved yet.</div>{{end}}
|
||||
</div>
|
||||
{{if .Cancelled}}
|
||||
<h3>Cancelled approvals (last 7 days)</h3>
|
||||
<div class="rel-grid">
|
||||
{{range .Cancelled}}
|
||||
<div><strong>{{.Layer}}</strong>: <code>{{.ID}}</code><br>
|
||||
<span class="c-warn">cancelled {{.Cancelled}} UTC{{if .Test}} — a TEST approval{{end}}</span><br>
|
||||
<span class="text-muted">no further box installs it; boxes that installed it keep it</span></div>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
<h3>What ring 0 runs now</h3>
|
||||
<div class="rel-grid">
|
||||
{{range .Candidates}}
|
||||
@@ -72,12 +83,12 @@
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Box</th><th>Ring / updates</th><th>Tunnel</th>
|
||||
<th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th>
|
||||
<th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th><th title="The root-owned config bundle: sudoers, wrappers, units (R-840)">Root files</th>
|
||||
<th class="grp">Guest Debian</th><th>Felhom release</th><th>Pending</th><th>Restart needed</th>
|
||||
<th class="grp">Docker</th><th>containerd</th><th>live-restore</th><th>Docker release</th>
|
||||
<th class="grp">Last OS leg</th>
|
||||
</tr>
|
||||
<tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="13">host</th><th class="grp" colspan="4">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
|
||||
<tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="14">host</th><th class="grp" colspan="4">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .Rows}}
|
||||
@@ -103,7 +114,7 @@
|
||||
{{template "sys_cell" .KernelRunning}}{{template "sys_cell" .KernelNextBoot}}{{template "sys_cell" .HostDebian}}
|
||||
{{template "sys_cell" .HostRelease}}{{template "sys_cell" .HostPending}}{{template "sys_cell" .HostNotCovered}}
|
||||
{{template "sys_cell" .Held}}{{template "sys_cell" .RebootSince}}{{template "sys_cell" .KernelPanic}}{{template "sys_cell" .Oops}}
|
||||
{{template "sys_cell" .CrashRestarts24h}}{{template "sys_cell" .Guard}}
|
||||
{{template "sys_cell" .CrashRestarts24h}}{{template "sys_cell" .Guard}}{{template "sys_cell" .Bundle}}
|
||||
<td class="grp {{if .GuestDebian.Class}}c-{{.GuestDebian.Class}}{{end}}">{{.GuestDebian.Text}}</td>
|
||||
{{template "sys_cell" .GuestRelease}}{{template "sys_cell" .GuestPending}}{{template "sys_cell" .GuestRestart}}
|
||||
<td class="grp {{if .Engine.Class}}c-{{.Engine.Class}}{{end}}">{{.Engine.Text}}</td>
|
||||
|
||||
Reference in New Issue
Block a user