diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index b4ca1f7..35a2fa1 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,35 @@ +## v0.102.0 — the hub refuses to vouch a version that cannot be installed (2026-08-09, R-273) + +**The guard owed since 2026-08-09 morning.** Agent v0.128.0 had been published as a package and never +git-tagged. It was vouched here. `felhom-host-install.sh` fetches an agent's config files from +`raw/tag/v/configs/`, so **every fresh install and every reinstall died at step 5 of 8 — as +root, on a virgin machine — for most of a day.** Nothing checked, and the moment of risk is this +handler: `handleSetArtifacts` is the sole UI path to `SetArtifactManifest`. + +**Two independent legs, because both failed within two days of each other.** The TAG was missing on +2026-08-08 (R-273); the PACKAGE was pruned out from under a still-tagged version on 2026-08-08/09 +(R-287). Checking either alone would have caught one of them. + +- **It asserts the path the installer really fetches.** `configs/felhom-mkfs-guarded.sh` is the FIRST + of the sixteen `fetch_raw` calls and is literally the file whose 404 broke Friday. A probe against + some other path that merely exists is how that failure stayed invisible; a test pins the constant. +- **The golden gets the package leg only** — it is fetched by version and has no config tree, so a tag + probe on it would assert something the installer never does. +- **"Could not verify" is also a refusal, and says so differently.** An unreachable registry refuses + with its own message rather than saving with a warning: a warning beside a success is read as a + success. **There is deliberately no override** — the registry is the operator's own server, so if it + cannot be reached the vouch can wait. +- **Ordering is load-bearing, and a failing test found it.** The probes run BEFORE `resolveArtifactSHA`, + whose failure flash reads *"version missing / Gitea unreachable / bad sha"* — three facts in one + message. Probing first means an unreachable registry is reported as unreachable. + +New in `internal/gitea`: `TagServesFile`, `PackageDownloadable`, and a three-valued `ProbeResult` +(yes / no / could-not-tell) so "absent" and "unanswerable" cannot collapse into each other. + +Five scenarios, each naming the wrong outcome it prevents (`artifact_installability_test.go`), and +three red-proofs: the tag check removed (scenario A then reports `artifacts_set` — **Friday's exact +defect returns**), the package check removed, and the inconclusive branch mapped onto the success path. + ## v0.101.0 — the artifact dropdown is memoised for 60 seconds (2026-08-08, R-267, operator ruling) v0.100.x took `/configuration` from 26.2 s to a mean of ~9.85 s by removing the serialisation. What diff --git a/hub/internal/gitea/gitea.go b/hub/internal/gitea/gitea.go index 91eaaa7..514895c 100644 --- a/hub/internal/gitea/gitea.go +++ b/hub/internal/gitea/gitea.go @@ -8,6 +8,7 @@ import ( "context" "encoding/json" "fmt" + "io" "net/http" "sort" "strconv" @@ -146,3 +147,67 @@ func compareSemver(a, b string) int { } return 0 } + +// --- Installability probes (R-273) -------------------------------------------------------------- +// +// WHY THESE EXIST. On 2026-08-08 agent v0.128.0 was published as a package and never git-tagged; +// the hub vouched it, and because felhom-host-install.sh fetches an agent's config files from +// `raw/tag/v/configs/`, EVERY fresh install and reinstall died at step 5 of 8, as root, on +// a virgin machine, for most of a day. Nothing checked at the moment of risk. These two probes are +// what the artifact-manifest save now runs before it writes. +// +// They deliberately answer three ways — yes / no / could-not-tell — because "could not verify" and +// "is missing" must not collapse into one another at the save boundary. + +// ProbeResult is a three-valued answer. Err non-nil means UNDETERMINED: the caller must refuse the +// save with a "could not verify" message rather than with a "missing" one. +type ProbeResult struct { + OK bool + Err error +} + +// TagServesFile reports whether `raw/tag//` resolves — i.e. whether the tag exists AND +// its tree carries the file. Both halves matter: a tag that exists but predates the config would +// 404 a box mid-install just as thoroughly as an absent tag. +// +// The caller passes the path the INSTALLER actually fetches. Asserting some other path that merely +// happens to exist is how Friday's failure stayed invisible. +func (c *Client) TagServesFile(ctx context.Context, repo, tag, path string) ProbeResult { + url := fmt.Sprintf("%s/%s/%s/raw/tag/%s/%s", c.baseURL, c.owner, repo, tag, path) + return c.probe(ctx, url) +} + +// PackageDownloadable reports whether a generic package file can actually be fetched. +func (c *Client) PackageDownloadable(ctx context.Context, pkg, version, file string) ProbeResult { + url := fmt.Sprintf("%s/api/packages/%s/generic/%s/%s/%s", c.baseURL, c.owner, pkg, version, file) + return c.probe(ctx, url) +} + +// probe does one GET and maps the outcome onto the three-valued answer. A GET rather than a HEAD: +// Gitea's raw and package routes do not answer HEAD consistently, and a probe that is wrong about +// its own method would be worse than no probe. Nothing reads the body. +func (c *Client) probe(ctx context.Context, url string) ProbeResult { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return ProbeResult{Err: err} + } + if c.token != "" { + req.SetBasicAuth(c.user, c.token) + } + resp, err := c.http.Do(req) + if err != nil { + return ProbeResult{Err: err} // transport failure -> UNDETERMINED, never "missing" + } + defer resp.Body.Close() + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<12)) + switch { + case resp.StatusCode == http.StatusOK: + return ProbeResult{OK: true} + case resp.StatusCode == http.StatusNotFound: + return ProbeResult{OK: false} // a real, determined "it is not there" + default: + // 5xx, 401, 403, a proxy error: the registry did not tell us the thing is absent, it told us + // it could not answer. Undetermined. + return ProbeResult{Err: fmt.Errorf("registry returned HTTP %d for %s", resp.StatusCode, url)} + } +} diff --git a/hub/internal/web/artifact_installability_test.go b/hub/internal/web/artifact_installability_test.go new file mode 100644 index 0000000..1d36dbd --- /dev/null +++ b/hub/internal/web/artifact_installability_test.go @@ -0,0 +1,241 @@ +package web + +import ( + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/gitea" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// The installability gate (R-273). On 2026-08-08 agent v0.128.0 was published as a package and +// never git-tagged; the hub vouched it, and every fresh install and reinstall died at step 5 of 8, +// as root, on a virgin machine, for most of a day. These tests pin the check at the moment of risk. +// +// Each scenario names the WRONG outcome it exists to prevent, because a test whose failure mode is +// unstated is a test nobody can weigh. + +// fakeRegistry serves the two URLs the gate probes. Anything not explicitly present 404s, which is +// the shape a pruned package or an unpushed tag really has. +type fakeRegistry struct { + tagFiles map[string]bool // "//" + packages map[string]bool // "/" + status int // when non-zero, every request answers with this instead (E) + // downloadGone models R-287: the version's METADATA still answers while the file itself is + // gone. Keyed "/"; consulted only on the download route. + downloadGone map[string]bool + // shaless models a version whose metadata answers but carries no sha256. + shaless map[string]bool + hits int +} + +func (f *fakeRegistry) start(t *testing.T) *gitea.Client { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + f.hits++ + if f.status != 0 { + w.WriteHeader(f.status) + return + } + p := strings.TrimPrefix(r.URL.Path, "/") + // Package METADATA (api/v1/packages//generic///files) — what + // resolveArtifactSHA calls before the gate ever runs. Serving it keeps these tests about + // the installability gate rather than about sha resolution. + if strings.HasPrefix(p, "api/v1/packages/") && strings.HasSuffix(p, "/files") { + seg := strings.Split(p, "/") + if len(seg) >= 8 && f.packages[seg[5]+"/"+seg[6]] { + sha := strings.Repeat("c", 64) + if f.shaless[seg[5]+"/"+seg[6]] { + sha = "" + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`[{"name":"` + seg[5] + `","sha256":"` + sha + `"}]`)) + return + } + w.WriteHeader(http.StatusNotFound) + return + } + // raw/tag: //raw/tag// + if i := strings.Index(p, "/raw/tag/"); i >= 0 { + repo := p[:i] + repo = repo[strings.Index(repo, "/")+1:] + rest := p[i+len("/raw/tag/"):] + slash := strings.Index(rest, "/") + if slash > 0 && f.tagFiles[repo+"/"+rest[:slash]+"/"+rest[slash+1:]] { + w.WriteHeader(http.StatusOK) + return + } + w.WriteHeader(http.StatusNotFound) + return + } + // packages: api/packages//generic/// + if strings.HasPrefix(p, "api/packages/") { + seg := strings.Split(p, "/") + if len(seg) >= 6 && f.packages[seg[4]+"/"+seg[5]] && !f.downloadGone[seg[4]+"/"+seg[5]] { + w.WriteHeader(http.StatusOK) + return + } + w.WriteHeader(http.StatusNotFound) + return + } + w.WriteHeader(http.StatusNotFound) + })) + t.Cleanup(srv.Close) + return gitea.New(srv.URL, "admin", "", "") +} + +// healthy is the registry as it should be: agent tagged + published, golden published. +func healthy() *fakeRegistry { + return &fakeRegistry{ + tagFiles: map[string]bool{"felhom-agent/v0.128.0/configs/felhom-mkfs-guarded.sh": true}, + packages: map[string]bool{"felhom-agent/0.128.0": true, "felhom-golden/0.210.0": true}, + } +} + +func saveArtifacts(t *testing.T, s *Server) *httptest.ResponseRecorder { + t.Helper() + form := url.Values{ + "agent_version": {"0.128.0"}, + "golden_version": {"0.210.0"}, + "agent_sha256": {strings.Repeat("a", 64)}, + "golden_sha256": {strings.Repeat("b", 64)}, + } + r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode())) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + w := httptest.NewRecorder() + s.handleSetArtifacts(w, r) + return w +} + +func flashOf(w *httptest.ResponseRecorder) string { + loc := w.Header().Get("Location") + if i := strings.Index(loc, "flash="); i >= 0 { + return loc[i+len("flash="):] + } + return "" +} + +func assertUnchanged(t *testing.T, st *store.Store, why string) { + t.Helper() + if m := st.GetArtifactManifest(); m.AgentVersion != "" || m.GoldenVersion != "" { + t.Fatalf("%s: manifest was WRITTEN anyway: %+v", why, m) + } +} + +// A — the tag is missing and the package is there. THE ONE THAT MATTERS: this is 2026-08-09 exactly. +// WRONG OUTCOME: saved, and every install dies five steps in, as root, on a virgin machine. +func TestInstallability_A_TagMissing(t *testing.T) { + s, st := newTestServer(t) + fr := healthy() + fr.tagFiles = map[string]bool{} // published, never tagged + s.SetGiteaClient(fr.start(t)) + + w := saveArtifacts(t, s) + if got := flashOf(w); got != "artifact_tag_missing" { + t.Fatalf("flash = %q, want artifact_tag_missing — Friday's defect is not being caught", got) + } + assertUnchanged(t, st, "A") +} + +// B — the tag is there and the package was pruned out from under it (R-287). +// WRONG OUTCOME: saved, and the installer 404s on the binary itself. +func TestInstallability_B_PackagePruned(t *testing.T) { + s, st := newTestServer(t) + fr := healthy() + // R-287's real shape: metadata still answers, the FILE does not. Modelled with a separate + // download map so the sha resolver succeeds and the gate's download leg is what convicts. + fr.downloadGone = map[string]bool{"felhom-agent/0.128.0": true} + s.SetGiteaClient(fr.start(t)) + + w := saveArtifacts(t, s) + if got := flashOf(w); got != "artifact_pkg_missing" { + t.Fatalf("flash = %q, want artifact_pkg_missing", got) + } + assertUnchanged(t, st, "B") +} + +// C — both artifacts are installable, but the registry can produce no checksum for one of them. +// The manifest must not record a version with an empty sha: that is vouching bytes nobody verified. +// +// NOTE ON WHERE THIS IS ENFORCED, because the first draft of this test was wrong. With a Gitea +// client configured, resolveArtifactSHA fetches the sha AUTHORITATIVELY and ignores anything +// submitted — so a blank field in the form cannot produce a blank stored sha, and the scenario as +// originally written ("submit an empty sha") modelled nothing real. The genuine shape is Gitea +// answering with no sha256, and the EXISTING refusal owns it. This test pins the guarantee rather +// than the mechanism: whatever refuses, the manifest must be unchanged. +// WRONG OUTCOME: the hub vouches a checksum for bytes it never saw. +func TestInstallability_C_NoChecksum(t *testing.T) { + s, st := newTestServer(t) + fr := healthy() + fr.shaless = map[string]bool{"felhom-agent/0.128.0": true} // metadata answers, with no sha256 + s.SetGiteaClient(fr.start(t)) + + w := saveArtifacts(t, s) + if got := flashOf(w); got == "artifacts_set" { + t.Fatalf("a version with no resolvable checksum SAVED — the hub vouched bytes nobody verified") + } + assertUnchanged(t, st, "C") +} + +// D — everything correct. The gate must be invisible on a good approval. +// WRONG OUTCOME: a new obstacle in front of a correct operator action. +func TestInstallability_D_HappyPathStillSaves(t *testing.T) { + s, st := newTestServer(t) + s.SetGiteaClient(healthy().start(t)) + + w := saveArtifacts(t, s) + if got := flashOf(w); got != "artifacts_set" { + t.Fatalf("flash = %q, want artifacts_set — a correct approval was blocked", got) + } + m := st.GetArtifactManifest() + if m.AgentVersion != "0.128.0" || m.GoldenVersion != "0.210.0" { + t.Fatalf("manifest not saved on the happy path: %+v", m) + } +} + +// E — the registry cannot answer. It must refuse, and it must say "could not verify" rather than +// "missing": those are different facts and the operator acts differently on each. +// WRONG OUTCOME: saved with a warning. A warning beside a success is read as a success. +func TestInstallability_E_RegistryUnreachable(t *testing.T) { + s, st := newTestServer(t) + fr := healthy() + fr.status = http.StatusBadGateway + s.SetGiteaClient(fr.start(t)) + + w := saveArtifacts(t, s) + got := flashOf(w) + if got == "artifacts_set" { + t.Fatalf("saved while the registry was unreachable — the warning-beside-a-success shape") + } + if got != "artifact_unverifiable" { + t.Fatalf("flash = %q, want artifact_unverifiable — an undetermined result must not be "+ + "reported as a missing artifact", got) + } + assertUnchanged(t, st, "E") +} + +// The gate must not fire when there is no registry client at all (the legacy manual path), or a +// hub with Gitea unconfigured could never vouch anything. +func TestInstallability_NoGiteaClient_StillSaves(t *testing.T) { + s, st := newTestServer(t) + w := saveArtifacts(t, s) + if got := flashOf(w); got != "artifacts_set" { + t.Fatalf("flash = %q, want artifacts_set with no Gitea client configured", got) + } + if st.GetArtifactManifest().AgentVersion != "0.128.0" { + t.Fatal("manifest not saved on the no-client path") + } +} + +// The probe must assert the path the INSTALLER fetches. A gate that probes some other path that +// merely exists is how 2026-08-09 stayed invisible. +func TestInstallability_ProbesTheInstallersOwnPath(t *testing.T) { + if installerProbeConfig != "configs/felhom-mkfs-guarded.sh" { + t.Fatalf("probe path = %q; it must be the FIRST config felhom-host-install.sh fetches "+ + "(step 5/8), which is the file whose 404 broke every install on 2026-08-09", + installerProbeConfig) + } +} diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index 2c6990d..d02ec0c 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -1110,6 +1110,12 @@ func normalizeSHA256(raw string) (string, bool) { // hash that can never match. var sha256HexRe = regexp.MustCompile(`^[0-9a-f]{64}$`) +// installerProbeConfig is the FIRST config felhom-host-install.sh fetches from a vouched agent's +// tag (step 5/8, `fetch_raw "configs/felhom-mkfs-guarded.sh"`). It is the exact file whose 404 broke +// every install on 2026-08-09, and it is what the installability gate asserts — the path the +// installer uses, never a path that merely exists. +const installerProbeConfig = "configs/felhom-mkfs-guarded.sh" + func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { http.Error(w, "Bad request", http.StatusBadRequest) @@ -1122,6 +1128,78 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/configuration?flash=artifact_ver_invalid", http.StatusSeeOther) return } + // --- INSTALLABILITY GATE (R-273) ----------------------------------------------------------- + // + // THE INCIDENT. On 2026-08-08 agent v0.128.0 was published as a package and never git-tagged. + // It was vouched here. felhom-host-install.sh fetches an agent's config files from + // `raw/tag/v/configs/`, so EVERY fresh install and every reinstall died at step 5 of 8 + // — as root, on a virgin machine — for most of a day. Nothing checked at the moment of risk, and + // the moment of risk is this handler: it is the sole UI path to SetArtifactManifest. + // + // TWO INDEPENDENT LEGS, because both have failed within two days of each other: + // the TAG — missing on 2026-08-08 (R-273) + // the PACKAGE — pruned out from under a still-tagged version on 2026-08-08/09 (R-287) + // Checking one would have caught one of them. + // + // IT ASSERTS THE PATH THE INSTALLER ACTUALLY FETCHES. `configs/felhom-mkfs-guarded.sh` is the + // FIRST of the sixteen `fetch_raw` calls (felhom-host-install.sh, step 5) and is literally the + // file whose 404 broke Friday. Probing some other path that merely exists is how that failure + // stayed invisible. + // + // IT RUNS BEFORE resolveArtifactSHA, and that ordering is load-bearing. The sha lookup fails + // with `artifact_sha_invalid`, whose text reads "version missing / Gitea unreachable / bad sha" + // — three different facts in one message. If it ran first, an unreachable registry would be + // reported to the operator as a possibly-missing artifact. Probing first means the operator is + // told which of those it actually is. (Found by scenario E failing against the first draft.) + // + // UNDETERMINED IS ALSO A REFUSAL, and it says something different. A warning beside a success is + // read as a success, and this project has the scars; so an unreachable registry refuses too, + // with its own message. There is NO OVERRIDE: the registry is the operator's own server, and if + // it cannot be reached then vouching is moot rather than urgent. + if s.gitea != nil { + ctx := r.Context() + type probeTarget struct{ label, pkg, file, version, repo string } + targets := []probeTarget{} + if agentVer != "" { + targets = append(targets, probeTarget{"agent", pkgAgent, fileAgent, agentVer, "felhom-agent"}) + } + if goldenVer != "" { + targets = append(targets, probeTarget{"golden", pkgGolden, fileGolden, goldenVer, ""}) + } + for _, t := range targets { + // Leg 1 — the tag, but only where a tag is what the installer uses. The golden is + // fetched as a package by version and has no config tree, so a tag probe on it would be + // asserting something the installer never does. + if t.repo != "" { + res := s.gitea.TagServesFile(ctx, t.repo, "v"+t.version, installerProbeConfig) + if res.Err != nil { + s.logger.Printf("[WARN] artifact vouch REFUSED: could not verify the %s tag v%s: %v", t.label, t.version, res.Err) + http.Redirect(w, r, "/configuration?flash=artifact_unverifiable", http.StatusSeeOther) + return + } + if !res.OK { + s.logger.Printf("[WARN] artifact vouch REFUSED: %s v%s has NO usable git tag — "+ + "raw/tag/v%s/%s does not resolve, so every install would 404 at step 5/8 (R-273)", + t.label, t.version, t.version, installerProbeConfig) + http.Redirect(w, r, "/configuration?flash=artifact_tag_missing", http.StatusSeeOther) + return + } + } + // Leg 2 — the artifact itself is fetchable. + res := s.gitea.PackageDownloadable(ctx, t.pkg, t.version, t.file) + if res.Err != nil { + s.logger.Printf("[WARN] artifact vouch REFUSED: could not verify the %s package %s: %v", t.label, t.version, res.Err) + http.Redirect(w, r, "/configuration?flash=artifact_unverifiable", http.StatusSeeOther) + return + } + if !res.OK { + s.logger.Printf("[WARN] artifact vouch REFUSED: %s package %s is NOT downloadable (R-287)", t.label, t.version) + http.Redirect(w, r, "/configuration?flash=artifact_pkg_missing", http.StatusSeeOther) + return + } + } + } + agentSHA, okAS := s.resolveArtifactSHA(r.Context(), pkgAgent, fileAgent, agentVer, r.FormValue("agent_sha256")) goldenSHA, okGS := s.resolveArtifactSHA(r.Context(), pkgGolden, fileGolden, goldenVer, r.FormValue("golden_sha256")) if !okAS || !okGS { diff --git a/hub/internal/web/templates/configuration.html b/hub/internal/web/templates/configuration.html index 9275c05..52302e2 100644 --- a/hub/internal/web/templates/configuration.html +++ b/hub/internal/web/templates/configuration.html @@ -50,6 +50,18 @@ {{if eq .Flash "golden_behind_fleet"}}
Refused: that golden is older than the controller the fleet already runs. A fresh install would land on stale application code — which is R-120, where new boxes shipped a controller that told customers the wrong thing about a missing backup drive. Manifest unchanged. Re-bake the golden on the current controller, publish it, then vouch it here.
{{end}} + {{if eq .Flash "artifact_tag_missing"}} +
Refused: that version has no usable git tag. The installer fetches an agent's config files from raw/tag/v<version>/configs/, so a version published without its tag makes every fresh install and reinstall fail at step 5 of 8 — as root, on a virgin machine. That is exactly what happened on 2026-08-09. Manifest unchanged. Fix it by pushing the tag for that release: git tag -a v<version> <released-commit> && git push origin v<version>, then vouch it here again.
+ {{end}} + {{if eq .Flash "artifact_pkg_missing"}} +
Refused: that version's artifact is not downloadable. The version is tagged but its package is not in the registry, so a box would 404 fetching the binary itself. Manifest unchanged. Publish it — bash scripts/release-agent.sh <version> for the agent, or re-bake and publish the golden — then vouch it here again.
+ {{end}} + {{if eq .Flash "artifact_unverifiable"}} +
Refused: could not verify — this does not mean anything is missing. The registry did not answer, so the hub cannot tell whether that version is installable. It refuses rather than saving with a warning, because a warning beside a success reads as a success. Manifest unchanged. Check that Gitea is up, then try again. There is deliberately no override: the registry is on your own server, so if it is unreachable the vouch can wait.
+ {{end}} + {{if eq .Flash "artifact_sha_missing"}} +
Refused: no checksum for a chosen version. The hub records the sha256 it reads from Gitea, and that lookup came back empty — so saving would vouch bytes nobody verified. Manifest unchanged.
+ {{end}} {{if eq .Flash "pw_changed"}}
Login password changed. It is already in effect — use it next time you sign in. Existing sessions stay logged in.
{{end}}