hub v0.102.0 — refuse to vouch a version that cannot be installed (R-273)

The guard owed since Friday morning. Agent v0.128.0 was published as a package
and never git-tagged; it was vouched here; and because felhom-host-install.sh
fetches an agent's configs from raw/tag/v<version>/configs/, every fresh install
and reinstall died at step 5 of 8, as root, on a virgin machine, for most of a
day. handleSetArtifacts is the sole UI path to SetArtifactManifest, so the check
belongs here and nowhere else.

TWO LEGS, because both failed inside two days: the TAG (missing, R-273) and the
PACKAGE (pruned from under a still-tagged version, R-287). Either alone catches
one of them.

It asserts configs/felhom-mkfs-guarded.sh -- the FIRST of the installer's sixteen
fetch_raw calls and literally the file whose 404 broke Friday. A test pins the
constant, because probing a path that merely exists is how it stayed invisible.
The golden gets the package leg only: it has no config tree, so a tag probe would
assert something the installer never does.

"Could not verify" refuses too, with its own message. No override -- the registry
is the operator's own server, so if it is unreachable the vouch can wait.

ORDERING IS LOAD-BEARING AND A FAILING TEST FOUND IT. The probes run before
resolveArtifactSHA, whose flash conflates "missing", "unreachable" and "bad sha".
Probing first means an unreachable registry is reported as unreachable.

Five scenarios each naming the wrong outcome; three red-proofs, mutations asserted
applied and reverted. With the tag check removed, scenario A reports artifacts_set
-- Friday's exact defect returns.
This commit is contained in:
2026-08-09 19:13:45 +02:00
parent 6088afcbed
commit b55fc17d82
5 changed files with 428 additions and 0 deletions
+65
View File
@@ -8,6 +8,7 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"sort"
"strconv"
@@ -146,3 +147,67 @@ func compareSemver(a, b string) int {
}
return 0
}
// --- Installability probes (R-273) --------------------------------------------------------------
//
// WHY THESE EXIST. On 2026-08-08 agent v0.128.0 was published as a package and never git-tagged;
// the hub vouched it, and because felhom-host-install.sh fetches an agent's config files from
// `raw/tag/v<version>/configs/`, EVERY fresh install and reinstall died at step 5 of 8, as root, on
// a virgin machine, for most of a day. Nothing checked at the moment of risk. These two probes are
// what the artifact-manifest save now runs before it writes.
//
// They deliberately answer three ways — yes / no / could-not-tell — because "could not verify" and
// "is missing" must not collapse into one another at the save boundary.
// ProbeResult is a three-valued answer. Err non-nil means UNDETERMINED: the caller must refuse the
// save with a "could not verify" message rather than with a "missing" one.
type ProbeResult struct {
OK bool
Err error
}
// TagServesFile reports whether `raw/tag/<tag>/<path>` resolves — i.e. whether the tag exists AND
// its tree carries the file. Both halves matter: a tag that exists but predates the config would
// 404 a box mid-install just as thoroughly as an absent tag.
//
// The caller passes the path the INSTALLER actually fetches. Asserting some other path that merely
// happens to exist is how Friday's failure stayed invisible.
func (c *Client) TagServesFile(ctx context.Context, repo, tag, path string) ProbeResult {
url := fmt.Sprintf("%s/%s/%s/raw/tag/%s/%s", c.baseURL, c.owner, repo, tag, path)
return c.probe(ctx, url)
}
// PackageDownloadable reports whether a generic package file can actually be fetched.
func (c *Client) PackageDownloadable(ctx context.Context, pkg, version, file string) ProbeResult {
url := fmt.Sprintf("%s/api/packages/%s/generic/%s/%s/%s", c.baseURL, c.owner, pkg, version, file)
return c.probe(ctx, url)
}
// probe does one GET and maps the outcome onto the three-valued answer. A GET rather than a HEAD:
// Gitea's raw and package routes do not answer HEAD consistently, and a probe that is wrong about
// its own method would be worse than no probe. Nothing reads the body.
func (c *Client) probe(ctx context.Context, url string) ProbeResult {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return ProbeResult{Err: err}
}
if c.token != "" {
req.SetBasicAuth(c.user, c.token)
}
resp, err := c.http.Do(req)
if err != nil {
return ProbeResult{Err: err} // transport failure -> UNDETERMINED, never "missing"
}
defer resp.Body.Close()
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<12))
switch {
case resp.StatusCode == http.StatusOK:
return ProbeResult{OK: true}
case resp.StatusCode == http.StatusNotFound:
return ProbeResult{OK: false} // a real, determined "it is not there"
default:
// 5xx, 401, 403, a proxy error: the registry did not tell us the thing is absent, it told us
// it could not answer. Undetermined.
return ProbeResult{Err: fmt.Errorf("registry returned HTTP %d for %s", resp.StatusCode, url)}
}
}
@@ -0,0 +1,241 @@
package web
import (
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// The installability gate (R-273). On 2026-08-08 agent v0.128.0 was published as a package and
// never git-tagged; the hub vouched it, and every fresh install and reinstall died at step 5 of 8,
// as root, on a virgin machine, for most of a day. These tests pin the check at the moment of risk.
//
// Each scenario names the WRONG outcome it exists to prevent, because a test whose failure mode is
// unstated is a test nobody can weigh.
// fakeRegistry serves the two URLs the gate probes. Anything not explicitly present 404s, which is
// the shape a pruned package or an unpushed tag really has.
type fakeRegistry struct {
tagFiles map[string]bool // "<repo>/<tag>/<path>"
packages map[string]bool // "<pkg>/<version>"
status int // when non-zero, every request answers with this instead (E)
// downloadGone models R-287: the version's METADATA still answers while the file itself is
// gone. Keyed "<pkg>/<version>"; consulted only on the download route.
downloadGone map[string]bool
// shaless models a version whose metadata answers but carries no sha256.
shaless map[string]bool
hits int
}
func (f *fakeRegistry) start(t *testing.T) *gitea.Client {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
f.hits++
if f.status != 0 {
w.WriteHeader(f.status)
return
}
p := strings.TrimPrefix(r.URL.Path, "/")
// Package METADATA (api/v1/packages/<owner>/generic/<pkg>/<ver>/files) — what
// resolveArtifactSHA calls before the gate ever runs. Serving it keeps these tests about
// the installability gate rather than about sha resolution.
if strings.HasPrefix(p, "api/v1/packages/") && strings.HasSuffix(p, "/files") {
seg := strings.Split(p, "/")
if len(seg) >= 8 && f.packages[seg[5]+"/"+seg[6]] {
sha := strings.Repeat("c", 64)
if f.shaless[seg[5]+"/"+seg[6]] {
sha = ""
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`[{"name":"` + seg[5] + `","sha256":"` + sha + `"}]`))
return
}
w.WriteHeader(http.StatusNotFound)
return
}
// raw/tag: <owner>/<repo>/raw/tag/<tag>/<path...>
if i := strings.Index(p, "/raw/tag/"); i >= 0 {
repo := p[:i]
repo = repo[strings.Index(repo, "/")+1:]
rest := p[i+len("/raw/tag/"):]
slash := strings.Index(rest, "/")
if slash > 0 && f.tagFiles[repo+"/"+rest[:slash]+"/"+rest[slash+1:]] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
return
}
// packages: api/packages/<owner>/generic/<pkg>/<version>/<file>
if strings.HasPrefix(p, "api/packages/") {
seg := strings.Split(p, "/")
if len(seg) >= 6 && f.packages[seg[4]+"/"+seg[5]] && !f.downloadGone[seg[4]+"/"+seg[5]] {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusNotFound)
}))
t.Cleanup(srv.Close)
return gitea.New(srv.URL, "admin", "", "")
}
// healthy is the registry as it should be: agent tagged + published, golden published.
func healthy() *fakeRegistry {
return &fakeRegistry{
tagFiles: map[string]bool{"felhom-agent/v0.128.0/configs/felhom-mkfs-guarded.sh": true},
packages: map[string]bool{"felhom-agent/0.128.0": true, "felhom-golden/0.210.0": true},
}
}
func saveArtifacts(t *testing.T, s *Server) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{
"agent_version": {"0.128.0"},
"golden_version": {"0.210.0"},
"agent_sha256": {strings.Repeat("a", 64)},
"golden_sha256": {strings.Repeat("b", 64)},
}
r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode()))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
w := httptest.NewRecorder()
s.handleSetArtifacts(w, r)
return w
}
func flashOf(w *httptest.ResponseRecorder) string {
loc := w.Header().Get("Location")
if i := strings.Index(loc, "flash="); i >= 0 {
return loc[i+len("flash="):]
}
return ""
}
func assertUnchanged(t *testing.T, st *store.Store, why string) {
t.Helper()
if m := st.GetArtifactManifest(); m.AgentVersion != "" || m.GoldenVersion != "" {
t.Fatalf("%s: manifest was WRITTEN anyway: %+v", why, m)
}
}
// A — the tag is missing and the package is there. THE ONE THAT MATTERS: this is 2026-08-09 exactly.
// WRONG OUTCOME: saved, and every install dies five steps in, as root, on a virgin machine.
func TestInstallability_A_TagMissing(t *testing.T) {
s, st := newTestServer(t)
fr := healthy()
fr.tagFiles = map[string]bool{} // published, never tagged
s.SetGiteaClient(fr.start(t))
w := saveArtifacts(t, s)
if got := flashOf(w); got != "artifact_tag_missing" {
t.Fatalf("flash = %q, want artifact_tag_missing — Friday's defect is not being caught", got)
}
assertUnchanged(t, st, "A")
}
// B — the tag is there and the package was pruned out from under it (R-287).
// WRONG OUTCOME: saved, and the installer 404s on the binary itself.
func TestInstallability_B_PackagePruned(t *testing.T) {
s, st := newTestServer(t)
fr := healthy()
// R-287's real shape: metadata still answers, the FILE does not. Modelled with a separate
// download map so the sha resolver succeeds and the gate's download leg is what convicts.
fr.downloadGone = map[string]bool{"felhom-agent/0.128.0": true}
s.SetGiteaClient(fr.start(t))
w := saveArtifacts(t, s)
if got := flashOf(w); got != "artifact_pkg_missing" {
t.Fatalf("flash = %q, want artifact_pkg_missing", got)
}
assertUnchanged(t, st, "B")
}
// C — both artifacts are installable, but the registry can produce no checksum for one of them.
// The manifest must not record a version with an empty sha: that is vouching bytes nobody verified.
//
// NOTE ON WHERE THIS IS ENFORCED, because the first draft of this test was wrong. With a Gitea
// client configured, resolveArtifactSHA fetches the sha AUTHORITATIVELY and ignores anything
// submitted — so a blank field in the form cannot produce a blank stored sha, and the scenario as
// originally written ("submit an empty sha") modelled nothing real. The genuine shape is Gitea
// answering with no sha256, and the EXISTING refusal owns it. This test pins the guarantee rather
// than the mechanism: whatever refuses, the manifest must be unchanged.
// WRONG OUTCOME: the hub vouches a checksum for bytes it never saw.
func TestInstallability_C_NoChecksum(t *testing.T) {
s, st := newTestServer(t)
fr := healthy()
fr.shaless = map[string]bool{"felhom-agent/0.128.0": true} // metadata answers, with no sha256
s.SetGiteaClient(fr.start(t))
w := saveArtifacts(t, s)
if got := flashOf(w); got == "artifacts_set" {
t.Fatalf("a version with no resolvable checksum SAVED — the hub vouched bytes nobody verified")
}
assertUnchanged(t, st, "C")
}
// D — everything correct. The gate must be invisible on a good approval.
// WRONG OUTCOME: a new obstacle in front of a correct operator action.
func TestInstallability_D_HappyPathStillSaves(t *testing.T) {
s, st := newTestServer(t)
s.SetGiteaClient(healthy().start(t))
w := saveArtifacts(t, s)
if got := flashOf(w); got != "artifacts_set" {
t.Fatalf("flash = %q, want artifacts_set — a correct approval was blocked", got)
}
m := st.GetArtifactManifest()
if m.AgentVersion != "0.128.0" || m.GoldenVersion != "0.210.0" {
t.Fatalf("manifest not saved on the happy path: %+v", m)
}
}
// E — the registry cannot answer. It must refuse, and it must say "could not verify" rather than
// "missing": those are different facts and the operator acts differently on each.
// WRONG OUTCOME: saved with a warning. A warning beside a success is read as a success.
func TestInstallability_E_RegistryUnreachable(t *testing.T) {
s, st := newTestServer(t)
fr := healthy()
fr.status = http.StatusBadGateway
s.SetGiteaClient(fr.start(t))
w := saveArtifacts(t, s)
got := flashOf(w)
if got == "artifacts_set" {
t.Fatalf("saved while the registry was unreachable — the warning-beside-a-success shape")
}
if got != "artifact_unverifiable" {
t.Fatalf("flash = %q, want artifact_unverifiable — an undetermined result must not be "+
"reported as a missing artifact", got)
}
assertUnchanged(t, st, "E")
}
// The gate must not fire when there is no registry client at all (the legacy manual path), or a
// hub with Gitea unconfigured could never vouch anything.
func TestInstallability_NoGiteaClient_StillSaves(t *testing.T) {
s, st := newTestServer(t)
w := saveArtifacts(t, s)
if got := flashOf(w); got != "artifacts_set" {
t.Fatalf("flash = %q, want artifacts_set with no Gitea client configured", got)
}
if st.GetArtifactManifest().AgentVersion != "0.128.0" {
t.Fatal("manifest not saved on the no-client path")
}
}
// The probe must assert the path the INSTALLER fetches. A gate that probes some other path that
// merely exists is how 2026-08-09 stayed invisible.
func TestInstallability_ProbesTheInstallersOwnPath(t *testing.T) {
if installerProbeConfig != "configs/felhom-mkfs-guarded.sh" {
t.Fatalf("probe path = %q; it must be the FIRST config felhom-host-install.sh fetches "+
"(step 5/8), which is the file whose 404 broke every install on 2026-08-09",
installerProbeConfig)
}
}
+78
View File
@@ -1110,6 +1110,12 @@ func normalizeSHA256(raw string) (string, bool) {
// hash that can never match.
var sha256HexRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
// installerProbeConfig is the FIRST config felhom-host-install.sh fetches from a vouched agent's
// tag (step 5/8, `fetch_raw "configs/felhom-mkfs-guarded.sh"`). It is the exact file whose 404 broke
// every install on 2026-08-09, and it is what the installability gate asserts — the path the
// installer uses, never a path that merely exists.
const installerProbeConfig = "configs/felhom-mkfs-guarded.sh"
func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
http.Error(w, "Bad request", http.StatusBadRequest)
@@ -1122,6 +1128,78 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/configuration?flash=artifact_ver_invalid", http.StatusSeeOther)
return
}
// --- INSTALLABILITY GATE (R-273) -----------------------------------------------------------
//
// THE INCIDENT. On 2026-08-08 agent v0.128.0 was published as a package and never git-tagged.
// It was vouched here. felhom-host-install.sh fetches an agent's config files from
// `raw/tag/v<version>/configs/`, so EVERY fresh install and every reinstall died at step 5 of 8
// — as root, on a virgin machine — for most of a day. Nothing checked at the moment of risk, and
// the moment of risk is this handler: it is the sole UI path to SetArtifactManifest.
//
// TWO INDEPENDENT LEGS, because both have failed within two days of each other:
// the TAG — missing on 2026-08-08 (R-273)
// the PACKAGE — pruned out from under a still-tagged version on 2026-08-08/09 (R-287)
// Checking one would have caught one of them.
//
// IT ASSERTS THE PATH THE INSTALLER ACTUALLY FETCHES. `configs/felhom-mkfs-guarded.sh` is the
// FIRST of the sixteen `fetch_raw` calls (felhom-host-install.sh, step 5) and is literally the
// file whose 404 broke Friday. Probing some other path that merely exists is how that failure
// stayed invisible.
//
// IT RUNS BEFORE resolveArtifactSHA, and that ordering is load-bearing. The sha lookup fails
// with `artifact_sha_invalid`, whose text reads "version missing / Gitea unreachable / bad sha"
// — three different facts in one message. If it ran first, an unreachable registry would be
// reported to the operator as a possibly-missing artifact. Probing first means the operator is
// told which of those it actually is. (Found by scenario E failing against the first draft.)
//
// UNDETERMINED IS ALSO A REFUSAL, and it says something different. A warning beside a success is
// read as a success, and this project has the scars; so an unreachable registry refuses too,
// with its own message. There is NO OVERRIDE: the registry is the operator's own server, and if
// it cannot be reached then vouching is moot rather than urgent.
if s.gitea != nil {
ctx := r.Context()
type probeTarget struct{ label, pkg, file, version, repo string }
targets := []probeTarget{}
if agentVer != "" {
targets = append(targets, probeTarget{"agent", pkgAgent, fileAgent, agentVer, "felhom-agent"})
}
if goldenVer != "" {
targets = append(targets, probeTarget{"golden", pkgGolden, fileGolden, goldenVer, ""})
}
for _, t := range targets {
// Leg 1 — the tag, but only where a tag is what the installer uses. The golden is
// fetched as a package by version and has no config tree, so a tag probe on it would be
// asserting something the installer never does.
if t.repo != "" {
res := s.gitea.TagServesFile(ctx, t.repo, "v"+t.version, installerProbeConfig)
if res.Err != nil {
s.logger.Printf("[WARN] artifact vouch REFUSED: could not verify the %s tag v%s: %v", t.label, t.version, res.Err)
http.Redirect(w, r, "/configuration?flash=artifact_unverifiable", http.StatusSeeOther)
return
}
if !res.OK {
s.logger.Printf("[WARN] artifact vouch REFUSED: %s v%s has NO usable git tag — "+
"raw/tag/v%s/%s does not resolve, so every install would 404 at step 5/8 (R-273)",
t.label, t.version, t.version, installerProbeConfig)
http.Redirect(w, r, "/configuration?flash=artifact_tag_missing", http.StatusSeeOther)
return
}
}
// Leg 2 — the artifact itself is fetchable.
res := s.gitea.PackageDownloadable(ctx, t.pkg, t.version, t.file)
if res.Err != nil {
s.logger.Printf("[WARN] artifact vouch REFUSED: could not verify the %s package %s: %v", t.label, t.version, res.Err)
http.Redirect(w, r, "/configuration?flash=artifact_unverifiable", http.StatusSeeOther)
return
}
if !res.OK {
s.logger.Printf("[WARN] artifact vouch REFUSED: %s package %s is NOT downloadable (R-287)", t.label, t.version)
http.Redirect(w, r, "/configuration?flash=artifact_pkg_missing", http.StatusSeeOther)
return
}
}
}
agentSHA, okAS := s.resolveArtifactSHA(r.Context(), pkgAgent, fileAgent, agentVer, r.FormValue("agent_sha256"))
goldenSHA, okGS := s.resolveArtifactSHA(r.Context(), pkgGolden, fileGolden, goldenVer, r.FormValue("golden_sha256"))
if !okAS || !okGS {
@@ -50,6 +50,18 @@
{{if eq .Flash "golden_behind_fleet"}}
<div class="flash flash-error"><strong>Refused: that golden is older than the controller the fleet already runs.</strong> A fresh install would land on stale application code — which is R-120, where new boxes shipped a controller that told customers the wrong thing about a missing backup drive. Manifest unchanged. Re-bake the golden on the current controller, publish it, then vouch it here.</div>
{{end}}
{{if eq .Flash "artifact_tag_missing"}}
<div class="flash flash-error"><strong>Refused: that version has no usable git tag.</strong> The installer fetches an agent's config files from <code>raw/tag/v&lt;version&gt;/configs/</code>, so a version published without its tag makes <em>every</em> fresh install and reinstall fail at step 5 of 8 — as root, on a virgin machine. That is exactly what happened on 2026-08-09. Manifest unchanged. Fix it by pushing the tag for that release: <code>git tag -a v&lt;version&gt; &lt;released-commit&gt; &amp;&amp; git push origin v&lt;version&gt;</code>, then vouch it here again.</div>
{{end}}
{{if eq .Flash "artifact_pkg_missing"}}
<div class="flash flash-error"><strong>Refused: that version's artifact is not downloadable.</strong> The version is tagged but its package is not in the registry, so a box would 404 fetching the binary itself. Manifest unchanged. Publish it — <code>bash scripts/release-agent.sh &lt;version&gt;</code> for the agent, or re-bake and publish the golden — then vouch it here again.</div>
{{end}}
{{if eq .Flash "artifact_unverifiable"}}
<div class="flash flash-error"><strong>Refused: could not verify — this does not mean anything is missing.</strong> The registry did not answer, so the hub cannot tell whether that version is installable. It refuses rather than saving with a warning, because a warning beside a success reads as a success. Manifest unchanged. Check that Gitea is up, then try again. There is deliberately no override: the registry is on your own server, so if it is unreachable the vouch can wait.</div>
{{end}}
{{if eq .Flash "artifact_sha_missing"}}
<div class="flash flash-error"><strong>Refused: no checksum for a chosen version.</strong> The hub records the sha256 it reads from Gitea, and that lookup came back empty — so saving would vouch bytes nobody verified. Manifest unchanged.</div>
{{end}}
{{if eq .Flash "pw_changed"}}
<div class="flash flash-success">Login password changed. It is already in effect — use it next time you sign in. Existing sessions stay logged in.</div>
{{end}}