From 107f74ea3cddad052f2910eca7a68295c0390844 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 10:01:35 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.68.0=20=E2=80=94=20auth=5Ffailed=20sel?= =?UTF-8?q?f-heal,=20consumed=5Fat=20honesty=20gauge,=20wrapper=20drift=20?= =?UTF-8?q?(R-39=20+=20R-50b(a))?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2). pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a new machine. A box whose credential PBS rejects escalates to a fresh mint, never a re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper: a 401 flap must not become a secret-minting chain. With the generation stamp this closes the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor hash moves, agent re-consumes. consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18 fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one, which is the mint/consume race R-39(a) already recorded. One event per distinct report, and an honestly-stuck box does not double-report (its unconsumed secret is the symptom being healed, not a contradiction). R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift — lighting every host amber on rollout day is how a warning becomes background noise. The delivery channel itself stays R-50b(b)/(c). Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet has self-updated. Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run at the assertion level: removing the auth_failed arm fails the escalation tests with reissues=0. --- hub/CHANGELOG.md | 64 ++++++ hub/internal/pbsdrheal/reconciler.go | 96 ++++++++- hub/internal/pbsdrheal/reconciler_test.go | 200 +++++++++++++++++- hub/internal/store/pbsdr.go | 43 +++- hub/internal/store/store.go | 18 +- hub/internal/web/configs.go | 17 +- hub/internal/web/hosts.go | 91 ++++++-- hub/internal/web/templates/configuration.html | 5 + .../web/templates/host_detail_body.html | 11 + hub/internal/web/wrapper_drift_test.go | 54 +++++ 10 files changed, 561 insertions(+), 38 deletions(-) create mode 100644 hub/internal/web/wrapper_drift_test.go diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 0365148..57be110 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,69 @@ # Felhom Hub — Changelog +## v0.68.0 — a credential re-issue finally re-arms the box (R-39 fleet fix) + wrapper drift is visible (R-50b(a)) (2026-07-21) + +**Coupling, stated honestly: this release is SAFE for agents at 0.90.0** — the new descriptor field +is an unknown JSON key to them; they drop it and behave exactly as today (inert, not breaking). +**The re-arm and auth-honesty guarantees require agent >= 0.91.0.** Raise MinAgent to 0.91.0 only +after the fleet's agents have self-updated. + +### The defect (R-39, fleet half) + +An ep0 credential re-issue re-keys the **secret of an existing token**. `token_id`, `fingerprint`, +`datastore` and `namespace` all come back byte-identical — only the side-table `host_pbs_secrets` +row rotates. The agent's re-apply trigger is a change in the **descriptor content hash** +(`felhom-agent internal/pbsdr/manager.go` `descriptorHash`). Same hash → the converged agent +short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential +→ **401 forever, while both tiers report `applied`**. Proven on the N100 2026-07-18: the agent's +`consumed-failed.json` hash was byte-identical to the `marker.json` written two minutes before the +re-issue. + +### The fix + +- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh **mint** + and by nothing else, stamped into the descriptor as `secret_generation`. That is now the only + field a re-key moves, and it is what re-arms the agent. + - **A re-stage deliberately does not advance it**: it re-arms the *same* secret, the descriptor + content genuinely has not changed, and a bump would cause a pointless agent refetch loop. + - `omitempty` is load-bearing — emitting a zero into every pre-existing descriptor would itself be + a fleet-wide spurious re-apply. + - **Deviation from the spec, deliberate:** the brief said to reuse "the new row's id … no schema + change". There is no row id — the table is keyed by `host_id` and UPSERTed last-write-wins, so a + new row never exists, and `created_at` collides for two mints in one second. An additive counter + column (existing idempotent `ALTER TABLE` idiom) is the only monotonic source available. +- **`pbsdrheal` gains an `auth_failed` trigger** — a NEW trigger in the existing machine, not a new + machine. A box whose credential PBS rejects is escalated to a fresh mint (never a re-stage: that + re-feeds the secret PBS just rejected), through the **existing damping** — a 401 flap must not + become a secret-minting chain. This closes the loop end to end: agent proves the 401 → hub re-keys + → generation advances → descriptor hash moves → agent re-consumes. +- **`consumed_at` honesty gauge** — a staged secret left **unconsumed** past a 15-minute grace while + the box reports `applied` is surfaced loudly with its own event. This is the exact July-18 + fingerprint and a disagreement **no single tier can detect alone**. Deliberately a *surface*, not + a heal: auto-re-issuing here would mint a second secret on top of an unconsumed one — the + mint/consume race R-39(a) already recorded. +- **Corrected a comment that stated a falsehood**: `ReissuePBSDR` claimed it refreshed the descriptor + "with the NEW token_id/fingerprint". That is false for a re-key, and believing it is why nobody + expected the descriptor to come back identical. + +### R-50b(a) — wrapper drift is answerable + +`ArtifactManifest.WrapperSHA256` + an operator field. Unlike the agent binary and the golden, the +PBS-DR wrapper is installed from `raw/branch/main` — unversioned, unpinned, absent from every +manifest — yet it is root-owned 0755 and the pinned sudoers vector. Agents (>= 0.91.0) report the +installed file's hash and the host page surfaces a mismatch. **An unknown on either side reads as +quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes noise. +This does not fix the delivery channel; that stays R-50b(b)/(c). + +### Tests + +Store-level generation monotonicity, per-host isolation and restage-leaves-it-alone; descriptor +byte-change, `omitempty` and sibling-key round-trip; a **flow-level** test driving `ReissuePBSDR` +against a fake that models a real re-key; `auth_failed` escalate/debounce/recovery; the honesty +gauge incl. its grace window, the restage edge, the consumed case and the honest-stuck case; wrapper +drift incl. both unknown directions. **Two red-proofs run at the assertion level** (not the +compiler): removing the generation stamp makes the flow test fail with both byte-identical blocks +printed; removing the `auth_failed` arm makes the escalation tests fail with `reissues=0`. + ## v0.67.0 — the hub stops keeping things to itself: auto-minted self-bind link, post-RESET staleness, unprovisioned-offsite warning (2026-07-18) Four small items, each one a case where the hub already knew something and said nothing. Green: diff --git a/hub/internal/pbsdrheal/reconciler.go b/hub/internal/pbsdrheal/reconciler.go index 2f11fbd..b8975c2 100644 --- a/hub/internal/pbsdrheal/reconciler.go +++ b/hub/internal/pbsdrheal/reconciler.go @@ -33,8 +33,15 @@ import ( // (the secret is untouched by verify-before-consume), self-heals when the tunnel recovers, and must // stay LOUD for the operator if it does not — re-staging a secret would not help it. const ( - stateWaitingSecret = "waiting_secret" + stateWaitingSecret = "waiting_secret" stateConsumedFailed = "consumed_failed" + // stateAuthFailed (R-39, agent >= 0.91.0) — the box HAS a credential, the descriptor is applied, + // and PBS rejects it with 401. Before 0.91.0 this state could not exist: the agent's verify loop + // read the credential file directly as non-root, always failed with "permission denied", and + // skipped — so an applied-and-dead tier was invisible to both tiers. It is healed like + // consumed_failed (escalate to a fresh mint), never by a re-stage: re-staging re-feeds the SAME + // secret PBS just rejected. + stateAuthFailed = "auth_failed" ) // Audit event types (store.SaveEvent; hub-internal, not gated by allowedEventTypes). Distinct per @@ -43,8 +50,18 @@ const ( eventRestaged = "pbsdr_selfheal_restaged" // re-armed the stored secret (routine) eventReissued = "pbsdr_selfheal_reissued" // no stored secret → minted a fresh one eventConsumedFailed = "pbsdr_selfheal_consumed_failed" // burned secret → minted a fresh one (a real problem was remediated) + eventAuthFailed = "pbsdr_selfheal_auth_failed" // PBS rejected the box's credential (401) → minted a fresh one + eventUnconsumed = "pbsdr_unconsumed_secret" // staged secret never consumed under an `applied` box (surfaced, NOT auto-healed) ) +// unconsumedGrace bounds how long a staged-but-unconsumed secret is NORMAL before it is a lie. +// +// A fresh mint (or a deliberate re-stage) is consumed on the agent's next tick — well inside one +// ~15-minute report cycle. Beyond this window, an unconsumed secret sitting under a box that reports +// `applied` is the exact fingerprint of the 2026-07-18 N100 failure: hub minted, agent +// short-circuited, both tiers green, box serving a revoked credential. +const unconsumedGrace = 15 * time.Minute + // Actions is the mutation seam — fakes in tests count calls without SSH/ep0. Restage flips a stored // secret's consumed flag (returns restaged=false when NO row exists → the caller escalates). Reissue // mints a fresh ep0 token + stores a fresh consume-once secret + bumps the descriptor. @@ -71,7 +88,9 @@ func (a storeActions) Reissue(ctx context.Context, customerID string) error { } // NewActions builds the production mutation seam. -func NewActions(st *store.Store, reissuer Reissuer) Actions { return storeActions{st: st, reissuer: reissuer} } +func NewActions(st *store.Store, reissuer Reissuer) Actions { + return storeActions{st: st, reissuer: reissuer} +} // debounceState tracks, per host, the last DISTINCT report observed and how many consecutive // distinct reports it has held the current stuck state — so a fresh box that briefly shows @@ -86,15 +105,18 @@ type debounceState struct { // Reconciler re-arms stuck PBS-DR hosts. DECLARATIVE + IDEMPOTENT: a tick over a converged fleet // writes nothing (Scenario C). It reads the hub DB (the source of truth) — never the box. type Reconciler struct { - store *store.Store - act Actions - interval time.Duration - debounceReports int // distinct stuck reports required before healing (default 2) - onlyHost string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout) - trigger chan struct{} - logger *log.Logger + store *store.Store + act Actions + interval time.Duration + debounceReports int // distinct stuck reports required before healing (default 2) + onlyHost string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout) + trigger chan struct{} + logger *log.Logger deb map[string]debounceState + // unconsumedSeen remembers the last report id already surfaced per host (Scenario F), so a + // sustained disagreement produces one event per fresh report rather than one per tick. + unconsumedSeen map[string]int64 } // NewReconciler builds the reconciler. interval defaults to 5m, debounceReports to 2. @@ -110,6 +132,7 @@ func NewReconciler(st *store.Store, act Actions, logger *log.Logger) *Reconciler trigger: make(chan struct{}, 1), logger: logger, deb: map[string]debounceState{}, + unconsumedSeen: map[string]int64{}, } } @@ -161,6 +184,14 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) { delete(r.deb, row.HostID) continue } + // R-39 consumed_at HONESTY (Scenario F). Deliberately a SURFACE, not a heal: the remediation + // for a genuinely stuck box is the auth_failed / waiting_secret / consumed_failed machinery + // above, driven by what the BOX reports. This check catches the disagreement itself — the hub + // staged a credential the box never took, while the box claims to be applied — which is a + // state no single tier can detect alone. Auto-re-issuing on it would mint a second secret on + // top of an unconsumed one: exactly the mint/consume race R-39(a) already recorded. + r.checkUnconsumed(row) + switch row.ReportedState { case stateWaitingSecret: if r.confirm(row) { @@ -170,6 +201,10 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) { if r.confirm(row) { r.healConsumedFailed(ctx, row) } + case stateAuthFailed: + if r.confirm(row) { + r.healAuthFailed(ctx, row) + } default: // applied | adopted | disabled | verify_failed | "" (no report) | anything else → no-op. delete(r.deb, row.HostID) @@ -183,6 +218,31 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) { } } +// checkUnconsumed surfaces the applied-but-never-consumed disagreement (Scenario F). One event per +// distinct report, so a sustained state does not spam the audit log every tick. +func (r *Reconciler) checkUnconsumed(row store.PBSDRHealRow) { + if row.SecretUnconsumedFor <= unconsumedGrace { + return // no staged secret, already consumed, or still inside the normal pickup window + } + // Only meaningful when the box claims to be converged. A box that honestly reports + // waiting_secret / consumed_failed / auth_failed is already being healed above and its unconsumed + // secret is the SYMPTOM being fixed, not a contradiction. + if row.ReportedState != "applied" && row.ReportedState != "adopted" { + return + } + if st, ok := r.unconsumedSeen[row.HostID]; ok && st == row.ReportID { + return // already surfaced for this exact report + } + if r.unconsumedSeen == nil { + r.unconsumedSeen = map[string]int64{} + } + r.unconsumedSeen[row.HostID] = row.ReportID + r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports pbs_dr=%s while a staged one-time secret has been UNCONSUMED for %s — the box is not using the credential the hub issued (R-39 fingerprint)", + row.HostID, row.CustomerID, row.ReportedState, row.SecretUnconsumedFor.Round(time.Minute)) + r.event(row.CustomerID, eventUnconsumed, "warning", + "PBS-DR honesty check: the box reports the DR tier as applied, but a one-time credential issued by the hub has never been consumed. The tier may be authenticating with a superseded credential.") +} + // confirm advances the per-host debounce and reports whether the stuck state has held across // >= debounceReports DISTINCT reports. A re-observed same report (same reportID) never advances the // streak — the debounce counts fresh evidence, not reconciler ticks. @@ -235,6 +295,24 @@ func (r *Reconciler) healConsumedFailed(ctx context.Context, row store.PBSDRHeal } } +// healAuthFailed escalates a box whose credential PBS rejects (401) to a fresh mint. This is the leg +// that closes the R-39 loop end to end: the agent now PROVES the credential is dead instead of +// silently skipping, the hub re-keys, the fresh mint advances the secret generation, the descriptor +// hash moves, and the agent finally re-consumes (Scenario A). Before this, an applied-and-401 tier +// stayed green forever and would have surfaced first at a real restore. +// +// A re-stage is deliberately NOT attempted: the stored secret IS the one PBS just rejected, so +// re-arming it would burn a tick and change nothing. Damping is the SHARED confirm() — a 401 flap +// must not turn into a secret-minting chain. +func (r *Reconciler) healAuthFailed(ctx context.Context, row store.PBSDRHealRow) { + r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports auth_failed (PBS rejects its credential) — escalating to Re-issue", row.HostID, row.CustomerID) + if r.reissue(ctx, row) { + r.event(row.CustomerID, eventAuthFailed, "warning", + "PBS-DR self-heal: a box reported auth_failed (the DR endpoint rejected its stored credential); re-issued fresh endpoint credentials so the agent can re-consume and converge.") + r.resetAfterHeal(row) + } +} + // reissue runs the escalation; returns true on success (the caller then records the audit event). func (r *Reconciler) reissue(ctx context.Context, row store.PBSDRHealRow) bool { if err := r.act.Reissue(ctx, row.CustomerID); err != nil { diff --git a/hub/internal/pbsdrheal/reconciler_test.go b/hub/internal/pbsdrheal/reconciler_test.go index f575dd6..b377416 100644 --- a/hub/internal/pbsdrheal/reconciler_test.go +++ b/hub/internal/pbsdrheal/reconciler_test.go @@ -12,6 +12,7 @@ import ( "path/filepath" "sync" "testing" + "time" "gitea.dooplex.hu/admin/felhom-hub/internal/store" ) @@ -75,6 +76,16 @@ func seedReport(t *testing.T, st *store.Store, hostID, customerID, state string) } } +// ageSecret back-dates a staged secret's created_at so grace-window behaviour is testable without +// sleeping. Writes the house SQLite datetime format. +func ageSecret(t *testing.T, st *store.Store, hostID string, by time.Duration) { + t.Helper() + when := time.Now().UTC().Add(-by).Format("2006-01-02 15:04:05") + if err := st.SetHostPBSSecretCreatedAtForTest(hostID, when); err != nil { + t.Fatalf("age secret %s: %v", hostID, err) + } +} + func boolStr(b bool) string { if b { return "true" @@ -228,7 +239,7 @@ func TestScenarioE_ConsumedFailedEscalates(t *testing.T) { // F — DR-OFF (disabled descriptor) and enabled-but-unprovisioned hosts are never in the work set. func TestScenarioF_OutOfScopeHostsUntouched(t *testing.T) { st := newHealStore(t) - seedHost(t, st, "hOff", "cOff", false, true) // descriptor disabled + seedHost(t, st, "hOff", "cOff", false, true) // descriptor disabled seedReport(t, st, "hOff", "cOff", stateWaitingSecret) seedHost(t, st, "hUnprov", "cUnprov", true, false) // enabled but never provisioned seedReport(t, st, "hUnprov", "cUnprov", stateWaitingSecret) @@ -274,3 +285,190 @@ func TestNoReHealOnSameReport(t *testing.T) { t.Fatalf("re-healed the same stuck report: restages=%d, want 1", fake.restages()) } } + +// R-39 — auth_failed is escalated to a fresh mint, never re-staged. +// +// This is the leg that closes the loop end to end: the agent (>=0.91.0) now PROVES the credential is +// rejected instead of silently skipping, and the hub re-keys, which advances the secret generation, +// which moves the descriptor hash, which finally re-arms the agent. A re-stage would re-feed the very +// secret PBS just rejected. +// +// COMPANION RED-PROOF (run + recorded): delete the `case stateAuthFailed` arm → the state falls to +// the default no-op branch and this test FAILS with reissues=0. +func TestR39_AuthFailedEscalatesToReissue(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hAF", "cAF", true, true) + seedReport(t, st, "hAF", "cAF", stateAuthFailed) + fake := &fakeActions{restageResult: true} // a re-stageable secret exists — it must STILL not be used + r := newRec(st, fake) + + r.reconcileOnce(context.Background()) + + if fake.restages() != 0 { + t.Errorf("restages=%d, want 0 — re-staging re-feeds the credential PBS just rejected", fake.restages()) + } + if fake.reissues() != 1 { + t.Fatalf("reissues=%d, want 1 — an applied-and-401 tier must be re-keyed, not left green", fake.reissues()) + } + if got := eventTypes(t, st, "cAF"); len(got) != 1 || got[0] != eventAuthFailed { + t.Errorf("events = %v, want [%s] (distinct auth_failed signal, not borrowed from consumed_failed)", got, eventAuthFailed) + } +} + +// The damper is the burn protection: a 401 flap must not become a secret-minting chain. auth_failed +// uses the SAME confirm() debounce as the other triggers — one report is not enough. +func TestR39_AuthFailedIsDebounced(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hAF2", "cAF2", true, true) + fake := &fakeActions{restageResult: true} + r := newRec(st, fake) + r.debounceReports = 2 // require two DISTINCT stuck reports + + seedReport(t, st, "hAF2", "cAF2", stateAuthFailed) + r.reconcileOnce(context.Background()) + if fake.reissues() != 0 { + t.Fatalf("re-issued on the FIRST auth_failed report (reissues=%d) — the damper is bypassed", fake.reissues()) + } + + seedReport(t, st, "hAF2", "cAF2", stateAuthFailed) // a second, distinct report + r.reconcileOnce(context.Background()) + if fake.reissues() != 1 { + t.Fatalf("reissues=%d after two distinct auth_failed reports, want 1", fake.reissues()) + } + + // And the same confirmed report must not re-heal on the next tick. + r.reconcileOnce(context.Background()) + if fake.reissues() != 1 { + t.Errorf("re-healed on a stale report: reissues=%d, want 1", fake.reissues()) + } +} + +// A host that recovers (auth_failed → applied) is a pure no-op, and its streak is forgotten. +func TestR39_AuthFailedRecoveryIsNoOp(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hAF3", "cAF3", true, true) + fake := &fakeActions{restageResult: true} + r := newRec(st, fake) + r.debounceReports = 2 // so a single stuck report is not yet actionable + + seedReport(t, st, "hAF3", "cAF3", stateAuthFailed) + r.reconcileOnce(context.Background()) // streak 1 — below the damper, nothing done + seedReport(t, st, "hAF3", "cAF3", "applied") // the box healed on its own + r.reconcileOnce(context.Background()) + + if fake.reissues() != 0 || fake.restages() != 0 { + t.Errorf("a recovered host was acted on: restages=%d reissues=%d, want 0/0", fake.restages(), fake.reissues()) + } + + // The streak must be FORGOTTEN, not merely paused: a later single auth_failed must start over + // rather than instantly tripping the damper it had half-filled before recovering. + seedReport(t, st, "hAF3", "cAF3", stateAuthFailed) + r.reconcileOnce(context.Background()) + if fake.reissues() != 0 { + t.Errorf("a stale pre-recovery streak survived and fired early: reissues=%d, want 0", fake.reissues()) + } +} + +// R-39 Scenario F — the consumed_at HONESTY gauge. +// +// The July-18 fingerprint is a disagreement no single tier can see: the hub staged a credential, the +// box never took it, and the box nonetheless reports `applied`. This surfaces it. It deliberately +// does NOT auto-heal — minting on top of an unconsumed secret is the mint/consume race R-39(a) +// already recorded. +// +// COMPANION RED-PROOF (run + recorded): delete the r.checkUnconsumed(row) call → no event is written +// and TestR39_UnconsumedSecretUnderAppliedIsSurfaced FAILS. +func TestR39_UnconsumedSecretUnderAppliedIsSurfaced(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hU", "cU", true, true) + seedReport(t, st, "hU", "cU", "applied") // the box claims converged + if _, err := st.SaveHostPBSSecret("hU", "never-taken"); err != nil { + t.Fatalf("mint: %v", err) + } + // Age the staged secret past the grace window. + ageSecret(t, st, "hU", 3*time.Hour) + + fake := &fakeActions{restageResult: true} + r := newRec(st, fake) + r.reconcileOnce(context.Background()) + + if got := eventTypes(t, st, "cU"); len(got) != 1 || got[0] != eventUnconsumed { + t.Fatalf("events = %v, want [%s]", got, eventUnconsumed) + } + // SURFACE, not heal: an applied box is not in any heal arm, so nothing was minted or re-staged. + if fake.reissues() != 0 || fake.restages() != 0 { + t.Errorf("the honesty gauge must not mutate: restages=%d reissues=%d, want 0/0", fake.restages(), fake.reissues()) + } + + // One event per DISTINCT report — a sustained disagreement must not spam the audit log. + r.reconcileOnce(context.Background()) + if got := eventTypes(t, st, "cU"); len(got) != 1 { + t.Errorf("re-surfaced on the same report: %d events, want 1", len(got)) + } +} + +// A freshly staged secret (or a just-restaged one) is NORMAL inside the grace window and must not +// alarm — RestageHostPBSSecret deliberately sets consumed_at back to NULL. +func TestR39_FreshlyStagedSecretDoesNotAlarm(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hF", "cF", true, true) + seedReport(t, st, "hF", "cF", "applied") + if _, err := st.SaveHostPBSSecret("hF", "just-minted"); err != nil { + t.Fatalf("mint: %v", err) + } + // A deliberate re-stage clears consumed_at — the Scenario-F edge case. + if _, err := st.ConsumeHostPBSSecret("hF"); err != nil { + t.Fatalf("consume: %v", err) + } + if _, err := st.RestageHostPBSSecret("hF"); err != nil { + t.Fatalf("restage: %v", err) + } + + r := newRec(st, &fakeActions{restageResult: true}) + r.reconcileOnce(context.Background()) + + if got := eventTypes(t, st, "cF"); len(got) != 0 { + t.Errorf("a just-staged secret alarmed inside the grace window: %v", got) + } +} + +// A CONSUMED secret never alarms, however old. +func TestR39_ConsumedSecretNeverAlarms(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hC", "cC", true, true) + seedReport(t, st, "hC", "cC", "applied") + if _, err := st.SaveHostPBSSecret("hC", "taken"); err != nil { + t.Fatalf("mint: %v", err) + } + if _, err := st.ConsumeHostPBSSecret("hC"); err != nil { + t.Fatalf("consume: %v", err) + } + ageSecret(t, st, "hC", 30*24*time.Hour) + + r := newRec(st, &fakeActions{restageResult: true}) + r.reconcileOnce(context.Background()) + if got := eventTypes(t, st, "cC"); len(got) != 0 { + t.Errorf("a consumed secret alarmed: %v", got) + } +} + +// A box that HONESTLY reports a stuck state is already being healed; its unconsumed secret is the +// symptom, not a contradiction, so the gauge stays quiet and only the heal event is written. +func TestR39_HonestStuckStateDoesNotDoubleReport(t *testing.T) { + st := newHealStore(t) + seedHost(t, st, "hH", "cH", true, true) + seedReport(t, st, "hH", "cH", stateWaitingSecret) + if _, err := st.SaveHostPBSSecret("hH", "staged"); err != nil { + t.Fatalf("mint: %v", err) + } + ageSecret(t, st, "hH", 3*time.Hour) + + r := newRec(st, &fakeActions{restageResult: true}) + r.reconcileOnce(context.Background()) + + for _, e := range eventTypes(t, st, "cH") { + if e == eventUnconsumed { + t.Error("an honestly-stuck box must not also raise the honesty gauge — it is the symptom being healed") + } + } +} diff --git a/hub/internal/store/pbsdr.go b/hub/internal/store/pbsdr.go index 409f1c4..6c7bcef 100644 --- a/hub/internal/store/pbsdr.go +++ b/hub/internal/store/pbsdr.go @@ -3,6 +3,7 @@ package store import ( "database/sql" "encoding/json" + "time" ) // PBS DR tier (SLICE 1): the HOST-scoped one-time PBS token secret — the host/agent twin of the @@ -102,6 +103,17 @@ type PBSDRHealRow struct { DescriptorProvisioned bool // desired_json pbs_dr.namespace != "" (was provisioned, not a bare enable) ReportedState string // latest report pbs_dr.state ("" = no report / no stanza) ReportID int64 // id of that latest host_report (0 = none); the debounce distinctness key + + // R-39 consumed_at honesty (Scenario F). SecretUnconsumedFor is how long an UNCONSUMED secret has + // been staged for this host (0 when none is staged, or when it has already been consumed). + // + // A staged-but-unconsumed secret sitting under a box that reports `applied` is the exact + // fingerprint of the 2026-07-18 N100 failure: the hub minted, the agent short-circuited, and both + // tiers reported success while the box served a revoked credential. Briefly unconsumed is NORMAL + // (a fresh mint, or a deliberate re-stage, is consumed on the agent's next tick) — only a + // SUSTAINED one is a lie, which is why the caller applies a grace window rather than alarming on + // presence alone. + SecretUnconsumedFor time.Duration } // PBSDRHealStates returns one row per host: its descriptor enable/provision flags + the agent's @@ -110,11 +122,13 @@ type PBSDRHealRow struct { // (the reconciler filters to enabled+provisioned hosts). func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) { rows, err := s.db.Query(` - SELECT h.host_id, h.customer_id, h.desired_json, latest.mx, hr.report_json + SELECT h.host_id, h.customer_id, h.desired_json, latest.mx, hr.report_json, + ps.created_at, ps.consumed_at FROM hosts h LEFT JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest ON latest.host_id = h.host_id - LEFT JOIN host_reports hr ON hr.id = latest.mx`) + LEFT JOIN host_reports hr ON hr.id = latest.mx + LEFT JOIN host_pbs_secrets ps ON ps.host_id = h.host_id`) if err != nil { return nil, err } @@ -123,8 +137,9 @@ func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) { for rows.Next() { var hostID, customerID, desiredJSON string var reportID sql.NullInt64 - var reportJSON sql.NullString - if err := rows.Scan(&hostID, &customerID, &desiredJSON, &reportID, &reportJSON); err != nil { + var reportJSON, secretCreatedAt, secretConsumedAt sql.NullString + if err := rows.Scan(&hostID, &customerID, &desiredJSON, &reportID, &reportJSON, + &secretCreatedAt, &secretConsumedAt); err != nil { return nil, err } r := PBSDRHealRow{HostID: hostID, CustomerID: customerID} @@ -151,7 +166,27 @@ func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) { r.ReportedState = rr.PBSDR.State } } + // Unconsumed-secret age: only when a secret is staged AND still unconsumed. + if secretCreatedAt.Valid && !secretConsumedAt.Valid { + // SQLite datetime('now') is UTC and format-varied — parseSQLiteTime is the house parser + // (it returns the zero time on an unparseable value, which we treat as "unknown", never + // as "infinitely stale"). + if created := parseSQLiteTime(secretCreatedAt.String); !created.IsZero() { + if age := time.Since(created); age > 0 { + r.SecretUnconsumedFor = age + } + } + } out = append(out, r) } return out, rows.Err() } + +// SetHostPBSSecretCreatedAtForTest back-dates a staged secret's created_at. TEST-ONLY seam: the +// grace-window behaviour of the consumed_at honesty gauge (R-39 Scenario F) is otherwise only +// observable by sleeping for 15 minutes. It touches nothing else — not the value, not consumed_at, +// not the generation. +func (s *Store) SetHostPBSSecretCreatedAtForTest(hostID, sqliteTime string) error { + _, err := s.db.Exec(`UPDATE host_pbs_secrets SET created_at = ? WHERE host_id = ?`, sqliteTime, hostID) + return err +} diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index c40bf1e..7e5ebe0 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -1476,6 +1476,17 @@ type ArtifactManifest struct { // box whose agent is below it (Part D) — mechanising the publish-train "agent BEFORE controller // floor" rule instead of leaving it to operator discipline. MinAgent string `json:"min_agent"` + // WrapperSHA256 is the sha256 of the PBS-DR apply wrapper (configs/felhom-pbs-apply) the operator + // has vouched (R-50b(a), v0.68.0). + // + // Unlike the agent binary and the golden, this artifact is installed from + // `raw/branch/main` by felhom-host-install.sh — UNVERSIONED, with no tag, no pin and no checksum. + // It is a root-owned 0755 file and the pinned sudoers vector for the PBS storage verbs, so "which + // wrapper is on this host?" was previously unanswerable from any manifest: two hosts installed a + // week apart could carry different privileged code while reporting the same agent version. + // Recording the hash here does not fix the delivery channel (that is R-50b(b)/(c)) — it makes + // DRIFT VISIBLE, which is the cheap honest first step. + WrapperSHA256 string `json:"wrapper_sha256"` } // hub_settings keys for the artifact manifest (BUNDLE slice). Stored as discrete key/value rows in @@ -1487,6 +1498,7 @@ const ( settingArtifactGoldenVersion = "artifact_golden_version" settingArtifactGoldenSHA256 = "artifact_golden_sha256" settingArtifactMinAgent = "artifact_min_agent" + settingArtifactWrapperSHA256 = "artifact_wrapper_sha256" // R-50b(a): the vouched felhom-pbs-apply hash ) // settingOperatorPasswordHash is the hub_settings key for the operator login password bcrypt hash, @@ -1534,6 +1546,7 @@ func (s *Store) GetArtifactManifest() ArtifactManifest { GoldenVersion: s.getSetting(settingArtifactGoldenVersion), GoldenSHA256: s.getSetting(settingArtifactGoldenSHA256), MinAgent: s.getSetting(settingArtifactMinAgent), + WrapperSHA256: s.getSetting(settingArtifactWrapperSHA256), } } @@ -1552,7 +1565,10 @@ func (s *Store) SetArtifactManifest(m ArtifactManifest) error { if err := s.setSetting(settingArtifactGoldenSHA256, m.GoldenSHA256); err != nil { return err } - return s.setSetting(settingArtifactMinAgent, m.MinAgent) + if err := s.setSetting(settingArtifactMinAgent, m.MinAgent); err != nil { + return err + } + return s.setSetting(settingArtifactWrapperSHA256, m.WrapperSHA256) } // EffectiveMinControllerVersion resolves the floor that actually applies to a customer: the diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index d7bda3d..b45e8a4 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -1019,6 +1019,12 @@ func normalizeSHA256(raw string) (string, bool) { // that version's sha256 from Gitea itself (never trusting a client-supplied checksum), so there is no // hand-copied sha to get wrong. When no Gitea client is configured (no registry creds) it falls back // to the submitted sha256 (legacy manual path). Empty version clears that artifact. +// sha256HexRe validates an operator-typed sha256 (R-50b(a) wrapper hash): exactly 64 lowercase hex +// characters. The agent/golden hashes are resolved from the package registry instead, so this is the +// only manifest field a human types by hand — and a truncated paste must be refused, not stored as a +// hash that can never match. +var sha256HexRe = regexp.MustCompile(`^[0-9a-f]{64}$`) + func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { http.Error(w, "Bad request", http.StatusBadRequest) @@ -1037,18 +1043,27 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther) return } + // R-50b(a): the PBS-DR wrapper hash is operator-typed, not resolved from the package registry — + // unlike the agent binary and the golden, this artifact is not published there at all. It is + // installed from raw/branch/main, which is exactly the drift this field makes visible. + wrapperSHA := strings.ToLower(strings.TrimSpace(r.FormValue("wrapper_sha256"))) + if wrapperSHA != "" && !sha256HexRe.MatchString(wrapperSHA) { + http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther) + return + } if err := s.store.SetArtifactManifest(store.ArtifactManifest{ AgentVersion: agentVer, AgentSHA256: agentSHA, GoldenVersion: goldenVer, GoldenSHA256: goldenSHA, MinAgent: minAgent, + WrapperSHA256: wrapperSHA, }); err != nil { s.logger.Printf("[ERROR] Failed to set artifact manifest: %v", err) http.Error(w, "Internal error", http.StatusInternalServerError) return } - s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q", agentVer, goldenVer, minAgent) + s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q wrapper_sha=%t", agentVer, goldenVer, minAgent, wrapperSHA != "") // Agent-plane immediate-sync (Direction-2a, v0.59.0): a MinAgent-floor / vouched-agent change is // a fleet-wide agent-plane intent shift. Fire-and-forget nudge every box so it re-reports at // once (the self-update train's signed op / floor re-evaluation lands in seconds, not ≤15 min). diff --git a/hub/internal/web/hosts.go b/hub/internal/web/hosts.go index 5219823..c8e8eb8 100644 --- a/hub/internal/web/hosts.go +++ b/hub/internal/web/hosts.go @@ -356,6 +356,49 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) { // hostDetailData assembles the view-model map the shared host_detail_body sub-template // renders — used by BOTH the standalone /hosts/{id} page and the customer page's Host tab // (v0.47.0). Booleans/counts only for DR/escrow; never api_key or blob contents. +// wrapperDrift compares the PBS-DR wrapper hash a host REPORTS against the one the operator vouched +// in the artifact manifest (R-50b(a), v0.68.0). +// +// The wrapper is a root-owned 0755 file installed from `raw/branch/main` — unversioned, unpinned and +// absent from every manifest until now, so two hosts installed a week apart could carry different +// privileged code while reporting the same agent version. This does not fix the delivery channel +// (R-50b(b)/(c)); it makes drift VISIBLE, which is what was missing. +// +// Returns ("", "") when either side is unknown: a hub that has not vouched a hash, or an agent below +// 0.91.0 that does not report one, is NOT drift — treating "unknown" as "mismatch" would light every +// host amber on the day this ships and teach the operator to ignore it. +func (s *Server) wrapperDrift(reportJSON string) (drift string, reported string) { + reported = parseReportedWrapperSHA(reportJSON) + return compareWrapperSHA(reported, s.store.GetArtifactManifest().WrapperSHA256), reported +} + +// compareWrapperSHA is the pure comparison: "" (quiet) when either side is unknown, else ok/mismatch. +func compareWrapperSHA(reported, vouched string) string { + if reported == "" || vouched == "" { + return "" + } + if !strings.EqualFold(reported, vouched) { + return "mismatch" + } + return "ok" +} + +// parseReportedWrapperSHA pulls host.wrapper_sha256 out of a host report ("" when absent). +func parseReportedWrapperSHA(reportJSON string) string { + if strings.TrimSpace(reportJSON) == "" { + return "" + } + var doc struct { + Host struct { + WrapperSHA256 string `json:"wrapper_sha256"` + } `json:"host"` + } + if json.Unmarshal([]byte(reportJSON), &doc) != nil { + return "" + } + return strings.ToLower(strings.TrimSpace(doc.Host.WrapperSHA256)) +} + func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]interface{} { status := s.hostStatus(host.LastReportAt) @@ -369,6 +412,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in reportJSON, _ := s.store.GetLatestHostReportJSON(host.CustomerID) vitals := parseHostVitals(reportJSON) + wrapperDrift, reportedWrapperSHA := s.wrapperDrift(reportJSON) storageTargets := parseHostStorageTargets(reportJSON) sort.Slice(storageTargets, func(i, j int) bool { return storageTargets[i].Name < storageTargets[j].Name }) // v0.51.0: capability chips — non-ok first (what the operator needs to see), then by name. @@ -395,28 +439,31 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in escrow, _ := s.store.GetHostEscrow(host.HostID) return map[string]interface{}{ - "HostID": host.HostID, - "CustomerID": host.CustomerID, - "CustomerName": s.customerName(host.CustomerID), - "AgentVersion": host.AgentVersion, - "CreatedAt": host.CreatedAt, - "Status": status, - "StatusLabel": hostStatusLabel(status), - "StatusClass": hostStatusClass(status), - "LastReportAt": host.LastReportAt, - "HasReport": host.LastReportAt != nil, - "RecoveryMode": host.InRecoveryMode(time.Now()), - "RecoveryUntil": host.RecoveryModeUntil, - "DesiredGeneration": host.DesiredGeneration, - "Vitals": vitals, - "Guests": guests, - "GuestRunning": guestRunning, - "GuestTotal": len(guests), - "StorageTargets": storageTargets, - "Capabilities": capabilities, - "NeedsDRMigration": capabilitiesNeedDRMigration(capabilities), - "DRPresent": drBundle != nil, - "EscrowPresent": escrow != nil, + "WrapperDrift": wrapperDrift, + "ReportedWrapperSHA": reportedWrapperSHA, + "VouchedWrapperSHA": s.store.GetArtifactManifest().WrapperSHA256, + "HostID": host.HostID, + "CustomerID": host.CustomerID, + "CustomerName": s.customerName(host.CustomerID), + "AgentVersion": host.AgentVersion, + "CreatedAt": host.CreatedAt, + "Status": status, + "StatusLabel": hostStatusLabel(status), + "StatusClass": hostStatusClass(status), + "LastReportAt": host.LastReportAt, + "HasReport": host.LastReportAt != nil, + "RecoveryMode": host.InRecoveryMode(time.Now()), + "RecoveryUntil": host.RecoveryModeUntil, + "DesiredGeneration": host.DesiredGeneration, + "Vitals": vitals, + "Guests": guests, + "GuestRunning": guestRunning, + "GuestTotal": len(guests), + "StorageTargets": storageTargets, + "Capabilities": capabilities, + "NeedsDRMigration": capabilitiesNeedDRMigration(capabilities), + "DRPresent": drBundle != nil, + "EscrowPresent": escrow != nil, // v0.60.0 Part B: retained superseded escrow blobs (data-first — old passphrases stay // R-recoverable). Operator-only surface. "SupersededEscrowCount": func() int { n, _ := s.store.CountSupersededEscrow(host.HostID); return n }(), diff --git a/hub/internal/web/templates/configuration.html b/hub/internal/web/templates/configuration.html index d3afc16..5a38ec1 100644 --- a/hub/internal/web/templates/configuration.html +++ b/hub/internal/web/templates/configuration.html @@ -180,6 +180,11 @@ The golden's controller CHANGELOG MinAgent:. The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. + +
+ + + sha256 of configs/felhom-pbs-apply (R-50b). Unlike the agent and golden, this root-owned wrapper is installed from raw/branch/main — unversioned and unpinned. Recording it here does not fix the channel; it makes host drift visible: agents report the installed file's hash and a mismatch is surfaced on the host. diff --git a/hub/internal/web/templates/host_detail_body.html b/hub/internal/web/templates/host_detail_body.html index 9046ef4..cd61142 100644 --- a/hub/internal/web/templates/host_detail_body.html +++ b/hub/internal/web/templates/host_detail_body.html @@ -22,6 +22,17 @@ Agent Version {{if .AgentVersion}}{{.AgentVersion}}{{else}}—{{end}}
+ {{if .WrapperDrift}} +
+ PBS wrapper + {{if eq .WrapperDrift "ok"}} + matches vouched {{slice .ReportedWrapperSHA 0 12}}… + {{else}} + DRIFT — installed {{slice .ReportedWrapperSHA 0 12}}…, vouched {{slice .VouchedWrapperSHA 0 12}}…
+ /usr/local/sbin/felhom-pbs-apply differs from the manifest. It is delivered unversioned from main (R-50b), so this host may be running privileged code from a different commit. + {{end}} +
+ {{end}}
Enrolled {{timeAgo .CreatedAt}} diff --git a/hub/internal/web/wrapper_drift_test.go b/hub/internal/web/wrapper_drift_test.go new file mode 100644 index 0000000..e24db9a --- /dev/null +++ b/hub/internal/web/wrapper_drift_test.go @@ -0,0 +1,54 @@ +package web + +import "testing" + +// R-50b(a) — wrapper drift must be VISIBLE, and "unknown" must never read as "mismatch". +// +// The wrapper is root-owned, 0755, and installed from raw/branch/main: unversioned, unpinned, and +// absent from every manifest until v0.68.0. This is the surface that makes drift answerable. +func TestParseReportedWrapperSHA(t *testing.T) { + cases := []struct { + name, report, want string + }{ + {"present", `{"host":{"wrapper_sha256":"AABBCC"}}`, "aabbcc"}, + {"lowercased and trimmed", `{"host":{"wrapper_sha256":" AaBb "}}`, "aabb"}, + {"absent key", `{"host":{"agent_version":"0.91.0"}}`, ""}, + {"no host stanza", `{"guests":[]}`, ""}, + {"empty report", ``, ""}, + {"malformed json", `{nope`, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := parseReportedWrapperSHA(tc.report); got != tc.want { + t.Errorf("parseReportedWrapperSHA(%q) = %q, want %q", tc.report, got, tc.want) + } + }) + } +} + +// The comparison itself. The load-bearing case is the pair of UNKNOWNS: an un-vouched hub, or an +// agent below 0.91.0 that reports nothing, must be quiet — lighting every host amber on rollout day +// is how a warning gets trained into background noise. +func TestWrapperDriftComparison(t *testing.T) { + const vouched = "1111111111111111111111111111111111111111111111111111111111111111" + const other = "2222222222222222222222222222222222222222222222222222222222222222" + + cases := []struct { + name, reported, vouched, want string + }{ + {"match", vouched, vouched, "ok"}, + {"match is case-insensitive", "AAAA", "aaaa", "ok"}, + {"mismatch", other, vouched, "mismatch"}, + {"agent reports nothing (pre-0.91.0) → quiet", "", vouched, ""}, + {"hub has vouched nothing → quiet", vouched, "", ""}, + {"neither side known → quiet", "", "", ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := compareWrapperSHA(tc.reported, tc.vouched) + if got != tc.want { + t.Errorf("compareWrapperSHA(%q, %q) = %q, want %q", tc.reported, tc.vouched, got, tc.want) + } + }) + } +}