v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))

Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
2026-07-21 10:12:31 +02:00
parent 8c55ac7fda
commit b2ca63ee9f
14 changed files with 784 additions and 8 deletions
+50
View File
@@ -141,6 +141,56 @@ func (m *Manager) DRConfigured() bool {
return m.loadMarker() != nil
}
// NoteAuthResult implements pbs.AuthSink (R-39 leg c): the credential probe's verdict for one
// storage, turned into the DR bridge's reported state.
//
// This is the leg that makes `applied` mean something. Until v0.91.0 the agent could not read the
// credential it had written (root-only path, no wrapper read verb), so a tier pinned to a superseded
// secret reported `applied` forever while every PBS request 401'd — and the hub, seeing `applied`,
// had no reason to re-key. Now a rejection becomes a LOUD `auth_failed` that pbsdrheal escalates to
// a fresh mint; the fresh mint advances the secret generation; the descriptor hash moves; and Apply
// finally re-consumes.
//
// Rules that keep it safe:
// - Only a REJECTION (401) sets the state. An unreachable PBS is UNKNOWN and must never re-key.
// - Only the storage this box's descriptor actually names is considered; a host may carry other
// PBS entries that are none of the DR tier's business.
// - Recovery is self-clearing: a subsequent successful probe restores the converged state from the
// marker, so the operator does not have to acknowledge a fault that fixed itself.
func (m *Manager) NoteAuthResult(storageID string, unauthorized bool, detail string) {
m.mu.Lock()
st := m.status
m.mu.Unlock()
// No descriptor seen yet, or this is not our storage → not our business.
if st == nil || st.StorageID == "" || storageID == "" || st.StorageID != storageID {
return
}
if unauthorized {
if st.State == "auth_failed" {
return // already loud; do not churn the report
}
m.logger.Error("pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD",
"storage_id", storageID, "previous_state", st.State)
m.setStatus(&hub.PBSDRStatus{
State: "auth_failed", StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: st.AppliedAt,
AuthFailed: true,
Message: detail + " — awaiting fresh credentials from the hub (automatic)",
})
return
}
// A clean probe clears a previously-loud auth failure by restoring the converged marker state.
if st.State == "auth_failed" && detail == "" {
mk := m.loadMarker()
restored := "applied"
appliedAt := st.AppliedAt
if mk != nil {
restored, appliedAt = mk.State, mk.AppliedAt
}
m.logger.Info("pbsdr: credential accepted again — clearing auth_failed", "storage_id", storageID, "state", restored)
m.setStatus(&hub.PBSDRStatus{State: restored, StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: appliedAt})
}
}
// descriptorHash is the idempotency key: sha256 of the canonical (struct-ordered) JSON.
func descriptorHash(b *hub.WirePBSDR) string {
j, _ := json.Marshal(b)
+175
View File
@@ -0,0 +1,175 @@
package pbsdr
import (
"context"
"encoding/json"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// jsonUnmarshal is aliased so the Scenario-C decode reads as intent, not plumbing.
var jsonUnmarshal = json.Unmarshal
// R-39, Scenario A — THE fix: a credential re-key must re-arm a converged agent.
//
// The 2026-07-18 N100 failure in one sentence: an ep0 re-issue re-keys the SECRET of an existing
// token, so token_id, fingerprint, datastore and namespace all come back byte-identical; the
// descriptor hash did not move; the converged agent short-circuited; the fresh secret was never
// consumed; and the box served a revoked credential while reporting `applied`. The hub now stamps a
// monotonic SecretGeneration into the descriptor, and because descriptorHash marshals THIS STRUCT,
// that is what finally moves the hash.
//
// COMPANION RED-PROOF (run + recorded): delete SecretGeneration from hub.WirePBSDR (or stop the hub
// from advancing it) → the two descriptors marshal identically, the marker short-circuit fires, and
// this test FAILS with consume calls stuck at 1. That reproduces the defect exactly.
func TestR39_ReKeyReArmsAConvergedAgent(t *testing.T) {
r := &fakeRunner{}
// found=false on the first apply (fresh create), then the entry exists but is NOT active — which
// is what PVE reports for a storage whose credential is rejected (storage_info catches the 401,
// leaves active=0). That is the state a re-key has to recover from.
st := &fakeStorage{found: false, active: []bool{true}}
c := &fakeConsumer{secret: "SECRET-GEN-1"}
m, _ := newTestManager(t, r, st, c)
block := testBlock()
block.SecretGeneration = 1
m.Apply(context.Background(), true, block)
if c.calls != 1 {
t.Fatalf("first apply consumed %d secrets, want 1", c.calls)
}
if s := m.Status(); s == nil || s.State != "applied" {
t.Fatalf("first apply status = %+v, want applied", s)
}
// A re-apply of the SAME descriptor must stay a no-op — the idempotency the marker exists for.
m.Apply(context.Background(), true, block)
if c.calls != 1 {
t.Fatalf("re-apply of an unchanged descriptor consumed again (calls=%d)", c.calls)
}
// THE RE-KEY. Everything an ep0 re-issue actually returns is unchanged; only the generation moves.
//
// The entry now EXISTS and reads INACTIVE — which is exactly what PVE reports for a PBS storage
// whose credential is rejected: storage_info wraps activate_storage/status in eval{}, warns, and
// leaves the pre-initialised active=0. (Verified against PVE's own source; it returns HTTP 200
// with active:0, never an API error — which is what lets Apply fall through to the recovery path
// instead of bailing out at the status probe.)
st.found = true
st.entry = &proxmox.StorageEntryConfig{Type: "pbs", Namespace: "peti"}
st.active = append(st.active, false, true) // rejected → inactive, healthy after the reconcile
c.secret = "SECRET-GEN-2"
rekeyed := testBlock()
m.Apply(context.Background(), true, rekeyed)
if c.calls != 2 {
t.Fatalf("the re-key did NOT re-arm the agent: consume calls=%d, want 2.\n"+
"The converged short-circuit fired because the descriptor hash did not move — this is the "+
"R-39 defect (N100, 2026-07-18).", c.calls)
}
if s := m.Status(); s == nil || (s.State != "applied" && s.State != "adopted") {
t.Fatalf("post-re-key status = %+v, want converged", s)
}
}
// The generation genuinely changes the hash — the mechanism the test above depends on. Stated
// separately so a failure points at the CAUSE rather than at the flow.
func TestR39_SecretGenerationMovesTheDescriptorHash(t *testing.T) {
a := testBlock()
a.SecretGeneration = 1
b := testBlock()
b.SecretGeneration = 2
if descriptorHash(a) == descriptorHash(b) {
t.Fatal("SecretGeneration does not move descriptorHash — a re-key stays invisible to a " +
"converged agent and the fresh secret is never consumed (R-39)")
}
// And an unchanged generation must NOT move it (or every report would re-apply).
c := testBlock()
c.SecretGeneration = 1
if descriptorHash(a) != descriptorHash(c) {
t.Fatal("identical descriptors hash differently — the agent would re-apply on every tick")
}
}
// Scenario C, from the agent side: a descriptor carrying an UNKNOWN field (what a pre-0.91.0 agent
// sees) must be ignored, not rejected. Asserted by decoding hub JSON that contains a key this build
// does not know about.
func TestR39_UnknownDescriptorFieldIsInert(t *testing.T) {
var b hub.WirePBSDR
raw := []byte(`{"enabled":true,"storage_id":"felhom-pbs","secret_generation":7,"some_future_key":"x"}`)
if err := jsonUnmarshal(raw, &b); err != nil {
t.Fatalf("a descriptor with an unknown key must decode, got %v", err)
}
if !b.Enabled || b.StorageID != "felhom-pbs" || b.SecretGeneration != 7 {
t.Fatalf("known fields lost while ignoring an unknown one: %+v", b)
}
}
// R-39 leg (c) — a rejected credential becomes a LOUD auth_failed, and recovers by itself.
//
// COMPANION RED-PROOF (run + recorded): make NoteAuthResult ignore `unauthorized` (the pre-fix
// Warn-and-skip shape) → the state stays `applied` and this test FAILS.
func TestR39_AuthFailureBecomesLoudAndSelfClears(t *testing.T) {
r := &fakeRunner{}
st := &fakeStorage{found: false, active: []bool{true}}
c := &fakeConsumer{secret: "S"}
m, _ := newTestManager(t, r, st, c)
block := testBlock()
block.SecretGeneration = 1
m.Apply(context.Background(), true, block)
if s := m.Status(); s.State != "applied" {
t.Fatalf("precondition: want applied, got %+v", s)
}
// PBS rejects the credential.
m.NoteAuthResult("felhom-pbs", true, "PBS rejected the stored credential (401)")
s := m.Status()
if s.State != "auth_failed" || !s.AuthFailed {
t.Fatalf("a rejected credential must be LOUD: status = %+v, want auth_failed", s)
}
if s.StorageID != "felhom-pbs" {
t.Errorf("auth_failed must name the storage, got %q", s.StorageID)
}
// A transport error is UNKNOWN, not dead — it must not clear a real fault either.
m.NoteAuthResult("felhom-pbs", false, "dial tcp: connection refused")
if m.Status().State != "auth_failed" {
t.Error("an unreachable PBS cleared a real 401 — a blip must not paper over a dead credential")
}
// A clean probe restores the converged state without operator action.
m.NoteAuthResult("felhom-pbs", false, "")
if got := m.Status().State; got != "applied" {
t.Errorf("recovery did not clear auth_failed: state = %q, want applied", got)
}
}
// Another host's storage must never move this bridge's state.
func TestR39_AuthResultForAnotherStorageIsIgnored(t *testing.T) {
r := &fakeRunner{}
st := &fakeStorage{found: false, active: []bool{true}}
m, _ := newTestManager(t, r, st, &fakeConsumer{secret: "S"})
block := testBlock()
block.SecretGeneration = 1
m.Apply(context.Background(), true, block)
m.NoteAuthResult("some-other-pbs", true, "401")
if got := m.Status().State; got != "applied" {
t.Errorf("an unrelated storage's 401 changed our state to %q", got)
}
}
// A bridge that has never seen a descriptor must not invent a state from a probe.
func TestR39_AuthResultBeforeAnyDescriptorIsIgnored(t *testing.T) {
m := NewManager(&fakeRunner{}, &fakeStorage{}, &fakeConsumer{}, t.TempDir(), "/etc/pve/priv/storage",
t.TempDir()+"/agent.json", slog.New(slog.NewTextHandler(io.Discard, nil)))
m.NoteAuthResult("felhom-pbs", true, "401")
if s := m.Status(); s != nil {
t.Errorf("status invented from a probe with no descriptor: %+v", s)
}
}
+244
View File
@@ -0,0 +1,244 @@
package pbsdr
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// R-39 leg (b), Scenario D — the `read` verb, tested as the ARTIFACT it is.
//
// This verb is the one that EXFILTRATES a file: the agent writes the PBS token secret through the
// root wrapper but could never read it back (/etc/pve/priv is 0700 root:www-data), so its verify
// loop was permanently blind to an `applied`-but-401 tier. Giving it a read path is right, but it
// means the wrapper now has a verb whose whole job is to print a secret — so its refusals are
// security-critical and are executed here under a real bash, not pattern-matched.
//
// COMPANION RED-PROOF (run + recorded): delete the storage-id charset guard at the top of the
// wrapper → TestWrapperRead_RefusesTraversal FAILS (the traversal case is no longer refused).
func wrapperPath(t *testing.T) string {
t.Helper()
p := filepath.Join("..", "..", "configs", "felhom-pbs-apply")
if _, err := os.Stat(p); err != nil {
t.Fatalf("wrapper not found: %v", err)
}
return p
}
// runWrapper executes the real script under bash and returns stdout, combined stderr and the code.
func runWrapper(t *testing.T, args ...string) (string, string, int) {
t.Helper()
cmd := exec.Command("bash", append([]string{wrapperPath(t)}, args...)...)
var out, errb strings.Builder
cmd.Stdout = &out
cmd.Stderr = &errb
err := cmd.Run()
code := 0
if ee, ok := err.(*exec.ExitError); ok {
code = ee.ExitCode()
} else if err != nil {
t.Fatalf("run wrapper: %v", err)
}
return out.String(), errb.String(), code
}
// Every traversal-shaped input is refused, non-zero, and prints NOTHING on stdout — a refusal that
// still emitted bytes would be the leak this test exists to prevent.
func TestWrapperRead_RefusesTraversal(t *testing.T) {
cases := []struct {
name string
args []string
}{
{"dotdot id", []string{"read", "../../etc/shadow", "/etc/pve/priv/storage"}},
{"id with slash", []string{"read", "a/b", "/etc/pve/priv/storage"}},
{"id starting with a dot", []string{"read", ".hidden", "/etc/pve/priv/storage"}},
{"secret dir outside the allowlist", []string{"read", "felhom-pbs", "/tmp"}},
{"secret dir traversal", []string{"read", "felhom-pbs", "/var/lib/felhom-agent/../../etc"}},
{"etc passwd as a dir", []string{"read", "passwd", "/etc"}},
{"missing secret-dir arg", []string{"read", "felhom-pbs"}},
{"too many args", []string{"read", "felhom-pbs", "/etc/pve/priv/storage", "extra"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
stdout, stderr, code := runWrapper(t, tc.args...)
if code == 0 {
t.Errorf("ACCEPTED a traversal-shaped read (rc=0): %v", tc.args)
}
if strings.TrimSpace(stdout) != "" {
t.Errorf("a refused read still wrote to stdout (%q) — that is the leak", stdout)
}
if !strings.Contains(stderr, "REFUSED") {
t.Errorf("refusal not announced on stderr: %q", stderr)
}
})
}
}
// The happy path: a secret under an allowed dir is printed verbatim to STDOUT and nowhere else.
//
// val_sdir hard-codes the /var/lib/felhom-agent prefix, which a test user cannot create. Rather than
// SKIP (a skipped test proves nothing) or weaken the production allowlist with a test escape hatch,
// these two cases run a COPY of the script with that one prefix constant relocated into t.TempDir().
// Only the allowlisted location moves; every guard — the id grammar, the traversal refusal, the
// resolved-path prefix assertion — is the real code. The refusal tests above still run the
// unmodified script.
func relocatedWrapper(t *testing.T, base string) string {
t.Helper()
raw, err := os.ReadFile(wrapperPath(t))
if err != nil {
t.Fatal(err)
}
src := strings.ReplaceAll(string(raw), "/var/lib/felhom-agent", base)
if src == string(raw) {
t.Fatal("relocation matched nothing — val_sdir no longer pins /var/lib/felhom-agent; revisit this test")
}
dst := filepath.Join(t.TempDir(), "felhom-pbs-apply")
if err := os.WriteFile(dst, []byte(src), 0o755); err != nil {
t.Fatal(err)
}
return dst
}
func runScript(t *testing.T, script string, args ...string) (string, string, int) {
t.Helper()
cmd := exec.Command("bash", append([]string{script}, args...)...)
var out, errb strings.Builder
cmd.Stdout = &out
cmd.Stderr = &errb
err := cmd.Run()
code := 0
if ee, ok := err.(*exec.ExitError); ok {
code = ee.ExitCode()
} else if err != nil {
t.Fatalf("run: %v", err)
}
return out.String(), errb.String(), code
}
func TestWrapperRead_PrintsSecretToStdoutOnly(t *testing.T) {
base := t.TempDir()
script := relocatedWrapper(t, base)
dir := filepath.Join(base, "pbs")
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
const secret = "tok-secret-abcdef0123456789"
if err := os.WriteFile(filepath.Join(dir, "felhom-pbs.pw"), []byte(secret+"\n"), 0o600); err != nil {
t.Fatal(err)
}
stdout, stderr, code := runScript(t, script, "read", "felhom-pbs", dir)
if code != 0 {
t.Fatalf("read failed rc=%d stderr=%q", code, stderr)
}
if strings.TrimSpace(stdout) != secret {
t.Errorf("stdout = %q, want the secret verbatim", stdout)
}
if strings.Contains(stderr, secret) {
t.Error("the secret leaked onto stderr, where sudo and the journal would capture it")
}
// The relocated copy must STILL refuse traversal — proving the guards travelled with it and the
// happy path above is not passing because the checks were relocated away.
if _, _, rc := runScript(t, script, "read", "../../etc/shadow", dir); rc == 0 {
t.Error("the relocated copy accepted a traversal id — the guards did not travel")
}
}
// A missing secret file is a clean refusal, never an empty success (an empty secret would build a
// client that 401s and be misdiagnosed as a revoked credential).
func TestWrapperRead_MissingFileRefuses(t *testing.T) {
base := t.TempDir()
script := relocatedWrapper(t, base)
dir := filepath.Join(base, "pbs")
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
stdout, stderr, code := runScript(t, script, "read", "felhom-pbs", dir)
if code == 0 {
t.Error("a missing secret file must refuse, not succeed with empty output")
}
if strings.TrimSpace(stdout) != "" {
t.Errorf("stdout = %q, want empty", stdout)
}
if !strings.Contains(stderr, "REFUSED") {
t.Errorf("stderr = %q, want a REFUSED line", stderr)
}
}
// The read verb must remain read-ONLY: no pvesm/pveum mutation may appear in its block.
func TestWrapperRead_IsSideEffectFree(t *testing.T) {
raw, err := os.ReadFile(wrapperPath(t))
if err != nil {
t.Fatal(err)
}
block := readVerbBlock(t, string(raw))
for _, forbidden := range []string{"pvesm ", "pveum ", "install ", "rm ", "place_copies"} {
if strings.Contains(block, forbidden) {
t.Errorf("the read verb performs a side effect (%q) — it must only print a file", forbidden)
}
}
}
// readVerbBlock isolates the `read)` case body, comment lines stripped (the WHY note names the very
// traversal strings under test, and a naive scan would flag the explanation as the defect — the
// vacuous-pass trap the reconcile guard already documents).
func readVerbBlock(t *testing.T, src string) string {
t.Helper()
start := strings.Index(src, "\nread)\n")
if start < 0 {
t.Fatal("could not locate the read) block in configs/felhom-pbs-apply")
}
rest := src[start+len("\nread)\n"):]
end := strings.Index(rest, "\n ;;")
if end < 0 {
t.Fatal("could not locate the end of the read) block")
}
var code []string
for _, line := range strings.Split(rest[:end], "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "#") {
continue
}
code = append(code, line)
}
return strings.Join(code, "\n")
}
// The prefix assertion, exercised against a file that ACTUALLY EXISTS outside the secret dir.
//
// The earlier traversal cases are refused by whichever guard fires first, so they cannot tell us
// which one is load-bearing — and indeed deleting the id charset guard alone does not open a hole,
// because readlink -f plus the prefix assertion still catch it. That layering is the design, but it
// means a single-guard red-proof passes vacuously. This case isolates the LAST line of defence: a
// real secret file one directory up, reachable only if BOTH the charset guard and the prefix
// assertion are gone.
//
// COMPANION RED-PROOF (run + recorded): delete the id charset guard AND the `case "$resolved" in
// "$sdir"/*)` prefix assertion → this test FAILS by printing the out-of-tree secret.
func TestWrapperRead_PrefixAssertionStopsEscapeToARealFile(t *testing.T) {
base := t.TempDir()
script := relocatedWrapper(t, base)
dir := filepath.Join(base, "pbs")
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
// A juicy file one level ABOVE the secret dir, inside the allowlisted prefix (so val_sdir is not
// the guard doing the work here).
const stolen = "NOT-FOR-THE-AGENT-0123456789"
if err := os.WriteFile(filepath.Join(base, "elsewhere.pw"), []byte(stolen), 0o600); err != nil {
t.Fatal(err)
}
stdout, stderr, code := runScript(t, script, "read", "../elsewhere", dir)
if code == 0 || strings.Contains(stdout, stolen) {
t.Errorf("the read verb escaped its secret dir and printed a file it must never reach.\n"+
" rc=%d stdout=%q", code, stdout)
}
if !strings.Contains(stderr, "REFUSED") {
t.Errorf("escape not refused loudly: stderr=%q", stderr)
}
}