From 6d7904786c814be90018dcc9e2541e34fde16935 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 13:41:12 +0200 Subject: [PATCH] agent v0.125.0: open the sealed bundle, return one field (R-199 links 7-8) Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not exist: that selftest writes the whole bundle JSON and its success message named "tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the offsite repository password into the same bundle. It now names what THIS bundle carried and what it did not. POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS token, not the WG key -- the controller is a trust tier down and needs none of them. R: in memory for one call, cleared on every path, never on disk, never in argv, never logged, never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness rather than a content scan, because a content scan is defeated by a later call overwriting the leaked file, which is how the first version of that test passed its own red-proof while R sat on disk. Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a code that does not open it (400, fail-closed at the KDF, nothing written). The wiring is asserted by an AST walk from func main() to the Options field, not by grep. --- CHANGELOG.md | 44 ++++ .../escrow_recover_wiring_test.go | 188 +++++++++++++++++ cmd/felhom-agent/main.go | 49 ++++- internal/escrow/recover.go | 74 +++++++ internal/escrow/recover_test.go | 189 ++++++++++++++++++ internal/hub/client.go | 47 +++++ internal/localapi/escrow_recover.go | 102 ++++++++++ internal/localapi/server.go | 23 +++ 8 files changed, 713 insertions(+), 3 deletions(-) create mode 100644 cmd/felhom-agent/escrow_recover_wiring_test.go create mode 100644 internal/escrow/recover.go create mode 100644 internal/escrow/recover_test.go create mode 100644 internal/localapi/escrow_recover.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e616ee..543bec1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,47 @@ +## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 7–8) + +**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed +since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the +recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest +writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token` — +an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the +offsite repository password into the same bundle. Anyone reading that output would conclude the +repository password was not there. It now names what THIS bundle actually carried and what it did not. + +**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the +customer's recovery code, the agent fetches this host's own sealed blob from the hub +(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and +returns **only the offsite restic repository password** plus its sha256. + +**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG +private key. The controller is a trust tier down and needs none of them; returning them would widen +the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not +recoverable later. + +**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately +absent from the controller image; the blob is a host-scoped object whose only writer is this agent +under the per-host key, so the read is that write's mirror. + +**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned +channel, is held in memory for one call, is cleared on the success path AND every failure path, is +never written to disk, never an argument in a process list, never logged at any level including +inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered +plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty +afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later +call overwriting the leaked file, which is exactly how the first version of that test passed its own +red-proof while R sat on disk. + +Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a +bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and +a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to +re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids. + +**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main` → `runDaemon` → +`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as +`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping +that makes cross-host retrieval impossible is a property of which key is used). This project's +built-but-never-wired count is six and links 6–7 were two of them; the fix must not become the seventh. + ## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190) **v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The diff --git a/cmd/felhom-agent/escrow_recover_wiring_test.go b/cmd/felhom-agent/escrow_recover_wiring_test.go new file mode 100644 index 0000000..064d8a7 --- /dev/null +++ b/cmd/felhom-agent/escrow_recover_wiring_test.go @@ -0,0 +1,188 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "strings" + "testing" +) + +// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by +// grepping for a string. +// +// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six, +// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree +// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no +// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass +// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag +// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be +// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`. + +func parseMain(t *testing.T) (*token.FileSet, *ast.File) { + t.Helper() + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments) + if err != nil { + t.Fatalf("parsing main.go: %v", err) + } + return fset, f +} + +// callsWithin returns the set of function names called (directly, by identifier or selector) inside +// the named top-level function. +func callsWithin(f *ast.File, fnName string) map[string]bool { + out := map[string]bool{} + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil { + continue + } + ast.Inspect(fd.Body, func(n ast.Node) bool { + ce, ok := n.(*ast.CallExpr) + if !ok { + return true + } + switch fn := ce.Fun.(type) { + case *ast.Ident: + out[fn.Name] = true + case *ast.SelectorExpr: + if x, ok := fn.X.(*ast.Ident); ok { + out[x.Name+"."+fn.Sel.Name] = true + } + out[fn.Sel.Name] = true + } + return true + }) + } + return out +} + +// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field. +func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) { + _, f := parseMain(t) + + // 1. main() reaches runDaemon. + if !callsWithin(f, "main")["runDaemon"] { + t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one") + } + // 2. runDaemon reaches buildLocalAPIServer. + if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] { + t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path") + } + + // 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and + // an escrow.OffsiteKeyRecoverer is constructed there. + var optionsHasField, recovererConstructed bool + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil { + continue + } + ast.Inspect(fd.Body, func(n ast.Node) bool { + cl, ok := n.(*ast.CompositeLit) + if !ok { + return true + } + sel, ok := cl.Type.(*ast.SelectorExpr) + if !ok { + return true + } + pkg, _ := sel.X.(*ast.Ident) + if pkg == nil { + return true + } + switch pkg.Name + "." + sel.Sel.Name { + case "localapi.Options": + for _, el := range cl.Elts { + kv, ok := el.(*ast.KeyValueExpr) + if !ok { + continue + } + if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" { + optionsHasField = true + } + } + case "escrow.OffsiteKeyRecoverer": + recovererConstructed = true + } + return true + }) + } + if !recovererConstructed { + t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " + + "production assembly point (the built-but-never-wired shape, seventh instance)") + } + if !optionsHasField { + t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " + + "exists and the route would answer 503 forever") + } +} + +// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different +// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH +// key is used. +func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) { + fset, f := parseMain(t) + var fetchUsesHubClient bool + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil { + continue + } + ast.Inspect(fd.Body, func(n ast.Node) bool { + ce, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := ce.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "FetchIdentityEscrow" { + return true + } + if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" { + fetchUsesHubClient = true + } else { + t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client", + fset.Position(ce.Pos())) + } + return true + }) + } + if !fetchUsesHubClient { + t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " + + "not wired, or it uses a client whose credentials are not this host's") + } +} + +// The route itself must be registered on the local API. A handler with no route is the same defect +// one layer down, and it has shipped here before. +func TestRecoverRouteIsRegistered(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0) + if err != nil { + t.Fatalf("parsing localapi/server.go: %v", err) + } + var registered bool + ast.Inspect(f, func(n ast.Node) bool { + ce, ok := n.(*ast.CallExpr) + if !ok || len(ce.Args) < 2 { + return true + } + sel, ok := ce.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "HandleFunc" { + return true + } + lit, ok := ce.Args[0].(*ast.BasicLit) + if !ok { + return true + } + if strings.Contains(lit.Value, "/escrow/recover-offsite-password") { + registered = true + } + return true + }) + if !registered { + t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " + + "exists and nothing can reach it") + } +} diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index f0b2b27..dbcdd1a 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -1099,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int return false }, } - localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) + localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) if localTokens != nil { defer localTokens.Close() } @@ -1677,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re // leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the // daemon — the host still reports/reconciles; only the controller channel is unavailable until // fixed. The opened token store is returned via outTokens so the caller can Close it. -func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server { +func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server { if !cfg.LocalAPI.Enabled() { return nil } @@ -1749,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St gaMode = proxmox.RunnerSudo } guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger) + // R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's + // own hub client (per-host key, self-scoped server-side), so the recoverer can never read another + // host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config + // cannot reach this line (the daemon exits above), so the seam is always live in production — + // which is the point: links 6 and 7 spent months existing without a caller. + escrowRecoverer := escrow.OffsiteKeyRecoverer{ + Fetch: func(ctx context.Context) ([]byte, bool, error) { + resp, ferr := hubClient.FetchIdentityEscrow(ctx) + if ferr != nil { + return nil, false, ferr + } + if !resp.Present || resp.IdentityEscrowB64 == "" { + return nil, false, nil + } + blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64) + if derr != nil { + return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)") + } + return blob, true, nil + }, + } srv, err := localapi.NewServer(localapi.Options{ + EscrowRecovery: escrowRecoverer, ListenAddr: cfg.LocalAPI.ListenAddr, Cert: cert, AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel @@ -2871,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger * fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err) return 1 } - fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest) + // R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was + // accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the + // offsite repository password into the same bundle. Anyone reading the old output would conclude the + // repository password was not there, and that is part of how the chain's extraction link came to be + // described as missing for a month. Name what was recovered from THIS bundle, and name what is + // absent, rather than reciting a fixed list. + recovered := []string{"tunnel_token", "pbs_token"} + var absent []string + if bundle.WGPrivateKey != "" { + recovered = append(recovered, "wg_private_key") + } else { + absent = append(absent, "wg_private_key") + } + if bundle.ResticRepoPassword != "" { + recovered = append(recovered, "restic_repo_password") + } else { + absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)") + } + fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest) + if len(absent) > 0 { + fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; ")) + } // S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME // identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a diff --git a/internal/escrow/recover.go b/internal/escrow/recover.go new file mode 100644 index 0000000..9bd8856 --- /dev/null +++ b/internal/escrow/recover.go @@ -0,0 +1,74 @@ +package escrow + +import ( + "context" + "errors" + "fmt" +) + +// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery +// code R, and hand back EXACTLY ONE field: the offsite restic repository password. +// +// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and +// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one +// trust tier down — needs none of them, and returning them would widen the blast radius of a +// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later. +// +// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a +// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper +// keeps that property: it takes R as an argument, passes it straight through, and holds no copy. +// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent). +// +// The errors below are DISTINCT on purpose. "no blob", "wrong code" and "the blob predates the field" +// are three different situations for the operator and only one of them is a fault. + +var ( + // ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run. + ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)") + // ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected + // for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be + // retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is + // not sent hunting for a mistyped recovery code that was typed correctly. + ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)") +) + +// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none". +// An interface-free func field keeps this package free of any dependency on the hub client. +type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error) + +// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R. +type OffsiteKeyRecoverer struct { + Fetch BlobFetcher +} + +// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password. +// +// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits +// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere. +// That property is the crypto's, not a check here, which is why this function has no "validate R" +// step to get wrong. +// +// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes. +func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) { + if r.Fetch == nil { + return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured") + } + if recoveryCode == "" { + return "", fmt.Errorf("escrow: the recovery code is required") + } + blob, present, err := r.Fetch(ctx) + if err != nil { + return "", fmt.Errorf("escrow: fetching the sealed bundle: %w", err) // carries no secret + } + if !present || len(blob) == 0 { + return "", ErrNoEscrowBlob + } + bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode) + if err != nil { + return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it + } + if bundle.ResticRepoPassword == "" { + return "", ErrNoResticPassword + } + return bundle.ResticRepoPassword, nil +} diff --git a/internal/escrow/recover_test.go b/internal/escrow/recover_test.go new file mode 100644 index 0000000..31aeb49 --- /dev/null +++ b/internal/escrow/recover_test.go @@ -0,0 +1,189 @@ +package escrow + +import ( + "context" + "errors" + "os" + "path/filepath" + "strings" + "testing" +) + +// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips +// elsewhere). These are the unit half of the session's question — "is the repository password +// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware. + +const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel" + +func sealBundle(t *testing.T, b IdentityBundle, r string) []byte { + t.Helper() + blob, err := WrapIdentityBundle(context.Background(), b, r) + if err != nil { + t.Fatalf("WrapIdentityBundle: %v", err) + } + return blob +} + +func fetcherFor(blob []byte) BlobFetcher { + return func(context.Context) ([]byte, bool, error) { return blob, true, nil } +} + +// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it +// is the REPOSITORY password rather than some other field of a bundle that also parses. +// +// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of +// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS. +// That mutation is the shape of the bug that would otherwise ship silently, because every one of +// those fields is a non-empty string that looks like a secret. +func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) { + ensureAge(t) + const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + blob := sealBundle(t, IdentityBundle{ + TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER", + PBSToken: "PBS-TOKEN-NOT-THE-ANSWER", + WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=", + ResticRepoPassword: repoPW, + }, testR) + + got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR) + if err != nil { + t.Fatalf("recover: %v", err) + } + if got != repoPW { + t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+ + "field of the bundle was returned", len(got), len(repoPW)) + } + // Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check + // above by coincidence. + for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} { + if got == other { + t.Fatalf("the recoverer returned the wrong bundle field") + } + } +} + +// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is +// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no +// validation step here to get wrong — the test pins that it stays that way. +func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) { + ensureAge(t) + const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff" + blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR) + + got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all") + if err == nil { + t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids") + } + if got != "" { + t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got)) + } + // The error may name the step; it may never name a secret. + for _, secret := range []string{repoPW, testR, "not the recovery code at all"} { + if strings.Contains(err.Error(), secret) { + t.Fatalf("the failure message leaked a secret: %v", err) + } + } +} + +// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before +// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly +// typed recovery code would be the wrong instruction. +func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) { + ensureAge(t) + blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR) + + _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR) + if !errors.Is(err, ErrNoResticPassword) { + t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err) + } +} + +// Scenario D at this layer — no blob is a clean, distinguishable answer. +func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) { + rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }} + _, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR) + if !errors.Is(err, ErrNoEscrowBlob) { + t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err) + } +} + +// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is +// run for real, and the whole tree is then walked: no file may contain R (or the recovered password), +// and the staging directory the unseal creates must be gone. +// +// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add +// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before +// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the +// walk finds it → this FAILS. +func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) { + ensureAge(t) + const repoPW = "1111111111111111111111111111111111111111111111111111111111111111" + tmp := t.TempDir() + t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk + + const wrongR = "wrong code entirely" + blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR) + if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil { + t.Fatalf("recover: %v", err) + } + // A failed unseal must leave nothing either — exercise both paths before walking. + _, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR) + + // THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later + // call OVERWRITING the leaked file with a different secret — which is exactly how the first + // version of this test passed its own red-proof while R sat on disk. Nothing in this test writes + // under TMPDIR, so after both calls the tree must contain no files at all. + var survivors []string + err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error { + if err != nil || info == nil || info.IsDir() || path == tmp { + return nil + } + survivors = append(survivors, strings.TrimPrefix(path, tmp)) + return nil + }) + if err != nil { + t.Fatal(err) + } + if len(survivors) > 0 { + t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+ + "recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors) + } + // Defence in depth: any secret that DOES appear anywhere is named, for every code used. + _ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error { + if err != nil || info == nil || info.IsDir() { + return nil + } + body, rerr := os.ReadFile(path) + if rerr != nil { + return nil + } + for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} { + if strings.Contains(string(body), secret) { + t.Errorf("%s survived on disk at %s", label, path) + } + } + return nil + }) + // And the staging directories are gone, not merely free of secrets. + entries, _ := os.ReadDir(tmp) + for _, e := range entries { + if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") { + t.Fatalf("an unseal staging directory survived: %s", e.Name()) + } + } +} + +// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be +// sent to re-read their recovery code because the hub was unreachable. +func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) { + rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { + return nil, false, errors.New("hub: connection refused") + }} + _, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR) + if err == nil || !strings.Contains(err.Error(), "fetching the sealed bundle") { + t.Fatalf("a fetch failure must say so, got %v", err) + } + if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) { + t.Fatal("a transport failure must not masquerade as a content verdict") + } +} diff --git a/internal/hub/client.go b/internal/hub/client.go index c5bd7ab..b1e9e4f 100644 --- a/internal/hub/client.go +++ b/internal/hub/client.go @@ -307,3 +307,50 @@ func tail(b []byte, max int) string { } return s } + +// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199). +// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet. +type IdentityEscrowResponse struct { + HostID string `json:"host_id"` + Present bool `json:"present"` + IdentityEscrowB64 string `json:"identity_escrow_b64"` +} + +// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the +// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they +// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds. +// +// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no +// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different +// endpoint with a different gate and is not reached from here. +// +// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged — +// only its length. +func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) { + if c.hostID == "" { + return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id") + } + url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow" + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err) + } + req.Header.Set("Authorization", "Bearer "+c.apiKey) + req.Header.Set("Accept", "application/json") + + resp, err := c.hc.Do(req) + if err != nil { + return nil, &TransportError{Err: err} + } + defer resp.Body.Close() + + raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)} + } + var out IdentityEscrowResponse + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err) + } + return &out, nil +} diff --git a/internal/localapi/escrow_recover.go b/internal/localapi/escrow_recover.go new file mode 100644 index 0000000..31b0993 --- /dev/null +++ b/internal/localapi/escrow_recover.go @@ -0,0 +1,102 @@ +package localapi + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "net/http" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/escrow" +) + +// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository +// password from the hub's sealed bundle, using the customer's recovery code R. +// +// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`) +// is an agent runtime dependency and is deliberately absent from the controller image; the sealed +// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is +// that write's mirror; and the controller is a trust tier down — it should receive one field, not a +// bundle it has no use for. +// +// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that +// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it: +// - arrives in the request body over the already-pinned local-API channel (the operator accepted +// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end); +// - is held in memory for the duration of one call and cleared on BOTH paths; +// - is never written to disk, never an argument in a process list, and never logged at any level, +// including inside an error; +// - is never echoed: no response this endpoint can emit contains it. +// +// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see +// `logRequests`. Do not add a body dump. +// +// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares +// (compare by hash, never by value). The password itself is present because the next link — placing a +// recovered password so the existing repository opens — needs it, and building a hash-only seam now +// would have to be torn out to add it. The controller's diagnostic reads only the hash. + +type recoverOffsitePasswordRequest struct { + VMID int `json:"vmid"` + // RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed. + RecoveryCode string `json:"recovery_code"` +} + +// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only +// the offsite repository password (plus its sha256, for hash-only comparison by the caller). +func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) { + var req recoverOffsitePasswordRequest + if !decodeBody(w, r, &req) { + return + } + if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) { + return + } + R := strings.TrimSpace(req.RecoveryCode) + req.RecoveryCode = "" // drop the decoded copy immediately + if R == "" { + writeErr(w, http.StatusBadRequest, "recovery_code is required") + return + } + if s.escrowRecovery == nil { + R = "" + writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)") + return + } + + ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second) + defer cancel() + s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid) + + pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R) + R = "" // cleared on BOTH paths, before anything else can happen + if err != nil { + // Each situation gets its own status and its own words. None of them names a secret. + switch { + case errors.Is(err, escrow.ErrNoEscrowBlob): + s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid) + writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run") + case errors.Is(err, escrow.ErrNoResticPassword): + s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid) + writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)") + default: + // Includes the fail-closed wrong-code case. The agent log records the STEP, never the code. + s.logger.Warn("local-api: offsite key recovery FAILED (wrong recovery code, or the blob could not be fetched)", "vmid", vmid, "err", err) + writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle, or the bundle could not be fetched — nothing was written") + } + return + } + + sum := sha256.Sum256([]byte(strings.TrimSpace(pw))) + // §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this + // concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing). + s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+ + "(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)", + "vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:])) + writeOK(w, map[string]any{ + "restic_repo_password": pw, + "restic_pw_sha256": hex.EncodeToString(sum[:]), + }) +} diff --git a/internal/localapi/server.go b/internal/localapi/server.go index 8e93f2c..67d8e28 100644 --- a/internal/localapi/server.go +++ b/internal/localapi/server.go @@ -111,6 +111,14 @@ type HostMetricsProvider interface { } // Options configures a Server. +// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and +// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the +// localapi package needs no hub-client dependency and the route is testable without crypto. +// R is an argument and is never retained by any implementation. +type EscrowRecoverer interface { + RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) +} + type Options struct { ListenAddr string // bridge IP:port Cert tls.Certificate @@ -210,6 +218,11 @@ type Options struct { // GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured". LogRing *applog.Ring Logger *slog.Logger + // EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind + // POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured" + // (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer. + EscrowRecovery EscrowRecoverer + } // defaultBackupCadence is the fallback /backup/due window when none is configured. @@ -273,6 +286,11 @@ type Server struct { netMountRoot string // the user-data namespace root for the network-mount role gate smbCredsDir string // where SMB creds files are written (out-of-band, 0600) escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password + // escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed + // identity blob from the hub, unseal it with the customer's recovery code, return ONLY the + // offsite repository password. OPTIONAL — nil (no hub client configured) makes + // POST /escrow/recover-offsite-password answer 503 rather than pretending. + escrowRecovery EscrowRecoverer intent IntentRecorder // slice 10 P3 (optional) guestBinds *GuestBindStore // F9 startup bind re-assert record (optional) formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional) @@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) { netMountRoot: storage.NetworkMountRoot, smbCredsDir: o.SmbCredsDir, escrowStagePath: o.EscrowStagePath, + escrowRecovery: o.EscrowRecovery, intent: o.Intent, guestBinds: o.GuestBinds, formatJobs: o.FormatJobs, @@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret)) // fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent. mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret)) + // R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle, + // using the customer recovery code supplied in the body. Returns that ONE field. See + // escrow_recover.go for R's handling rules — they are the tightest in this package. + mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword)) // Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony // job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.