Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6d7904786c | |||
| 856a127cd6 |
@@ -1,3 +1,70 @@
|
||||
## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 7–8)
|
||||
|
||||
**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed
|
||||
since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the
|
||||
recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest
|
||||
writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token` —
|
||||
an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||
offsite repository password into the same bundle. Anyone reading that output would conclude the
|
||||
repository password was not there. It now names what THIS bundle actually carried and what it did not.
|
||||
|
||||
**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the
|
||||
customer's recovery code, the agent fetches this host's own sealed blob from the hub
|
||||
(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and
|
||||
returns **only the offsite restic repository password** plus its sha256.
|
||||
|
||||
**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG
|
||||
private key. The controller is a trust tier down and needs none of them; returning them would widen
|
||||
the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not
|
||||
recoverable later.
|
||||
|
||||
**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately
|
||||
absent from the controller image; the blob is a host-scoped object whose only writer is this agent
|
||||
under the per-host key, so the read is that write's mirror.
|
||||
|
||||
**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned
|
||||
channel, is held in memory for one call, is cleared on the success path AND every failure path, is
|
||||
never written to disk, never an argument in a process list, never logged at any level including
|
||||
inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered
|
||||
plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty
|
||||
afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later
|
||||
call overwriting the leaked file, which is exactly how the first version of that test passed its own
|
||||
red-proof while R sat on disk.
|
||||
|
||||
Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a
|
||||
bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and
|
||||
a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to
|
||||
re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids.
|
||||
|
||||
**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main` → `runDaemon` →
|
||||
`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as
|
||||
`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping
|
||||
that makes cross-host retrieval impossible is a property of which key is used). This project's
|
||||
built-but-never-wired count is six and links 6–7 were two of them; the fix must not become the seventh.
|
||||
|
||||
## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190)
|
||||
|
||||
**v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The
|
||||
capability reported degraded for "one cycle" — meaning the probe call that performed the repair. But
|
||||
`probeAll` is invoked **independently** by the periodic self-check log and by the collector building a
|
||||
host-report. On the demo box the repairing call was the log's (`09:39:34`, journal shows
|
||||
`GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED` and `degraded=1`), and the host-report built three
|
||||
seconds later found the grant present and sent **`ok`**. The agent's journal had the record; the hub
|
||||
had nothing; the operator would have learned nothing.
|
||||
|
||||
That is the exact silence R-190 is about, re-created inside its own mitigation — and every unit test
|
||||
passed while it was true.
|
||||
|
||||
**The fix is a latch on TIME rather than on call count.** A confirmed repair is reported for
|
||||
`storeGrantRepairReportWindow` (20 minutes), which comfortably exceeds the 900 s host-report interval,
|
||||
so at least one report must carry the transition. It clears on its own — a permanently degraded
|
||||
capability would be its own false alarm — and it is per tier.
|
||||
|
||||
**Two hollow tests were caught and fixed on the way**, both the same shape this repo keeps finding: a
|
||||
test asserting a value it constructed itself, and a test asserting the latch HELPER rather than the
|
||||
path that consumes it — whose red-proof duly passed. The decisions now live in
|
||||
`storeGrantHealthyVerdict` and `storeGrantRepairedVerdict`, and the tests call those.
|
||||
|
||||
## v0.124.0 — a lost storage grant repairs itself, and says that it was lost (2026-08-04, R-190)
|
||||
|
||||
**R-190 is a grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24** — with a
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
|
||||
// grepping for a string.
|
||||
//
|
||||
// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
|
||||
// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
|
||||
// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
|
||||
// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
|
||||
// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
|
||||
// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
|
||||
// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
|
||||
|
||||
func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
|
||||
t.Helper()
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
|
||||
if err != nil {
|
||||
t.Fatalf("parsing main.go: %v", err)
|
||||
}
|
||||
return fset, f
|
||||
}
|
||||
|
||||
// callsWithin returns the set of function names called (directly, by identifier or selector) inside
|
||||
// the named top-level function.
|
||||
func callsWithin(f *ast.File, fnName string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn := ce.Fun.(type) {
|
||||
case *ast.Ident:
|
||||
out[fn.Name] = true
|
||||
case *ast.SelectorExpr:
|
||||
if x, ok := fn.X.(*ast.Ident); ok {
|
||||
out[x.Name+"."+fn.Sel.Name] = true
|
||||
}
|
||||
out[fn.Sel.Name] = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
|
||||
func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
|
||||
_, f := parseMain(t)
|
||||
|
||||
// 1. main() reaches runDaemon.
|
||||
if !callsWithin(f, "main")["runDaemon"] {
|
||||
t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
|
||||
}
|
||||
// 2. runDaemon reaches buildLocalAPIServer.
|
||||
if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||
t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
|
||||
}
|
||||
|
||||
// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
|
||||
// an escrow.OffsiteKeyRecoverer is constructed there.
|
||||
var optionsHasField, recovererConstructed bool
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
cl, ok := n.(*ast.CompositeLit)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := cl.Type.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
pkg, _ := sel.X.(*ast.Ident)
|
||||
if pkg == nil {
|
||||
return true
|
||||
}
|
||||
switch pkg.Name + "." + sel.Sel.Name {
|
||||
case "localapi.Options":
|
||||
for _, el := range cl.Elts {
|
||||
kv, ok := el.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
|
||||
optionsHasField = true
|
||||
}
|
||||
}
|
||||
case "escrow.OffsiteKeyRecoverer":
|
||||
recovererConstructed = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !recovererConstructed {
|
||||
t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
|
||||
"production assembly point (the built-but-never-wired shape, seventh instance)")
|
||||
}
|
||||
if !optionsHasField {
|
||||
t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
|
||||
"exists and the route would answer 503 forever")
|
||||
}
|
||||
}
|
||||
|
||||
// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
|
||||
// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
|
||||
// key is used.
|
||||
func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
|
||||
fset, f := parseMain(t)
|
||||
var fetchUsesHubClient bool
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
|
||||
return true
|
||||
}
|
||||
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
|
||||
fetchUsesHubClient = true
|
||||
} else {
|
||||
t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
|
||||
fset.Position(ce.Pos()))
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !fetchUsesHubClient {
|
||||
t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
|
||||
"not wired, or it uses a client whose credentials are not this host's")
|
||||
}
|
||||
}
|
||||
|
||||
// The route itself must be registered on the local API. A handler with no route is the same defect
|
||||
// one layer down, and it has shipped here before.
|
||||
func TestRecoverRouteIsRegistered(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parsing localapi/server.go: %v", err)
|
||||
}
|
||||
var registered bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok || len(ce.Args) < 2 {
|
||||
return true
|
||||
}
|
||||
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "HandleFunc" {
|
||||
return true
|
||||
}
|
||||
lit, ok := ce.Args[0].(*ast.BasicLit)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
|
||||
registered = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !registered {
|
||||
t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
|
||||
"exists and nothing can reach it")
|
||||
}
|
||||
}
|
||||
+117
-8
@@ -451,6 +451,21 @@ func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Conf
|
||||
return out
|
||||
}
|
||||
|
||||
// storeGrantRepairReportWindow is how long after a repair the capability keeps reporting the
|
||||
// transition. It MUST exceed the hub report interval, or the record never reaches the operator.
|
||||
//
|
||||
// FOUND BY THE LIVE RUN, NOT BY THE TESTS (2026-08-04). The first implementation reported degraded
|
||||
// for exactly "one cycle" — the probe call that did the repair. But `probeAll` is invoked
|
||||
// INDEPENDENTLY by the startup/periodic self-check log and by the collector building a host report,
|
||||
// so the repairing call was the LOG's, and the report built three seconds later found the grant
|
||||
// present and reported `ok`. The agent's journal had the record; the hub had nothing; the operator
|
||||
// would have learned nothing. That is precisely the silence R-190 is about, re-created inside its own
|
||||
// mitigation.
|
||||
//
|
||||
// A latch on TIME rather than on call count fixes it: 20 minutes comfortably exceeds the 900 s report
|
||||
// interval, so at least one host-report must carry the transition, and it still clears on its own.
|
||||
const storeGrantRepairReportWindow = 20 * time.Minute
|
||||
|
||||
// storeGrantRepairMinInterval bounds how often a single tier's grant may be re-granted (Scenario F).
|
||||
//
|
||||
// A storage can be unreadable for reasons an ACL cannot fix — the storage is gone, PVE is wedged,
|
||||
@@ -464,10 +479,37 @@ const storeGrantRepairMinInterval = time.Hour
|
||||
// restart re-arms the repair, which is correct — a restart is exactly when a box should re-check
|
||||
// everything it depends on.
|
||||
type storeGrantRepairer struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
|
||||
log *slog.Logger
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
|
||||
log *slog.Logger
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||
}
|
||||
|
||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||
func (r *storeGrantRepairer) noteRepaired(target string, now time.Time) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.repaired == nil {
|
||||
r.repaired = map[string]time.Time{}
|
||||
}
|
||||
r.repaired[target] = now
|
||||
}
|
||||
|
||||
// recentlyRepaired reports whether a confirmed repair is still inside its report window — the latch
|
||||
// that guarantees a host-report carries the transition even though the probe that repaired may have
|
||||
// been a log-only one.
|
||||
func (r *storeGrantRepairer) recentlyRepaired(target string, now time.Time) bool {
|
||||
if r == nil {
|
||||
return false
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
t, ok := r.repaired[target]
|
||||
return ok && now.Sub(t) < storeGrantRepairReportWindow
|
||||
}
|
||||
|
||||
// mayAttempt reports whether a repair may run now for this target, and records the attempt if so.
|
||||
@@ -544,9 +586,16 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
defer cancel()
|
||||
privs, err := px.Permissions(pctx, "/storage/"+targetID)
|
||||
s = storeGrantVerdict(targetID, critical, privs, err)
|
||||
if err != nil || s.Status != capability.StatusDegraded {
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
if s.Status != capability.StatusDegraded {
|
||||
// Healthy — but if this tier was repaired moments ago, keep REPORTING the transition until a
|
||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||
// comment).
|
||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||
}
|
||||
|
||||
// ── R-190 mitigation: the grant is missing — repair it, and SAY that it was missing ──────────
|
||||
//
|
||||
@@ -593,6 +642,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||
repair.noteRepaired(targetID, time.Now())
|
||||
s = storeGrantRepairedVerdict(targetID, critical)
|
||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
@@ -600,6 +650,22 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
return s
|
||||
}
|
||||
|
||||
// storeGrantHealthyVerdict decides what a HEALTHY probe reports — which is not always "ok".
|
||||
//
|
||||
// Split out so the tests exercise this decision rather than a copy of it. An earlier version of this
|
||||
// guard lived inline and its red-proof PASSED, because the test asserted the latch helper instead of
|
||||
// the path that consumes it — the same hollow shape this file has now caught twice.
|
||||
//
|
||||
// If the tier was repaired inside the report window, the transition is reported even though the grant
|
||||
// is present: the probe that repaired may have been a log-only one, and without this the host-report
|
||||
// carries `ok` and the operator never learns the permission vanished (measured live 2026-08-04).
|
||||
func storeGrantHealthyVerdict(targetID string, critical bool, healthy capability.Status, repairedRecently bool) capability.Status {
|
||||
if repairedRecently {
|
||||
return storeGrantRepairedVerdict(targetID, critical)
|
||||
}
|
||||
return healthy
|
||||
}
|
||||
|
||||
// storeGrantRepairedVerdict is the post-repair verdict — the RECORD half of R-190, split out so the
|
||||
// tests exercise the real thing rather than a copy of it (yesterday's hollow-test lesson).
|
||||
//
|
||||
@@ -1033,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
return false
|
||||
},
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localTokens != nil {
|
||||
defer localTokens.Close()
|
||||
}
|
||||
@@ -1611,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
|
||||
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
|
||||
// fixed. The opened token store is returned via outTokens so the caller can Close it.
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
if !cfg.LocalAPI.Enabled() {
|
||||
return nil
|
||||
}
|
||||
@@ -1683,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
gaMode = proxmox.RunnerSudo
|
||||
}
|
||||
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
|
||||
// R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's
|
||||
// own hub client (per-host key, self-scoped server-side), so the recoverer can never read another
|
||||
// host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config
|
||||
// cannot reach this line (the daemon exits above), so the seam is always live in production —
|
||||
// which is the point: links 6 and 7 spent months existing without a caller.
|
||||
escrowRecoverer := escrow.OffsiteKeyRecoverer{
|
||||
Fetch: func(ctx context.Context) ([]byte, bool, error) {
|
||||
resp, ferr := hubClient.FetchIdentityEscrow(ctx)
|
||||
if ferr != nil {
|
||||
return nil, false, ferr
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return nil, false, nil
|
||||
}
|
||||
blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if derr != nil {
|
||||
return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)")
|
||||
}
|
||||
return blob, true, nil
|
||||
},
|
||||
}
|
||||
srv, err := localapi.NewServer(localapi.Options{
|
||||
EscrowRecovery: escrowRecoverer,
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
Cert: cert,
|
||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||
@@ -2805,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger *
|
||||
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
|
||||
return 1
|
||||
}
|
||||
fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest)
|
||||
// R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was
|
||||
// accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||
// offsite repository password into the same bundle. Anyone reading the old output would conclude the
|
||||
// repository password was not there, and that is part of how the chain's extraction link came to be
|
||||
// described as missing for a month. Name what was recovered from THIS bundle, and name what is
|
||||
// absent, rather than reciting a fixed list.
|
||||
recovered := []string{"tunnel_token", "pbs_token"}
|
||||
var absent []string
|
||||
if bundle.WGPrivateKey != "" {
|
||||
recovered = append(recovered, "wg_private_key")
|
||||
} else {
|
||||
absent = append(absent, "wg_private_key")
|
||||
}
|
||||
if bundle.ResticRepoPassword != "" {
|
||||
recovered = append(recovered, "restic_repo_password")
|
||||
} else {
|
||||
absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)")
|
||||
}
|
||||
fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest)
|
||||
if len(absent) > 0 {
|
||||
fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; "))
|
||||
}
|
||||
|
||||
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
|
||||
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
|
||||
|
||||
@@ -353,3 +353,65 @@ func TestMainWiresTheGrantRepair(t *testing.T) {
|
||||
"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
|
||||
}
|
||||
}
|
||||
|
||||
// The transition must survive a probe that is NOT the one feeding the hub.
|
||||
//
|
||||
// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
|
||||
// production while every unit test passed: `probeAll` is called independently by the self-check LOG
|
||||
// and by the collector building a host-report. The repairing call was the log's; the report three
|
||||
// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
|
||||
// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
|
||||
//
|
||||
// --- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
|
||||
// storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
|
||||
// the host-report would carry ok and the operator would never learn the grant vanished
|
||||
//
|
||||
// Restored.
|
||||
func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||
r := newRepairer(&fakeRepairRunner{})
|
||||
// Jittered, never landing on the window boundary.
|
||||
repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
|
||||
r.noteRepaired("felhom-backup", repairedAt)
|
||||
|
||||
// The DECISION a later probe makes — the production function, not the helper it calls. An
|
||||
// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
|
||||
// the latch's USE left the helper untouched.
|
||||
healthy := probeWith(permGranted, "felhom-backup", true)
|
||||
if healthy.Status != capability.StatusOK {
|
||||
t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
|
||||
}
|
||||
got := storeGrantHealthyVerdict("felhom-backup", true,
|
||||
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
|
||||
if got.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
|
||||
"would carry ok and the operator would never learn the grant vanished", got.Status)
|
||||
}
|
||||
if !strings.Contains(got.Feature, "RESTORED") {
|
||||
t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
|
||||
}
|
||||
// Outside the window it reports plain ok again.
|
||||
late := storeGrantHealthyVerdict("felhom-backup", true,
|
||||
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
|
||||
if late.Status != capability.StatusOK {
|
||||
t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
|
||||
"would be its own false alarm", late.Status)
|
||||
}
|
||||
if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
|
||||
t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
|
||||
}
|
||||
// ...and it clears on its own rather than latching a box degraded forever.
|
||||
if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
|
||||
t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
|
||||
}
|
||||
// It is per tier.
|
||||
if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
|
||||
t.Fatal("one tier's repair must not latch another tier's status")
|
||||
}
|
||||
// The window MUST exceed the report interval — the property, asserted rather than assumed.
|
||||
if storeGrantRepairReportWindow <= 15*time.Minute {
|
||||
t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
|
||||
"be missed entirely", storeGrantRepairReportWindow)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery
|
||||
// code R, and hand back EXACTLY ONE field: the offsite restic repository password.
|
||||
//
|
||||
// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and
|
||||
// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one
|
||||
// trust tier down — needs none of them, and returning them would widen the blast radius of a
|
||||
// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later.
|
||||
//
|
||||
// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a
|
||||
// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper
|
||||
// keeps that property: it takes R as an argument, passes it straight through, and holds no copy.
|
||||
// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent).
|
||||
//
|
||||
// The errors below are DISTINCT on purpose. "no blob", "wrong code" and "the blob predates the field"
|
||||
// are three different situations for the operator and only one of them is a fault.
|
||||
|
||||
var (
|
||||
// ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run.
|
||||
ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)")
|
||||
// ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected
|
||||
// for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be
|
||||
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
|
||||
// not sent hunting for a mistyped recovery code that was typed correctly.
|
||||
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
|
||||
)
|
||||
|
||||
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
|
||||
// An interface-free func field keeps this package free of any dependency on the hub client.
|
||||
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
|
||||
|
||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||
type OffsiteKeyRecoverer struct {
|
||||
Fetch BlobFetcher
|
||||
}
|
||||
|
||||
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
|
||||
//
|
||||
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
|
||||
// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere.
|
||||
// That property is the crypto's, not a check here, which is why this function has no "validate R"
|
||||
// step to get wrong.
|
||||
//
|
||||
// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes.
|
||||
func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) {
|
||||
if r.Fetch == nil {
|
||||
return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured")
|
||||
}
|
||||
if recoveryCode == "" {
|
||||
return "", fmt.Errorf("escrow: the recovery code is required")
|
||||
}
|
||||
blob, present, err := r.Fetch(ctx)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("escrow: fetching the sealed bundle: %w", err) // carries no secret
|
||||
}
|
||||
if !present || len(blob) == 0 {
|
||||
return "", ErrNoEscrowBlob
|
||||
}
|
||||
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
|
||||
if err != nil {
|
||||
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
}
|
||||
if bundle.ResticRepoPassword == "" {
|
||||
return "", ErrNoResticPassword
|
||||
}
|
||||
return bundle.ResticRepoPassword, nil
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips
|
||||
// elsewhere). These are the unit half of the session's question — "is the repository password
|
||||
// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware.
|
||||
|
||||
const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
|
||||
|
||||
func sealBundle(t *testing.T, b IdentityBundle, r string) []byte {
|
||||
t.Helper()
|
||||
blob, err := WrapIdentityBundle(context.Background(), b, r)
|
||||
if err != nil {
|
||||
t.Fatalf("WrapIdentityBundle: %v", err)
|
||||
}
|
||||
return blob
|
||||
}
|
||||
|
||||
func fetcherFor(blob []byte) BlobFetcher {
|
||||
return func(context.Context) ([]byte, bool, error) { return blob, true, nil }
|
||||
}
|
||||
|
||||
// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it
|
||||
// is the REPOSITORY password rather than some other field of a bundle that also parses.
|
||||
//
|
||||
// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of
|
||||
// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS.
|
||||
// That mutation is the shape of the bug that would otherwise ship silently, because every one of
|
||||
// those fields is a non-empty string that looks like a secret.
|
||||
func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
blob := sealBundle(t, IdentityBundle{
|
||||
TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER",
|
||||
PBSToken: "PBS-TOKEN-NOT-THE-ANSWER",
|
||||
WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
|
||||
ResticRepoPassword: repoPW,
|
||||
}, testR)
|
||||
|
||||
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if err != nil {
|
||||
t.Fatalf("recover: %v", err)
|
||||
}
|
||||
if got != repoPW {
|
||||
t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+
|
||||
"field of the bundle was returned", len(got), len(repoPW))
|
||||
}
|
||||
// Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check
|
||||
// above by coincidence.
|
||||
for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} {
|
||||
if got == other {
|
||||
t.Fatalf("the recoverer returned the wrong bundle field")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is
|
||||
// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no
|
||||
// validation step here to get wrong — the test pins that it stays that way.
|
||||
func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||
|
||||
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all")
|
||||
if err == nil {
|
||||
t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids")
|
||||
}
|
||||
if got != "" {
|
||||
t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got))
|
||||
}
|
||||
// The error may name the step; it may never name a secret.
|
||||
for _, secret := range []string{repoPW, testR, "not the recovery code at all"} {
|
||||
if strings.Contains(err.Error(), secret) {
|
||||
t.Fatalf("the failure message leaked a secret: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before
|
||||
// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly
|
||||
// typed recovery code would be the wrong instruction.
|
||||
func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) {
|
||||
ensureAge(t)
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR)
|
||||
|
||||
_, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if !errors.Is(err, ErrNoResticPassword) {
|
||||
t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D at this layer — no blob is a clean, distinguishable answer.
|
||||
func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) {
|
||||
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
|
||||
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if !errors.Is(err, ErrNoEscrowBlob) {
|
||||
t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is
|
||||
// run for real, and the whole tree is then walked: no file may contain R (or the recovered password),
|
||||
// and the staging directory the unseal creates must be gone.
|
||||
//
|
||||
// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add
|
||||
// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before
|
||||
// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the
|
||||
// walk finds it → this FAILS.
|
||||
func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "1111111111111111111111111111111111111111111111111111111111111111"
|
||||
tmp := t.TempDir()
|
||||
t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk
|
||||
|
||||
const wrongR = "wrong code entirely"
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||
if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil {
|
||||
t.Fatalf("recover: %v", err)
|
||||
}
|
||||
// A failed unseal must leave nothing either — exercise both paths before walking.
|
||||
_, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR)
|
||||
|
||||
// THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later
|
||||
// call OVERWRITING the leaked file with a different secret — which is exactly how the first
|
||||
// version of this test passed its own red-proof while R sat on disk. Nothing in this test writes
|
||||
// under TMPDIR, so after both calls the tree must contain no files at all.
|
||||
var survivors []string
|
||||
err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil || info == nil || info.IsDir() || path == tmp {
|
||||
return nil
|
||||
}
|
||||
survivors = append(survivors, strings.TrimPrefix(path, tmp))
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(survivors) > 0 {
|
||||
t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+
|
||||
"recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors)
|
||||
}
|
||||
// Defence in depth: any secret that DOES appear anywhere is named, for every code used.
|
||||
_ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil || info == nil || info.IsDir() {
|
||||
return nil
|
||||
}
|
||||
body, rerr := os.ReadFile(path)
|
||||
if rerr != nil {
|
||||
return nil
|
||||
}
|
||||
for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} {
|
||||
if strings.Contains(string(body), secret) {
|
||||
t.Errorf("%s survived on disk at %s", label, path)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// And the staging directories are gone, not merely free of secrets.
|
||||
entries, _ := os.ReadDir(tmp)
|
||||
for _, e := range entries {
|
||||
if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") {
|
||||
t.Fatalf("an unseal staging directory survived: %s", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be
|
||||
// sent to re-read their recovery code because the hub was unreachable.
|
||||
func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) {
|
||||
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) {
|
||||
return nil, false, errors.New("hub: connection refused")
|
||||
}}
|
||||
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if err == nil || !strings.Contains(err.Error(), "fetching the sealed bundle") {
|
||||
t.Fatalf("a fetch failure must say so, got %v", err)
|
||||
}
|
||||
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
|
||||
t.Fatal("a transport failure must not masquerade as a content verdict")
|
||||
}
|
||||
}
|
||||
@@ -307,3 +307,50 @@ func tail(b []byte, max int) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199).
|
||||
// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet.
|
||||
type IdentityEscrowResponse struct {
|
||||
HostID string `json:"host_id"`
|
||||
Present bool `json:"present"`
|
||||
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||
}
|
||||
|
||||
// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the
|
||||
// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they
|
||||
// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds.
|
||||
//
|
||||
// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no
|
||||
// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different
|
||||
// endpoint with a different gate and is not reached from here.
|
||||
//
|
||||
// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged —
|
||||
// only its length.
|
||||
func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) {
|
||||
if c.hostID == "" {
|
||||
return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id")
|
||||
}
|
||||
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, &TransportError{Err: err}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||
}
|
||||
var out IdentityEscrowResponse
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err)
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
)
|
||||
|
||||
// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository
|
||||
// password from the hub's sealed bundle, using the customer's recovery code R.
|
||||
//
|
||||
// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`)
|
||||
// is an agent runtime dependency and is deliberately absent from the controller image; the sealed
|
||||
// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is
|
||||
// that write's mirror; and the controller is a trust tier down — it should receive one field, not a
|
||||
// bundle it has no use for.
|
||||
//
|
||||
// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that
|
||||
// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it:
|
||||
// - arrives in the request body over the already-pinned local-API channel (the operator accepted
|
||||
// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end);
|
||||
// - is held in memory for the duration of one call and cleared on BOTH paths;
|
||||
// - is never written to disk, never an argument in a process list, and never logged at any level,
|
||||
// including inside an error;
|
||||
// - is never echoed: no response this endpoint can emit contains it.
|
||||
//
|
||||
// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see
|
||||
// `logRequests`. Do not add a body dump.
|
||||
//
|
||||
// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares
|
||||
// (compare by hash, never by value). The password itself is present because the next link — placing a
|
||||
// recovered password so the existing repository opens — needs it, and building a hash-only seam now
|
||||
// would have to be torn out to add it. The controller's diagnostic reads only the hash.
|
||||
|
||||
type recoverOffsitePasswordRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
// RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed.
|
||||
RecoveryCode string `json:"recovery_code"`
|
||||
}
|
||||
|
||||
// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only
|
||||
// the offsite repository password (plus its sha256, for hash-only comparison by the caller).
|
||||
func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
var req recoverOffsitePasswordRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
R := strings.TrimSpace(req.RecoveryCode)
|
||||
req.RecoveryCode = "" // drop the decoded copy immediately
|
||||
if R == "" {
|
||||
writeErr(w, http.StatusBadRequest, "recovery_code is required")
|
||||
return
|
||||
}
|
||||
if s.escrowRecovery == nil {
|
||||
R = ""
|
||||
writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid)
|
||||
|
||||
pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R)
|
||||
R = "" // cleared on BOTH paths, before anything else can happen
|
||||
if err != nil {
|
||||
// Each situation gets its own status and its own words. None of them names a secret.
|
||||
switch {
|
||||
case errors.Is(err, escrow.ErrNoEscrowBlob):
|
||||
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
|
||||
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
|
||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||
default:
|
||||
// Includes the fail-closed wrong-code case. The agent log records the STEP, never the code.
|
||||
s.logger.Warn("local-api: offsite key recovery FAILED (wrong recovery code, or the blob could not be fetched)", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle, or the bundle could not be fetched — nothing was written")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
|
||||
// §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this
|
||||
// concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing).
|
||||
s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+
|
||||
"(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)",
|
||||
"vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:]))
|
||||
writeOK(w, map[string]any{
|
||||
"restic_repo_password": pw,
|
||||
"restic_pw_sha256": hex.EncodeToString(sum[:]),
|
||||
})
|
||||
}
|
||||
@@ -111,6 +111,14 @@ type HostMetricsProvider interface {
|
||||
}
|
||||
|
||||
// Options configures a Server.
|
||||
// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and
|
||||
// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the
|
||||
// localapi package needs no hub-client dependency and the route is testable without crypto.
|
||||
// R is an argument and is never retained by any implementation.
|
||||
type EscrowRecoverer interface {
|
||||
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error)
|
||||
}
|
||||
|
||||
type Options struct {
|
||||
ListenAddr string // bridge IP:port
|
||||
Cert tls.Certificate
|
||||
@@ -210,6 +218,11 @@ type Options struct {
|
||||
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
|
||||
LogRing *applog.Ring
|
||||
Logger *slog.Logger
|
||||
// EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind
|
||||
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
|
||||
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
|
||||
EscrowRecovery EscrowRecoverer
|
||||
|
||||
}
|
||||
|
||||
// defaultBackupCadence is the fallback /backup/due window when none is configured.
|
||||
@@ -273,6 +286,11 @@ type Server struct {
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
|
||||
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
|
||||
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
@@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
@@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
|
||||
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
|
||||
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
|
||||
// R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle,
|
||||
// using the customer recovery code supplied in the body. Returns that ONE field. See
|
||||
// escrow_recover.go for R's handling rules — they are the tightest in this package.
|
||||
mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
|
||||
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.
|
||||
|
||||
Reference in New Issue
Block a user