4 Commits

Author SHA1 Message Date
admin a2e914f683 v0.126.0: a fetch failure is not a wrong recovery code (R-224)
gates / gates (push) Successful in 7s
A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.

The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.

escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.

A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.

Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.

29 packages ok, vet clean, agent gates OK.
2026-08-06 07:55:15 +02:00
admin 0404f60e6a pre-push: refuse a push from a clone outside the felhom workspace (R-204 rider)
gates / gates (push) Successful in 10s
The workspace root is already documented (workspace-CLAUDE.md, the workspace-root
CLAUDE.md 'stay inside it') and work drifted into a home directory anyway. A rule
that has failed once as a reminder is not fixed by writing it down again, so it is
now asserted where it can bite.

A push is the right trigger: throwaway clones under /tmp for probes and red-proofs
never push, so nothing legitimate breaks. Symlinks are resolved on both sides; an
absent workspace root SKIPS the check rather than failing it, so this cannot brick
a legitimate clone on another machine. The only bypass is the documented
--no-verify, whose use is already reportable.

Identical in all four repos.
2026-08-05 10:46:39 +02:00
admin 3f5f61b716 docs: R-199 links 6-8 — CONTEXT + REPORT (proven live on demo-felhom)
gates / gates (push) Successful in 7s
2026-08-04 13:56:38 +02:00
admin 6d7904786c agent v0.125.0: open the sealed bundle, return one field (R-199 links 7-8)
gates / gates (push) Successful in 7s
Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.

POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.

R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.

Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).

The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
2026-08-04 13:41:12 +02:00
12 changed files with 1076 additions and 192 deletions
+35
View File
@@ -29,6 +29,41 @@ root=$(git rev-parse --show-toplevel 2>/dev/null) || {
}
cd "$root" || exit 1
# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
# Refuse a push from a clone outside the felhom workspace.
#
# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
# fixed by writing it down again — it has to be asserted where it can bite.
#
# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
#
# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
# failed — this hook must not brick a legitimate clone on a different host.
#
# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
root_real=$(pwd -P) || root_real=""
if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
case "$root_real/" in
"$ws_real"/*) : ;; # inside the workspace — proceed
*)
echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
echo " clone: $root_real" >&2
echo " expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
echo " Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
echo " and state that you did in the session report." >&2
exit 1
;;
esac
fi
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
+81
View File
@@ -1,3 +1,84 @@
## v0.126.0 — a fetch failure is not a wrong recovery code (2026-08-06, R-224)
**A hub the agent could not reach was reported to the customer as a bad recovery code.** Measured live
on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and a **correct,
current** recovery code, the customer was told their code did not open their package — **in 0.0556 s**,
against ~1.0 s for a genuine unseal. No unseal was attempted. F4 produced the same message in 0.0299 s
with this agent stopped.
**The discriminator existed here the whole time and this boundary threw it away.** `recover.go` fails
at four distinguishable points; the local-api handler had cases for two of them and a `default` that
answered *"the recovery code did not open the sealed bundle, or the bundle could not be fetched"*
one sentence for two situations, only one of which is the customer's doing.
**The fix is a value, not a log line.** `escrow.ErrBundleFetch` joins the fetch leg's error, and the
handler routes it to **502** with its own words: *"the sealed recovery bundle could not be fetched from
the hub — the recovery code was NOT used and nothing was written."* 502 rather than 4xx because the
request was not bad; an upstream dependency failed. The `default` now carries **only** the fail-closed
wrong-code case and says so without the "or".
Four situations, four statuses — **502** fetch failed · **400** the bundle was fetched and refused the
code · **404** the hub holds no bundle · **409** the bundle predates the repository-password field.
The controller classifies on the STATUS and must never parse these sentences.
⚠ **A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.**
`TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct` has said since v0.125.0 that *"the operator must
not be sent to re-read their recovery code because the hub was unreachable"* — and it passed
throughout, because it asserted this package's error **string** one layer below where the merge
happened, and a string is something no caller can branch on. It now asserts the sentinel, and its
consequence-level twin asserts the STATUS at the boundary the customer's message is derived from.
**Prefer the test that asserts the consequence over the one that asserts the mechanism.**
Tests: `recover_test.go` (fetch classifies as `ErrBundleFetch`; a wrong code does **not**; an absent
blob keeps its own identity) and `localapi/escrow_recover_class_test.go` (each situation's status, and
a standalone assertion that fetch-failure and wrong-code never share one). **Red-proofs:** removing the
`%w` join fails the sentinel test; deleting the handler case makes both answer `400` with the
wrong-code sentence — the exact pre-fix code, and the exact defect CAMPAIGN-11 measured.
## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 78)
**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed
since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the
recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest
writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token`
an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the
offsite repository password into the same bundle. Anyone reading that output would conclude the
repository password was not there. It now names what THIS bundle actually carried and what it did not.
**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the
customer's recovery code, the agent fetches this host's own sealed blob from the hub
(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and
returns **only the offsite restic repository password** plus its sha256.
**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG
private key. The controller is a trust tier down and needs none of them; returning them would widen
the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not
recoverable later.
**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately
absent from the controller image; the blob is a host-scoped object whose only writer is this agent
under the per-host key, so the read is that write's mirror.
**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned
channel, is held in memory for one call, is cleared on the success path AND every failure path, is
never written to disk, never an argument in a process list, never logged at any level including
inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered
plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty
afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later
call overwriting the leaked file, which is exactly how the first version of that test passed its own
red-proof while R sat on disk.
Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a
bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and
a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to
re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids.
**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main``runDaemon`
`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as
`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping
that makes cross-host retrieval impossible is a property of which key is used). This project's
built-but-never-wired count is six and links 67 were two of them; the fix must not become the seventh.
## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190)
**v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The
+34
View File
@@ -3,6 +3,40 @@
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
> entry = current); the end-of-task detail lives in `REPORT.md`.
## R-199 (v0.125.0) — links 68 of the recovery chain, assembled and walked
`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
(`hub.Client.FetchIdentityEscrow``GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
repository password plus its sha256.
**Rules that must not erode:**
- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
at age's KDF, nothing written).
- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
`main``runDaemon``buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
`Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
Links 6 and 7 were two of this project's six built-but-never-wired instances.
**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
recovered password, reopens a repository, or restores a file.
**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
into the same bundle — anyone reading it would conclude the password was not there. It now names what
THIS bundle carried and what it did not.
## Current
- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
+37 -189
View File
@@ -1,205 +1,53 @@
# REPORT — R-185: a tier the box cannot READ must say so
# REPORT — felhom-agent v0.126.0: a fetch failure is not a wrong recovery code (R-224)
**Date:** 2026-08-03 · **Repos:** `felhom-agent` **v0.122.0 → v0.123.0** (`fe14bc6`) · `felhom.eu`
installer **1.23.0 → 1.24.0** (`688470c`, tag `installer-v1.24.0`, manifest bump `311dc06`) ·
**no hub change and no hub bump** — the hub already alerts on a degraded critical capability, which is
why that mechanism was chosen.
**Scope: this repo's half of R-224.** The controller half ships as felhom-controller v0.202.0.
---
## Why the agent changed at all
## 1. Baselines, re-read on arrival
The task that commissioned this work scoped `felhom-agent` as **untouched**. It could not be. Its
Scenario A (a hub outage must not blame the customer's code) and Scenario C (a genuine mistype must be
told to re-check the ten words) are **mutually unsatisfiable** while this agent answers both with one
HTTP 400 and one sentence. No value available to the controller separates them. The task's own §5
anticipates this — *"if the step is not recoverable from the value, make it so, and say what that
cost"* — and §4.3 says the source outranks the register's recorded shape. **The cost is this version,
a publish, and a `MinAgent` coupling on the controller side.**
| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-agent` | `0b28eae7bb14` | `v0.122.0` | **yes** |
| `felhom.eu` | `7a5694341d59` | installer `1.23.0`, both `--ref=installer-v1.23.0` (lines 327, 372) | **yes** |
## What changed
Highest register ID in use **R-189**; R-190+ confirmed free by grep, and none was needed.
## 2. Part 0 — the measurements, before anything was designed against them
**The row's three-way observation, reproduced unchanged:**
| leg | result |
| File | Change |
|---|---|
| root, `pvesh … /storage/felhom-backup/content` | **3 archives** — 6.1 / 6.2 / 6.3 GB, dated 08-01, 08-02, 08-03 |
| the **agent's token**, same endpoint | `{"data":[]}` |
| the agent's token, `/storage/local/content` | **8 entries** — the token works where it is granted |
| `internal/escrow/recover.go` | new `ErrBundleFetch` sentinel; the fetch leg joins it with `%w: %w` so the cause survives for the operator log |
| `internal/localapi/escrow_recover.go` | new `case errors.Is(err, escrow.ErrBundleFetch)`**502** with its own words; the `default` now carries only the wrong-code case and drops the "or" |
| `internal/escrow/recover_test.go` | three new tests; the pre-existing `FetchErrorIsDistinct` re-pointed from a string to the sentinel, with the reason it failed to protect |
| `internal/localapi/escrow_recover_class_test.go` | new — the consequence-level test: four situations, four statuses |
So the token is the variable, not the storage. Two further checks removed the obvious alternative
explanation: guest **9201 IS in the `felhom` pool** (so `VM.Backup` is not the discriminator), and
`pveum acl list` showed ACL rows for `/storage/{local,local-lvm,felhom-pbs}` and **none** for
`/storage/felhom-backup`.
**Four statuses:** `502` fetch failed (the code was **not used**) · `400` fetched and refused ·
`404` no bundle · `409` bundle predates the field.
**The permission query, asked by the token itself — and the obvious reading is wrong:**
## The finding this turned up
```
/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
```
**A green test named the defect and did not prevent it.** `TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct`
has asserted since v0.125.0 that *"the operator must not be sent to re-read their recovery code because
the hub was unreachable"*. It passed throughout, because it checked this package's error **string** one
layer below the local-api `default` that did the merging — and a string is not something a caller can
branch on. **Mechanism asserted, consequence unpinned**; the project's own rule names this exact case.
It is also a comment-vs-code entry: `recover.go`'s header said the errors were *"DISTINCT on purpose"*
and named **three** situations while a fourth was silently folded into one of them.
The ungranted path answers **neither empty nor 403**. It answers with the privileges **inherited**
from the box-wide `/` grant. A probe asking *"did the path come back?"* — or *"does it hold
`Datastore.Audit`?"* — would have reported the blinded storage **healthy**. This is exactly what §3
required to be measured rather than assumed, and it changed the design: the probe tests
`Datastore.AllocateSpace` specifically, and a red-proof pins that choice.
## Green gate
## 3. The probe
`go build ./...` clean · `go vet ./...` clean · `go test ./...`**29 packages ok** ·
`python3 scripts/agent_gates.py --fast` → all gates OK.
`Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's own token**
(asking as root answers a different question and always says yes). `storeGrantStatuses` emits one
`capability.Status` per configured tier.
**Red-proofs, each demonstrated failing then restored:**
**Deviation from §5/§8.1, stated because a recommendation not followed gets a line:** the spec asked
for the sudo `Prober` to be minimally generalised. This repo already has the better-established
pattern for exactly this — `poolReadStatus`, composed **around** the prober, with the comment *"an API
read does not belong inside the sudo-policy probe"* (v0.62.0, audit A1). The probe follows that
precedent instead. `capability.Status` is untouched either way, which is the constraint that mattered.
**Decisions:**
- **The probed set comes from the box's own `BackupTiers()`**, never a fixed list — a hardcoded probe
list is the defect reproduced inside the fix.
- **Critical** (§8.3): the hub alerts only on critical, so a non-critical entry would ride the report
and alert nobody — the same silence with extra steps. **Except** the `local` fallback target, which
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
does not page. Turning an ordinary documented setup into an alert is how a signal becomes something
an operator archives unread.
- **It never consults content**, so it cannot alarm on a newborn tier by construction — a stronger
guarantee than gating on emptiness would be.
- **It never reports ok when it could not ask.** Unreachable PVE is degraded: a self-check that fails
open converts *"I do not know"* into *"fine"*.
## 4. The installer — the root cause was not where the row or the task expected
Both assumed `PVE_STORAGES` (the fixed grant list) was the culprit. **It is not.**
`configure_backup_target` has two arms:
- **Case A** creates the storage and calls `felhom-backup-target-apply grant` in the same breath — a
box that builds its own target has always been correct.
- **The Scenario-F arm***"the target already exists, leave it exactly as it is"* — **returned
without granting**.
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
surviving a reinstall — which is both demo boxes) pointed `local_backup_target` at a storage its own
token could not read. The reuse arm now ensures the ACL through the same guarded wrapper.
**Scenario F is unviolated:** the storage DEFINITION is still untouched. Granting the role the agent is
supposed to have on the target this same script is about to write into `agent.json` is finishing the
job, not retargeting the box; `pveum acl modify` is idempotent, so a box that already has it is
unchanged and a box whose token was rotated gets it back.
**`$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES`,** and the comment now says why: that
list is granted in step 4/5, *before* `configure_backup_target` runs in step 6, and `--acl-storages`
entries are preflight-checked for existence. Adding it there would grant on a storage that may not yet
exist and would split ownership of the decision across two places.
**A gate now asserts it:** every arm of `configure_backup_target` that resolves the target must also
grant on it — the check that would have caught this.
## 5. Live validation, in order
| # | evidence |
| Mutation | Result |
|---|---|
| 1 | Part 0's measurements above, taken **before** any change |
| 2 | **The signal that has never existed**, on the still-blind box: `capability DEGRADED … capability=pve:store-grant:felhom-backup … reason="the agent token lacks Datastore.AllocateSpace on /storage/felhom-backup (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested" critical=true`, with `ok=69 total=70 degraded=1`. The hub: `Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)` and **`Operator email sent`** |
| 3 | Grant applied (user **and** token — a privsep token's rights are the intersection); the token then lists **3 archives** where it listed none, and the permission answer becomes `{"Datastore.AllocateSpace":1,"Datastore.Allocate":1}` |
| 4 | `capabilities self-check ok=70 total=70 degraded=0`; the hub: `degraded → ok (agent_capability_recovered)` |
| 5 | **The host tier is a due-check candidate for the first time on that box**: `tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" proven=""` — and the settle rule applies to it exactly as to the others, selecting the **08-02** archive because the 08-03 one has not settled 24 h |
| 6 | The served installer over HTTPS: `SCRIPT_VERSION="1.24.0"`, and the served bytes carry the fix itself, not merely the version |
| remove the `%w: %w` join (pre-R-224 wrap) | `FetchFailureIsClassifiedAsFetch` **FAILS** |
| delete the `ErrBundleFetch` handler case | fetch answers `400 "the recovery code did not open the sealed bundle"`**the exact defect**, and both status tests **FAIL** |
## 6. The other machines
## Not changed
- **demo-hp CARRIES THE SAME DRIFT — and was fixed.** `local_backup_target=felhom-backup`, ACL rows for
`local`, `local-lvm`, `felhom-pbs` only. §8.6 assumed a single affected box; the same one-line,
additive, path-scoped, idempotent grant applies to the other, and leaving a known-blind backup tier
on a Tier-0 box after finding it would be this row happening twice. Granted (user + token); its
token now lists **4 archives**. It still runs agent `0.120.0`, so it has no probe yet — that arrives
when you vouch.
- **The tester's box was NOT touched** (Tier 2, protected). **What is known without connecting to it:**
it very likely carries the same drift — the mechanism is the Scenario-F reuse arm, which fires on
any box whose target pre-dated its install, and its target was moved by the very runbook that
creates that condition. It is due for reinstall, and installer 1.24.0 fixes it on the way in.
## 7. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...` — rc=0, plus `agent_gates.py` and
`repo_gates.py` all OK. Test runs and commits were always separate commands.
| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestStoreGrant_ForbiddenStorageIsDegradedAndNamed` | degraded, critical, naming storage **and** role | probe removed from `probeAll` | **FAIL**`main.go never calls storeGrantStatuses` (via the seam test); with the wrong-privilege mutation: `must be DEGRADED, not "ok"` |
| A | `TestStoreGrant_InheritedPrivilegesAreNotAGrant` | the measured trap: inherited ≠ granted | probe `Datastore.Audit` instead | **FAIL**`checking for the wrong privilege reports a blinded storage healthy; got "ok"` |
| B | `TestStoreGrant_GrantedButEmptyIsHealthy` | a readable-but-empty tier is healthy | — (it never reads content, so emptiness cannot reach it) | pass |
| B | `TestStoreGrant_TheFallbackTargetIsNotCritical` | `local` is reported but does not page | gating removed (`return true`) | **FAIL**`must not page the operator about an ordinary, documented configuration` |
| C | `TestStoreGrant_ForbiddenAndNewbornAreDistinguishable` | different status **and** different capability id | — | pass |
| — | `TestStoreGrant_UnreachablePVEIsDegradedNotOK` | unknown ≠ ok | — | pass |
| F | `hostinstall_gates.py` backup-target assertion | every resolving arm also grants | reuse arm reverted | **FAIL**`resolves the backup target in 2 place(s) but grants in only 1` |
| H | `TestMainWiresTheStoreGrantProbe` | **AST** of `main.go` | call commented out | **FAIL** — a `strings.Contains` check would have passed |
**A hollow test caught and fixed before it shipped:** the first draft of `storegrant_test.go`
re-implemented the verdict branch inside the test. It passed, and would have kept passing while
production diverged. The decision was extracted into `storeGrantVerdict` and the tests now call it.
**Scenario B's red-proof, honestly:** the spec asked for "degrade on an empty content listing" as the
mutation. That is not a mutation of this code — the probe never looks at content, which is a stronger
guarantee than gating on emptiness. The gating red-proof above (`storeGrantCritical`) is the one that
exercises the guard that does exist, and it fails as required.
## 8. Files, commits, tag
`internal/proxmox/query.go` (`Permissions`), `cmd/felhom-agent/main.go` (`storeGrantStatuses`,
`storeGrantVerdict`, `storeGrantCritical`, `storeGrantRequiredPriv`, wiring),
`cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `REPORT.md`.
`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`,
`manifests/webpage.yaml`, `CONTEXT.md`, `STATUS.md`, `documentation/architecture/00-capability-map.md`,
`documentation/backlog/OPEN-ITEMS.md`, `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.
**Commits** — `felhom-agent`: `fe14bc6` (v0.123.0). `felhom.eu`: `688470c` (installer 1.24.0), `311dc06`
(manifest refs), `e3187c8` (docs). **Installer tag:** `installer-v1.24.0`.
## 9. Deployment
Agent released through `release-agent.sh` — tag `v0.123.0`, sha256
`74910135ac4feb1b7f0ad4dbd1541d965cbc0fe70d4f47b62ebf7e4bfb962453`, round-trip verified. The
**published bytes** were downloaded and deployed: the running binary's sha matches the published one.
`felhom-agent --version`**0.123.0**, `systemctl is-active` → active, prior kept as `.bak-0.122.0`.
**NOT VOUCHED** — that stays the operator's act.
## 10. Registers
- **R-185 → CLOSED** (shipped + proven live on both demo boxes), with the corrected root cause
recorded on the row.
- No new IDs minted; `ROADMAP.md` contains no R-185 row, so there was nothing to collapse.
- **The capability map's whole-guest row was OPTIMISTIC and now says so:** every live restore-test it
cited is on the OFFSITE tier, and the HOST tier was not merely unproven but *unprovable* on both
demo boxes. It now records that, the closure, and that it will carry a host-tier live proof when one
runs.
- The vzdump-target-move runbook's item 5 **predicted this** and is annotated, not rewritten: it
expected a 403 on backup, and the reason it did not surface that way is that `vzdump` writes through
a root path, so backups kept landing while the agent's *read* stayed blind.
- `CONTEXT.md`: agent-side entry, plus `felhom.eu` **S-21** (empty ≠ forbidden; the measured trap) and
**S-22** (the Scenario-F arm must finish the job).
## 11. Teardown
**Nothing was provisioned.** No scratch storage, no fixture grant, no probe tag, no scratch package
version. The two ACL grants are the intended durable change; the only other mutation was the
installer label, which is reversible by moving the tag.
## 12. Observations — noticed, recorded, NOT acted on
- **Both demo boxes are now due for a host-tier restore-test**, which has never run on either. The
scheduler will pick it up within 6 h unattended (a ~6 GB local restore — fast, and cheaper than the
offsite ones). Expected, not a defect, and the first host-tier proof this fleet will have.
- **`--acl-storages` semantics are unchanged and the automatic grant does not consult it.** If an
operator passes `--acl-storages` deliberately excluding the backup target, the target is still
granted by the resolution path. That is the correct precedence — a box cannot function with an
unreadable backup target — but it is a place where an override is not absolute, and it is written
here rather than left to be discovered.
- **`storeGrantRequiredPriv` is a single privilege**, chosen from measurement. If PVE ever changes
which privilege gates content listing, the probe would report healthy while the tier is blind. The
test asserts the constant's value so a change forces a re-measurement, but nothing detects a change
on PVE's side.
- **Ten pre-existing `gofmt`-unclean files** remain in the agent repo (unchanged from yesterday's
observation); every file touched here is clean.
No Proxmox surface, no privileged path, no report/hub contract, no config schema. The route's
success path, its scoping and its R-handling discipline (`R = ""` on both paths, never logged, never
persisted) are untouched.
@@ -0,0 +1,188 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
// grepping for a string.
//
// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
if err != nil {
t.Fatalf("parsing main.go: %v", err)
}
return fset, f
}
// callsWithin returns the set of function names called (directly, by identifier or selector) inside
// the named top-level function.
func callsWithin(f *ast.File, fnName string) map[string]bool {
out := map[string]bool{}
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok {
return true
}
switch fn := ce.Fun.(type) {
case *ast.Ident:
out[fn.Name] = true
case *ast.SelectorExpr:
if x, ok := fn.X.(*ast.Ident); ok {
out[x.Name+"."+fn.Sel.Name] = true
}
out[fn.Sel.Name] = true
}
return true
})
}
return out
}
// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
// 1. main() reaches runDaemon.
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
}
// 2. runDaemon reaches buildLocalAPIServer.
if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
}
// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
// an escrow.OffsiteKeyRecoverer is constructed there.
var optionsHasField, recovererConstructed bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok {
return true
}
pkg, _ := sel.X.(*ast.Ident)
if pkg == nil {
return true
}
switch pkg.Name + "." + sel.Sel.Name {
case "localapi.Options":
for _, el := range cl.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
optionsHasField = true
}
}
case "escrow.OffsiteKeyRecoverer":
recovererConstructed = true
}
return true
})
}
if !recovererConstructed {
t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
"production assembly point (the built-but-never-wired shape, seventh instance)")
}
if !optionsHasField {
t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
"exists and the route would answer 503 forever")
}
}
// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
// key is used.
func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
fset, f := parseMain(t)
var fetchUsesHubClient bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := ce.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
return true
}
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
fetchUsesHubClient = true
} else {
t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
fset.Position(ce.Pos()))
}
return true
})
}
if !fetchUsesHubClient {
t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
"not wired, or it uses a client whose credentials are not this host's")
}
}
// The route itself must be registered on the local API. A handler with no route is the same defect
// one layer down, and it has shipped here before.
func TestRecoverRouteIsRegistered(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
if err != nil {
t.Fatalf("parsing localapi/server.go: %v", err)
}
var registered bool
ast.Inspect(f, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok || len(ce.Args) < 2 {
return true
}
sel, ok := ce.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "HandleFunc" {
return true
}
lit, ok := ce.Args[0].(*ast.BasicLit)
if !ok {
return true
}
if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
registered = true
}
return true
})
if !registered {
t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
"exists and nothing can reach it")
}
}
+46 -3
View File
@@ -1099,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return false
},
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localTokens != nil {
defer localTokens.Close()
}
@@ -1677,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
// fixed. The opened token store is returned via outTokens so the caller can Close it.
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
if !cfg.LocalAPI.Enabled() {
return nil
}
@@ -1749,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
gaMode = proxmox.RunnerSudo
}
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
// R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's
// own hub client (per-host key, self-scoped server-side), so the recoverer can never read another
// host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config
// cannot reach this line (the daemon exits above), so the seam is always live in production —
// which is the point: links 6 and 7 spent months existing without a caller.
escrowRecoverer := escrow.OffsiteKeyRecoverer{
Fetch: func(ctx context.Context) ([]byte, bool, error) {
resp, ferr := hubClient.FetchIdentityEscrow(ctx)
if ferr != nil {
return nil, false, ferr
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return nil, false, nil
}
blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if derr != nil {
return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)")
}
return blob, true, nil
},
}
srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr,
Cert: cert,
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
@@ -2871,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger *
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
return 1
}
fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest)
// R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was
// accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the
// offsite repository password into the same bundle. Anyone reading the old output would conclude the
// repository password was not there, and that is part of how the chain's extraction link came to be
// described as missing for a month. Name what was recovered from THIS bundle, and name what is
// absent, rather than reciting a fixed list.
recovered := []string{"tunnel_token", "pbs_token"}
var absent []string
if bundle.WGPrivateKey != "" {
recovered = append(recovered, "wg_private_key")
} else {
absent = append(absent, "wg_private_key")
}
if bundle.ResticRepoPassword != "" {
recovered = append(recovered, "restic_repo_password")
} else {
absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)")
}
fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest)
if len(absent) > 0 {
fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; "))
}
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
+95
View File
@@ -0,0 +1,95 @@
package escrow
import (
"context"
"errors"
"fmt"
)
// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery
// code R, and hand back EXACTLY ONE field: the offsite restic repository password.
//
// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and
// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one
// trust tier down — needs none of them, and returning them would widen the blast radius of a
// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later.
//
// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a
// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper
// keeps that property: it takes R as an argument, passes it straight through, and holds no copy.
// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent).
//
// The errors below are DISTINCT on purpose. "could not fetch", "no blob", "wrong code" and "the blob
// predates the field" are FOUR different situations for the operator and only one of them is a fault.
//
// ⚠ THERE WERE THREE, AND THE FOURTH WAS THE DEFECT (R-224, 2026-08-06). This comment said "three"
// and named "no blob", "wrong code" and "predates the field" — while a FAILED FETCH was wrapped as an
// anonymous error and fell through the caller's `default` branch into the wrong-code message. So a
// hub that could not be reached was reported to the customer as a bad recovery code.
//
// Measured live on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and
// a CORRECT current recovery code, the customer was told the code did not open their package — in
// 0.0556 s, when a real unseal costs ~1 s of scrypt. The agent's own log carried the truth the whole
// time (`escrow: fetching the sealed bundle: hub: transport error: … no route to host`) and the HTTP
// boundary threw it away.
//
// The discriminator therefore has to be a VALUE, not a log line — that is what ErrBundleFetch is.
var (
// ErrBundleFetch — the sealed bundle could not be FETCHED (the hub refused, was unreachable, or
// the transport failed). **The recovery code was never used**, so nothing about it is known and
// nothing may be said about it. Wraps the underlying cause for the operator log; carries no secret.
ErrBundleFetch = errors.New("escrow: the sealed bundle could not be fetched")
// ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run.
ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)")
// ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected
// for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
// not sent hunting for a mistyped recovery code that was typed correctly.
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
)
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
// An interface-free func field keeps this package free of any dependency on the hub client.
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct {
Fetch BlobFetcher
}
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
//
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere.
// That property is the crypto's, not a check here, which is why this function has no "validate R"
// step to get wrong.
//
// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes.
func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) {
if r.Fetch == nil {
return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured")
}
if recoveryCode == "" {
return "", fmt.Errorf("escrow: the recovery code is required")
}
blob, present, err := r.Fetch(ctx)
if err != nil {
// R-224: joined with ErrBundleFetch so the caller can classify by VALUE. The cause stays
// wrapped for the operator log; neither carries a secret. Before this, the fetch failure was
// an anonymous error and the local-api handler's `default` branch reported it to the customer
// as a wrong recovery code.
return "", fmt.Errorf("%w: %w", ErrBundleFetch, err)
}
if !present || len(blob) == 0 {
return "", ErrNoEscrowBlob
}
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
if err != nil {
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
}
if bundle.ResticRepoPassword == "" {
return "", ErrNoResticPassword
}
return bundle.ResticRepoPassword, nil
}
+267
View File
@@ -0,0 +1,267 @@
package escrow
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips
// elsewhere). These are the unit half of the session's question — "is the repository password
// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware.
const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
func sealBundle(t *testing.T, b IdentityBundle, r string) []byte {
t.Helper()
blob, err := WrapIdentityBundle(context.Background(), b, r)
if err != nil {
t.Fatalf("WrapIdentityBundle: %v", err)
}
return blob
}
func fetcherFor(blob []byte) BlobFetcher {
return func(context.Context) ([]byte, bool, error) { return blob, true, nil }
}
// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it
// is the REPOSITORY password rather than some other field of a bundle that also parses.
//
// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of
// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS.
// That mutation is the shape of the bug that would otherwise ship silently, because every one of
// those fields is a non-empty string that looks like a secret.
func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) {
ensureAge(t)
const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
blob := sealBundle(t, IdentityBundle{
TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER",
PBSToken: "PBS-TOKEN-NOT-THE-ANSWER",
WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
ResticRepoPassword: repoPW,
}, testR)
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
if err != nil {
t.Fatalf("recover: %v", err)
}
if got != repoPW {
t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+
"field of the bundle was returned", len(got), len(repoPW))
}
// Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check
// above by coincidence.
for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} {
if got == other {
t.Fatalf("the recoverer returned the wrong bundle field")
}
}
}
// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is
// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no
// validation step here to get wrong — the test pins that it stays that way.
func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) {
ensureAge(t)
const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all")
if err == nil {
t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids")
}
if got != "" {
t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got))
}
// The error may name the step; it may never name a secret.
for _, secret := range []string{repoPW, testR, "not the recovery code at all"} {
if strings.Contains(err.Error(), secret) {
t.Fatalf("the failure message leaked a secret: %v", err)
}
}
}
// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before
// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly
// typed recovery code would be the wrong instruction.
func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) {
ensureAge(t)
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR)
_, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoResticPassword) {
t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err)
}
}
// Scenario D at this layer — no blob is a clean, distinguishable answer.
func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) {
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoEscrowBlob) {
t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err)
}
}
// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is
// run for real, and the whole tree is then walked: no file may contain R (or the recovered password),
// and the staging directory the unseal creates must be gone.
//
// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add
// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before
// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the
// walk finds it → this FAILS.
func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) {
ensureAge(t)
const repoPW = "1111111111111111111111111111111111111111111111111111111111111111"
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk
const wrongR = "wrong code entirely"
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil {
t.Fatalf("recover: %v", err)
}
// A failed unseal must leave nothing either — exercise both paths before walking.
_, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR)
// THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later
// call OVERWRITING the leaked file with a different secret — which is exactly how the first
// version of this test passed its own red-proof while R sat on disk. Nothing in this test writes
// under TMPDIR, so after both calls the tree must contain no files at all.
var survivors []string
err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
if err != nil || info == nil || info.IsDir() || path == tmp {
return nil
}
survivors = append(survivors, strings.TrimPrefix(path, tmp))
return nil
})
if err != nil {
t.Fatal(err)
}
if len(survivors) > 0 {
t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+
"recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors)
}
// Defence in depth: any secret that DOES appear anywhere is named, for every code used.
_ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
if err != nil || info == nil || info.IsDir() {
return nil
}
body, rerr := os.ReadFile(path)
if rerr != nil {
return nil
}
for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} {
if strings.Contains(string(body), secret) {
t.Errorf("%s survived on disk at %s", label, path)
}
}
return nil
})
// And the staging directories are gone, not merely free of secrets.
entries, _ := os.ReadDir(tmp)
for _, e := range entries {
if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") {
t.Fatalf("an unseal staging directory survived: %s", e.Name())
}
}
}
// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be
// sent to re-read their recovery code because the hub was unreachable.
//
// ⚠ THIS TEST WAS GREEN THROUGHOUT THE DEFECT IT DESCRIBES (R-224, 2026-08-06). Its sentence is
// exactly right and it did not prevent anything, for two reasons worth keeping:
//
// 1. **It asserted the MECHANISM, one layer below the consequence.** It checked this package's error
// STRING. The merge happened one layer up, in the local-api handler's `default` branch, which
// answered a fetch failure with "the recovery code did not open the sealed bundle". The customer
// never sees this string; they see that one. The project's own rule — prefer the test that asserts
// the CONSEQUENCE (does the customer get blamed?) over the one that asserts the MECHANISM (is the
// error distinct here?) — names this case precisely.
// 2. **It asserted on TEXT.** `strings.Contains(err.Error(), …)` cannot be consumed by a caller, so
// it pinned something no production code could branch on. The distinction it checked was real and
// unusable.
//
// It now asserts the SENTINEL, which is what the handler branches on, and its consequence-level twin
// lives in `internal/localapi/escrow_recover_class_test.go` where the status is asserted.
func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) {
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) {
return nil, false, errors.New("hub: connection refused")
}}
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
if err == nil || !errors.Is(err, ErrBundleFetch) {
t.Fatalf("a fetch failure must classify as ErrBundleFetch, got %v", err)
}
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
t.Fatal("a transport failure must not masquerade as a content verdict")
}
}
// ── R-224 — A FAILED FETCH IS NOT A WRONG CODE ──────────────────────────────────────────────────
//
// CAMPAIGN-11 F3 measured the consequence of these two being indistinguishable: with the hub
// firewalled off and a CORRECT current recovery code, the customer was told the code did not open
// their package, in 0.0556 s — no unseal was attempted at all.
//
// The pair below is the whole point. Asserting only the first would pass with a `return ErrBundleFetch`
// stuck on every error path, which is the same defect pointing the other way.
func TestRecoverOffsiteRepoPassword_FetchFailureIsClassifiedAsFetch(t *testing.T) {
boom := errors.New("hub: transport error: dial tcp 37.191.56.193:443: connect: no route to host")
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, boom }}
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
if err == nil {
t.Fatal("a failing fetch must return an error")
}
// RED-PROOF: drop the `%w: %w` join in RecoverOffsiteRepoPassword (return the bare wrapped cause,
// as it was before R-224) → this FAILS, and the local-api handler falls back to the wrong-code
// message exactly as it did on 2026-08-05.
if !errors.Is(err, ErrBundleFetch) {
t.Fatalf("a failed fetch must classify as ErrBundleFetch, got %v", err)
}
// The underlying cause survives for the operator log.
if !errors.Is(err, boom) {
t.Fatalf("the fetch cause must stay wrapped for the operator, got %v", err)
}
// And it must NOT be mistaken for either of the bundle-content situations.
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
t.Fatalf("a transport failure is neither of the bundle-content errors: %v", err)
}
}
// The other half: a genuinely wrong code must NOT classify as a fetch failure, or the fix trades one
// misattribution for its mirror image and the customer is told the hub is down when they mistyped.
func TestRecoverOffsiteRepoPassword_WrongCodeIsNotAFetchFailure(t *testing.T) {
ensureAge(t)
blob := sealBundle(t, IdentityBundle{ResticRepoPassword: "0123456789abcdef"}, testR)
r := OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}
_, err := r.RecoverOffsiteRepoPassword(context.Background(),
"wrong horse battery staple sedative anaconda wobbly kingdom placard yodel")
if err == nil {
t.Fatal("a wrong recovery code must fail closed")
}
if errors.Is(err, ErrBundleFetch) {
t.Fatalf("a wrong code must NOT classify as a fetch failure, got %v", err)
}
}
// A clean "the hub holds nothing" keeps its own identity too — it is not a fetch failure, and the
// customer must not be told the hub was unreachable when it answered perfectly well.
func TestRecoverOffsiteRepoPassword_AbsentBlobIsNotAFetchFailure(t *testing.T) {
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoEscrowBlob) {
t.Fatalf("an absent blob must stay ErrNoEscrowBlob, got %v", err)
}
if errors.Is(err, ErrBundleFetch) {
t.Fatalf("an absent blob is not a fetch FAILURE, got %v", err)
}
}
+47
View File
@@ -307,3 +307,50 @@ func tail(b []byte, max int) string {
}
return s
}
// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199).
// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet.
type IdentityEscrowResponse struct {
HostID string `json:"host_id"`
Present bool `json:"present"`
IdentityEscrowB64 string `json:"identity_escrow_b64"`
}
// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the
// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they
// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds.
//
// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no
// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different
// endpoint with a different gate and is not reached from here.
//
// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged —
// only its length.
func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) {
if c.hostID == "" {
return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id")
}
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return nil, &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
var out IdentityEscrowResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err)
}
return &out, nil
}
+124
View File
@@ -0,0 +1,124 @@
package localapi
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"net/http"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository
// password from the hub's sealed bundle, using the customer's recovery code R.
//
// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`)
// is an agent runtime dependency and is deliberately absent from the controller image; the sealed
// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is
// that write's mirror; and the controller is a trust tier down — it should receive one field, not a
// bundle it has no use for.
//
// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that
// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it:
// - arrives in the request body over the already-pinned local-API channel (the operator accepted
// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end);
// - is held in memory for the duration of one call and cleared on BOTH paths;
// - is never written to disk, never an argument in a process list, and never logged at any level,
// including inside an error;
// - is never echoed: no response this endpoint can emit contains it.
//
// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see
// `logRequests`. Do not add a body dump.
//
// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares
// (compare by hash, never by value). The password itself is present because the next link — placing a
// recovered password so the existing repository opens — needs it, and building a hash-only seam now
// would have to be torn out to add it. The controller's diagnostic reads only the hash.
type recoverOffsitePasswordRequest struct {
VMID int `json:"vmid"`
// RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed.
RecoveryCode string `json:"recovery_code"`
}
// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only
// the offsite repository password (plus its sha256, for hash-only comparison by the caller).
func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) {
var req recoverOffsitePasswordRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
R := strings.TrimSpace(req.RecoveryCode)
req.RecoveryCode = "" // drop the decoded copy immediately
if R == "" {
writeErr(w, http.StatusBadRequest, "recovery_code is required")
return
}
if s.escrowRecovery == nil {
R = ""
writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid)
pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R)
R = "" // cleared on BOTH paths, before anything else can happen
if err != nil {
// Each situation gets its own status and its own words. None of them names a secret.
switch {
// ── R-224 (2026-08-06) — THE FETCH FAILURE IS NOT A WRONG CODE. ────────────────────────
//
// This case did not exist, and its absence is the defect. A failed fetch fell through to the
// `default` below and was answered with "the recovery code did not open the sealed bundle" —
// so a hub that could not be reached was reported to the customer as a bad recovery code, on
// the one screen whose whole purpose is to be believed about their backups.
//
// Measured live 2026-08-05 (CAMPAIGN-11 F3 and F4): a CORRECT current code returned that
// message in 0.0556 s with the hub firewalled off, and in 0.0299 s with this agent stopped —
// against ~1.0 s for a genuine unseal. No unseal was attempted in either case.
//
// 502 rather than 400: 4xx says "your request was bad", and the request was not bad — an
// upstream dependency failed. The status is the machine-readable half; the controller
// classifies on it and must never parse this sentence.
//
// ⚠ THE CODE WAS NOT USED. Nothing may be said about it — not that it was wrong, and not
// that it was right.
case errors.Is(err, escrow.ErrBundleFetch):
s.logger.Warn("local-api: offsite key recovery: the sealed bundle could not be FETCHED — the recovery code was never used", "vmid", vmid, "err", err)
writeErr(w, http.StatusBadGateway, "the sealed recovery bundle could not be fetched from the hub — the recovery code was NOT used and nothing was written")
case errors.Is(err, escrow.ErrNoEscrowBlob):
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
default:
// The fail-closed WRONG-CODE case, and only it: the bundle was fetched and `age -d`
// refused it. Every other situation above has its own status. The agent log records the
// STEP, never the code.
s.logger.Warn("local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code", "vmid", vmid, "err", err)
writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle — nothing was written")
}
return
}
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
// §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this
// concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing).
s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+
"(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)",
"vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:]))
writeOK(w, map[string]any{
"restic_repo_password": pw,
"restic_pw_sha256": hex.EncodeToString(sum[:]),
})
}
@@ -0,0 +1,99 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-224 — THE STATUS IS THE DISCRIMINATOR, and this test asserts the CONSEQUENCE (what the HTTP
// boundary answers) rather than the mechanism (that the sentinel exists).
//
// The controller one trust tier down classifies on the STATUS and must never parse the sentence. So
// the contract this pins is: four distinguishable situations, four distinct statuses, and the
// wrong-code message reachable ONLY from a real refusal.
//
// Before R-224 the first and last rows both answered 400 with the same sentence — which is how
// CAMPAIGN-11 F3 told a customer holding a CORRECT code that it did not open their package.
type fakeRecoverer struct{ err error }
func (f fakeRecoverer) RecoverOffsiteRepoPassword(context.Context, string) (string, error) {
if f.err != nil {
return "", f.err
}
return "0123456789abcdef0123456789abcdef", nil
}
func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
cases := []struct {
name string
err error
wantStatus int
// mustNotSay guards the specific misattribution each status exists to prevent.
mustNotSay []string
}{
{
name: "fetch failed — the code was NEVER used",
err: errors.Join(escrow.ErrBundleFetch, errors.New("hub: transport error: no route to host")),
wantStatus: 502,
mustNotSay: []string{"did not open"},
},
{
name: "wrong code — the bundle WAS fetched and refused it",
err: errors.New("escrow: the recovery code did not unwrap the identity escrow"),
wantStatus: 400,
mustNotSay: []string{"could not be fetched"},
},
{
name: "the hub holds no bundle",
err: escrow.ErrNoEscrowBlob,
wantStatus: 404,
mustNotSay: []string{"did not open"},
},
{
name: "the bundle predates the repository-password field",
err: escrow.ErrNoResticPassword,
wantStatus: 409,
mustNotSay: []string{"could not be fetched"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowRecovery = fakeRecoverer{err: tc.err}
w := do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`)
if w.Code != tc.wantStatus {
t.Fatalf("status: got %d, want %d — body=%s", w.Code, tc.wantStatus, w.Body.String())
}
for _, phrase := range tc.mustNotSay {
if strings.Contains(w.Body.String(), phrase) {
t.Fatalf("the %d answer must not say %q — body=%s", tc.wantStatus, phrase, w.Body.String())
}
}
})
}
}
// The pair that matters most, stated as its own assertion so a regression cannot hide inside a table:
// a fetch failure and a wrong code must never answer with the SAME status. Collapsing them is the
// whole of R-224.
func TestRecoverOffsitePassword_FetchFailureAndWrongCodeDiffer(t *testing.T) {
status := func(err error) int {
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowRecovery = fakeRecoverer{err: err}
return do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`).Code
}
fetch := status(errors.Join(escrow.ErrBundleFetch, errors.New("no route to host")))
wrong := status(errors.New("escrow: the recovery code did not unwrap the identity escrow"))
// RED-PROOF: delete the ErrBundleFetch case from handleRecoverOffsitePassword → both become 400
// → this FAILS. That is the exact pre-R-224 code, and the exact defect CAMPAIGN-11 measured.
if fetch == wrong {
t.Fatalf("a failed fetch and a wrong code must not share a status (both %d)", fetch)
}
}
+23
View File
@@ -111,6 +111,14 @@ type HostMetricsProvider interface {
}
// Options configures a Server.
// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and
// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the
// localapi package needs no hub-client dependency and the route is testable without crypto.
// R is an argument and is never retained by any implementation.
type EscrowRecoverer interface {
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error)
}
type Options struct {
ListenAddr string // bridge IP:port
Cert tls.Certificate
@@ -210,6 +218,11 @@ type Options struct {
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
LogRing *applog.Ring
Logger *slog.Logger
// EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
EscrowRecovery EscrowRecoverer
}
// defaultBackupCadence is the fallback /backup/due window when none is configured.
@@ -273,6 +286,11 @@ type Server struct {
netMountRoot string // the user-data namespace root for the network-mount role gate
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
escrowRecovery EscrowRecoverer
intent IntentRecorder // slice 10 P3 (optional)
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
@@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) {
netMountRoot: storage.NetworkMountRoot,
smbCredsDir: o.SmbCredsDir,
escrowStagePath: o.EscrowStagePath,
escrowRecovery: o.EscrowRecovery,
intent: o.Intent,
guestBinds: o.GuestBinds,
formatJobs: o.FormatJobs,
@@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
// R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle,
// using the customer recovery code supplied in the body. Returns that ONE field. See
// escrow_recover.go for R's handling rules — they are the tightest in this package.
mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword))
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.