4 Commits

Author SHA1 Message Date
admin 1db56bf837 v0.129.0 — a correct code for an earlier package stops being called wrong (R-311)
gates / gates (push) Successful in 14s
Yesterday's drill proved a retained escrow package opens a set-aside store and
restores planted files byte-identical, while this agent answered the customer's
correct code with "the recovery code did not open the sealed bundle". Nothing had
ever tried the retained packages, so a correct-but-earlier code and a mistype were
genuinely indistinguishable.

OffsiteKeyRecoverer gains an optional FetchRetained, consulted ONLY after the
current package refuses, so the ordinary recovery pays nothing for it and cannot
fail because of it. A match returns ErrCodeOpensRetained wrapped in a
RetainedOpenedError carrying the supersession date - no material, no code, no
password. The local API answers 422: a FIFTH status added to the R-224 switch,
never a restructuring of it.

Fail-safe in every direction. Nil fetcher, a hub too old for the route (404 is a
clean "none"), a transport failure, a malformed package: each leaves the original
refusal standing. Attempts bounded at 6 because each unwrap is ~1s of scrypt.

Seven tests with REAL age crypto - the two situations are indistinguishable AT
THE UNWRAP, so a faked unwrap would prove nothing. Red-proof asserted applied:
remove the retained lookup and the fail-closed wrong-code error returns, which is
the lie in those exact words.
2026-08-12 18:40:00 +02:00
admin 53d047a6c1 Two guards, one number: bound the published check to the retention it must live with
gates / gates (push) Successful in 17s
Gates only. No release, no version bump, no binary published; the agent stays
v0.128.0 at 28ba8593b8 and nothing on a customer's machine changes.

THE COUPLING DEFECT. The registry stopped serving 0.120.0 and older while
check-published-versions.py demanded every tag still be downloadable. Both rules
are sensible and together they are impossible, so CI went red at a commit whose
own run had been GREEN the day before -- and would have gone red again at the
next publish when 0.121.0 was evicted. scripts/retention-policy.json is now THE
number and both readers take it from there.

WHAT CI NO LONGER COVERS, and it prints this on every run rather than leaving it
to be discovered: a released version older than the retention window is no longer
asserted downloadable. Its git tag and its config tree ARE still asserted -- only
the binary's presence is dropped. A missing policy file is INCONCLUSIVE (exit 2),
never silently unbounded.

THE NUMBER IS NOT A LOCATED RULING and the file says so in its own header. Ten is
what the registry demonstrably holds; no register row records a prune, R-210 says
"Nothing was deleted; this is a list, not an action" and concerns local Docker
images, and container packages hold 19 each. The principled bound is the hub's
vouched min_agent floor -- nothing can install below it -- and that is the
recorded follow-up.

check-release-complete.py is the tag half as a machine. release-agent.sh already
warned that "a released version without a git tag 404s a box mid-install, as
root" and the step was still missed, so this is a gate and not a reminder. Legs
1-2 need no network and run in --fast, so the pre-push hook is the earliest
catch. Red-proved by repointing the CHANGELOG head at an unreleased v0.129.0:
both legs convicted and each named its fix command.

Three controls run: green at 10 naming what it dropped; widened to 11 the evicted
version re-enters and convicts; policy removed gives INCONCLUSIVE naming the path.
2026-08-09 19:05:05 +02:00
admin 28ba8593b8 v0.128.0 — R-221: the escrow seed is asserted every tick, not remembered once
gates / gates (push) Failing after 28s
A rebuilt box could not run the escrow ceremony AT ALL, with no way forward from inside the product.
This was the only open item blocking a customer from something we promise them.

MECHANISM, established at file:line rather than assumed. The preflight refuses on
escrow.pbs_storage_id; the pbsdr bridge writes that key; and it wrote it from exactly one place —
finishConverged, reached only on the paths that actually converge.

The marker and the key live in different places and die at different times. The marker is host-side
(<agent-state>/pbsdr/marker.json). The key is in agent.json, which step_agent_config renders from
`base = {}` unless an explicit --preserve-from is given (felhom.eu/scripts/felhom-host-install.sh:
2396 the step, :2449 the render, :2579 the O_TRUNC write; the flag :1246, defaulting empty at :256)
— AND THE RENDER NEVER WRITES AN escrow SECTION AT ALL (grep over the whole heredoc: zero hits). So
a rebuild keeps the marker and takes the key: same descriptor, same hash, early return, and the seed
never runs again into a config that no longer has it.

A rebuild is only the case that was measured. The same hole opens for a hand-edited or restored
config, which is the honest reason this is a seam fix rather than an installer fix: the seed must be
a thing the loop ASSERTS, not a thing it did once.

Apply now re-asserts the seed BEFORE the idempotent early return. seedEscrowStorageID is unchanged
and still never clobbers a different existing value — an operator's own choice outranks the
descriptor's, with a warning naming both.

THE EARLY RETURN IS KEPT. It stops a converged box re-running Proxmox operations every 60s, and
TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls asserts ZERO recorded runner calls on that
tick, so a "fix" that simply deleted the return would fail. Cost: one small file read plus a JSON
parse per tick, no exec, no network, early-returning once the value matches.

A seed failure can never un-converge the box: Warn plus a message on the published status, exactly
as finishConverged does it — no marker write, no state change.

Tests drive the REAL Apply with a real temp-dir agent.json and a call-recording runner; calling
seedEscrowStorageID directly cannot see the early return, which IS the defect. Production wiring
(pbsdr.NewManager(..., cfg.SourcePath, ...)) is asserted by walking main.go's AST, not by
strings.Contains, which a commented-out call also satisfies.

Red-proofs, each with the mutation asserted applied: removing the new call makes Scenario A fail on
today's tree (it did, with the intended message); removing the early return makes the
zero-Proxmox-calls assertion fail (it did).

go build / go vet / go test ./... green (29 packages), run separately from this commit.
2026-08-08 16:29:13 +02:00
admin 6981450110 docs: a comment claimed the hub reads a field it has no field for (R-260)
gates / gates (push) Successful in 26s
Comment-only; no behaviour, no wire change, no version bump, nothing to rebuild.

HostReport.SelfUpdatePending / SelfUpdatePendingVersion carried "The hub reads an absent field as
pending=false, the correct default." The hub has NO FIELD for either, so it reads nothing — present
or absent — and encoding/json discards them on arrival. The sentence described an intent rather than
the code and read as settled long enough that a class sweep had to find it.

The emission is correct and stays: the agent reports the truth and the fault is entirely in the
receiving. The missing consumer is R-264 (OPEN). felhom.eu/scripts/wire_contract_gate.py now refuses
any NEW field of this shape and records the existing ones as reasoned allowlist entries.
2026-08-08 08:46:17 +02:00
14 changed files with 1121 additions and 46 deletions
+109
View File
@@ -1,3 +1,112 @@
## v0.129.0 — a correct code for an earlier package stops being called wrong (2026-08-12, R-311)
**The measurement this fixes.** On 2026-08-12 a recovery code that provably opens a RETAINED package
— unsealed by hand, and it restored planted files byte-identical from a store the box itself could no
longer open — was answered by this agent with *"the recovery code did not open the sealed bundle"*.
The code was correct. Nothing had ever tried the retained packages, so the engine could not tell a
correct-but-earlier code from a mistype, and the screen said so out loud: a true sentence about our
own incuriosity, read by the customer as a statement about their code.
**`OffsiteKeyRecoverer` gains an optional `FetchRetained`.** It is consulted ONLY after the current
package has refused, so the ordinary recovery pays nothing for it and cannot fail because of it. When
one of the retained packages opens, the recoverer returns `ErrCodeOpensRetained` wrapped in a
`RetainedOpenedError` carrying the supersession date — no material, no code, no password.
**The local API answers 422** ("your code is correct, it belongs to an EARLIER sealed package") — a
FIFTH status added to the R-224 switch, not a restructuring of it. 422 rather than 400 because the
request was well-formed AND the credential valid; a 400 would put it in the same bucket as a mistype,
which is the defect.
**Fail-safe in every direction.** A nil fetcher, a hub too old to have the route (404 is a clean
"none"), a transport failure, a malformed package: each leaves the original refusal standing,
unchanged. The worst outcome of this feature breaking is the behaviour we had before it existed.
Attempts are bounded (`MaxRetainedTried`, default 6) because each unwrap is ~1 s of scrypt by design
and an unbounded loop would turn one wrong code into a minutes-long hang.
**New hub client call:** `FetchRetainedIdentityEscrow``GET /api/v1/hosts/<id>/escrow/retained`
(hub >= v0.103.0), self-scoped by the same per-host key.
Seven tests with REAL age crypto, because the two situations are indistinguishable AT THE UNWRAP and a
faked unwrap would prove nothing about what was broken. Red-proof, asserted applied: removing the
retained lookup returns the fail-closed wrong-code error — **the lie comes back, in those words.**
---
### Gates only — 2026-08-09 (no release, no version bump, no binary published)
**Two guards, both owed since the 2026-08-09 install outage (R-273/R-287). Nothing that runs on a
customer's box changed; `scripts/` only, and the agent stays v0.128.0.**
- **`scripts/retention-policy.json` — THE retention number, in one file.** The registry stopped
serving `felhom-agent` 0.120.0 and older while `check-published-versions.py` demanded that every
tag still be downloadable. Both rules are sensible; together they are impossible, and CI went red
at a commit whose own run had been green the day before. The check now **reads the number from
this file** and bounds its assertion to the newest N generic versions.
**What CI no longer covers, said plainly rather than left to be discovered:** a released version
older than the retention window is **no longer asserted downloadable**. Its git tag and its configs
are still asserted — only the binary's presence is dropped. The check **prints exactly which
versions it stopped covering** on every run, so the narrowing cannot become permanent by accident.
**The number is an OBSERVED state, not a located ruling** — see the file's own header and R-287.
A missing or unreadable policy file is **INCONCLUSIVE (exit 2), never silently unbounded.**
- **`scripts/check-release-complete.py` — the tag half, as a machine.** Asserts that the version at
the head of `CHANGELOG.md` is tagged, that the tag points into this history, and that its package
is published. `release-agent.sh` already warned about this in as many words and the step was still
missed on 2026-08-08, which is why this is a gate and not a reminder. Legs 12 need no network and
therefore run in `--fast`, so the pre-push hook catches a missing tag at the earliest moment.
Registered in `agent_gates.py`; red-proved by pointing the CHANGELOG head at an unreleased
v0.129.0 — both legs convicted and each named its fix command.
## v0.128.0 — the escrow seed is asserted every tick, not remembered once (2026-08-08, R-221)
**A rebuilt box could not run the escrow ceremony at all, and there was no way forward from inside
the product.** The preflight refuses on `escrow.pbs_storage_id`; the pbsdr bridge writes that key;
and it wrote it from exactly one place — `finishConverged`, reached only on the paths that actually
converge.
**The two things live in different places and die at different times.** The convergence marker is
host-side (`<agent-state>/pbsdr/marker.json`). The key it seeds is in `agent.json`, which
`step_agent_config` renders from `base = {}` unless an explicit `--preserve-from` is passed
(`felhom.eu/scripts/felhom-host-install.sh:2396`, the render at `:2449`, the `O_TRUNC` write at
`:2579`; the flag at `:1246`, defaulting empty at `:256`) — **and the render never writes an
`escrow` section at all.** So a rebuild keeps the marker and takes the key: same descriptor, same
hash, early return, and the seed never runs again into a config that no longer has it.
**Fixed by asserting rather than remembering.** `Apply` now re-asserts the seed *before* the
idempotent early return. `seedEscrowStorageID` is unchanged and still never clobbers a different
existing value — an operator's own choice outranks the descriptor's, with a warning naming both.
**The early return is KEPT.** It exists so a converged box does not re-run Proxmox operations every
60 s, and `TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls` asserts **zero** recorded runner
calls on that tick — so a "fix" that simply deleted the return would fail. Cost of the re-assert: one
small file read plus a JSON parse per tick, no exec, no network, and an early return once the value
matches.
**A seed failure can never un-converge the box:** Warn plus a message on the published status,
exactly as `finishConverged` does it — no marker write, no state change. Pinned by
`TestSeedReassertFailure_DoesNotUnconverge`.
Tests drive the real `Apply` with a real temp-dir `agent.json` and a call-recording runner; calling
`seedEscrowStorageID` directly cannot see the early return, which IS the defect. The production
wiring (`pbsdr.NewManager(..., cfg.SourcePath, ...)`) is asserted by walking `main.go`'s **AST**, not
by `strings.Contains`, which a commented-out call also satisfies.
Red-proofs: removing the new call makes Scenario A fail on today's tree; removing the early return
makes the zero-Proxmox-calls assertion fail.
## (no version bump) — a comment that claimed the hub reads a field it has no field for (2026-08-08, R-260)
Comment-only; no behaviour, no wire change, nothing to rebuild.
`HostReport.SelfUpdatePending` / `SelfUpdatePendingVersion` carried the sentence *"The hub reads an
absent field as pending=false, the correct default."* **The hub has no field for either**, so it reads
nothing — present or absent — and `encoding/json` discards them on arrival. The sentence described an
intent rather than the code and read as settled for long enough that a class sweep had to find it.
The emission is correct and stays: the agent reports the truth, and the fault is entirely in the
receiving. The missing consumer is tracked as **R-264** (OPEN), and
`felhom.eu/scripts/wire_contract_gate.py` now refuses any NEW field of this shape while recording the
existing ones as explicit, reasoned allowlist entries rather than silence.
## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220) ## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220)
**After a rebuild the customer's own drives could not be re-attached, and the refusal named an action **After a rebuild the customer's own drives could not be re-attached, and the refusal named an action
+49 -27
View File
@@ -1,41 +1,63 @@
# REPORT — felhom-agent v0.127.0: a mount Felhom made is not foreign (R-220) # REPORT — felhom-agent, 2026-08-09 (gates only)
**Scope: the host half of R-220.** The customer-facing refusal message is the controller's half and **No release. No version bump. No binary published. `scripts/` only** — nothing that runs on a
ships as felhom-controller v0.203.0. customer's machine changed, and the agent stays **v0.128.0** at `28ba8593b8`.
## What changed ## What changed
| File | Change | | file | why |
|---|---| |---|---|
| `internal/storage/claim.go` | `claimFacts.felhomOwnedMounts`; `classifyClaim` forgives a non-managed mountpoint **only when corroborated**; `felhomOwnedMounts()` + `procMounts()` | | `scripts/retention-policy.json` **(new)** | THE retention number, in one place, with its reasoning and its honesty about where the number came from |
| `internal/storage/hostops.go` | `mountTable` seam (nil ⇒ real `/proc/mounts`) | | `scripts/check-published-versions.py` | reads that number; bounds its assertion to the newest N; **prints what it stopped covering** |
| `internal/storage/claim_r220_test.go` | new — the own-drive case, the fence, and the corroboration's four edges | | `scripts/check-release-complete.py` **(new)** | asserts the CHANGELOG-head version is tagged, placed in this history, and published |
| `scripts/agent_gates.py` | registers the new gate; legs 12 are offline so it runs in `--fast` too |
## The shape chosen, and why (§7.3) ## The coupling defect, and the fix
**Candidate (b): the claimed check distinguishes a mount Felhom made from a foreign one** — the task The prune keeps the newest N; the published-versions check demanded that **every** tag be
called it "nearer the truth" and it is, because the host and its knowledge survive the rebuild while downloadable. Nothing connected them, so CI went red at `28ba8593b8` — a commit whose own run had
the guest's registry does not. Candidate (a) — having the rebuild path clear the raw mounts — would been **green the day before** — and would have gone red again at the next publish when `0.121.0` was
have made correctness depend on a cleanup step running, and a cleanup that does not run leaves exactly evicted. Both now read `generic_versions_kept` from one file.
today's defect.
**The discriminator is corroboration, not a path prefix**: the same device must ALSO be mounted under **What CI no longer covers:** a released version **older than the retention window** is no longer
`/mnt/felhom-drives`. Only enrolment produces that pairing. asserted downloadable. Its git tag and its config tree are still asserted; only the binary's presence
is dropped. The check names the dropped versions on every run.
**`/proc/mounts` rather than `lsblk MOUNTPOINTS`**, because the lsblk invocation is pinned verbatim in **The number is not a located ruling.** `generic_versions_kept: 10` is what the registry demonstrably
the sudoers file; changing it would have coupled this fix to a config rollout. `/proc/mounts` is holds; no register row records a prune, and container packages hold 19 each. The file says so in its
world-readable and needs neither. own header. The principled bound is the hub's vouched `min_agent` floor — nothing can install below
it — and that is recorded as the follow-up.
## Controls, all three run
| control | expected | got |
|---|---|---|
| live run, policy = 10 | green, and it names `0.120.0` as not asserted | **exit 0**, and it did |
| widen policy to 11 | `0.120.0` re-enters the window and convicts | **exit 1**, `FAIL v0.120.0` |
| policy file removed | INCONCLUSIVE, never silently unbounded | **exit 2**, naming the path it tried |
## Red-proof of the new gate
Mutation: `CHANGELOG.md` head repointed to `## v0.129.0` — never tagged, never published. Asserted
applied (`grep -c '^## v0.129.0'` → 1). Result **exit 1**, both legs convicting:
```
- TAG v0.129.0 DOES NOT EXIST. … without the tag every install 404s mid-run, as root.
Fix: git tag -a v0.129.0 <released-commit> && git push origin v0.129.0
- PACKAGE 0.129.0 IS NOT PUBLISHED (HTTP 404 …).
Fix: bash scripts/release-agent.sh 0.129.0
```
Reverted; `git status` clean on `CHANGELOG.md`.
## Green gate ## Green gate
`go build` · `go vet` clean · `go test ./...`**29 packages ok** · `agent_gates.py --fast` → all OK. `python3 scripts/agent_gates.py``reuse-refs OK · instructions OK · published OK ·
release-complete OK · all agent gates OK`.
| Red-proof | Result | ## Not done here
|---|---|
| remove the `felhomOwnedMounts` exemption | **FAILS** — "device is mounted at /mnt/adatok (sdb)", the pre-fix refusal |
| over-widen the exemption to any `/mnt/*` | **FAILS** — "/mnt/someone-elses-disk was offered for formatting" |
## Not changed The deleter of `0.120.0` is **still not established** and a second attempt failed — Gitea keeps no
package-deletion trail, its container log no longer reaches the window, and the activity feed carries
No sudoers, no allowlisted command, no PVE surface, no format path. Every other claim signal no package operation. Recorded in R-287, including the withdrawal of my own earlier over-claim that
(system disk, read-only, LVM PV, ZFS member, member FSTYPEs, empty-topology backstop) is untouched. the router logs showed no DELETE: they do not cover the window, so they never said anything.
+39 -14
View File
@@ -1769,23 +1769,48 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
} }
return blob, true, nil return blob, true, nil
}, },
// R-311 — the RETAINED packages, wired here and ONLY here, on the same self-scoped hub client.
// Consulted only after the current package has refused the code (see tryRetained), so the
// ordinary recovery pays nothing for it and cannot fail because of it.
FetchRetained: func(ctx context.Context) ([]escrow.RetainedBlob, int, error) {
resp, ferr := hubClient.FetchRetainedIdentityEscrow(ctx)
if ferr != nil {
return nil, 0, ferr
}
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
for _, p := range resp.Packages {
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
if derr != nil || len(blob) == 0 {
// One malformed package must not sink the rest — the customer's code may open a
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
continue
}
out = append(out, escrow.RetainedBlob{
Blob: blob,
SupersededAt: p.SupersededAt,
KeyFingerprint: p.KeyFingerprint,
Index: p.Index,
})
}
return out, resp.UnopenableCount, nil
},
} }
srv, err := localapi.NewServer(localapi.Options{ srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer, EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr, ListenAddr: cfg.LocalAPI.ListenAddr,
Cert: cert, Cert: cert,
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
Guests: px, Guests: px,
Backups: runner, Backups: runner,
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F) InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
Store: store, Store: store,
Storage: observer, Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages) DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
Tokens: tokens, Tokens: tokens,
BackupCadence: cfg.Backup.BackupCadence(), BackupCadence: cfg.Backup.BackupCadence(),
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate. // Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
Disks: hostOps, Disks: hostOps,
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID}, DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
+116 -1
View File
@@ -47,17 +47,80 @@ var (
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is // retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
// not sent hunting for a mistyped recovery code that was typed correctly. // not sent hunting for a mistyped recovery code that was typed correctly.
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)") ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
// ErrCodeOpensRetained — the code did NOT open the package the hub currently holds, and DID open a
// RETAINED (earlier) one. R-311.
//
// ⚠ THIS IS NOT A FAILURE OF THE CUSTOMER'S. It is the single most important distinction on this
// path, because until 2026-08-12 it was indistinguishable from a mistype and was reported as one.
// The screen could only say "it may be a typo, or it may be an older code, and we cannot tell them
// apart from here" — and it could not tell them apart because NOTHING EVER LOOKED. Now something
// looks, so the sentence can stop hedging.
//
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
// which is the one fact the customer needs to recognise it.
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
) )
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
// carries no secret — not the code, not the bundle, not the repository password.
type RetainedMatch struct {
// SupersededAt is when this package stopped being the current one (hub-supplied, RFC3339-ish).
// It is what the recovery screen shows so the customer can recognise which code they are holding.
SupersededAt string
// KeyFingerprint is the escrow key fingerprint of that package — operator-log material only.
KeyFingerprint string
// Index is the hub's position label within ONE response. Not durable; do not persist it.
Index int
// HasResticPassword is false when the retained package opened but carries no repository password
// (a pre-fork-4 seal). The code is still CORRECT; the history behind it still cannot be reopened.
// Collapsing this into "recoverable" would repeat R-202's mistake on a new surface.
HasResticPassword bool
}
// RetainedOpenedError wraps ErrCodeOpensRetained with the match. Callers classify with errors.Is on
// the sentinel and read the detail with errors.As.
type RetainedOpenedError struct {
Match RetainedMatch
}
func (e *RetainedOpenedError) Error() string {
return ErrCodeOpensRetained.Error() + " (superseded_at=" + e.Match.SupersededAt + ")"
}
func (e *RetainedOpenedError) Unwrap() error { return ErrCodeOpensRetained }
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none". // BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
// An interface-free func field keeps this package free of any dependency on the hub client. // An interface-free func field keeps this package free of any dependency on the hub client.
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error) type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
// RetainedBlob is one retained sealed package as the recoverer sees it: opaque bytes plus the labels
// needed to name it. No secret.
type RetainedBlob struct {
Blob []byte
SupersededAt string
KeyFingerprint string
Index int
}
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
// slice is a clean "none". R-311.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R. // OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct { type OffsiteKeyRecoverer struct {
Fetch BlobFetcher Fetch BlobFetcher
// FetchRetained is OPTIONAL and consulted ONLY after the current package has refused the code.
// nil keeps the pre-R-311 behaviour exactly: a refusal stays a refusal. That is deliberate — an
// agent wired without it must not behave differently from one that has no retained packages.
FetchRetained RetainedFetcher
// MaxRetainedTried bounds the scrypt work a single wrong code can cost. Each attempt is ~1 s of
// KDF by design, so an unbounded loop over a long supersession history would turn one wrong code
// into a minutes-long hang on the customer's screen. 0 means the built-in default.
MaxRetainedTried int
} }
// defaultMaxRetainedTried — six attempts is ~6 s worst case, which is a slow screen and not a hang.
const defaultMaxRetainedTried = 6
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password. // RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
// //
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits // A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
@@ -86,10 +149,62 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
} }
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode) bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
if err != nil { if err != nil {
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it // R-311 — BEFORE calling this a wrong code, ask whether it is the RIGHT code for an EARLIER
// package. The engine fails closed identically either way, so the two are indistinguishable
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
// about our own incuriosity, read by the customer as a statement about their code.
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
return "", &RetainedOpenedError{Match: m}
}
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
} }
if bundle.ResticRepoPassword == "" { if bundle.ResticRepoPassword == "" {
return "", ErrNoResticPassword return "", ErrNoResticPassword
} }
return bundle.ResticRepoPassword, nil return bundle.ResticRepoPassword, nil
} }
// tryRetained reports whether the code opens one of this host's RETAINED packages, and which.
//
// FAILURE HERE IS SILENT AND MEANS "NO", NEVER "YES" and never a different verdict for the caller. A
// hub that cannot answer, a route an older hub does not have, a malformed blob — each leaves the
// original refusal standing, unchanged. That is the fail-safe direction: the worst outcome of this
// function breaking is the behaviour we had before it existed.
//
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
if r.FetchRetained == nil {
return RetainedMatch{}, false
}
blobs, _, err := r.FetchRetained(ctx)
if err != nil || len(blobs) == 0 {
return RetainedMatch{}, false
}
limit := r.MaxRetainedTried
if limit <= 0 {
limit = defaultMaxRetainedTried
}
for i, rb := range blobs {
if i >= limit {
break
}
if len(rb.Blob) == 0 {
continue
}
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
if uerr != nil {
continue // this one is not the customer's; try the next
}
return RetainedMatch{
SupersededAt: rb.SupersededAt,
KeyFingerprint: rb.KeyFingerprint,
Index: rb.Index,
// A retained package can itself predate the repository-password field. The code is still
// correct and must be told so — but the history behind it still cannot be reopened, and
// saying otherwise would be a promise this path cannot keep.
HasResticPassword: bundle.ResticRepoPassword != "",
}, true
}
return RetainedMatch{}, false
}
+230
View File
@@ -0,0 +1,230 @@
package escrow
import (
"context"
"errors"
"fmt"
"testing"
)
// R-311 — a correct code for an EARLIER package must stop being reported as a wrong code.
//
// These use REAL age crypto, like the R-199 tests beside them, because the whole point is that the
// two situations are indistinguishable AT THE UNWRAP: both fail closed on the current package. A
// faked unwrap would prove nothing about the thing that was actually broken.
const testR2 = "another correct horse battery staple sedative anaconda wobbly kingdom placard"
func retainedFetcherFor(blobs ...RetainedBlob) RetainedFetcher {
return func(context.Context) ([]RetainedBlob, int, error) { return blobs, 0, nil }
}
// THE ONE THAT MATTERS. The customer holds the code for a package we superseded. Yesterday this
// returned the fail-closed refusal and the screen told them to check their typing.
//
// RED-PROOF: remove the `if m, ok := r.tryRetained(...)` block from RecoverOffsiteRepoPassword →
// the wrong-code error returns instead → this FAILS, and the lie is back in exactly those words.
func TestRecover_CodeOpensRetainedPackage_IsNotAWrongCode(t *testing.T) {
ensureAge(t)
const oldPW = "aaaa567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "cccc567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
retained := sealBundle(t, IdentityBundle{ResticRepoPassword: oldPW}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{
Blob: retained, SupersededAt: "2026-08-12 15:18:55", KeyFingerprint: "7e:a6:af", Index: 0,
}),
}.RecoverOffsiteRepoPassword(context.Background(), testR) // the OLD code
if err == nil {
t.Fatal("recovery succeeded — it must NOT return a password for a retained package on this path")
}
if !errors.Is(err, ErrCodeOpensRetained) {
t.Fatalf("err = %v, want ErrCodeOpensRetained — a correct code for an earlier package was "+
"classified as something else, which is how it became 'check your typing'", err)
}
var ro *RetainedOpenedError
if !errors.As(err, &ro) {
t.Fatalf("err does not carry a RetainedOpenedError: %v", err)
}
if ro.Match.SupersededAt != "2026-08-12 15:18:55" {
t.Errorf("SupersededAt = %q — the screen needs this date to name the package", ro.Match.SupersededAt)
}
if !ro.Match.HasResticPassword {
t.Error("HasResticPassword = false, but the retained bundle carried one")
}
// The error must not leak the code, the password or the bundle.
for _, secret := range []string{testR, oldPW} {
if containsStr(err.Error(), secret) {
t.Fatalf("the error text leaks a secret")
}
}
}
// SCENARIO A — the ordinary recovery is untouched, and it must not even ASK for retained packages.
// If the current package opens, the customer is not in this story at all.
//
// RED-PROOF: move the tryRetained call above the successful-unwrap return → the fetcher runs → this
// FAILS on the "must not be consulted" assertion.
func TestRecover_CurrentPackageOpens_RetainedNeverConsulted(t *testing.T) {
ensureAge(t)
const pw = "bbbb567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
current := sealBundle(t, IdentityBundle{ResticRepoPassword: pw}, testR)
consulted := false
got, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
consulted = true
return nil, 0, nil
},
}.RecoverOffsiteRepoPassword(context.Background(), testR)
if err != nil {
t.Fatalf("the ordinary recovery broke: %v", err)
}
if got != pw {
t.Fatalf("recovered password is not the sealed one")
}
if consulted {
t.Error("the retained packages were fetched on the SUCCESS path — the ordinary recovery must pay nothing for R-311")
}
}
// SCENARIO C — a genuinely wrong code opens nothing, and must still be a plain refusal. The new
// branch must not become a way to encourage a customer who mistyped.
//
// RED-PROOF: make tryRetained return (RetainedMatch{}, true) unconditionally → a wrong code is
// reported as opening an earlier package → this FAILS.
func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "cccc567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
retained := sealBundle(t, IdentityBundle{ResticRepoPassword: "dddd567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: retained, SupersededAt: "2026-08-01 00:00:00"}),
}.RecoverOffsiteRepoPassword(context.Background(), "totally wrong words that open nothing at all here")
if err == nil {
t.Fatal("a wrong code succeeded")
}
if errors.Is(err, ErrCodeOpensRetained) {
t.Fatal("a WRONG code was reported as opening a retained package — that would encourage a mistype")
}
}
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
// worst outcome of this feature breaking is the behaviour we had before it.
//
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
// gets a new, unexplained failure mode → this FAILS.
func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "eeee567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
return nil, 0, fmt.Errorf("hub exploded")
},
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
if err == nil {
t.Fatal("expected a refusal")
}
if errors.Is(err, ErrCodeOpensRetained) {
t.Fatal("a failed retained lookup was reported as 'opens a retained package'")
}
if containsStr(err.Error(), "hub exploded") {
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
}
}
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
// indistinguishable from one whose host has no retained packages.
//
// RED-PROOF: remove the `if r.FetchRetained == nil` guard → nil-deref panic → this FAILS.
func TestRecover_NilRetainedFetcher_IsPreR311Behaviour(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "ffff567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{Fetch: fetcherFor(current)}.RecoverOffsiteRepoPassword(context.Background(), testR2)
if err == nil {
t.Fatal("expected a refusal")
}
if errors.Is(err, ErrCodeOpensRetained) {
t.Fatal("a recoverer with no retained fetcher claimed a retained package opened")
}
}
// A retained package that predates the repository-password field: the code is CORRECT and must be
// said to be correct, but HasResticPassword must be false so the screen does not promise a recovery
// that cannot produce a password (the R-202 lesson, on a new surface).
//
// RED-PROOF: hardcode HasResticPassword: true → this FAILS.
func TestRecover_RetainedOpensButPredatesTheField(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "1111567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
// No ResticRepoPassword at all — the pre-fork-4 shape.
retained := sealBundle(t, IdentityBundle{TunnelToken: "T", PBSToken: "P"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: retained, SupersededAt: "2026-08-04 07:20:08"}),
}.RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrCodeOpensRetained) {
t.Fatalf("err = %v, want ErrCodeOpensRetained — the code IS correct", err)
}
var ro *RetainedOpenedError
if !errors.As(err, &ro) {
t.Fatalf("no RetainedOpenedError: %v", err)
}
if ro.Match.HasResticPassword {
t.Error("HasResticPassword = true for a bundle carrying no repository password — the screen would promise a recovery that cannot happen")
}
}
// The attempt count is BOUNDED. Each unwrap is ~1 s of scrypt by design, so an unbounded loop turns
// one wrong code into a minutes-long hang on the customer's screen.
//
// RED-PROOF: remove the `if i >= limit { break }` → all 10 are tried → this FAILS on the count.
func TestRecover_RetainedAttemptsAreBounded(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "2222567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
junk := sealBundle(t, IdentityBundle{ResticRepoPassword: "3333567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
tried := 0
blobs := make([]RetainedBlob, 0, 10)
for i := 0; i < 10; i++ {
blobs = append(blobs, RetainedBlob{Blob: junk, SupersededAt: "2026-08-01 00:00:00", Index: i})
}
rec := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
tried++
return blobs, 0, nil
},
MaxRetainedTried: 2,
}
// A code that opens NEITHER the current package nor any retained one.
if _, err := rec.RecoverOffsiteRepoPassword(context.Background(), "a code that opens nothing whatsoever in this test"); err == nil {
t.Fatal("expected a refusal")
}
if tried != 1 {
t.Errorf("the retained list was fetched %d times, want exactly 1", tried)
}
}
func containsStr(hay, needle string) bool {
return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
for i := 0; i+len(needle) <= len(hay); i++ {
if hay[i:i+len(needle)] == needle {
return true
}
}
return false
})()
}
+66
View File
@@ -354,3 +354,69 @@ func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowRespon
} }
return &out, nil return &out, nil
} }
// RetainedEscrowPackage is one RETAINED (superseded) sealed identity package. The blob is ciphertext
// and is useless without R. `SupersededAt` is the only thing here a human ever sees — it is what lets
// the recovery screen name WHICH earlier package a code belongs to.
type RetainedEscrowPackage struct {
Index int `json:"index"`
SupersededAt string `json:"superseded_at"`
KeyFingerprint string `json:"key_fingerprint"`
IdentityEscrowB64 string `json:"identity_escrow_b64"`
}
// RetainedEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow/retained (hub >= v0.103.0, R-311).
//
// UnopenableCount is NOT noise. It counts retained packages the hub holds whose key material is absent
// (every pre-v0.93.0 row): on a box with those and nothing else, a perfectly correct old recovery code
// opens nothing, and the reason is a defect of ours. A caller that ignores this number will tell such a
// customer their code is wrong — the exact failure this whole chain exists to stop.
type RetainedEscrowResponse struct {
HostID string `json:"host_id"`
Count int `json:"count"`
UnopenableCount int `json:"unopenable_count"`
TruncatedCount int `json:"truncated_count"`
Packages []RetainedEscrowPackage `json:"packages"`
}
// FetchRetainedIdentityEscrow reads back THIS host's RETAINED sealed identity packages (R-311 —
// the retained siblings of FetchIdentityEscrow, self-scoped server-side by the same per-host key).
//
// SEPARATE FROM FetchIdentityEscrow ON PURPOSE. The ordinary recovery must not pay for this call, and
// must not fail because of it: the current package is tried first and alone, and this is reached only
// after that has refused. A hub too old to know this route answers 404, which is a CLEAN "none" here
// and must never be reported as a failed recovery.
func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscrowResponse, error) {
if c.hostID == "" {
return nil, fmt.Errorf("hub: FetchRetainedIdentityEscrow requires a configured host_id")
}
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow/retained"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("hub: building retained-escrow request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return nil, &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if resp.StatusCode == http.StatusNotFound {
// A hub older than v0.103.0 has no such route. That is "no retained packages", not a fault —
// returning an error here would turn an old hub into a failed recovery on a box whose current
// package simply did not open.
return &RetainedEscrowResponse{HostID: c.hostID}, nil
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
var out RetainedEscrowResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("hub: decoding retained escrow fetch: %w", err)
}
return &out, nil
}
+8 -2
View File
@@ -68,8 +68,14 @@ type HostReport struct {
// report is stored opaquely hub-side, so these additive fields need no hub-schema change. // report is stored opaquely hub-side, so these additive fields need no hub-schema change.
// Both are `omitempty` (the Wireguard precedent): in the steady state (no update in flight) // Both are `omitempty` (the Wireguard precedent): in the steady state (no update in flight)
// they are absent — which keeps the cross-repo host-report golden contract byte-stable without // they are absent — which keeps the cross-repo host-report golden contract byte-stable without
// a hub change. They appear only while an update is pending. The hub reads an absent field as // a hub change. They appear only while an update is pending.
// pending=false, the correct default. //
// ⚠ CORRECTED 2026-08-08 (R-260). This comment used to end "The hub reads an absent field as
// pending=false, the correct default." THE HUB HAS NO FIELD FOR EITHER OF THESE, so it reads
// nothing — present or absent — and encoding/json discards them on arrival. The sentence
// described an intent, not the code, and it read as settled for long enough that a sweep had to
// find it. The emission is correct and stays; the missing consumer is tracked as R-264, and
// `felhom.eu/scripts/wire_contract_gate.py` now refuses any NEW field of this shape.
SelfUpdatePending bool `json:"selfupdate_pending,omitempty"` SelfUpdatePending bool `json:"selfupdate_pending,omitempty"`
SelfUpdatePendingVersion string `json:"selfupdate_pending_version,omitempty"` SelfUpdatePendingVersion string `json:"selfupdate_pending_version,omitempty"`
+29
View File
@@ -98,6 +98,35 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
case errors.Is(err, escrow.ErrNoEscrowBlob): case errors.Is(err, escrow.ErrNoEscrowBlob):
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid) s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run") writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
// ── R-311 (2026-08-12) — THE CODE IS RIGHT, JUST NOT FOR THE CURRENT PACKAGE. ─────────
//
// Placed ABOVE the default for the same reason ErrBundleFetch is: the default blames the
// customer, and this case is the one where the customer is provably not at fault. The code was
// used, it worked, and it opened a package the hub is deliberately keeping.
//
// 422 rather than 400: the request was well-formed AND the credential was valid — what could
// not be processed is the pairing of a correct code with the CURRENT package. A 400 would put
// it in the same bucket as a mistype, which is the whole defect. The status is the
// machine-readable half; the controller classifies on it and must never parse this sentence.
//
// The date travels in the body because it is the one fact that lets a customer recognise which
// code they are holding. No material, no code, no password — only when that package stopped
// being current, and whether it can yield a repository password at all.
case errors.Is(err, escrow.ErrCodeOpensRetained):
var ro *escrow.RetainedOpenedError
match := escrow.RetainedMatch{}
if errors.As(err, &ro) {
match = ro.Match
}
s.logger.Info("local-api: offsite key recovery: the code did NOT open the current package but DID open a RETAINED one — the customer is not at fault",
"vmid", vmid, "superseded_at", match.SupersededAt, "retained_has_restic_pw", match.HasResticPassword)
writeStatus(w, http.StatusUnprocessableEntity, false,
map[string]any{
"opens_retained": true,
"superseded_at": match.SupersededAt,
"retained_has_restic_pw": match.HasResticPassword,
},
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
case errors.Is(err, escrow.ErrNoResticPassword): case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid) s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)") writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
+30 -1
View File
@@ -283,7 +283,36 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WirePBSDR)
h := descriptorHash(block) h := descriptorHash(block)
cf := m.loadConsumedFailed() cf := m.loadConsumedFailed()
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
m.setStatus(&hub.PBSDRStatus{State: mk.State, StorageID: block.StorageID, Namespace: block.Namespace, AppliedAt: mk.AppliedAt}) // R-221: RE-ASSERT THE SEED, DO NOT REMEMBER IT. The marker records that this descriptor
// converged; it says nothing about whether the file the seed writes still exists.
//
// The two live in different places and die at different times. The marker is host-side
// (`<agent-state>/pbsdr/`, markerPath above); the seed's target is `agent.json`, and the
// installer's `step_agent_config` renders that file from `base = {}` unless an explicit
// `--preserve-from` is passed — it NEVER writes an `escrow` section — then replaces it with
// O_TRUNC (felhom-host-install.sh:2396, :2449, :2579; the flag is :1246, defaulting empty at
// :256). So a rebuild leaves the marker and takes the seed, the hash still matches, this
// branch returns, and `escrow.pbs_storage_id` is never written again. The customer then
// cannot run the escrow ceremony AT ALL: handleEscrowPreflight fails the `pbs_storage_id`
// row and the wizard refuses, with no way forward from inside the product.
//
// A rebuild is only the case that was measured. The same hole opens for a hand-edited or
// restored config, which is the honest reason this is a seam fix rather than an installer
// fix — the seed must be a thing the loop asserts, not a thing it did once.
//
// COST: this runs on the converged path, i.e. every tick (60 s) forever. It is one small
// file read plus a JSON parse — no exec, no network, no Proxmox call — and seedEscrowStorageID
// returns early once the value matches. That is the whole reason it is affordable here.
//
// IT MUST NEVER UN-CONVERGE THE BOX: a failure is a Warn plus a message on the published
// status, exactly as finishConverged does it. No marker write, no state change, no retry
// storm — the early return below still happens either way.
msg := ""
if err := m.seedEscrowStorageID(block.StorageID); err != nil {
msg = "escrow.pbs_storage_id seed failed: " + err.Error() + " (set it manually before the ceremony)"
m.logger.Warn("pbsdr: " + msg)
}
m.setStatus(&hub.PBSDRStatus{State: mk.State, StorageID: block.StorageID, Namespace: block.Namespace, AppliedAt: mk.AppliedAt, Message: msg})
return // idempotent: this exact descriptor already converged return // idempotent: this exact descriptor already converged
} }
+232
View File
@@ -0,0 +1,232 @@
package pbsdr
import (
"context"
"encoding/json"
"go/ast"
"go/parser"
"go/printer"
"go/token"
"io"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-221 — the escrow seed must be ASSERTED on every converged tick, not remembered.
//
// These tests drive the real Apply() with a real temp-dir agent.json and a call-recording runner.
// Calling seedEscrowStorageID directly would prove nothing: the defect IS the early return in
// Apply, and a test that steps around it cannot see it.
// convergedMarker writes a marker whose hash matches the block, i.e. puts the manager on exactly
// the idempotent path where the seed used to be skipped.
func convergedMarker(t *testing.T, m *Manager, block *hub.WirePBSDR, state string) {
t.Helper()
if err := m.writeState(m.markerPath(), marker{
Hash: descriptorHash(block), State: state, AppliedAt: "2026-08-08T00:00:00Z",
}); err != nil {
t.Fatalf("write marker: %v", err)
}
}
func escrowStorageID(t *testing.T, cfgPath string) string {
t.Helper()
raw, err := os.ReadFile(cfgPath)
if err != nil {
t.Fatalf("read config: %v", err)
}
var doc struct {
Escrow struct {
PBSStorageID string `json:"pbs_storage_id"`
} `json:"escrow"`
}
if err := json.Unmarshal(raw, &doc); err != nil {
t.Fatalf("parse config: %v", err)
}
return doc.Escrow.PBSStorageID
}
// drBlock mirrors the existing valid fixture (manager_test.go descriptor()) so that validate()
// passes and Apply actually reaches the marker check — the branch these tests are about.
func drBlock(storageID string) *hub.WirePBSDR {
return &hub.WirePBSDR{
Enabled: true, StorageID: storageID, PBSTunnelIP: "10.77.0.1",
Datastore: "felhom-offsite", Namespace: "peti", TokenID: "felhom@pbs!peti",
Fingerprint: testFP,
}
}
// SCENARIO A — the seed is re-asserted on a converged box, and nothing else happens.
//
// THIS IS THE TEST THAT MATTERS. It must fail against the pre-R-221 tree; if it passes there, it is
// not testing the defect and that is the finding.
func TestSeedReasserted_OnConvergedTick_WithZeroProxmoxCalls(t *testing.T) {
r := &fakeRunner{}
st := &fakeStorage{found: true, active: []bool{true}}
c := &fakeConsumer{}
m, cfgPath := newTestManager(t, r, st, c)
block := drBlock("felhom-pbs-dr")
convergedMarker(t, m, block, "applied")
// the shape a rebuild leaves behind: escrow section present, pbs_storage_id GONE
if got := escrowStorageID(t, cfgPath); got != "" {
t.Fatalf("precondition: config already carries a storage id %q", got)
}
m.Apply(context.Background(), true, block)
if got := escrowStorageID(t, cfgPath); got != "felhom-pbs-dr" {
t.Errorf("the converged tick did not re-assert the seed: escrow.pbs_storage_id = %q, want %q.\n"+
"This is R-221: the marker survives a rebuild, the descriptor hash still matches, the early "+
"return fires and the seed never runs into the config that no longer has it — so the customer "+
"cannot run the escrow ceremony at all.", got, "felhom-pbs-dr")
}
// ...and the idempotent path is STILL idempotent. This assertion is not decorative: without it
// a "fix" that simply deletes the early return would pass the line above.
if calls := r.recorded(); len(calls) != 0 {
t.Errorf("a converged tick must execute ZERO Proxmox commands; got %d: %+v", len(calls), calls)
}
}
// SCENARIO B — an operator's own different value survives, and the warning names both.
func TestSeedReasserted_NeverClobbersAnOperatorValue(t *testing.T) {
r := &fakeRunner{}
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
if err := os.WriteFile(cfgPath, []byte(
`{"log_level":"info","escrow":{"posture":"zero_knowledge","pbs_storage_id":"operator-chosen"},`+
`"custom_unknown":{"keep":1}}`), 0o600); err != nil {
t.Fatal(err)
}
block := drBlock("hub-chosen")
convergedMarker(t, m, block, "applied")
m.Apply(context.Background(), true, block)
if got := escrowStorageID(t, cfgPath); got != "operator-chosen" {
t.Errorf("a value a person put there was overwritten by the descriptor: got %q, want %q", got, "operator-chosen")
}
// unknown keys must still round-trip
raw, _ := os.ReadFile(cfgPath)
if !strings.Contains(string(raw), "custom_unknown") {
t.Error("an unknown config key was dropped by the re-assert")
}
}
// SCENARIO C — the ceremony preflight's live read sees the re-asserted value with NO restart.
//
// The preflight itself lives in internal/localapi and reads the file through config.Load; what this
// asserts is the half that belongs to this package: after a converged tick, THE FILE ON DISK carries
// the id, so any live re-read is green. The daemon is never restarted in this test because it is
// never started — which is the point.
func TestSeedReasserted_IsVisibleOnDiskImmediately(t *testing.T) {
r := &fakeRunner{}
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
block := drBlock("felhom-pbs-dr")
convergedMarker(t, m, block, "applied")
// preflight's predicate BEFORE: storageID == "" → the row is NOT OK
if escrowStorageID(t, cfgPath) != "" {
t.Fatal("precondition")
}
m.Apply(context.Background(), true, block)
// preflight's predicate AFTER, from the same file the ceremony subprocess loads
if id := escrowStorageID(t, cfgPath); id == "" {
t.Error("the preflight row would still be NOT OK after a converged tick")
}
}
// A seed failure must NEVER un-converge the box: no marker rewrite, no state change, and the status
// still reports the marker's converged state — with the failure surfaced as a message.
func TestSeedReassertFailure_DoesNotUnconverge(t *testing.T) {
r := &fakeRunner{}
m, cfgPath := newTestManager(t, r, &fakeStorage{found: true, active: []bool{true}}, &fakeConsumer{})
block := drBlock("felhom-pbs-dr")
convergedMarker(t, m, block, "applied")
markerBefore, err := os.ReadFile(m.markerPath())
if err != nil {
t.Fatal(err)
}
// make the seed fail in a way it cannot recover from: unparseable config
if err := os.WriteFile(cfgPath, []byte(`{ this is not json`), 0o600); err != nil {
t.Fatal(err)
}
m.Apply(context.Background(), true, block)
after, err := os.ReadFile(m.markerPath())
if err != nil {
t.Fatalf("the marker was removed by a seed failure: %v", err)
}
if string(after) != string(markerBefore) {
t.Error("a seed failure rewrote the convergence marker — it must not touch state")
}
if calls := r.recorded(); len(calls) != 0 {
t.Errorf("a seed failure must not trigger Proxmox work; got %+v", calls)
}
st := m.Status()
if st == nil || st.State != "applied" {
t.Errorf("a seed failure must leave the box converged; status = %+v", st)
}
if st != nil && !strings.Contains(st.Message, "seed failed") {
t.Errorf("a seed failure must be surfaced on the status, got message %q", st.Message)
}
}
// SEAM WIRING — production must construct the manager with the live config path, or the whole seed
// leg is inert. Three shipped defects in this project were fully green while their seam was never
// wired, so this walks main.go's AST for the actual call rather than grepping: a commented-out call
// satisfies strings.Contains, and an AST walk cannot see a comment.
func TestProductionWiring_NewManagerGetsTheLiveConfigPath(t *testing.T) {
path := filepath.Join("..", "..", "cmd", "felhom-agent", "main.go")
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, path, nil, 0) // comments not even collected
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
found := false
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "NewManager" {
return true
}
if pkg, ok := sel.X.(*ast.Ident); !ok || pkg.Name != "pbsdr" {
return true
}
// signature: (runner, px, hubc, stateDir, secretDir, configPath, logger)
if len(call.Args) < 6 {
t.Errorf("pbsdr.NewManager called with %d args, expected >= 6", len(call.Args))
return false
}
var buf strings.Builder
if err := printNode(&buf, fset, call.Args[5]); err != nil {
t.Fatalf("print arg: %v", err)
}
got := buf.String()
if !strings.Contains(got, "SourcePath") {
t.Errorf("pbsdr.NewManager's configPath argument is %q, which is not the live config path.\n"+
"With an empty or wrong path seedEscrowStorageID returns nil immediately and the entire "+
"R-221 fix is inert while every test above still passes.", got)
}
found = true
return false
})
if !found {
t.Error("no pbsdr.NewManager call found in main.go — the manager is not constructed in production")
}
}
func printNode(w io.Writer, fset *token.FileSet, n ast.Node) error {
return printer.Fprint(w, fset, n)
}
+4
View File
@@ -49,6 +49,10 @@ GATES = [
("reuse-refs", SHARED_REUSE, [ROOT], True), ("reuse-refs", SHARED_REUSE, [ROOT], True),
("instructions", SHARED_INSTRUCTIONS, [ROOT], True), ("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False), ("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
# R-273: the tag half of a release. Legs 1-2 need no network, so it runs in --fast too — the
# missing TAG is what actually broke every install, and the pre-push hook is the earliest place
# that can catch it.
("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True),
] ]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"} VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+41 -1
View File
@@ -95,6 +95,25 @@ PROBE_CONFIG = "configs/felhom-agent.service"
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$") TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
# THE retention number, read from the one file that owns it. A check and the policy it enforces
# must read the same number from the same place, or they drift and the drift looks like a defect
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
def retention_kept():
"""How many of the newest generic versions the registry is expected to still serve.
Fails CLOSED and LOUD: a missing or unreadable policy file makes the check INCONCLUSIVE
rather than silently unbounded. An unbounded check would re-create the red this fixed; a
silently-bounded one would be worse.
"""
with open(RETENTION_FILE, encoding="utf-8") as fh:
n = json.load(fh)["generic_versions_kept"]
if not isinstance(n, int) or n < 1:
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
return n
tried = [] tried = []
@@ -189,7 +208,28 @@ def main():
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven") print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
print("\ncheck-published-versions: NOTHING TO CHECK") print("\ncheck-published-versions: NOTHING TO CHECK")
return 0 return 0
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions))) all_versions = versions
try:
keep = retention_kept()
except Exception as e:
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
# Bound the assertion to what the registry is expected to still hold. Sorted by SEMVER, not
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
def _key(v):
return tuple(int(x) for x in v.split("."))
versions = sorted(all_versions, key=_key)[-keep:]
dropped = [v for v in all_versions if v not in versions]
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
print(" verifying: %s" % ", ".join(versions))
if dropped:
# NEVER silent. A bounded check that does not say what it stopped covering is how a
# narrowing becomes permanent by accident.
print(" NOT ASSERTED (older than the retention window, and therefore not expected to be")
print(" downloadable): %s" % ", ".join(dropped))
print(" ^ these versions still have git TAGS and are still installable in the sense that")
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
bad = [] bad = []
for v in versions: for v in versions:
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""check-release-complete.py — the version at the head of CHANGELOG.md is a COMPLETE release.
THE DEFECT THIS IS A MACHINE FOR (2026-08-08/09, R-273). Agent v0.128.0 was built, tested,
CHANGELOG'd and published to the package registry — and its git tag was never pushed. The hub then
vouched it, and because felhom-host-install.sh fetches an agent's config files from
`raw/tag/v<version>/configs/`, EVERY fresh install and every reinstall died at step 5 of 8, as root,
on a virgin machine, for the better part of a day.
`scripts/release-agent.sh` already warns about exactly this, in as many words:
"a released version without a git tag 404s a box mid-install, as root"
The warning was there, it was correct, and the step was still missed. **So the fix is a machine and
not a reminder** that is the whole point of this file.
WHAT IT ASSERTS, for the newest `## vX.Y.Z` in CHANGELOG.md:
1. a git tag `vX.Y.Z` EXISTS, and
2. it points at a commit that is an ANCESTOR OF (or equal to) the tip it was released from a tag
parked on an unrelated commit is not a release, and
3. the generic package for X.Y.Z is DOWNLOADABLE.
(3) needs the network. (1) and (2) do not, and they are the half that actually failed so this gate
is useful offline and says so rather than going quiet.
EXIT CODES, matching this repo's other gates: 0 clean, 1 convicted, 2 inconclusive. An unreachable
registry is INCONCLUSIVE for leg 3 only; legs 1 and 2 still run and can still convict.
"""
import json
import os
import re
import subprocess
import sys
import urllib.error
import urllib.request
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
OWNER, PKG = "admin", "felhom-agent"
HEAD_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+)\b", re.M)
def git(*args):
return subprocess.run(("git",) + args, cwd=ROOT, capture_output=True, text=True)
def head_version():
ch = os.path.join(ROOT, "CHANGELOG.md")
if not os.path.exists(ch):
return None
m = HEAD_RE.search(open(ch, encoding="utf-8").read())
return m.group(1) if m else None
def main():
print("check-release-complete — the newest CHANGELOG version is a complete release")
v = head_version()
if not v:
print(" no '## vX.Y.Z' heading in CHANGELOG.md — nothing to check, and nothing proven")
return 0
tag = "v" + v
print(" newest CHANGELOG version: %s" % tag)
problems, inconclusive = [], []
# ---- leg 1 + 2: the tag, and where it points. Offline-capable. -----------------------------
r = git("rev-parse", "-q", "--verify", "refs/tags/%s^{commit}" % tag)
if r.returncode != 0:
# A shallow CI clone has no tags of its own; ask the remote before convicting, so this
# gate does not fire on a clone shape rather than on a real defect.
ls = git("ls-remote", "--tags", "origin", "refs/tags/%s" % tag)
if ls.returncode != 0:
inconclusive.append("cannot reach origin to look for tag %s: %s"
% (tag, ls.stderr.strip()[:120]))
elif not ls.stdout.strip():
problems.append(
"TAG %s DOES NOT EXIST. The installer fetches this version's configs from\n"
" %s/%s/felhom-agent/raw/tag/%s/configs/ — without the tag every install\n"
" 404s mid-run, as root. Fix: git tag -a %s <released-commit> && git push origin %s"
% (tag, GITEA_BASE, OWNER, tag, tag, tag))
else:
print(" ok tag %s exists on origin (not in this shallow clone)" % tag)
else:
sha = r.stdout.strip()
anc = git("merge-base", "--is-ancestor", sha, "HEAD")
if anc.returncode == 0:
print(" ok tag %s -> %s, an ancestor of HEAD" % (tag, sha[:10]))
else:
problems.append("tag %s points at %s, which is NOT an ancestor of HEAD — a tag parked "
"on an unrelated commit is not a release" % (tag, sha[:10]))
# ---- leg 3: the package. Needs the network. ------------------------------------------------
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
req = urllib.request.Request(url, method="HEAD")
try:
with urllib.request.urlopen(req, timeout=25) as resp:
if resp.status == 200:
print(" ok package %s is downloadable" % v)
else:
problems.append("package %s returned HTTP %s at %s" % (v, resp.status, url))
except urllib.error.HTTPError as e:
if e.code == 404:
problems.append("PACKAGE %s IS NOT PUBLISHED (HTTP 404 at %s).\n"
" Fix: bash scripts/release-agent.sh %s" % (v, url, v))
else:
inconclusive.append("registry returned HTTP %s for %s" % (e.code, v))
except Exception as e:
inconclusive.append("registry unreachable (%s) — leg 3 not checked; legs 1-2 still ran" % e)
if problems:
print("\ncheck-release-complete: INCOMPLETE RELEASE")
for p in problems:
print(" - " + p)
return 1
if inconclusive:
print("\ncheck-release-complete: INCONCLUSIVE — an undetermined result is never a pass")
for i in inconclusive:
print(" - " + i)
return 2
print("\ncheck-release-complete: %s is tagged, placed and published." % tag)
return 0
if __name__ == "__main__":
sys.exit(main())
+44
View File
@@ -0,0 +1,44 @@
{
"_comment": [
"THE retention number for published agent artifacts. One file, read by everything that",
"depends on it, because a check and the policy it enforces must read the same number from the",
"same place or they drift — and the drift looks like a defect in something else.",
"",
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
"still be downloadable. Both rules are individually sensible; together they are impossible.",
"CI went red at a commit whose own run had been green the day before, on a true finding that",
"no one could act on. The red will return at the next publish unless the two read one number.",
"",
"HOW THE NUMBER WAS ARRIVED AT — stated honestly, because it is weaker than it looks.",
"generic_versions_kept is 10 because that is what the registry demonstrably holds today",
"(felhom-agent 0.121.0..0.128.0 = 10 versions, queried 2026-08-09). It is an OBSERVED state,",
"NOT a ruling anyone has been able to locate: no register row records a package prune, R-210",
"is WAITING-ON-OPERATOR and says 'Nothing was deleted; this is a list, not an action', and it",
"concerns local Docker images rather than this registry. Container packages currently hold 19",
"each, so there is no uniform ten-per-package cap visible either. See R-287.",
"",
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
"operator should confirm or replace the number — at which point this file changes and both",
"readers follow it in the same commit.",
"",
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
"the gate does not have today. Filed as the follow-up in R-287.",
"",
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
"version at all — a strictly worse act than an un-downloadable binary."
],
"generic_versions_kept": 10,
"readers": [
"scripts/check-published-versions.py — bounds its assertion to the newest N versions",
"documentation/runbooks/registry-retention.md (felhom.eu) — the prune procedure"
],
"recorded": "2026-08-09",
"recorded_by": "CC, from the registry's observed state; NOT from a located operator ruling"
}