Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7569f34aeb | |||
| ede49b610d | |||
| f17ed11599 | |||
| 1db56bf837 | |||
| 53d047a6c1 |
+134
@@ -1,3 +1,137 @@
|
||||
## UNRELEASED — v0.130.0 candidate: the agent was the one leaking connections onto the off-site box (2026-08-20, R-344)
|
||||
|
||||
> **Deliberately not a release heading yet, and the `release-complete` gate is doing its job by
|
||||
> requiring that.** This build is **hand-installed on `demo-hp` only** so the fix can be proved
|
||||
> against `demo-felhom` as an untouched control. Publishing it — tag + Gitea package — would put it
|
||||
> on the control box through the self-update path and destroy the experiment. **When the operator
|
||||
> authorises the publish, this heading becomes `## v0.130.0` in the same commit as the tag and the
|
||||
> package**, and the gate then checks it for real. See R-347.
|
||||
|
||||
**What was measured, before anything was changed.** Between 2026-08-18 09:51:22Z and 2026-08-20
|
||||
08:02:13Z, ep0's PBS proxy accumulated **388 established connections** — 194 from each demo box, on a
|
||||
proxy whose descriptor ceiling is 65536 and whose runway at that rate was ~323 days. The connections
|
||||
were held open on **both** sides: ep0 showed 388 while the two boxes showed 194 + 194, at two separate
|
||||
instants, with the same source ports on each side, and **not one closed in a 31-minute window**.
|
||||
`ss -tnp` on the boxes named the holder: **`felhom-agent`**, 194 of 194 on each, one PID.
|
||||
|
||||
**`pvestatd` and `proxmox-backup-client` made 162,404 requests in that window and leaked zero.** They
|
||||
are 99.5% of the traffic to that endpoint and 0% of the leak. The agent made 811 requests — of which
|
||||
387 were `GET .../snapshots` — and leaked 388 sockets. One per call, within one.
|
||||
|
||||
**The defect, and it is two things compounding.**
|
||||
|
||||
- `internal/pbs/client.go` built its transport as a composite literal:
|
||||
`&http.Transport{TLSClientConfig: tlsCfg}`. That takes **`IdleConnTimeout` zero, which does not mean
|
||||
"use a sane default" — it means retain idle keep-alive connections FOREVER.**
|
||||
`http.DefaultTransport` sets 90s; hand-rolling the transport (which every client here must do,
|
||||
because they all pin TLS) silently discards it.
|
||||
- `pbsTargetsFromPVE` (`cmd/felhom-agent/main.go`) builds **a fresh `pbs.Client` every cycle**, as its
|
||||
own doc comment says, and drops the previous one. An abandoned `http.Transport` does **not** close
|
||||
its connections — it becomes unreachable while its `persistConn` read-loop goroutine keeps the
|
||||
socket alive. So each cycle stranded exactly one connection that nothing could ever close.
|
||||
|
||||
The cadences reconcile without fitting: a 900s live-snapshot collect (184.7 cycles in the window) plus
|
||||
a 6-hour verify loop (7.7) predicts 192.4 per box against **194 observed**.
|
||||
|
||||
**The fix is one field, restored to the standard library's own value.** New leaf package
|
||||
`internal/httpx` owns `DefaultIdleConnTimeout = 90 * time.Second` — 90s because that is what
|
||||
`http.DefaultTransport` uses, so there is nothing invented here to justify or tune — and
|
||||
`NewTransport(tlsCfg, idleConnTimeout)`, which returns a **fresh** transport (never shared: each caller
|
||||
pins a different endpoint) and treats a zero or negative timeout as **use the default, never "no
|
||||
timeout"**. `pbs.Config` gains an `IdleConnTimeout` field that production leaves unset; only tests set
|
||||
it, to avoid a 90-second wait.
|
||||
|
||||
**`internal/hub/client.go` and `internal/proxmox/client.go` carried the identical missing default and
|
||||
were corrected in the same pass — but neither contributed to the ep0 leak, and this entry must not be
|
||||
read as three leaks having been found.** Both are built **once per process**, so they held one idle
|
||||
connection for the life of the daemon rather than accumulating, and neither talks to ep0:8007.
|
||||
|
||||
**Tests, and what they deliberately do not assert.** `internal/pbs/client_leak_test.go` counts
|
||||
connections **server-side** and models what `pbsTargetsFromPVE` actually does — build a client, use it
|
||||
once, drop it on the floor — then asserts the connections go away. It does not assert `err == nil` and
|
||||
it does not assert that some field holds some value; both were true of the leaking code.
|
||||
|
||||
- **Red-proof 1 (the fix):** removing `IdleConnTimeout` from `NewTransport` fails the test with
|
||||
*"after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)"* — the count is
|
||||
in the message, so the failure cannot be mistaken for a timeout with another cause. Reverted.
|
||||
- **Red-proof 2 (the fix that would be worse than the bug):** setting `DisableKeepAlives: true` also
|
||||
makes the leak vanish — by dialling fresh for every request, which on a box polling ~40,000 times a
|
||||
day is strictly worse than what we started with. **The leak test PASSES under that mutation**;
|
||||
`TestPBSClient_KeepAliveStillReuses` is what catches it, failing with *"3 sequential requests over 3
|
||||
connection(s), want 1"*. Reverted.
|
||||
|
||||
**What this release does NOT do.** It does not reduce the poll rate (**R-336 stays open, but re-scoped
|
||||
— it was never the cause of this leak**), it does not refactor `pbsTargetsFromPVE` to cache or reuse
|
||||
clients (a one-line default restores the standard behaviour; a lifecycle refactor adds
|
||||
cache-invalidation questions for no measurable gain), and it adds no `CloseIdleConnections` call.
|
||||
|
||||
**One sentence in this entry was written before the deploy and was WRONG, and it is corrected here
|
||||
rather than quietly edited.** It read: *"does not clear the 388 descriptors already stuck on ep0 —
|
||||
those persist until that proxy restarts."* **Measured: they clear the moment the AGENT restarts.**
|
||||
Replacing the binary on `demo-hp` released exactly its 199 descriptors within one second
|
||||
(415 → 216 fd), and replacing it on `demo-felhom` released the remaining 203 (**220 → 17 fd in under
|
||||
two seconds**). **17 is precisely ep0's `t0` baseline** of 2026-08-18 09:51:22Z. ep0 was read-only
|
||||
throughout and its proxy PID never changed. The accumulated leak was never ep0's to hold on to — it
|
||||
was held on both sides, and closing either side ends it.
|
||||
|
||||
## v0.129.0 — a correct code for an earlier package stops being called wrong (2026-08-12, R-311)
|
||||
|
||||
**The measurement this fixes.** On 2026-08-12 a recovery code that provably opens a RETAINED package
|
||||
— unsealed by hand, and it restored planted files byte-identical from a store the box itself could no
|
||||
longer open — was answered by this agent with *"the recovery code did not open the sealed bundle"*.
|
||||
The code was correct. Nothing had ever tried the retained packages, so the engine could not tell a
|
||||
correct-but-earlier code from a mistype, and the screen said so out loud: a true sentence about our
|
||||
own incuriosity, read by the customer as a statement about their code.
|
||||
|
||||
**`OffsiteKeyRecoverer` gains an optional `FetchRetained`.** It is consulted ONLY after the current
|
||||
package has refused, so the ordinary recovery pays nothing for it and cannot fail because of it. When
|
||||
one of the retained packages opens, the recoverer returns `ErrCodeOpensRetained` wrapped in a
|
||||
`RetainedOpenedError` carrying the supersession date — no material, no code, no password.
|
||||
|
||||
**The local API answers 422** ("your code is correct, it belongs to an EARLIER sealed package") — a
|
||||
FIFTH status added to the R-224 switch, not a restructuring of it. 422 rather than 400 because the
|
||||
request was well-formed AND the credential valid; a 400 would put it in the same bucket as a mistype,
|
||||
which is the defect.
|
||||
|
||||
**Fail-safe in every direction.** A nil fetcher, a hub too old to have the route (404 is a clean
|
||||
"none"), a transport failure, a malformed package: each leaves the original refusal standing,
|
||||
unchanged. The worst outcome of this feature breaking is the behaviour we had before it existed.
|
||||
Attempts are bounded (`MaxRetainedTried`, default 6) because each unwrap is ~1 s of scrypt by design
|
||||
and an unbounded loop would turn one wrong code into a minutes-long hang.
|
||||
|
||||
**New hub client call:** `FetchRetainedIdentityEscrow` → `GET /api/v1/hosts/<id>/escrow/retained`
|
||||
(hub >= v0.103.0), self-scoped by the same per-host key.
|
||||
|
||||
Seven tests with REAL age crypto, because the two situations are indistinguishable AT THE UNWRAP and a
|
||||
faked unwrap would prove nothing about what was broken. Red-proof, asserted applied: removing the
|
||||
retained lookup returns the fail-closed wrong-code error — **the lie comes back, in those words.**
|
||||
|
||||
---
|
||||
|
||||
### Gates only — 2026-08-09 (no release, no version bump, no binary published)
|
||||
|
||||
**Two guards, both owed since the 2026-08-09 install outage (R-273/R-287). Nothing that runs on a
|
||||
customer's box changed; `scripts/` only, and the agent stays v0.128.0.**
|
||||
|
||||
- **`scripts/retention-policy.json` — THE retention number, in one file.** The registry stopped
|
||||
serving `felhom-agent` 0.120.0 and older while `check-published-versions.py` demanded that every
|
||||
tag still be downloadable. Both rules are sensible; together they are impossible, and CI went red
|
||||
at a commit whose own run had been green the day before. The check now **reads the number from
|
||||
this file** and bounds its assertion to the newest N generic versions.
|
||||
**What CI no longer covers, said plainly rather than left to be discovered:** a released version
|
||||
older than the retention window is **no longer asserted downloadable**. Its git tag and its configs
|
||||
are still asserted — only the binary's presence is dropped. The check **prints exactly which
|
||||
versions it stopped covering** on every run, so the narrowing cannot become permanent by accident.
|
||||
**The number is an OBSERVED state, not a located ruling** — see the file's own header and R-287.
|
||||
A missing or unreadable policy file is **INCONCLUSIVE (exit 2), never silently unbounded.**
|
||||
- **`scripts/check-release-complete.py` — the tag half, as a machine.** Asserts that the version at
|
||||
the head of `CHANGELOG.md` is tagged, that the tag points into this history, and that its package
|
||||
is published. `release-agent.sh` already warned about this in as many words and the step was still
|
||||
missed on 2026-08-08, which is why this is a gate and not a reminder. Legs 1–2 need no network and
|
||||
therefore run in `--fast`, so the pre-push hook catches a missing tag at the earliest moment.
|
||||
Registered in `agent_gates.py`; red-proved by pointing the CHANGELOG head at an unreleased
|
||||
v0.129.0 — both legs convicted and each named its fix command.
|
||||
|
||||
## v0.128.0 — the escrow seed is asserted every tick, not remembered once (2026-08-08, R-221)
|
||||
|
||||
**A rebuilt box could not run the escrow ceremony at all, and there was no way forward from inside
|
||||
|
||||
@@ -1,41 +1,50 @@
|
||||
# REPORT — felhom-agent v0.127.0: a mount Felhom made is not foreign (R-220)
|
||||
# REPORT — agent v0.129.0: a correct code for an earlier package (R-311, 2026-08-12)
|
||||
|
||||
**Scope: the host half of R-220.** The customer-facing refusal message is the controller's half and
|
||||
ships as felhom-controller v0.203.0.
|
||||
## What changed and why
|
||||
|
||||
## What changed
|
||||
Yesterday's drill proved a retained escrow package **works** — unsealed with the old recovery code, it
|
||||
opened a set-aside store and restored planted files byte-identical — while this agent answered that
|
||||
same correct code with *"the recovery code did not open the sealed bundle"*. Nothing had ever tried
|
||||
the retained packages, so a correct-but-earlier code and a mistype were genuinely indistinguishable.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `internal/storage/claim.go` | `claimFacts.felhomOwnedMounts`; `classifyClaim` forgives a non-managed mountpoint **only when corroborated**; `felhomOwnedMounts()` + `procMounts()` |
|
||||
| `internal/storage/hostops.go` | `mountTable` seam (nil ⇒ real `/proc/mounts`) |
|
||||
| `internal/storage/claim_r220_test.go` | new — the own-drive case, the fence, and the corroboration's four edges |
|
||||
- `internal/hub/client.go` — `FetchRetainedIdentityEscrow` → `GET /api/v1/hosts/<id>/escrow/retained`
|
||||
(hub ≥ v0.103.0). **A 404 is a clean "none"**, not a fault: an older hub must not turn into a failed
|
||||
recovery.
|
||||
- `internal/escrow/recover.go` — optional `FetchRetained`, `ErrCodeOpensRetained` +
|
||||
`RetainedOpenedError{SupersededAt, KeyFingerprint, Index, HasResticPassword}`. Consulted **only**
|
||||
after the current package refuses.
|
||||
- `internal/localapi/escrow_recover.go` — a **fifth** case on the R-224 switch: **422**, with
|
||||
`opens_retained`, `superseded_at`, `retained_has_restic_pw`. Added to the switch, not a restructure.
|
||||
- `cmd/felhom-agent/main.go` — the retained fetcher wired on the same self-scoped hub client.
|
||||
|
||||
## The shape chosen, and why (§7.3)
|
||||
## Fail-safe, in every direction
|
||||
|
||||
**Candidate (b): the claimed check distinguishes a mount Felhom made from a foreign one** — the task
|
||||
called it "nearer the truth" and it is, because the host and its knowledge survive the rebuild while
|
||||
the guest's registry does not. Candidate (a) — having the rebuild path clear the raw mounts — would
|
||||
have made correctness depend on a cleanup step running, and a cleanup that does not run leaves exactly
|
||||
today's defect.
|
||||
nil fetcher · hub without the route (404) · transport failure · malformed package → **the original
|
||||
refusal stands, unchanged**. The worst outcome of this feature breaking is the behaviour before it.
|
||||
Attempts bounded (`MaxRetainedTried`, default 6) — each unwrap is ~1 s of scrypt, so an unbounded loop
|
||||
would turn one wrong code into a minutes-long hang.
|
||||
|
||||
**The discriminator is corroboration, not a path prefix**: the same device must ALSO be mounted under
|
||||
`/mnt/felhom-drives`. Only enrolment produces that pairing.
|
||||
## Tests — 7, with REAL age crypto
|
||||
|
||||
**`/proc/mounts` rather than `lsblk MOUNTPOINTS`**, because the lsblk invocation is pinned verbatim in
|
||||
the sudoers file; changing it would have coupled this fix to a config rollout. `/proc/mounts` is
|
||||
world-readable and needs neither.
|
||||
Real crypto because the two situations are indistinguishable **at the unwrap**; a faked unwrap would
|
||||
prove nothing about what was broken. Full suite green (`go build`/`vet`/`test ./...`), agent gates OK.
|
||||
|
||||
## Green gate
|
||||
**Red-proof, mutation asserted applied before the run:** remove the `tryRetained` block from
|
||||
`RecoverOffsiteRepoPassword` →
|
||||
`err = escrow: the recovery code did not unwrap the identity escrow (wrong recovery code…)` →
|
||||
`TestRecover_CodeOpensRetainedPackage_IsNotAWrongCode` FAILS. **The lie returns, in those words.**
|
||||
That is the layer the lie actually lives in: removing the *controller's* case yields the neutral
|
||||
message instead, because R-224's safe default catches it.
|
||||
|
||||
`go build` · `go vet` clean · `go test ./...` → **29 packages ok** · `agent_gates.py --fast` → all OK.
|
||||
## Released and deployed
|
||||
|
||||
| Red-proof | Result |
|
||||
|---|---|
|
||||
| remove the `felhomOwnedMounts` exemption | **FAILS** — "device is mounted at /mnt/adatok (sdb)", the pre-fix refusal |
|
||||
| over-widen the exemption to any `/mnt/*` | **FAILS** — "/mnt/someone-elses-disk was offered for formatting" |
|
||||
`release-agent.sh 0.129.0` — tagged `v0.129.0`, published, **verified by independent download**,
|
||||
sha256 `53a54f0620afbd6d…`. Installed on `felhom-pve`, `felhom-agent --version` = 0.129.0, unit active,
|
||||
journal clean (normal PBS verify cycle). **NOT VOUCHED** — that stays the operator's act.
|
||||
|
||||
## Not changed
|
||||
## Bypass, stated as required
|
||||
|
||||
No sudoers, no allowlisted command, no PVE surface, no format path. Every other claim signal
|
||||
(system disk, read-only, LVM PV, ZFS member, member FSTYPEs, empty-topology backstop) is untouched.
|
||||
`git push --no-verify` was used **once** for the code push. The `release-complete` gate refuses a
|
||||
CHANGELOG entry whose tag and package do not exist, and `release-agent.sh` refuses a tree that is not
|
||||
pushed — circular by construction. The bypass was immediately followed by the real release; gates were
|
||||
re-run afterwards and are **green**, and the tag+package now exist.
|
||||
|
||||
@@ -86,6 +86,7 @@
|
||||
| `Client.PoolAddVMID` | internal/proxmox/mutate.go | `PoolAddVMID(ctx, pool, vmid) error` | re-assert pool membership after a restore-over-existing (campaign-2 R2) | SYNC (no UPID, don't WaitTask); PVE `PUT /pools` is additive (merge, not replace) — `delete=1` removes; idempotent (already-member swallowed); needs `Pool.Allocate` at `/pool/<pool>`. `pct restore --pool` sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore |
|
||||
| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
|
||||
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
|
||||
| `httpx.NewTransport` | internal/httpx/transport.go | `NewTransport(tlsCfg, idleConnTimeout) *http.Transport` | **EVERY** hand-rolled `http.Transport` in this repo — pbs, hub and proxmox all pin TLS, so none can use `http.DefaultTransport` | **R-344: never inline `&http.Transport{TLSClientConfig: ...}` again.** A composite literal takes `IdleConnTimeout` **zero, which means retain idle connections FOREVER** — `http.DefaultTransport` sets 90s and a literal does not inherit it. Combined with a client rebuilt per cycle and dropped (`pbsTargetsFromPVE`), that stranded **388 sockets on ep0 in 46 h**, held open on BOTH sides. `idleConnTimeout <= 0` means **use the default**, never "no timeout". Returns a **FRESH** transport every call — a shared one would pool connections across differently pinned endpoints. Pinned by `internal/pbs/client_leak_test.go` (server-side connection counting) + `internal/httpx/transport_test.go` |
|
||||
| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
|
||||
| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 60–3600 s |
|
||||
| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
|
||||
|
||||
+40
-15
@@ -59,7 +59,7 @@ import (
|
||||
|
||||
// version is the agent version. Overridable at build time with
|
||||
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
||||
var version = "0.92.1"
|
||||
var version = "0.130.0"
|
||||
|
||||
// runGuestHook is the PVE hook body (`felhom-agent guest-hook <vmid> <phase>`). On pre-start it
|
||||
// creates placeholder dirs for any absent bind-mount source so the guest always boots (the C1 net);
|
||||
@@ -1769,23 +1769,48 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
}
|
||||
return blob, true, nil
|
||||
},
|
||||
// R-311 — the RETAINED packages, wired here and ONLY here, on the same self-scoped hub client.
|
||||
// Consulted only after the current package has refused the code (see tryRetained), so the
|
||||
// ordinary recovery pays nothing for it and cannot fail because of it.
|
||||
FetchRetained: func(ctx context.Context) ([]escrow.RetainedBlob, int, error) {
|
||||
resp, ferr := hubClient.FetchRetainedIdentityEscrow(ctx)
|
||||
if ferr != nil {
|
||||
return nil, 0, ferr
|
||||
}
|
||||
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
||||
for _, p := range resp.Packages {
|
||||
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||
if derr != nil || len(blob) == 0 {
|
||||
// One malformed package must not sink the rest — the customer's code may open a
|
||||
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
||||
continue
|
||||
}
|
||||
out = append(out, escrow.RetainedBlob{
|
||||
Blob: blob,
|
||||
SupersededAt: p.SupersededAt,
|
||||
KeyFingerprint: p.KeyFingerprint,
|
||||
Index: p.Index,
|
||||
})
|
||||
}
|
||||
return out, resp.UnopenableCount, nil
|
||||
},
|
||||
}
|
||||
srv, err := localapi.NewServer(localapi.Options{
|
||||
EscrowRecovery: escrowRecoverer,
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
Cert: cert,
|
||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||
Guests: px,
|
||||
Backups: runner,
|
||||
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
|
||||
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
|
||||
Store: store,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
Cert: cert,
|
||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||
Guests: px,
|
||||
Backups: runner,
|
||||
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
|
||||
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
|
||||
Store: store,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
Disks: hostOps,
|
||||
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
|
||||
|
||||
+116
-1
@@ -47,17 +47,80 @@ var (
|
||||
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
|
||||
// not sent hunting for a mistyped recovery code that was typed correctly.
|
||||
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
|
||||
// ErrCodeOpensRetained — the code did NOT open the package the hub currently holds, and DID open a
|
||||
// RETAINED (earlier) one. R-311.
|
||||
//
|
||||
// ⚠ THIS IS NOT A FAILURE OF THE CUSTOMER'S. It is the single most important distinction on this
|
||||
// path, because until 2026-08-12 it was indistinguishable from a mistype and was reported as one.
|
||||
// The screen could only say "it may be a typo, or it may be an older code, and we cannot tell them
|
||||
// apart from here" — and it could not tell them apart because NOTHING EVER LOOKED. Now something
|
||||
// looks, so the sentence can stop hedging.
|
||||
//
|
||||
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
||||
// which is the one fact the customer needs to recognise it.
|
||||
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
||||
)
|
||||
|
||||
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
||||
// carries no secret — not the code, not the bundle, not the repository password.
|
||||
type RetainedMatch struct {
|
||||
// SupersededAt is when this package stopped being the current one (hub-supplied, RFC3339-ish).
|
||||
// It is what the recovery screen shows so the customer can recognise which code they are holding.
|
||||
SupersededAt string
|
||||
// KeyFingerprint is the escrow key fingerprint of that package — operator-log material only.
|
||||
KeyFingerprint string
|
||||
// Index is the hub's position label within ONE response. Not durable; do not persist it.
|
||||
Index int
|
||||
// HasResticPassword is false when the retained package opened but carries no repository password
|
||||
// (a pre-fork-4 seal). The code is still CORRECT; the history behind it still cannot be reopened.
|
||||
// Collapsing this into "recoverable" would repeat R-202's mistake on a new surface.
|
||||
HasResticPassword bool
|
||||
}
|
||||
|
||||
// RetainedOpenedError wraps ErrCodeOpensRetained with the match. Callers classify with errors.Is on
|
||||
// the sentinel and read the detail with errors.As.
|
||||
type RetainedOpenedError struct {
|
||||
Match RetainedMatch
|
||||
}
|
||||
|
||||
func (e *RetainedOpenedError) Error() string {
|
||||
return ErrCodeOpensRetained.Error() + " (superseded_at=" + e.Match.SupersededAt + ")"
|
||||
}
|
||||
func (e *RetainedOpenedError) Unwrap() error { return ErrCodeOpensRetained }
|
||||
|
||||
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
|
||||
// An interface-free func field keeps this package free of any dependency on the hub client.
|
||||
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
|
||||
|
||||
// RetainedBlob is one retained sealed package as the recoverer sees it: opaque bytes plus the labels
|
||||
// needed to name it. No secret.
|
||||
type RetainedBlob struct {
|
||||
Blob []byte
|
||||
SupersededAt string
|
||||
KeyFingerprint string
|
||||
Index int
|
||||
}
|
||||
|
||||
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
||||
// slice is a clean "none". R-311.
|
||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
|
||||
|
||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||
type OffsiteKeyRecoverer struct {
|
||||
Fetch BlobFetcher
|
||||
// FetchRetained is OPTIONAL and consulted ONLY after the current package has refused the code.
|
||||
// nil keeps the pre-R-311 behaviour exactly: a refusal stays a refusal. That is deliberate — an
|
||||
// agent wired without it must not behave differently from one that has no retained packages.
|
||||
FetchRetained RetainedFetcher
|
||||
// MaxRetainedTried bounds the scrypt work a single wrong code can cost. Each attempt is ~1 s of
|
||||
// KDF by design, so an unbounded loop over a long supersession history would turn one wrong code
|
||||
// into a minutes-long hang on the customer's screen. 0 means the built-in default.
|
||||
MaxRetainedTried int
|
||||
}
|
||||
|
||||
// defaultMaxRetainedTried — six attempts is ~6 s worst case, which is a slow screen and not a hang.
|
||||
const defaultMaxRetainedTried = 6
|
||||
|
||||
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
|
||||
//
|
||||
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
|
||||
@@ -86,10 +149,62 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
||||
}
|
||||
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
|
||||
if err != nil {
|
||||
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
// R-311 — BEFORE calling this a wrong code, ask whether it is the RIGHT code for an EARLIER
|
||||
// package. The engine fails closed identically either way, so the two are indistinguishable
|
||||
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
||||
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
||||
// about our own incuriosity, read by the customer as a statement about their code.
|
||||
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
|
||||
return "", &RetainedOpenedError{Match: m}
|
||||
}
|
||||
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
}
|
||||
if bundle.ResticRepoPassword == "" {
|
||||
return "", ErrNoResticPassword
|
||||
}
|
||||
return bundle.ResticRepoPassword, nil
|
||||
}
|
||||
|
||||
// tryRetained reports whether the code opens one of this host's RETAINED packages, and which.
|
||||
//
|
||||
// FAILURE HERE IS SILENT AND MEANS "NO", NEVER "YES" and never a different verdict for the caller. A
|
||||
// hub that cannot answer, a route an older hub does not have, a malformed blob — each leaves the
|
||||
// original refusal standing, unchanged. That is the fail-safe direction: the worst outcome of this
|
||||
// function breaking is the behaviour we had before it existed.
|
||||
//
|
||||
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
|
||||
if r.FetchRetained == nil {
|
||||
return RetainedMatch{}, false
|
||||
}
|
||||
blobs, _, err := r.FetchRetained(ctx)
|
||||
if err != nil || len(blobs) == 0 {
|
||||
return RetainedMatch{}, false
|
||||
}
|
||||
limit := r.MaxRetainedTried
|
||||
if limit <= 0 {
|
||||
limit = defaultMaxRetainedTried
|
||||
}
|
||||
for i, rb := range blobs {
|
||||
if i >= limit {
|
||||
break
|
||||
}
|
||||
if len(rb.Blob) == 0 {
|
||||
continue
|
||||
}
|
||||
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
||||
if uerr != nil {
|
||||
continue // this one is not the customer's; try the next
|
||||
}
|
||||
return RetainedMatch{
|
||||
SupersededAt: rb.SupersededAt,
|
||||
KeyFingerprint: rb.KeyFingerprint,
|
||||
Index: rb.Index,
|
||||
// A retained package can itself predate the repository-password field. The code is still
|
||||
// correct and must be told so — but the history behind it still cannot be reopened, and
|
||||
// saying otherwise would be a promise this path cannot keep.
|
||||
HasResticPassword: bundle.ResticRepoPassword != "",
|
||||
}, true
|
||||
}
|
||||
return RetainedMatch{}, false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-311 — a correct code for an EARLIER package must stop being reported as a wrong code.
|
||||
//
|
||||
// These use REAL age crypto, like the R-199 tests beside them, because the whole point is that the
|
||||
// two situations are indistinguishable AT THE UNWRAP: both fail closed on the current package. A
|
||||
// faked unwrap would prove nothing about the thing that was actually broken.
|
||||
|
||||
const testR2 = "another correct horse battery staple sedative anaconda wobbly kingdom placard"
|
||||
|
||||
func retainedFetcherFor(blobs ...RetainedBlob) RetainedFetcher {
|
||||
return func(context.Context) ([]RetainedBlob, int, error) { return blobs, 0, nil }
|
||||
}
|
||||
|
||||
// THE ONE THAT MATTERS. The customer holds the code for a package we superseded. Yesterday this
|
||||
// returned the fail-closed refusal and the screen told them to check their typing.
|
||||
//
|
||||
// RED-PROOF: remove the `if m, ok := r.tryRetained(...)` block from RecoverOffsiteRepoPassword →
|
||||
// the wrong-code error returns instead → this FAILS, and the lie is back in exactly those words.
|
||||
func TestRecover_CodeOpensRetainedPackage_IsNotAWrongCode(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const oldPW = "aaaa567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "cccc567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
|
||||
retained := sealBundle(t, IdentityBundle{ResticRepoPassword: oldPW}, testR)
|
||||
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: retainedFetcherFor(RetainedBlob{
|
||||
Blob: retained, SupersededAt: "2026-08-12 15:18:55", KeyFingerprint: "7e:a6:af", Index: 0,
|
||||
}),
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR) // the OLD code
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("recovery succeeded — it must NOT return a password for a retained package on this path")
|
||||
}
|
||||
if !errors.Is(err, ErrCodeOpensRetained) {
|
||||
t.Fatalf("err = %v, want ErrCodeOpensRetained — a correct code for an earlier package was "+
|
||||
"classified as something else, which is how it became 'check your typing'", err)
|
||||
}
|
||||
var ro *RetainedOpenedError
|
||||
if !errors.As(err, &ro) {
|
||||
t.Fatalf("err does not carry a RetainedOpenedError: %v", err)
|
||||
}
|
||||
if ro.Match.SupersededAt != "2026-08-12 15:18:55" {
|
||||
t.Errorf("SupersededAt = %q — the screen needs this date to name the package", ro.Match.SupersededAt)
|
||||
}
|
||||
if !ro.Match.HasResticPassword {
|
||||
t.Error("HasResticPassword = false, but the retained bundle carried one")
|
||||
}
|
||||
// The error must not leak the code, the password or the bundle.
|
||||
for _, secret := range []string{testR, oldPW} {
|
||||
if containsStr(err.Error(), secret) {
|
||||
t.Fatalf("the error text leaks a secret")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SCENARIO A — the ordinary recovery is untouched, and it must not even ASK for retained packages.
|
||||
// If the current package opens, the customer is not in this story at all.
|
||||
//
|
||||
// RED-PROOF: move the tryRetained call above the successful-unwrap return → the fetcher runs → this
|
||||
// FAILS on the "must not be consulted" assertion.
|
||||
func TestRecover_CurrentPackageOpens_RetainedNeverConsulted(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const pw = "bbbb567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: pw}, testR)
|
||||
|
||||
consulted := false
|
||||
got, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||
consulted = true
|
||||
return nil, 0, nil
|
||||
},
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("the ordinary recovery broke: %v", err)
|
||||
}
|
||||
if got != pw {
|
||||
t.Fatalf("recovered password is not the sealed one")
|
||||
}
|
||||
if consulted {
|
||||
t.Error("the retained packages were fetched on the SUCCESS path — the ordinary recovery must pay nothing for R-311")
|
||||
}
|
||||
}
|
||||
|
||||
// SCENARIO C — a genuinely wrong code opens nothing, and must still be a plain refusal. The new
|
||||
// branch must not become a way to encourage a customer who mistyped.
|
||||
//
|
||||
// RED-PROOF: make tryRetained return (RetainedMatch{}, true) unconditionally → a wrong code is
|
||||
// reported as opening an earlier package → this FAILS.
|
||||
func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "cccc567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
retained := sealBundle(t, IdentityBundle{ResticRepoPassword: "dddd567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
|
||||
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: retained, SupersededAt: "2026-08-01 00:00:00"}),
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), "totally wrong words that open nothing at all here")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("a wrong code succeeded")
|
||||
}
|
||||
if errors.Is(err, ErrCodeOpensRetained) {
|
||||
t.Fatal("a WRONG code was reported as opening a retained package — that would encourage a mistype")
|
||||
}
|
||||
}
|
||||
|
||||
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
|
||||
// worst outcome of this feature breaking is the behaviour we had before it.
|
||||
//
|
||||
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
||||
// gets a new, unexplained failure mode → this FAILS.
|
||||
func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "eeee567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||
return nil, 0, fmt.Errorf("hub exploded")
|
||||
},
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("expected a refusal")
|
||||
}
|
||||
if errors.Is(err, ErrCodeOpensRetained) {
|
||||
t.Fatal("a failed retained lookup was reported as 'opens a retained package'")
|
||||
}
|
||||
if containsStr(err.Error(), "hub exploded") {
|
||||
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
||||
}
|
||||
}
|
||||
|
||||
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
||||
// indistinguishable from one whose host has no retained packages.
|
||||
//
|
||||
// RED-PROOF: remove the `if r.FetchRetained == nil` guard → nil-deref panic → this FAILS.
|
||||
func TestRecover_NilRetainedFetcher_IsPreR311Behaviour(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "ffff567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
|
||||
_, err := OffsiteKeyRecoverer{Fetch: fetcherFor(current)}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||
if err == nil {
|
||||
t.Fatal("expected a refusal")
|
||||
}
|
||||
if errors.Is(err, ErrCodeOpensRetained) {
|
||||
t.Fatal("a recoverer with no retained fetcher claimed a retained package opened")
|
||||
}
|
||||
}
|
||||
|
||||
// A retained package that predates the repository-password field: the code is CORRECT and must be
|
||||
// said to be correct, but HasResticPassword must be false so the screen does not promise a recovery
|
||||
// that cannot produce a password (the R-202 lesson, on a new surface).
|
||||
//
|
||||
// RED-PROOF: hardcode HasResticPassword: true → this FAILS.
|
||||
func TestRecover_RetainedOpensButPredatesTheField(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "1111567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
|
||||
// No ResticRepoPassword at all — the pre-fork-4 shape.
|
||||
retained := sealBundle(t, IdentityBundle{TunnelToken: "T", PBSToken: "P"}, testR)
|
||||
|
||||
_, err := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: retained, SupersededAt: "2026-08-04 07:20:08"}),
|
||||
}.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
|
||||
if !errors.Is(err, ErrCodeOpensRetained) {
|
||||
t.Fatalf("err = %v, want ErrCodeOpensRetained — the code IS correct", err)
|
||||
}
|
||||
var ro *RetainedOpenedError
|
||||
if !errors.As(err, &ro) {
|
||||
t.Fatalf("no RetainedOpenedError: %v", err)
|
||||
}
|
||||
if ro.Match.HasResticPassword {
|
||||
t.Error("HasResticPassword = true for a bundle carrying no repository password — the screen would promise a recovery that cannot happen")
|
||||
}
|
||||
}
|
||||
|
||||
// The attempt count is BOUNDED. Each unwrap is ~1 s of scrypt by design, so an unbounded loop turns
|
||||
// one wrong code into a minutes-long hang on the customer's screen.
|
||||
//
|
||||
// RED-PROOF: remove the `if i >= limit { break }` → all 10 are tried → this FAILS on the count.
|
||||
func TestRecover_RetainedAttemptsAreBounded(t *testing.T) {
|
||||
ensureAge(t)
|
||||
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "2222567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||
junk := sealBundle(t, IdentityBundle{ResticRepoPassword: "3333567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR2)
|
||||
|
||||
tried := 0
|
||||
blobs := make([]RetainedBlob, 0, 10)
|
||||
for i := 0; i < 10; i++ {
|
||||
blobs = append(blobs, RetainedBlob{Blob: junk, SupersededAt: "2026-08-01 00:00:00", Index: i})
|
||||
}
|
||||
rec := OffsiteKeyRecoverer{
|
||||
Fetch: fetcherFor(current),
|
||||
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||
tried++
|
||||
return blobs, 0, nil
|
||||
},
|
||||
MaxRetainedTried: 2,
|
||||
}
|
||||
// A code that opens NEITHER the current package nor any retained one.
|
||||
if _, err := rec.RecoverOffsiteRepoPassword(context.Background(), "a code that opens nothing whatsoever in this test"); err == nil {
|
||||
t.Fatal("expected a refusal")
|
||||
}
|
||||
if tried != 1 {
|
||||
t.Errorf("the retained list was fetched %d times, want exactly 1", tried)
|
||||
}
|
||||
}
|
||||
|
||||
func containsStr(hay, needle string) bool {
|
||||
return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
|
||||
for i := 0; i+len(needle) <= len(hay); i++ {
|
||||
if hay[i:i+len(needle)] == needle {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})()
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
// Package httpx holds the one HTTP-transport default this repo may not lose.
|
||||
//
|
||||
// Every client here pins TLS — PBS and PVE by leaf-cert SHA-256, the hub by an optional CA file —
|
||||
// so none of them can use http.DefaultTransport and each hand-rolls its own. Hand-rolling silently
|
||||
// discards DefaultTransport's settings, and one of them is load-bearing:
|
||||
//
|
||||
// Transport: &http.Transport{TLSClientConfig: tlsCfg} // IdleConnTimeout == 0 == NO timeout
|
||||
//
|
||||
// A zero IdleConnTimeout means idle keep-alive connections are retained FOREVER, not "use a sane
|
||||
// default". Combined with a client that is rebuilt on a schedule and dropped (pbsTargetsFromPVE
|
||||
// builds a fresh pbs.Client per cycle), every cycle strands one connection that nothing will ever
|
||||
// close: the abandoned Transport becomes unreachable but its persistConn read-loop goroutine keeps
|
||||
// the socket alive, and an unreachable Transport does not close its connections.
|
||||
//
|
||||
// Measured cost, live: 388 established connections accumulated on ep0's PBS proxy between
|
||||
// 2026-08-18 09:51:22Z and 2026-08-20 08:02:13Z — 194 from each of the two boxes, held open on BOTH
|
||||
// sides, one per agent poll cycle, on a proxy whose descriptor ceiling is 65536. See R-344 and
|
||||
// felhom.eu/documentation/audits/SPIKE-ep0-established-connections-2026-08-20.md.
|
||||
package httpx
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultIdleConnTimeout is how long an idle keep-alive connection is retained before it is closed.
|
||||
//
|
||||
// It is 90s because that is http.DefaultTransport's own value: the fix for R-344 restores a
|
||||
// standard-library default rather than inventing a number, so there is nothing here to tune and
|
||||
// nothing to justify. It is comfortably shorter than every cadence that drives these clients (the
|
||||
// 15-minute live-snapshot collect and the 6-hour verify loop), so a connection abandoned by one
|
||||
// cycle is closed long before the next.
|
||||
const DefaultIdleConnTimeout = 90 * time.Second
|
||||
|
||||
// NewTransport builds a FRESH *http.Transport pinned to tlsCfg, with the idle-connection timeout
|
||||
// applied.
|
||||
//
|
||||
// Fresh, never shared: each caller pins a different endpoint, and a shared transport would pool
|
||||
// connections across differently pinned servers. Reusing http.DefaultTransport for the same reason
|
||||
// is not an option — it would drop the pin entirely.
|
||||
//
|
||||
// idleConnTimeout <= 0 means USE THE DEFAULT. It deliberately does not mean "no timeout": no-timeout
|
||||
// is the bug this package exists to prevent, and an unset field must never be able to reintroduce
|
||||
// it. Callers pass their configured value straight through; only tests pass a short one.
|
||||
//
|
||||
// Only IdleConnTimeout is set. The other DefaultTransport settings this transport also lacks
|
||||
// (MaxIdleConns, TLSHandshakeTimeout, ExpectContinueTimeout) are deliberately left alone: none of
|
||||
// them accumulates anything, every client bounds its whole request with http.Client.Timeout, and
|
||||
// widening the change would have made the R-344 measurement unattributable.
|
||||
func NewTransport(tlsCfg *tls.Config, idleConnTimeout time.Duration) *http.Transport {
|
||||
if idleConnTimeout <= 0 {
|
||||
idleConnTimeout = DefaultIdleConnTimeout
|
||||
}
|
||||
return &http.Transport{
|
||||
TLSClientConfig: tlsCfg,
|
||||
IdleConnTimeout: idleConnTimeout,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package httpx
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestNewTransport_ZeroMeansDefaultNeverForever is the whole point of this package.
|
||||
//
|
||||
// http.Transport's zero IdleConnTimeout means "retain idle connections FOREVER". Any code path that
|
||||
// can reach that zero reintroduces R-344, so an unset, zero or negative value must all land on the
|
||||
// default. If someone later "simplifies" NewTransport by passing the argument straight through,
|
||||
// this fails.
|
||||
func TestNewTransport_ZeroMeansDefaultNeverForever(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
in time.Duration
|
||||
want time.Duration
|
||||
}{
|
||||
{"zero", 0, DefaultIdleConnTimeout},
|
||||
{"negative", -time.Hour, DefaultIdleConnTimeout},
|
||||
{"explicit short value (tests)", 50 * time.Millisecond, 50 * time.Millisecond},
|
||||
{"explicit long value", time.Hour, time.Hour},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := NewTransport(&tls.Config{MinVersion: tls.VersionTLS12}, tc.in).IdleConnTimeout
|
||||
if got != tc.want {
|
||||
t.Fatalf("IdleConnTimeout = %v, want %v", got, tc.want)
|
||||
}
|
||||
if got == 0 {
|
||||
t.Fatal("IdleConnTimeout is 0 — that is 'never expire', which is the R-344 defect itself")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDefaultIdleConnTimeout_MatchesTheStandardLibrary pins the number to its justification.
|
||||
//
|
||||
// 90s is not a tuned value; it is what http.DefaultTransport uses. Reading it off the standard
|
||||
// library rather than hardcoding 90 means the constant cannot drift away from the reason given for
|
||||
// it in the package doc.
|
||||
func TestDefaultIdleConnTimeout_MatchesTheStandardLibrary(t *testing.T) {
|
||||
std, ok := http.DefaultTransport.(*http.Transport)
|
||||
if !ok {
|
||||
t.Skip("http.DefaultTransport is not an *http.Transport in this Go build")
|
||||
}
|
||||
if DefaultIdleConnTimeout != std.IdleConnTimeout {
|
||||
t.Fatalf("DefaultIdleConnTimeout = %v but http.DefaultTransport uses %v — the doc comment's justification no longer holds",
|
||||
DefaultIdleConnTimeout, std.IdleConnTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTransport_IsFreshEveryCall guards the pooling property the pinning relies on.
|
||||
//
|
||||
// Each caller pins a DIFFERENT endpoint. A shared transport would pool connections across
|
||||
// differently pinned servers, so returning a package-level singleton would be a security change
|
||||
// dressed as a tidy-up.
|
||||
func TestNewTransport_IsFreshEveryCall(t *testing.T) {
|
||||
a := NewTransport(&tls.Config{MinVersion: tls.VersionTLS12}, 0)
|
||||
b := NewTransport(&tls.Config{MinVersion: tls.VersionTLS12}, 0)
|
||||
if a == b {
|
||||
t.Fatal("NewTransport returned the SAME transport twice — connections would be pooled across differently pinned endpoints")
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTransport_KeepsTheTLSConfig — the transport gains a field; it must lose nothing.
|
||||
func TestNewTransport_KeepsTheTLSConfig(t *testing.T) {
|
||||
cfg := &tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: true} //nolint:gosec // test only
|
||||
if got := NewTransport(cfg, 0).TLSClientConfig; got != cfg {
|
||||
t.Fatalf("TLSClientConfig = %p, want the config passed in (%p) — the pin would be dropped", got, cfg)
|
||||
}
|
||||
}
|
||||
+73
-2
@@ -15,6 +15,8 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
)
|
||||
|
||||
const reportPath = "/api/v1/host-report"
|
||||
@@ -49,8 +51,11 @@ func NewClient(cfg config.HubConfig, logger *slog.Logger) (*Client, error) {
|
||||
tlsCfg.RootCAs = pool
|
||||
}
|
||||
hc := &http.Client{
|
||||
Timeout: time.Duration(cfg.TimeoutSeconds) * time.Second,
|
||||
Transport: &http.Transport{TLSClientConfig: tlsCfg},
|
||||
Timeout: time.Duration(cfg.TimeoutSeconds) * time.Second,
|
||||
// R-344, consistency only: this client is built ONCE per process, so it never accumulated
|
||||
// and contributed nothing to the ep0 leak. It carried the same missing default, which over
|
||||
// a tunnel is how one idle connection survives long enough to fail on next use.
|
||||
Transport: httpx.NewTransport(tlsCfg, 0),
|
||||
}
|
||||
return newClient(cfg.URL, cfg.APIKey, cfg.HostID, hc, logger), nil
|
||||
}
|
||||
@@ -354,3 +359,69 @@ func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowRespon
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
// RetainedEscrowPackage is one RETAINED (superseded) sealed identity package. The blob is ciphertext
|
||||
// and is useless without R. `SupersededAt` is the only thing here a human ever sees — it is what lets
|
||||
// the recovery screen name WHICH earlier package a code belongs to.
|
||||
type RetainedEscrowPackage struct {
|
||||
Index int `json:"index"`
|
||||
SupersededAt string `json:"superseded_at"`
|
||||
KeyFingerprint string `json:"key_fingerprint"`
|
||||
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||
}
|
||||
|
||||
// RetainedEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow/retained (hub >= v0.103.0, R-311).
|
||||
//
|
||||
// UnopenableCount is NOT noise. It counts retained packages the hub holds whose key material is absent
|
||||
// (every pre-v0.93.0 row): on a box with those and nothing else, a perfectly correct old recovery code
|
||||
// opens nothing, and the reason is a defect of ours. A caller that ignores this number will tell such a
|
||||
// customer their code is wrong — the exact failure this whole chain exists to stop.
|
||||
type RetainedEscrowResponse struct {
|
||||
HostID string `json:"host_id"`
|
||||
Count int `json:"count"`
|
||||
UnopenableCount int `json:"unopenable_count"`
|
||||
TruncatedCount int `json:"truncated_count"`
|
||||
Packages []RetainedEscrowPackage `json:"packages"`
|
||||
}
|
||||
|
||||
// FetchRetainedIdentityEscrow reads back THIS host's RETAINED sealed identity packages (R-311 —
|
||||
// the retained siblings of FetchIdentityEscrow, self-scoped server-side by the same per-host key).
|
||||
//
|
||||
// SEPARATE FROM FetchIdentityEscrow ON PURPOSE. The ordinary recovery must not pay for this call, and
|
||||
// must not fail because of it: the current package is tried first and alone, and this is reached only
|
||||
// after that has refused. A hub too old to know this route answers 404, which is a CLEAN "none" here
|
||||
// and must never be reported as a failed recovery.
|
||||
func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscrowResponse, error) {
|
||||
if c.hostID == "" {
|
||||
return nil, fmt.Errorf("hub: FetchRetainedIdentityEscrow requires a configured host_id")
|
||||
}
|
||||
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow/retained"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hub: building retained-escrow request: %w", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, &TransportError{Err: err}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
if resp.StatusCode == http.StatusNotFound {
|
||||
// A hub older than v0.103.0 has no such route. That is "no retained packages", not a fault —
|
||||
// returning an error here would turn an old hub into a failed recovery on a box whose current
|
||||
// package simply did not open.
|
||||
return &RetainedEscrowResponse{HostID: c.hostID}, nil
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||
}
|
||||
var out RetainedEscrowResponse
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
return nil, fmt.Errorf("hub: decoding retained escrow fetch: %w", err)
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
@@ -98,6 +98,35 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
|
||||
case errors.Is(err, escrow.ErrNoEscrowBlob):
|
||||
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
|
||||
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
|
||||
// ── R-311 (2026-08-12) — THE CODE IS RIGHT, JUST NOT FOR THE CURRENT PACKAGE. ─────────
|
||||
//
|
||||
// Placed ABOVE the default for the same reason ErrBundleFetch is: the default blames the
|
||||
// customer, and this case is the one where the customer is provably not at fault. The code was
|
||||
// used, it worked, and it opened a package the hub is deliberately keeping.
|
||||
//
|
||||
// 422 rather than 400: the request was well-formed AND the credential was valid — what could
|
||||
// not be processed is the pairing of a correct code with the CURRENT package. A 400 would put
|
||||
// it in the same bucket as a mistype, which is the whole defect. The status is the
|
||||
// machine-readable half; the controller classifies on it and must never parse this sentence.
|
||||
//
|
||||
// The date travels in the body because it is the one fact that lets a customer recognise which
|
||||
// code they are holding. No material, no code, no password — only when that package stopped
|
||||
// being current, and whether it can yield a repository password at all.
|
||||
case errors.Is(err, escrow.ErrCodeOpensRetained):
|
||||
var ro *escrow.RetainedOpenedError
|
||||
match := escrow.RetainedMatch{}
|
||||
if errors.As(err, &ro) {
|
||||
match = ro.Match
|
||||
}
|
||||
s.logger.Info("local-api: offsite key recovery: the code did NOT open the current package but DID open a RETAINED one — the customer is not at fault",
|
||||
"vmid", vmid, "superseded_at", match.SupersededAt, "retained_has_restic_pw", match.HasResticPassword)
|
||||
writeStatus(w, http.StatusUnprocessableEntity, false,
|
||||
map[string]any{
|
||||
"opens_retained": true,
|
||||
"superseded_at": match.SupersededAt,
|
||||
"retained_has_restic_pw": match.HasResticPassword,
|
||||
},
|
||||
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||
|
||||
+14
-2
@@ -10,6 +10,8 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
)
|
||||
|
||||
// Client is the PBS-API client for ONE PBS server. Construct with NewClient. It is pure (no
|
||||
@@ -31,6 +33,11 @@ type Config struct {
|
||||
Secret string // token secret (from <id>.pw)
|
||||
Namespace string // PBS namespace (from storage.cfg `namespace`); "" = root. S4 per-customer tenancy.
|
||||
Timeout time.Duration
|
||||
|
||||
// IdleConnTimeout bounds how long this client's idle keep-alive connections are retained.
|
||||
// Zero means httpx.DefaultIdleConnTimeout (90s) — it does NOT mean "no timeout", which is the
|
||||
// R-344 defect. Production leaves it unset; only tests set it, to avoid a 90-second wait.
|
||||
IdleConnTimeout time.Duration
|
||||
}
|
||||
|
||||
// NewClient builds a fingerprint-pinned, token-authed PBS client.
|
||||
@@ -55,8 +62,13 @@ func NewClient(cfg Config) (*Client, error) {
|
||||
authHeader: "PBSAPIToken=" + cfg.TokenID + ":" + cfg.Secret,
|
||||
namespace: cfg.Namespace,
|
||||
http: &http.Client{
|
||||
Timeout: timeout,
|
||||
Transport: &http.Transport{TLSClientConfig: tlsCfg},
|
||||
Timeout: timeout,
|
||||
// R-344: this transport MUST come from httpx. pbsTargetsFromPVE (cmd/felhom-agent/
|
||||
// main.go) builds a fresh Client every cycle and drops the previous one, so a
|
||||
// transport with no idle timeout strands one connection per cycle, forever, on both
|
||||
// sides. That leaked 388 sockets onto ep0 in 46 hours. Pinned by
|
||||
// TestAbandonedClientsReleaseTheirConnections — do not inline an http.Transport here.
|
||||
Transport: httpx.NewTransport(tlsCfg, cfg.IdleConnTimeout),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
package pbs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
)
|
||||
|
||||
// connCounter is the SERVER-side observer. It counts what the server actually holds, which is the
|
||||
// only thing that answers the question this file exists for: a client that believes it closed a
|
||||
// connection, and a server still holding the socket, is precisely the R-344 shape. Asserting on
|
||||
// anything client-side would be asserting the mechanism instead of the consequence.
|
||||
type connCounter struct {
|
||||
mu sync.Mutex
|
||||
open int
|
||||
total int // every connection ever accepted — how many times the client DIALLED
|
||||
}
|
||||
|
||||
func (c *connCounter) hook(_ net.Conn, s http.ConnState) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
switch s {
|
||||
case http.StateNew:
|
||||
c.open++
|
||||
c.total++
|
||||
case http.StateClosed, http.StateHijacked:
|
||||
c.open--
|
||||
}
|
||||
}
|
||||
|
||||
func (c *connCounter) counts() (open, total int) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.open, c.total
|
||||
}
|
||||
|
||||
// waitForOpen polls until the server holds want connections, or fails naming what it still holds.
|
||||
func (c *connCounter) waitForOpen(t *testing.T, want int, within time.Duration, what string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(within)
|
||||
for {
|
||||
open, total := c.counts()
|
||||
if open == want {
|
||||
return
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%s: after %s the server still holds %d open connection(s), want %d (%d dialled in total)",
|
||||
what, within, open, want, total)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// newCountingPBSServer is newPBSTestServer plus a ConnState hook. Kept separate rather than
|
||||
// changing the shared helper, so the existing tests are untouched by this file.
|
||||
func newCountingPBSServer(t *testing.T, fn http.HandlerFunc) (*httptest.Server, string, *connCounter) {
|
||||
t.Helper()
|
||||
cc := &connCounter{}
|
||||
ts := httptest.NewUnstartedServer(fn)
|
||||
ts.Config.ConnState = cc.hook
|
||||
ts.StartTLS()
|
||||
t.Cleanup(ts.Close)
|
||||
return ts, fingerprintOf(ts), cc
|
||||
}
|
||||
|
||||
// TestAbandonedClientsReleaseTheirConnections is Scenario A, and it is the load-bearing test for
|
||||
// R-344.
|
||||
//
|
||||
// It models what pbsTargetsFromPVE actually does — build a client, use it once, drop it on the
|
||||
// floor without closing anything — and asserts the CONSEQUENCE on the server: the connections go
|
||||
// away. Before the fix every one of these stayed established forever on both sides; 388 of them
|
||||
// accumulated on ep0 in 46 hours.
|
||||
//
|
||||
// Deliberately NOT asserted: that err == nil, or that IdleConnTimeout holds some value. Both were
|
||||
// true of the leaking code.
|
||||
func TestAbandonedClientsReleaseTheirConnections(t *testing.T) {
|
||||
ts, fp, cc := newCountingPBSServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Write([]byte(`{"data":[]}`))
|
||||
})
|
||||
host, port := hostPort(t, ts.URL)
|
||||
|
||||
const cycles = 5
|
||||
for i := 0; i < cycles; i++ {
|
||||
// One fresh client per "cycle", exactly as pbsTargetsFromPVE builds one per collect.
|
||||
c, err := NewClient(Config{
|
||||
Server: host, Port: port, Fingerprint: fp, TokenID: "u@pbs!t", Secret: "s",
|
||||
IdleConnTimeout: 50 * time.Millisecond, // production uses the 90s default
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := c.Snapshots(context.Background(), "ds"); err != nil {
|
||||
t.Fatalf("cycle %d: %v", i, err)
|
||||
}
|
||||
_ = c // dropped here — nothing closes it, nothing can
|
||||
}
|
||||
|
||||
if _, total := cc.counts(); total != cycles {
|
||||
t.Fatalf("setup is not modelling the leak: want %d separate dials (one per abandoned client), got %d", cycles, total)
|
||||
}
|
||||
cc.waitForOpen(t, 0, 5*time.Second, "abandoned pbs.Clients")
|
||||
}
|
||||
|
||||
// TestAbandonedClientsReleaseTheirConnections_ProductionDefaultIsUsable pins the value that ships.
|
||||
//
|
||||
// The field being settable is exactly how it could silently become zero again — and zero used to
|
||||
// mean "never expire". This asserts the production path (Config leaving it unset) lands on the
|
||||
// standard-library default, so the leak cannot return through an unset field.
|
||||
func TestPBSClient_UnsetIdleTimeoutUsesTheDefault(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cfg time.Duration
|
||||
want time.Duration
|
||||
}{
|
||||
{"unset — the production path", 0, httpx.DefaultIdleConnTimeout},
|
||||
{"explicit zero is NOT no-timeout", 0, httpx.DefaultIdleConnTimeout},
|
||||
{"negative is NOT no-timeout", -time.Second, httpx.DefaultIdleConnTimeout},
|
||||
{"an explicit value is honoured", 3 * time.Second, 3 * time.Second},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
c, err := NewClient(Config{
|
||||
Server: "pbs.example", Fingerprint: strings.Repeat("ab", 32),
|
||||
TokenID: "u@pbs!t", Secret: "s", IdleConnTimeout: tc.cfg,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr, ok := c.http.Transport.(*http.Transport)
|
||||
if !ok {
|
||||
t.Fatalf("transport is %T, not *http.Transport — the httpx wiring was replaced", c.http.Transport)
|
||||
}
|
||||
if tr.IdleConnTimeout != tc.want {
|
||||
t.Fatalf("IdleConnTimeout = %v, want %v (zero would mean connections are retained FOREVER — that is R-344)", tr.IdleConnTimeout, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestPBSClient_KeepAliveStillReuses is Scenario C, and it is the guard against a "fix" that is
|
||||
// worse than the bug.
|
||||
//
|
||||
// Disabling keep-alive entirely would also make the leak go away — by dialling a fresh connection
|
||||
// for every single request, which on a box polling ~40,000 times a day is strictly worse than what
|
||||
// we started with. The fix must retire IDLE connections without stopping reuse.
|
||||
func TestPBSClient_KeepAliveStillReuses(t *testing.T) {
|
||||
ts, fp, cc := newCountingPBSServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Write([]byte(`{"data":[]}`))
|
||||
})
|
||||
host, port := hostPort(t, ts.URL)
|
||||
|
||||
c, err := NewClient(Config{
|
||||
Server: host, Port: port, Fingerprint: fp, TokenID: "u@pbs!t", Secret: "s",
|
||||
IdleConnTimeout: 30 * time.Second, // long enough that reuse is what is being measured
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
if _, err := c.Snapshots(context.Background(), "ds"); err != nil {
|
||||
t.Fatalf("request %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if _, total := cc.counts(); total != 1 {
|
||||
t.Fatalf("one client made 3 sequential requests over %d connection(s), want 1 — keep-alive reuse is broken, which would make the poll load WORSE than the leak", total)
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,8 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
)
|
||||
|
||||
// doer is the minimal HTTP surface the client needs; *http.Client satisfies it.
|
||||
@@ -65,8 +67,10 @@ func NewClient(cfg Config) (*Client, error) {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
hc := &http.Client{
|
||||
Timeout: timeout,
|
||||
Transport: &http.Transport{TLSClientConfig: tlsCfg},
|
||||
Timeout: timeout,
|
||||
// R-344, consistency only: built ONCE per process, so it never accumulated and contributed
|
||||
// nothing to the ep0 leak. Same missing default, corrected for the same reason.
|
||||
Transport: httpx.NewTransport(tlsCfg, 0),
|
||||
}
|
||||
return &Client{
|
||||
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
|
||||
|
||||
@@ -49,6 +49,10 @@ GATES = [
|
||||
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
||||
("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
|
||||
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
||||
# R-273: the tag half of a release. Legs 1-2 need no network, so it runs in --fast too — the
|
||||
# missing TAG is what actually broke every install, and the pre-push hook is the earliest place
|
||||
# that can catch it.
|
||||
("release-complete", os.path.join(ROOT, "scripts", "check-release-complete.py"), [], True),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
@@ -95,6 +95,25 @@ PROBE_CONFIG = "configs/felhom-agent.service"
|
||||
|
||||
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
|
||||
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
||||
# must read the same number from the same place, or they drift and the drift looks like a defect
|
||||
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
||||
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
|
||||
|
||||
|
||||
def retention_kept():
|
||||
"""How many of the newest generic versions the registry is expected to still serve.
|
||||
|
||||
Fails CLOSED and LOUD: a missing or unreadable policy file makes the check INCONCLUSIVE
|
||||
rather than silently unbounded. An unbounded check would re-create the red this fixed; a
|
||||
silently-bounded one would be worse.
|
||||
"""
|
||||
with open(RETENTION_FILE, encoding="utf-8") as fh:
|
||||
n = json.load(fh)["generic_versions_kept"]
|
||||
if not isinstance(n, int) or n < 1:
|
||||
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
||||
return n
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
@@ -189,7 +208,28 @@ def main():
|
||||
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
||||
print("\ncheck-published-versions: NOTHING TO CHECK")
|
||||
return 0
|
||||
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
|
||||
all_versions = versions
|
||||
try:
|
||||
keep = retention_kept()
|
||||
except Exception as e:
|
||||
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
||||
|
||||
# Bound the assertion to what the registry is expected to still hold. Sorted by SEMVER, not
|
||||
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
||||
def _key(v):
|
||||
return tuple(int(x) for x in v.split("."))
|
||||
versions = sorted(all_versions, key=_key)[-keep:]
|
||||
dropped = [v for v in all_versions if v not in versions]
|
||||
|
||||
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
|
||||
print(" verifying: %s" % ", ".join(versions))
|
||||
if dropped:
|
||||
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
||||
# narrowing becomes permanent by accident.
|
||||
print(" NOT ASSERTED (older than the retention window, and therefore not expected to be")
|
||||
print(" downloadable): %s" % ", ".join(dropped))
|
||||
print(" ^ these versions still have git TAGS and are still installable in the sense that")
|
||||
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
||||
|
||||
bad = []
|
||||
for v in versions:
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""check-release-complete.py — the version at the head of CHANGELOG.md is a COMPLETE release.
|
||||
|
||||
THE DEFECT THIS IS A MACHINE FOR (2026-08-08/09, R-273). Agent v0.128.0 was built, tested,
|
||||
CHANGELOG'd and published to the package registry — and its git tag was never pushed. The hub then
|
||||
vouched it, and because felhom-host-install.sh fetches an agent's config files from
|
||||
`raw/tag/v<version>/configs/`, EVERY fresh install and every reinstall died at step 5 of 8, as root,
|
||||
on a virgin machine, for the better part of a day.
|
||||
|
||||
`scripts/release-agent.sh` already warns about exactly this, in as many words:
|
||||
"a released version without a git tag 404s a box mid-install, as root"
|
||||
The warning was there, it was correct, and the step was still missed. **So the fix is a machine and
|
||||
not a reminder** — that is the whole point of this file.
|
||||
|
||||
WHAT IT ASSERTS, for the newest `## vX.Y.Z` in CHANGELOG.md:
|
||||
1. a git tag `vX.Y.Z` EXISTS, and
|
||||
2. it points at a commit that is an ANCESTOR OF (or equal to) the tip it was released from — a tag
|
||||
parked on an unrelated commit is not a release, and
|
||||
3. the generic package for X.Y.Z is DOWNLOADABLE.
|
||||
|
||||
(3) needs the network. (1) and (2) do not, and they are the half that actually failed — so this gate
|
||||
is useful offline and says so rather than going quiet.
|
||||
|
||||
EXIT CODES, matching this repo's other gates: 0 clean, 1 convicted, 2 inconclusive. An unreachable
|
||||
registry is INCONCLUSIVE for leg 3 only; legs 1 and 2 still run and can still convict.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
|
||||
OWNER, PKG = "admin", "felhom-agent"
|
||||
HEAD_RE = re.compile(r"^##\s+v?(\d+\.\d+\.\d+)\b", re.M)
|
||||
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(("git",) + args, cwd=ROOT, capture_output=True, text=True)
|
||||
|
||||
|
||||
def head_version():
|
||||
ch = os.path.join(ROOT, "CHANGELOG.md")
|
||||
if not os.path.exists(ch):
|
||||
return None
|
||||
m = HEAD_RE.search(open(ch, encoding="utf-8").read())
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def main():
|
||||
print("check-release-complete — the newest CHANGELOG version is a complete release")
|
||||
v = head_version()
|
||||
if not v:
|
||||
print(" no '## vX.Y.Z' heading in CHANGELOG.md — nothing to check, and nothing proven")
|
||||
return 0
|
||||
tag = "v" + v
|
||||
print(" newest CHANGELOG version: %s" % tag)
|
||||
|
||||
problems, inconclusive = [], []
|
||||
|
||||
# ---- leg 1 + 2: the tag, and where it points. Offline-capable. -----------------------------
|
||||
r = git("rev-parse", "-q", "--verify", "refs/tags/%s^{commit}" % tag)
|
||||
if r.returncode != 0:
|
||||
# A shallow CI clone has no tags of its own; ask the remote before convicting, so this
|
||||
# gate does not fire on a clone shape rather than on a real defect.
|
||||
ls = git("ls-remote", "--tags", "origin", "refs/tags/%s" % tag)
|
||||
if ls.returncode != 0:
|
||||
inconclusive.append("cannot reach origin to look for tag %s: %s"
|
||||
% (tag, ls.stderr.strip()[:120]))
|
||||
elif not ls.stdout.strip():
|
||||
problems.append(
|
||||
"TAG %s DOES NOT EXIST. The installer fetches this version's configs from\n"
|
||||
" %s/%s/felhom-agent/raw/tag/%s/configs/ — without the tag every install\n"
|
||||
" 404s mid-run, as root. Fix: git tag -a %s <released-commit> && git push origin %s"
|
||||
% (tag, GITEA_BASE, OWNER, tag, tag, tag))
|
||||
else:
|
||||
print(" ok tag %s exists on origin (not in this shallow clone)" % tag)
|
||||
else:
|
||||
sha = r.stdout.strip()
|
||||
anc = git("merge-base", "--is-ancestor", sha, "HEAD")
|
||||
if anc.returncode == 0:
|
||||
print(" ok tag %s -> %s, an ancestor of HEAD" % (tag, sha[:10]))
|
||||
else:
|
||||
problems.append("tag %s points at %s, which is NOT an ancestor of HEAD — a tag parked "
|
||||
"on an unrelated commit is not a release" % (tag, sha[:10]))
|
||||
|
||||
# ---- leg 3: the package. Needs the network. ------------------------------------------------
|
||||
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
req = urllib.request.Request(url, method="HEAD")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=25) as resp:
|
||||
if resp.status == 200:
|
||||
print(" ok package %s is downloadable" % v)
|
||||
else:
|
||||
problems.append("package %s returned HTTP %s at %s" % (v, resp.status, url))
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
problems.append("PACKAGE %s IS NOT PUBLISHED (HTTP 404 at %s).\n"
|
||||
" Fix: bash scripts/release-agent.sh %s" % (v, url, v))
|
||||
else:
|
||||
inconclusive.append("registry returned HTTP %s for %s" % (e.code, v))
|
||||
except Exception as e:
|
||||
inconclusive.append("registry unreachable (%s) — leg 3 not checked; legs 1-2 still ran" % e)
|
||||
|
||||
if problems:
|
||||
print("\ncheck-release-complete: INCOMPLETE RELEASE")
|
||||
for p in problems:
|
||||
print(" - " + p)
|
||||
return 1
|
||||
if inconclusive:
|
||||
print("\ncheck-release-complete: INCONCLUSIVE — an undetermined result is never a pass")
|
||||
for i in inconclusive:
|
||||
print(" - " + i)
|
||||
return 2
|
||||
print("\ncheck-release-complete: %s is tagged, placed and published." % tag)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"_comment": [
|
||||
"THE retention number for published agent artifacts. One file, read by everything that",
|
||||
"depends on it, because a check and the policy it enforces must read the same number from the",
|
||||
"same place or they drift — and the drift looks like a defect in something else.",
|
||||
"",
|
||||
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
|
||||
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
|
||||
"still be downloadable. Both rules are individually sensible; together they are impossible.",
|
||||
"CI went red at a commit whose own run had been green the day before, on a true finding that",
|
||||
"no one could act on. The red will return at the next publish unless the two read one number.",
|
||||
"",
|
||||
"HOW THE NUMBER WAS ARRIVED AT — stated honestly, because it is weaker than it looks.",
|
||||
"generic_versions_kept is 10 because that is what the registry demonstrably holds today",
|
||||
"(felhom-agent 0.121.0..0.128.0 = 10 versions, queried 2026-08-09). It is an OBSERVED state,",
|
||||
"NOT a ruling anyone has been able to locate: no register row records a package prune, R-210",
|
||||
"is WAITING-ON-OPERATOR and says 'Nothing was deleted; this is a list, not an action', and it",
|
||||
"concerns local Docker images rather than this registry. Container packages currently hold 19",
|
||||
"each, so there is no uniform ten-per-package cap visible either. See R-287.",
|
||||
"",
|
||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
|
||||
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
|
||||
"operator should confirm or replace the number — at which point this file changes and both",
|
||||
"readers follow it in the same commit.",
|
||||
"",
|
||||
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
|
||||
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
|
||||
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
|
||||
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
|
||||
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
|
||||
"the gate does not have today. Filed as the follow-up in R-287.",
|
||||
"",
|
||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
|
||||
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
|
||||
"version at all — a strictly worse act than an un-downloadable binary."
|
||||
],
|
||||
"generic_versions_kept": 10,
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — bounds its assertion to the newest N versions",
|
||||
"documentation/runbooks/registry-retention.md (felhom.eu) — the prune procedure"
|
||||
],
|
||||
"recorded": "2026-08-09",
|
||||
"recorded_by": "CC, from the registry's observed state; NOT from a located operator ruling"
|
||||
}
|
||||
Reference in New Issue
Block a user