agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed
gates / gates (push) Successful in 14s

Released via scripts/release-agent.sh: tag v0.130.0 at 7569f34, sha256
a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3,
verified by independent download and reproducible byte for byte with
-trimpath -buildvcs=false.

Vouched agent 0.129.0 -> 0.130.0 in the Day-0 manifest. Only the agent
fields changed: min_agent stays 0.129.0 because it states what the GOLDEN
CONTROLLER requires, and raising it would have HELD the floor for every
box below 0.130.0. Global floor untouched at 0.216.0 -- and on hub
v0.106.0 it is a separate form with its own action, so publish-train
rule 2's hazard no longer exists in the shape its incident describes.

No --no-verify: the CHANGELOG heading was flipped only after the tag and
package existed, so release-complete passes on the real artifact.

R-349: the fleet was running a DIFFERENT binary under the same version
name -- the proof deploy was a hand build, the release is -trimpath.
Self-update could never have corrected it, because every version check
compares the string. Both boxes reinstalled from the downloaded package.
The proper fix exists in miniature as wrapper_sha256 and was never
extended to the agent's own binary.

R-350: I printed the hub password into the session transcript via
curl -w '%{redirect_url}' -- the hub answers 303 and curl re-attaches the
credential. Not in git, not in any committed file, not in the evidence
directory. Rotation is the operator's call.

ep0 closes at fd 17, ESTAB 0, CLOSE-WAIT 0 -- its t0 baseline -- and was
read-only for this entire arc.
This commit is contained in:
2026-08-20 12:54:54 +02:00
parent 47268ad1c4
commit 910fd91124
6 changed files with 184 additions and 9 deletions
+51 -3
View File
@@ -1,8 +1,10 @@
# REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20) # REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20)
**Outcome: the fix works, measured three independent ways, and ep0 is back to its baseline of 17 file **Outcome: the fix works, measured three independent ways; ep0 is back to its baseline of 17 file
descriptors from 415.** Both demo boxes now run agent **0.130.0**. **Nothing is published** — that is descriptors from 415; and 0.130.0 is now published and vouched.** Both demo boxes run the **byte-exact
the one decision left, filed as R-347. published artifact**. R-347 is CLOSED. Two new findings came out of the release itself — **R-349**
(the fleet was briefly running a different binary under the same version name) and **R-350** (I printed
the hub password into the transcript; rotation is your call).
## 1. Confirmed baselines ## 1. Confirmed baselines
@@ -179,6 +181,52 @@ t=10:40:23Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
`CLOSE-WAIT 0`, `ESTAB` in the low single digits, `fd` at the baseline, proxy PID **551655** — the same `CLOSE-WAIT 0`, `ESTAB` in the low single digits, `fd` at the baseline, proxy PID **551655** — the same
process that has been running since 2026-08-18 09:51:04, never restarted by this work. process that has been running since 2026-08-18 09:51:04, never restarted by this work.
## 11b. The release (R-347, CLOSED)
`bash scripts/release-agent.sh 0.130.0` — the one documented way (R-115): build, tag, publish, and
**verify by independent download**.
| | |
|---|---|
| tag | `v0.130.0` at `7569f34` |
| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
| size | 14,141,158 bytes |
| reproducible | **yes, checked** — `-trimpath -buildvcs=false` rebuild matches byte for byte |
**Vouched** in the Day-0 artifact manifest: agent 0.129.0 → **0.130.0** + its sha. **Only the agent
fields changed.**
- **`min_agent` left at 0.129.0** — it states what the **golden controller** needs. Raising it to
0.130.0 would have made the hub **HOLD the floor** for every box below 0.130.0, which is the
opposite of shipping a fix.
- **Global floor never touched** (0.216.0). On hub v0.106.0 it is a *separate form with its own
action*, so publish-train rule 2's "save the floor last" hazard no longer exists in the shape its
incident describes — the rule's reasoning holds, its mechanism has moved.
- After: no `floor held`, no `*_unreachable`, both boxes reporting 0.130.0, artifact downloading
anonymously at the vouched sha.
**No `--no-verify` in the train.** The heading was flipped to `## v0.130.0` only after tag and package
existed. Flipping first and bypassing would have produced a red CI run and an alarm mail for a release
that worked — R-168's failure mode.
## 11c. Two findings from the release
**R-349 — the fleet was running a different binary under the same version name.** The proof deploy was
a hand build; the release builds `-trimpath -buildvcs=false`. Same source, same version string,
different bytes (`256e0829…` vs `a56a92a7…`). **Self-update could never have corrected it** — the boxes
already reported 0.130.0, so the vouched version looked installed. Every version check in the system
compares the *string*. Fixed by installing the **downloaded** artifact on both. The proper fix already
exists in miniature: `wrapper_sha256` does exactly this drift detection for the PBS wrapper and was
never extended to the agent's own binary.
**R-350 — I printed the hub password into the transcript.** Confirming the vouch used
`curl -w '%{redirect_url}'`; the hub answers 303 and curl re-attaches the basic-auth credential to the
redirect target it prints. **Not in git, not in any committed file** (checked by content), not in the
evidence directory — it is in the session transcript on DooPlex. Every other call printed only the
length; this came through curl's own formatting. **Rotation is your call** — I did not do it
unilaterally, and I can do it file-to-file without printing the new value if you want. The reusable
half: `%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential.
## 12. Observations ## 12. Observations
- **The closure refactor is not worth doing — recommend leaving it.** With the idle timeout restored an - **The closure refactor is not worth doing — recommend leaving it.** With the idle timeout restored an
+11 -5
View File
@@ -132,11 +132,17 @@ record with no machine** — created 13 August, no host, no backups, nothing to
off-site box is back to 17 open connections, its normal resting number, down from 415.** All of the off-site box is back to 17 open connections, its normal resting number, down from 415.** All of the
built-up connections released themselves when the agents restarted; the off-site box was only ever built-up connections released themselves when the agents restarted; the off-site box was only ever
read from, never touched. *(register: R-344)* read from, never touched. *(register: R-344)*
- **The fix is on the two demo machines by hand and NOT published yet** (R-347). A machine installed - **PUBLISHED the same day, on your word** (R-347, closed). Agent **0.130.0** is released, and the hub
from today's image still gets the old, leaking agent. That was deliberate — publishing it mid-test now hands it to any new machine. Both demo machines run the exact published copy. Nothing else on
would have contaminated the comparison — and the reason has now expired. **It is not urgent:** a new that screen was changed — in particular the controller floor was left alone, and the "minimum agent"
machine would take the better part of a year to matter, and any agent update clears the build-up. setting too, because raising that would have **stopped** machines getting updates rather than
**Publishing is your call**, and it needs the operator-only artifact screen at the end. helping them.
- **Two things the release itself turned up.** (1) The machines were briefly running a *different*
build of the same version number — harmless here, but nothing in the system would ever have noticed,
because everything compares the version *name*. Now corrected, and filed so it cannot repeat
(R-349). (2) **I printed the hub password into my own session log** while confirming the change
(R-350). It is not in git and not in any saved file — but it is in the log on this machine.
**Changing it is your call**; I can do it without ever showing the new one. Ask and I will.
- **The off-site box was updated, and it did not help — as expected** (R-341). On your ruling we - **The off-site box was updated, and it did not help — as expected** (R-341). On your ruling we
installed the newer backup software for the practice, having first read its release notes and found installed the newer backup software for the practice, having first read its release notes and found
**nothing** about the fault we have. The update went cleanly and everything works, but the leak **nothing** about the fault we have. The update went cleanly and everything works, but the leak
@@ -500,3 +500,84 @@ stays populated. Filed as R-348.
`[::ffff:10.77.0.2]:port`, and the pattern expected `10.77.0.2:`. Caught 15 minutes in by noticing `[::ffff:10.77.0.2]:port`, and the pattern expected `10.77.0.2:`. Caught 15 minutes in by noticing
that a 0/0 split could not sum to 199. Fixed, then **one sample was proved by hand before committing that a 0/0 split could not sum to 199. Fixed, then **one sample was proved by hand before committing
the window to it** — the check that should have happened first. the window to it** — the check that should have happened first.
---
# Published — 2026-08-20, on the operator's word (R-347 CLOSED)
Released through `scripts/release-agent.sh 0.130.0`, the one documented way (R-115), which builds,
tags, publishes and **verifies by independent download** rather than by its own say-so.
| | |
|---|---|
| version / tag | **0.130.0** / `v0.130.0` at `7569f34` |
| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
| size | 14,141,158 bytes |
| reproducible | **yes, checked** — a rebuild with `-trimpath -buildvcs=false` matches byte for byte (R-186) |
| anonymous fetch | matches the vouched sha |
## The vouch, and what was deliberately NOT changed
Day-0 artifact manifest: `agent_version` 0.129.0 → **0.130.0**, `agent_sha256` updated. **Everything
else re-sent unchanged**, and each for a reason:
- **`min_agent` stays 0.129.0.** It expresses what the **golden controller** requires, not what the
newest agent is. Raising it to 0.130.0 would have made the hub **HOLD the controller floor** for
every box not yet on 0.130.0 — the exact opposite of shipping a fix, and the manifest's own help
text says so: *"The hub HOLDS the floor for any box whose agent is below this."*
- **`golden_version` / `golden_sha256` / `wrapper_sha256`** — untouched; no controller was released.
- **The global floor was never touched** (still 0.216.0). **And on hub v0.106.0 it could not have been
by accident:** it is a *separate form with its own action* (`/configuration/global-floor`), so
publish-train rule 2's "the manifest screen carries the live DB floor — save it last" hazard no
longer exists in the shape its incident describes. Worth knowing before the next train; the rule's
reasoning still holds, its mechanism has moved.
Verified after: no `floor held` line for either box, no `*_unreachable`, both boxes reporting 0.130.0.
**No `--no-verify` anywhere in the train.** The CHANGELOG heading was flipped from
`## UNRELEASED — v0.130.0 candidate` to `## v0.130.0` **only after** the tag and package existed, so
`release-complete` passes on the real artifact. The ordering was: release at the UNRELEASED-heading
commit, then flip. The alternative — flipping first and bypassing the gate — would have produced a
genuinely red CI run and an alarm mail for a release that worked, which is R-168's failure mode.
## The trap this train exposed — R-349
**Both boxes were running a different binary under the same version name, and nothing would ever have
noticed.** The proof deploy used a hand build (`go build -ldflags …`); the release builds with
`-trimpath -buildvcs=false` for reproducibility. Same source, same version string, **different bytes**:
`256e0829…` on the boxes against `a56a92a7…` published.
The sharp edge is that **self-update cannot correct it**: the boxes already reported `0.130.0`, so the
vouched version looked installed and nothing would have happened, indefinitely. Every version check in
the system — the hub, `--version`, the artifact manifest — compares the version **string**, so the
divergence is invisible to all of them.
Corrected by installing the **downloaded** artifact (not a local rebuild — the boxes get the bytes a
fresh install would get) on both. Both now report `a56a92a7…`.
**The proper fix already exists in miniature:** `wrapper_sha256` makes exactly this drift visible for
the PBS wrapper — *"agents report the installed file's hash and a mismatch is surfaced on the host
page"*. It was simply never extended to the agent's own binary. R-349.
## A mistake of mine in this train — R-350
Confirming the vouch used `curl -w '%{redirect_url}'`. The hub answers the POST with a **303**, and
curl renders the redirect target **with the basic-auth credentials re-attached** — so the hub operator
password was printed in cleartext into the session transcript.
It is **not** in git, not in any committed file (checked by content, not by assumption), and not in
this evidence directory; it is in the Claude Code transcript on DooPlex. Every other call in the
session printed only the password's length — this arrived through curl's output formatting, which is
why the usual discipline missed it. Rotation is recommended and is the operator's call; the reusable
half is that **`%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential** —
confirm a redirect with `%{http_code}` and read the flash from a follow-up GET.
## Closing state
```
ep0 10:52:26Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
```
**fd 17 is ep0's `t0` baseline**, and it returns there between poll cycles. The proxy is the same
process that has run since 2026-08-18 09:51:04 — **ep0 was read-only for this entire arc**, from the
spike through the fix to the release, and was never restarted, reconfigured or upgraded by any of it.
@@ -0,0 +1,19 @@
=== final verification, 2026-08-20T10:50:03+00:00 ===
--- ep0 settle (expect fd back toward 17, CLOSE-WAIT 0) ---
t=10:50:04Z pid=551655 fd=21 estab=4 ctrl(.2)=2 fix(.3)=2 CLOSE-WAIT=0
t=10:51:15Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
t=10:52:26Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
--- hub: agent version per host, and any floor-held ---
demo-felhom-8363b5
0.130.0
demo-hp-bb76ea
0.130.0
0.129.0
--- hub log: floor held / unreachable since the vouch ---
(empty = none)
--- published artifact is downloadable as an anonymous client would fetch it ---
anonymous GET sha256: a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
tag on origin : 7edfea9aa9
@@ -0,0 +1,19 @@
=== reconciling the fleet onto the VOUCHED artifact ===
downloaded from the package registry, not rebuilt locally — the boxes get the bytes a fresh install would get
UTC: 2026-08-20T10:49:30+00:00
--- ep0 before ---
t=10:49:31Z pid=551655 fd=20 estab=3 ctrl(.2)=2 fix(.3)=1 CLOSE-WAIT=0
--- demo-hp ---
staged sha : a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
now running: felhom-agent 0.130.0 sha=a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
service : active
--- felhom-pve ---
staged sha : a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
now running: felhom-agent 0.130.0 sha=a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
service : active
--- ep0 after both restarts ---
t=10:49:37Z pid=551655 fd=21 estab=4 ctrl(.2)=2 fix(.3)=2 CLOSE-WAIT=0
File diff suppressed because one or more lines are too long