agent 0.130.0 published and vouched; R-347 closed, R-349 + R-350 filed

Released via scripts/release-agent.sh: tag v0.130.0 at 7569f34, sha256 a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3, verified by independent download and reproducible byte for byte with -trimpath -buildvcs=false. Vouched agent 0.129.0 -> 0.130.0 in the Day-0 manifest. Only the agent fields changed: min_agent stays 0.129.0 because it states what the GOLDEN CONTROLLER requires, and raising it would have HELD the floor for every box below 0.130.0. Global floor untouched at 0.216.0 -- and on hub v0.106.0 it is a separate form with its own action, so publish-train rule 2's hazard no longer exists in the shape its incident describes. No --no-verify: the CHANGELOG heading was flipped only after the tag and package existed, so release-complete passes on the real artifact. R-349: the fleet was running a DIFFERENT binary under the same version name -- the proof deploy was a hand build, the release is -trimpath. Self-update could never have corrected it, because every version check compares the string. Both boxes reinstalled from the downloaded package. The proper fix exists in miniature as wrapper_sha256 and was never extended to the agent's own binary. R-350: I printed the hub password into the session transcript via curl -w '%{redirect_url}' -- the hub answers 303 and curl re-attaches the credential. Not in git, not in any committed file, not in the evidence directory. Rotation is the operator's call. ep0 closes at fd 17, ESTAB 0, CLOSE-WAIT 0 -- its t0 baseline -- and was read-only for this entire arc.
2026-08-20 12:54:54 +02:00
parent 47268ad1c4
commit 910fd91124
6 changed files with 184 additions and 9 deletions
@@ -1,8 +1,10 @@
 # REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20)

-**Outcome: the fix works, measured three independent ways, and ep0 is back to its baseline of 17 file
-descriptors from 415.** Both demo boxes now run agent **0.130.0**. **Nothing is published** — that is
-the one decision left, filed as R-347.
+**Outcome: the fix works, measured three independent ways; ep0 is back to its baseline of 17 file
+descriptors from 415; and 0.130.0 is now published and vouched.** Both demo boxes run the **byte-exact
+published artifact**. R-347 is CLOSED. Two new findings came out of the release itself — **R-349**
+(the fleet was briefly running a different binary under the same version name) and **R-350** (I printed
+the hub password into the transcript; rotation is your call).

 ## 1. Confirmed baselines

@@ -179,6 +181,52 @@ t=10:40:23Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
 `CLOSE-WAIT 0`, `ESTAB` in the low single digits, `fd` at the baseline, proxy PID **551655** — the same
 process that has been running since 2026-08-18 09:51:04, never restarted by this work.

+## 11b. The release (R-347, CLOSED)
+
+`bash scripts/release-agent.sh 0.130.0` — the one documented way (R-115): build, tag, publish, and
+**verify by independent download**.
+
+| | |
+|---|---|
+| tag | `v0.130.0` at `7569f34` |
+| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
+| size | 14,141,158 bytes |
+| reproducible | **yes, checked** — `-trimpath -buildvcs=false` rebuild matches byte for byte |
+
+**Vouched** in the Day-0 artifact manifest: agent 0.129.0 → **0.130.0** + its sha. **Only the agent
+fields changed.**
+
+- **`min_agent` left at 0.129.0** — it states what the **golden controller** needs. Raising it to
+  0.130.0 would have made the hub **HOLD the floor** for every box below 0.130.0, which is the
+  opposite of shipping a fix.
+- **Global floor never touched** (0.216.0). On hub v0.106.0 it is a *separate form with its own
+  action*, so publish-train rule 2's "save the floor last" hazard no longer exists in the shape its
+  incident describes — the rule's reasoning holds, its mechanism has moved.
+- After: no `floor held`, no `*_unreachable`, both boxes reporting 0.130.0, artifact downloading
+  anonymously at the vouched sha.
+
+**No `--no-verify` in the train.** The heading was flipped to `## v0.130.0` only after tag and package
+existed. Flipping first and bypassing would have produced a red CI run and an alarm mail for a release
+that worked — R-168's failure mode.
+
+## 11c. Two findings from the release
+
+**R-349 — the fleet was running a different binary under the same version name.** The proof deploy was
+a hand build; the release builds `-trimpath -buildvcs=false`. Same source, same version string,
+different bytes (`256e0829…` vs `a56a92a7…`). **Self-update could never have corrected it** — the boxes
+already reported 0.130.0, so the vouched version looked installed. Every version check in the system
+compares the *string*. Fixed by installing the **downloaded** artifact on both. The proper fix already
+exists in miniature: `wrapper_sha256` does exactly this drift detection for the PBS wrapper and was
+never extended to the agent's own binary.
+
+**R-350 — I printed the hub password into the transcript.** Confirming the vouch used
+`curl -w '%{redirect_url}'`; the hub answers 303 and curl re-attaches the basic-auth credential to the
+redirect target it prints. **Not in git, not in any committed file** (checked by content), not in the
+evidence directory — it is in the session transcript on DooPlex. Every other call printed only the
+length; this came through curl's own formatting. **Rotation is your call** — I did not do it
+unilaterally, and I can do it file-to-file without printing the new value if you want. The reusable
+half: `%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential.
+
 ## 12. Observations

 - **The closure refactor is not worth doing — recommend leaving it.** With the idle timeout restored an
@@ -132,11 +132,17 @@ record with no machine** — created 13 August, no host, no backups, nothing to
  off-site box is back to 17 open connections, its normal resting number, down from 415.** All of the
  built-up connections released themselves when the agents restarted; the off-site box was only ever
  read from, never touched. *(register: R-344)*
- **The fix is on the two demo machines by hand and NOT published yet** (R-347). A machine installed
-  from today's image still gets the old, leaking agent. That was deliberate — publishing it mid-test
-  would have contaminated the comparison — and the reason has now expired. **It is not urgent:** a new
-  machine would take the better part of a year to matter, and any agent update clears the build-up.
-  **Publishing is your call**, and it needs the operator-only artifact screen at the end.
+- **PUBLISHED the same day, on your word** (R-347, closed). Agent **0.130.0** is released, and the hub
+  now hands it to any new machine. Both demo machines run the exact published copy. Nothing else on
+  that screen was changed — in particular the controller floor was left alone, and the "minimum agent"
+  setting too, because raising that would have **stopped** machines getting updates rather than
+  helping them.
+- **Two things the release itself turned up.** (1) The machines were briefly running a *different*
+  build of the same version number — harmless here, but nothing in the system would ever have noticed,
+  because everything compares the version *name*. Now corrected, and filed so it cannot repeat
+  (R-349). (2) **I printed the hub password into my own session log** while confirming the change
+  (R-350). It is not in git and not in any saved file — but it is in the log on this machine.
+  **Changing it is your call**; I can do it without ever showing the new one. Ask and I will.
 - **The off-site box was updated, and it did not help — as expected** (R-341). On your ruling we
  installed the newer backup software for the practice, having first read its release notes and found
  **nothing** about the fault we have. The update went cleanly and everything works, but the leak
@@ -500,3 +500,84 @@ stays populated. Filed as R-348.
  `[::ffff:10.77.0.2]:port`, and the pattern expected `10.77.0.2:`. Caught 15 minutes in by noticing
  that a 0/0 split could not sum to 199. Fixed, then **one sample was proved by hand before committing
  the window to it** — the check that should have happened first.
+
+---
+
+# Published — 2026-08-20, on the operator's word (R-347 CLOSED)
+
+Released through `scripts/release-agent.sh 0.130.0`, the one documented way (R-115), which builds,
+tags, publishes and **verifies by independent download** rather than by its own say-so.
+
+| | |
+|---|---|
+| version / tag | **0.130.0** / `v0.130.0` at `7569f34` |
+| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
+| size | 14,141,158 bytes |
+| reproducible | **yes, checked** — a rebuild with `-trimpath -buildvcs=false` matches byte for byte (R-186) |
+| anonymous fetch | matches the vouched sha |
+
+## The vouch, and what was deliberately NOT changed
+
+Day-0 artifact manifest: `agent_version` 0.129.0 → **0.130.0**, `agent_sha256` updated. **Everything
+else re-sent unchanged**, and each for a reason:
+
+- **`min_agent` stays 0.129.0.** It expresses what the **golden controller** requires, not what the
+  newest agent is. Raising it to 0.130.0 would have made the hub **HOLD the controller floor** for
+  every box not yet on 0.130.0 — the exact opposite of shipping a fix, and the manifest's own help
+  text says so: *"The hub HOLDS the floor for any box whose agent is below this."*
+- **`golden_version` / `golden_sha256` / `wrapper_sha256`** — untouched; no controller was released.
+- **The global floor was never touched** (still 0.216.0). **And on hub v0.106.0 it could not have been
+  by accident:** it is a *separate form with its own action* (`/configuration/global-floor`), so
+  publish-train rule 2's "the manifest screen carries the live DB floor — save it last" hazard no
+  longer exists in the shape its incident describes. Worth knowing before the next train; the rule's
+  reasoning still holds, its mechanism has moved.
+
+Verified after: no `floor held` line for either box, no `*_unreachable`, both boxes reporting 0.130.0.
+
+**No `--no-verify` anywhere in the train.** The CHANGELOG heading was flipped from
+`## UNRELEASED — v0.130.0 candidate` to `## v0.130.0` **only after** the tag and package existed, so
+`release-complete` passes on the real artifact. The ordering was: release at the UNRELEASED-heading
+commit, then flip. The alternative — flipping first and bypassing the gate — would have produced a
+genuinely red CI run and an alarm mail for a release that worked, which is R-168's failure mode.
+
+## The trap this train exposed — R-349
+
+**Both boxes were running a different binary under the same version name, and nothing would ever have
+noticed.** The proof deploy used a hand build (`go build -ldflags …`); the release builds with
+`-trimpath -buildvcs=false` for reproducibility. Same source, same version string, **different bytes**:
+`256e0829…` on the boxes against `a56a92a7…` published.
+
+The sharp edge is that **self-update cannot correct it**: the boxes already reported `0.130.0`, so the
+vouched version looked installed and nothing would have happened, indefinitely. Every version check in
+the system — the hub, `--version`, the artifact manifest — compares the version **string**, so the
+divergence is invisible to all of them.
+
+Corrected by installing the **downloaded** artifact (not a local rebuild — the boxes get the bytes a
+fresh install would get) on both. Both now report `a56a92a7…`.
+
+**The proper fix already exists in miniature:** `wrapper_sha256` makes exactly this drift visible for
+the PBS wrapper — *"agents report the installed file's hash and a mismatch is surfaced on the host
+page"*. It was simply never extended to the agent's own binary. R-349.
+
+## A mistake of mine in this train — R-350
+
+Confirming the vouch used `curl -w '%{redirect_url}'`. The hub answers the POST with a **303**, and
+curl renders the redirect target **with the basic-auth credentials re-attached** — so the hub operator
+password was printed in cleartext into the session transcript.
+
+It is **not** in git, not in any committed file (checked by content, not by assumption), and not in
+this evidence directory; it is in the Claude Code transcript on DooPlex. Every other call in the
+session printed only the password's length — this arrived through curl's output formatting, which is
+why the usual discipline missed it. Rotation is recommended and is the operator's call; the reusable
+half is that **`%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential** —
+confirm a redirect with `%{http_code}` and read the flash from a follow-up GET.
+
+## Closing state
+
+```
+ep0  10:52:26Z  pid=551655  fd=17  estab=0  ctrl(.2)=0  fix(.3)=0  CLOSE-WAIT=0
+```
+
+**fd 17 is ep0's `t0` baseline**, and it returns there between poll cycles. The proxy is the same
+process that has run since 2026-08-18 09:51:04 — **ep0 was read-only for this entire arc**, from the
+spike through the fix to the release, and was never restarted, reconfigured or upgraded by any of it.
@@ -0,0 +1,19 @@
+=== final verification, 2026-08-20T10:50:03+00:00 ===
+--- ep0 settle (expect fd back toward 17, CLOSE-WAIT 0) ---
+t=10:50:04Z pid=551655 fd=21 estab=4 ctrl(.2)=2 fix(.3)=2 CLOSE-WAIT=0
+t=10:51:15Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
+t=10:52:26Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
+
+--- hub: agent version per host, and any floor-held ---
+ demo-felhom-8363b5 
+ 0.130.0 
+ demo-hp-bb76ea 
+ 0.130.0 
+ 0.129.0 
+
+--- hub log: floor held / unreachable since the vouch ---
+(empty = none)
+
+--- published artifact is downloadable as an anonymous client would fetch it ---
+  anonymous GET sha256: a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
+  tag on origin       : 7edfea9aa9
@@ -0,0 +1,19 @@
+=== reconciling the fleet onto the VOUCHED artifact ===
+downloaded from the package registry, not rebuilt locally — the boxes get the bytes a fresh install would get
+UTC: 2026-08-20T10:49:30+00:00
+
+--- ep0 before ---
+t=10:49:31Z pid=551655 fd=20 estab=3 ctrl(.2)=2 fix(.3)=1 CLOSE-WAIT=0
+
+--- demo-hp ---
+  staged sha : a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
+  now running: felhom-agent 0.130.0  sha=a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
+  service    : active
+
+--- felhom-pve ---
+  staged sha : a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
+  now running: felhom-agent 0.130.0  sha=a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
+  service    : active
+
+--- ep0 after both restarts ---
+t=10:49:37Z pid=551655 fd=21 estab=4 ctrl(.2)=2 fix(.3)=2 CLOSE-WAIT=0