R-267 closed: 26.2s -> 5.4s cold / 0.14s warm, and two corrections to my own measurements

Registry pruned to the newest 10 per package on the operator's confirmed rule. 33 deletions, all
HTTP 204; the live-vouched golden 0.210.0, agent 0.128.0 and floor 0.127.0 were asserted into the
KEEP set BEFORE any DELETE was issued and verified still fetchable after.

TWO CORRECTIONS TO WHAT I REPORTED EARLIER, both recorded rather than quietly dropped:

1. 'Only 50 generic versions exist' was NOT a count, it was a PAGE LIMIT. ?limit=1000 returns at most
   50, and the 50 I measured was exactly the cap. Three older agent versions (0.81.0/0.80.0/0.79.0)
   only became visible after the first 30 deletions moved them onto page one. An unpaginated listing
   is not evidence of a total — this repo's own 'an empty listing is not evidence of emptiness' rule,
   walked into while measuring it.

2. The operator's 'reduce the number of artifacts' was the better call and my measurement said
   otherwise. I reported it helps sub-linearly and is not the lever. Measured after: trimming to
   10+10 took the COLD load from 13.4s to 5.4s, a 2.5x improvement on exactly the path the memo
   cannot help, because the fan-out is per-version.

drill-r50 runs agent 0.113.0, now deleted; flagged before deleting, disposable nested drill VM, only
its re-download path is gone.

documentation/backlog/OPEN-ITEMS.md

@@ -494,7 +494,7 @@ applied.** The one that matters: Scenario A **fails against today's tree** with
 |---|---|---|
 | **R-266** | **A failed root `statfs` still reaches the hub as a 0-of-0 disk, and the hub cannot tell that from an empty one.** Split out of R-259 on 2026-08-08 so that fixing the CUSTOMER-facing half could not be mistaken for fixing the wire. `report/builder.go:93-95` copies `sysInfo.DiskTotalGB` / `DiskUsedGB` / `DiskPercent` into `r.Storage[0]` (`Mount: "/"`), and those are exactly the zeros a failed `statfs` leaves behind — the controller now KNOWS the measurement failed (`SystemInfo.DiskKnown`, controller v0.210.0) and the report still does not carry it. **Deliberately not fixed here, for a reason that is now structural rather than a preference:** adding a field to that report is a change to a declared wire, which since G-1 means the receiving side must model it in the same session (`scripts/wire_contract_gate.py` refuses otherwise) — a two-repo change with a hub bump, and this session deliberately touched no hub code. **RANKED LOW, and the reason is that the consequence is bounded:** the hub bands host storage on `disk_percent`, so a failed read presents as 0% used — the *quiet* direction. It cannot raise a false "nearly full" alarm; it can only fail to raise a true one, and only while the root filesystem is unreadable, which is a state with louder symptoms of its own. **Fix shape when it is taken:** carry `disk_known` on the storage entry and have the hub's fill checker skip an unknown reading rather than band it — never treat absent as 0 | **READY** — owner Viktor |
 
-| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.0–0.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.20–3.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 7–11 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"* — **NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0.** Memoised 60 s in memory (operator ruling). **Measured after: cold 13.4 s, warm 0.24–0.33 s** — the operator sees ~0.25 s except at most once a minute. The three serialisation legs (v0.100.0–0.100.2) took 26.2 s → 9.85 s mean; the memo takes the warm path to a quarter-second. NOT persisted — see the CHANGELOG for why a copy in `hub_settings` would be a second source of truth. Artifact pruning is tracked separately below |
+| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.0–0.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.20–3.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 7–11 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"* — **NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0 + a registry prune.** **Final: cold 5.4 s, warm 0.14 s** (was 26.2 s). Three serialisation legs took it to 9.85 s mean, the 60 s in-memory memo took the warm path to a quarter-second, and the prune halved what remains of the cold path. **⚠ TWO CORRECTIONS TO THIS ROW'S OWN EARLIER TEXT, because both were wrong and both mattered.** **(1) "Only 50 generic versions exist" WAS NOT A COUNT, IT WAS A PAGE LIMIT.** `?type=generic&limit=1000` returns at most 50; the 50 I measured was exactly the cap, and three older agent versions (0.81.0, 0.80.0, 0.79.0) only became visible after the first 30 deletions moved them onto page one. **An unpaginated listing is not evidence of a total** — this repo's own "an empty listing is not evidence of emptiness" rule, walked into while measuring. **(2) THE OPERATOR'S "REDUCE THE NUMBER OF ARTIFACTS" WAS THE BETTER CALL AND MY MEASUREMENT SAID OTHERWISE.** I reported it helps "sub-linearly" and "is not the lever". Measured after: trimming to 10+10 took the COLD load from 13.4 s to 5.4 s — a 2.5× improvement on the path the memo cannot help, because the fan-out is per-version. Recorded rather than quietly dropped (the R-96 standing rule). **Pruned to the newest 10 per package on the operator's rule**, with the live-vouched golden/agent/floor asserted into the KEEP set before a single DELETE was issued; 33 deletions, all HTTP 204, and golden 0.210.0 / agent 0.128.0 / agent 0.127.0 verified still fetchable afterwards. `drill-r50` runs agent 0.113.0, now deleted — flagged to the operator first; it is a disposable nested drill VM and only its re-download path is gone |
 
 | **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **READY** — owner Viktor |