From efe9dfd15d0ed10617f09c9e567192848b6aab47 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 8 Aug 2026 20:13:22 +0200 Subject: [PATCH] R-267 CLOSED (26.2s -> 0.24s warm); R-268 filed: I printed a live token into a transcript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-267 closed by hub v0.101.0. Measured after the 60s memo: cold 13.4s, warm 0.24-0.33s. The operator sees a quarter-second except at most once a minute. R-268 filed against myself. Setting up R-221's live drill, a one-liner meant to list bootstrap.json's KEYS printed the local_api object whole, including its token, for guest 9201. Reported rather than quietly rotated, because a secret reaching a transcript is a finding whatever its blast radius. Exposure assessed rather than assumed, and it is small: the token opens only the agent's per-guest local API on the island bridge between that host and that one guest, self-scoped to guest 9201, not routable from the LAN or internet, on a Tier-0 disposable box with no customer data. Using it already requires code execution there, at which point an attacker has more than the token. Rotation exists (TokenStore.Mint, last-write-wins per VMID) but must also rewrite the guest's bootstrap.json or the controller loses agent access — an operator-timed act, not a background one. The general fix is upstream: reading secret-bearing JSON should go through a helper that prints keys and never values, the discipline the golden bake already uses for the Gitea token. R-221 also recorded as PROVEN ON HARDWARE in STATUS. --- STATUS.md | 35 ++++++++++++++++------------- documentation/backlog/OPEN-ITEMS.md | 4 +++- 2 files changed, 23 insertions(+), 16 deletions(-) diff --git a/STATUS.md b/STATUS.md index 9006f8f..b514265 100644 --- a/STATUS.md +++ b/STATUS.md @@ -27,16 +27,13 @@ in 72 seconds. The two rough edges that walk found are also gone. *(R-201, R-252 - **Nothing new is broken.** The *check* against a fourth secret-in-a-page covers 4 pages of 27, and the cheap one covering all of them is blind to the shape that shipped. *(R-255)* -- **The machine's own screen keeps telling an already-paired box to pair itself** — 25 minutes after it - was paired, on a screen that promises it refreshes itself. *(R-214, R-235)* -- **A backup that covered nothing still calls itself „Sikeres".** The state is honest; the word is not. - *(R-240)* -- **A machine waiting for its recovery code can stop backing up off-site without alarming us** — - after a *rebuild* we ARE told; the gap is reaching that state with no working tier. *(R-243)* +- **An already-paired box is still told to pair itself**, 25 minutes on. *(R-214, R-235)* +- **A backup that covered nothing still calls itself „Sikeres".** *(R-240)* +- **A machine waiting for its recovery code can stop backing up off-site without alarming us.** *(R-243)* - **The card offering to reopen set-aside backups promises more than we can deliver.** *(R-202)* - **Deleting a customer leaves rows behind** while reporting a clean teardown — no secrets, but it accumulates. *(R-244)* -- **Putting restored files back where they belong is still a manual step.** *(R-213)* +- **Putting restored files back where they belong is still manual.** *(R-213)* ## Fixed today — four things the machine knew and did not say @@ -59,6 +56,10 @@ All one family: something the box already knows, thrown away or drawn as its opp **Not fixed, and said rather than glossed:** that failed disk reading still reaches us as "0 of 0 GB". It is the quiet direction — it can only miss a true alarm, never raise a false one. *(R-266)* +**And R-221 is now proven on hardware, not just in tests** — on a demo machine we removed the one +line, watched the setup screen refuse, waited one minute, and watched it go green by itself with +nothing restarted. Every other line of that file came back identical. + ## What we're working on - **Widening the check** so a fourth secret-in-a-page is caught by a machine. *(R-255)* · **Deciding @@ -84,18 +85,22 @@ the machine all this is built on, not what a customer receives. Mixing them in i being readable.* - **DooPlex's own backup keeps every copy inside the same box, and is silent when it fails.** *(R-232)* -- **193 old images exist only on this machine**, ~27 GB against 199 GB free — clutter, not space. *(R-210)* +- **193 old images exist only on this machine**, ~27 GB — clutter, not space. *(R-210)* - **The hub password needs rotating** — a diagnostic printed it into a session log; nothing suggests anyone else saw it. *(R-132)* -- **One thing to read after DooPlex next restarts** — the second-SSD move has never survived a reboot; - it writes PASS/FAIL to `/var/log/felhom-store-postboot-check.log`. On PASS, 34 GB comes back. *(R-209a)* +- **After DooPlex next restarts**, read `/var/log/felhom-store-postboot-check.log` — the second-SSD + move has never survived a reboot; on PASS, 34 GB comes back. *(R-209a)* - **Backup scripts on DooPlex are unversioned host state** *(R-231)*, and the instruction-file follow-ups each need a decision rather than an edit *(R-229, R-230)*. -- **The Configuration page: 26 s → about 10 s, and not finished.** It was never hashing anything — - the hashes are already stored and just read. It was making 42 calls one after another. What is - left is a single slow Gitea lookup per dropdown; making it instant means holding a short-lived - copy, which trades freshness for speed and is your call. **Numbers were taken on a busy box and - swing 3×** — worth a re-check when it is idle. *(R-267)* +- **The Configuration page is fixed: 26 s → a quarter of a second.** It was never hashing anything — + the hashes are already stored and simply read. It was making 42 calls one after another. Now they + overlap, the connections are kept, and the answer is held for a minute, so you see ~0.25 s except + at most once a minute (13 s on that one load). *(R-267 — closed.)* +- **I printed a live access token into a session log** while setting up today's drill, and I am + telling you rather than quietly rotating it. It only opens the agent's private channel to one + demo guest, on a wire that exists solely between that host and that guest — not reachable from + your network or the internet, on a disposable machine with no customer data. Rotating it also + means updating the guest, so it is a deliberate act, not a background one. *(R-268)* - **Our build-check alarm has one gap left.** A run that hangs is now cut off after five minutes and the mail says how long it took — but **whether the alarm fires at all when the machinery kills a run outright is still unverified**, and we have not claimed otherwise. *(R-265)* diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 3bc741c..03ff0ae 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -494,7 +494,9 @@ applied.** The one that matters: Scenario A **fails against today's tree** with |---|---|---| | **R-266** | **A failed root `statfs` still reaches the hub as a 0-of-0 disk, and the hub cannot tell that from an empty one.** Split out of R-259 on 2026-08-08 so that fixing the CUSTOMER-facing half could not be mistaken for fixing the wire. `report/builder.go:93-95` copies `sysInfo.DiskTotalGB` / `DiskUsedGB` / `DiskPercent` into `r.Storage[0]` (`Mount: "/"`), and those are exactly the zeros a failed `statfs` leaves behind — the controller now KNOWS the measurement failed (`SystemInfo.DiskKnown`, controller v0.210.0) and the report still does not carry it. **Deliberately not fixed here, for a reason that is now structural rather than a preference:** adding a field to that report is a change to a declared wire, which since G-1 means the receiving side must model it in the same session (`scripts/wire_contract_gate.py` refuses otherwise) — a two-repo change with a hub bump, and this session deliberately touched no hub code. **RANKED LOW, and the reason is that the consequence is bounded:** the hub bands host storage on `disk_percent`, so a failed read presents as 0% used — the *quiet* direction. It cannot raise a false "nearly full" alarm; it can only fail to raise a true one, and only while the root filesystem is unreadable, which is a state with louder symptoms of its own. **Fix shape when it is taken:** carry `disk_known` on the storage entry and have the hub's fill checker skip an unknown reading rather than band it — never treat absent as 0 | **READY** — owner Viktor | -| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.0–0.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.20–3.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 7–11 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"* — **NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **READY** — owner Viktor | +| **R-267** | **The Configuration page is 2.6× faster and is still ~10 s, and the remaining cost is ONE Gitea call whose latency swings 20× with load.** Reported by the operator as *"almost minutes to load"*, with the reasonable guess that it hashes artifacts on page load. **THAT GUESS DOES NOT HOLD and the code already said so** — Gitea stores each file's sha256 and `gitea.FileSHA256` reads it as metadata (*"the artifact bytes are never downloaded"*). The cost was latency × count. **Fixed in hub v0.100.0–0.100.2, three legs, each found by refusing to accept a number that did not match the arithmetic:** (1) the per-version sha lookups were **serial** — 2 packages × (1 search + 20 lookups) = 42 sequential calls; now concurrent, bounded 8. (2) The two dropdowns resolved **one after the other**; now side by side. (3) The client used `http.DefaultTransport`, whose **`MaxIdleConnsPerHost` is 2**, so under a 16-way fan-out nearly every call paid a fresh TCP setup *and* a fresh authentication — and authentication is the expensive half (`/api/v1/version` unauthenticated **0.03 s** vs an authenticated package call **0.24 s**). **Measured: 26.2 s → mean 9.85 s over 8 samples (min 5.13, max 18.13).** **THE REMAINING COST IS THE PACKAGE SEARCH** `/api/v1/packages/admin?type=generic&q=…&limit=100`, one per dropdown: **0.20–3.8 s each depending on load**, and running the two concurrently does not help (3.68 s for both together vs 3.82 s for one alone — Gitea appears to serialise them). By contrast **16 concurrent file-metadata calls take 0.58 s**. **⚠ EVERY NUMBER HERE IS CONTAMINATED and that is stated rather than hidden:** they were taken on DooPlex at load average 7–11 while this same session was building images, running two Go suites and baking a golden. The same search measured **3.8 s** in-cluster and **0.44 s** from the host ninety seconds later. **Re-measure on an idle box before deciding anything.** **The operator's two proposals, answered on the measurement:** *"reduce the number of artifacts"* — only **50 generic versions exist in total** (33 agent + 17 golden) and `limit=25` costs 0.20 s against `limit=100`'s 0.44 s, so pruning helps **somewhat and sub-linearly**; it is worth doing for its own sake (it pairs with R-210) but it is not the lever. *"hash on creation, store in the DB"* — **NOT RECOMMENDED, and the reason is a rule this project already holds**: Gitea IS the store, and a copy in `hub_settings` would be a second source of truth that can drift from the registry it describes, while the operator reads exactly that value to confirm what they are about to vouch. `golden_currency_gate.py` records the same reasoning for the vouched version. **The lever that would actually work, and the trade-off that makes it a DECISION rather than an implementation:** cache the version list + shas **in memory** with a short TTL and refresh in the background — the page becomes instant and bounded-stale, at the cost that a just-published artifact does not appear for up to the TTL. That is an operator call about how fresh the dropdown must be, and it is why this row is open rather than closed. **Cheaper interim, also a decision:** drop the dropdown cap from 20 to ~8, which cuts the fan-out but hides older versions from a rollback | **CLOSED 2026-08-08 — hub v0.101.0.** Memoised 60 s in memory (operator ruling). **Measured after: cold 13.4 s, warm 0.24–0.33 s** — the operator sees ~0.25 s except at most once a minute. The three serialisation legs (v0.100.0–0.100.2) took 26.2 s → 9.85 s mean; the memo takes the warm path to a quarter-second. NOT persisted — see the CHANGELOG for why a copy in `hub_settings` would be a second source of truth. Artifact pruning is tracked separately below | + +| **R-268** | **A live per-guest local-API token was printed into a session transcript.** Done by me on 2026-08-08 while setting up R-221's live drill: a `python3` one-liner meant to list `bootstrap.json`'s keys printed the `local_api` object whole, including its `token`, for guest **9201 on demo-felhom**. **Reported immediately rather than quietly rotated**, because this project's rule is that a secret reaching a transcript is a finding whatever its blast radius. **THE EXPOSURE, ASSESSED RATHER THAN ASSUMED — it is small, and saying so is not the same as excusing it.** The token authorises only the agent's **per-guest** local API on `169.254.253.1:8443`, which listens on the island bridge `vmbr9` — an L2 segment between the PVE host and that one guest. It is not routable from the LAN or the internet, it is self-scoped to guest 9201 (the agent authorises against the token's own guest, never a caller-supplied id), and using it already requires code execution on that host or guest, at which point an attacker has strictly more than the token. The box is **Tier 0, disposable, no customer data**. **ROTATION IS AVAILABLE BUT IS NOT A ONE-LINER, which is why it was not done unannounced:** `localapi.TokenStore.Mint(vmid)` is last-write-wins per VMID (`tokenstore.go:127`, the store keeps only hashes), but the new plaintext must also be written into the guest's `/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access — so it is an operator-timed act, not a background one. **The honest general fix is upstream of the incident:** reading a secret-bearing JSON should go through a helper that prints keys and never values, the same discipline the bake uses for the Gitea token (file → file, `grep -c` on the shape, never the value). Two of this fortnight's findings (R-249, R-132) are the same class from the other direction | **READY** — owner Viktor | **Explicitly still open, untouched by this session:** R-246, R-255, R-256, R-257, R-261, R-262, R-263, **R-264** (the twenty-one undecided facts — a design session of its own), R-240, R-243,