admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338)
gates / gates (push) Failing after 12s
Details

Browse Source 
Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one
incident, and not on either customer box: ep0's proxmox-backup-proxy was
active, holding its listening socket, and accepting nothing.

Root cause: accept() returning EMFILE. The process held exactly 1024 fds
-- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets
and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had
overflowed (Recv-Q 1025), so every client timed out. It was wedged from
its own loopback too, which is what moved this from a network problem to
a process problem.

Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written
to weekly, that leak reached the ceiling in 14 days of uptime.

Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from
both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed
backups through the product path -- POST /backup?target=felhom-pbs on each
agent's local API, not a hand-run vzdump.

  demo-felhom  ct/9201/2026-08-18T03:57:43Z  4.10 GB  36.4s
  demo-hp      ct/9201/2026-08-18T03:58:43Z  4.29 GB  41.5s

Both host reports now carry felhom-pbs success=true, so the hub is green on
the evidence rather than on a restart having been performed. No data lost,
no backup skipped: the daily local tier was never affected and the PBS tier
is weekly, so the window cost exactly one attempt.

Evidence copied off ep0 BEFORE the restart, per standing rule 5.

Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling
is mitigation, not a cure), R-337 (a status endpoint that trailed its own
artifact by minutes then caught up -- WATCHING, downgraded from the defect
I first wrote, because it self-corrected), R-338 (demo-hp is not on the
R-50 island at all and nodes.md says it is; its local API is bound to the
customer LAN).

R-334 updated: still open, now one version wider (controller 0.216.0 vs
golden 0.214.0). golden-currency is the only failing gate and is inherited
-- it reads files this session did not touch -- so this push used
--no-verify, stated per .claude/rules/gates.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN
This commit is contained in:
authentik Default Admin
2026-08-18 06:09:38 +02:00
parent ea16a21bff
commit ebfd0967c1
7 changed files with 559 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+103 -337
View File
@@ -1,353 +1,119 @@
# REPORT — the third near-homograph, a machine told to be quiet, and a picture you can query (2026-08-13, late)
# REPORT — a listening socket that served nobody (2026-08-18, early)
**Shipped: hub v0.105.0, deployed and verified.** All three parts complete; **nothing dropped**.
**`demo-hp` was not touched** — see §8. No controller change, no agent change, no wire change, so no
bake and no approval.
**Trigger:** two `whole_guest_backup_failed` alert mails, 04:30 and 04:32 CEST.
**Outcome:** root cause found on **ep0**, fixed, both missed backups re-driven and landed.
**No repo code changed** — this was an operational run. Documentation, register and evidence only.
---
## 1. Part 1 — the enumeration, the name, and the reasoning
## 1. What was wrong
**Enumerated before editing. Five customer-facing sites, all in the hub:**
Both alerts were the same incident and neither was on a customer box. `demo-felhom` and `demo-hp`
each failed their `felhom-pbs` tier with `Can't connect to 10.77.0.1:8007 (Connection timed out)` —
the one thing they share, the Hetzner offsite PBS endpoint.
| `file:line` | what it is |
|---|---|
| `hub/internal/web/selfbind.go:255` | the binding page's lead sentence |
| `hub/internal/web/selfbind.go:256` | the failure banner (said „a jelszót" — generic, but naming the secret) |
| `hub/internal/web/selfbind.go:261` | the field label — „Visszaállító jelszó" |
| `hub/internal/web/selfbind.go:263` | the hint under the field |
| `hub/internal/notify/templates.go:331` | the self-bind e-mail, item 2 |
**No halt. The name appears nowhere in `felhom-controller` or `felhom-agent`** — the only hits there
are comments and a test asserting the *already-retired* „Visszaállító **kód**" is absent, which is a
different secret. Operator surfaces call it *"Retrieval Password"* in English and the installer uses
`FELHOM_RETRIEVAL_PASSPHRASE` as a shell identifier; neither is customer copy and neither was touched.
### The name: **„Tulajdonosi jelmondat"**
Checked against the table rather than against a habit. The phrase **proves the account owns the box
being bound — it restores nothing, so the old name was simply false.**
**Both of your suggestions are rejected, and the reasons are the argument:**
- **„Fiókjelszó" is worse than the trap it fixes.** There *is* an account password — the dashboard
login. This name would collide with a **different real secret**, trading one homograph for a worse
one.
- **„Összekötési jelszó" recreates the trap structurally.** The other factor on this very page and in
the same mail is the **„Párosító kód"** (`selfbind.go:258`, `templates.go:330`). Naming this one
after the same act would leave the two factors a customer types **in one sitting** separated only by
*kód*-versus-*jelszó* — which is precisely the „Visszaállító kód"/„Visszaállító jelszó" shape being
removed. It matches the page's verb, and that is exactly the problem.
**„Tulajdonosi jelmondat" is distinct on BOTH axes:**
| | stem | noun |
|---|---|---|
| „**Beállító** kód" | Beállító | kód |
| „**Helyreállítási** kód" | Helyreállítási | kód |
| „**Párosító** kód" (the other factor) | Párosító | kód |
| „**Tulajdonosi** jelmondat" | **Tulajdonosi** | **jelmondat** |
*If a plainer noun is ever wanted, „Tulajdonosi jelszó" is a one-word change — the **stem** is what
carries the separation.*
### As shipped, bytes confirmed
| string | hex |
|---|---|
| `Tulajdonosi jelmondat` | `54756c616a646f6e6f7369206a656c6d6f6e646174` |
| `tulajdonosi jelmondatodat` | `74756c616a646f6e6f7369206a656c6d6f6e6461746f646174` |
| `tulajdonosi jelmondatot` | `74756c616a646f6e6f7369206a656c6d6f6e6461746f74` |
| `Ez igazolja, hogy a fiók a tiéd` | `457a206967617a6f6c6a612c20686f67792061206669c3b36b2061207469c3a964` |
Mojibake check clean in both files. **Naming only:** the form field is still `name="passphrase"`, and
`TestSelfBindPassphrase_StillAcceptedAfterTheRename` drives the real handler with the same messy human
spacing (`" Alpha Beta gamma-delta epsilon "`) and asserts the appliance still binds. The whole
pre-existing self-bind suite (A, B, C1, C4, D, E, F, mint × 2) stayed green.
---
## 2. Does any customer-facing document name the old phrase? **No.**
Checked: `documentation/pilot/PETI-tester-agreement.md` does not mention the phrase **at all**;
`RUNBOOK-onboarding-draft-v4.md` and `RUNBOOK-byo-deployment.md` name it by its **English operator**
name ("retrieval passphrase"). The ISO/installer references are shell identifiers.
**So nothing printed is stranded.** The one honest caveat: the phrase reaches a customer
**out-of-band** — `RUNBOOK-byo-deployment.md:11`, *"delivered to the box owner over a secure channel"*
— so the only stale copy of the old name is whatever was said in a message or on the telephone. A
person holding such a message would see „Tulajdonosi jelmondat" on the page, be told *"az öt szóból
álló kifejezés, amelyet a beállításkor kaptál"*, and be holding exactly that: **the description
identifies the thing even where the name has moved.** That was the reason for keeping the hint
sentence and extending it with *"Ez igazolja, hogy a fiók a tiéd."*
---
## 3. Part 1b — the hub guard, and the control that found a bug in itself
`scripts/hub_copy_gate.py`, registered as gate 9 in `repo_gates.py`. **Two checks, deliberately
different:**
1. **Retired names — banned outright**, across **all 95** hub `.go`/`.html` files, no allowlist. A
name a different secret now owns is never correct anywhere. Comments are stripped, because the
register rows and the code comments that record these decisions must quote the retired names.
2. **Retrieval stems — registered, not banned**, in four declared customer surfaces
(`notify/templates.go`, `web/selfbind.go`, `api/handler.go`, `notify/dispatcher.go`). A missing
declared surface is a **FAILURE, never a skip**. The allowlist is **empty, and that is a
measurement**: the hub makes no retrieval promise today.
### plant → convict → remove → pass
On ep0, `proxmox-backup-proxy` was `active`, held its listening socket, and **served nobody**:
```
hub-copy gate SELFTEST
1. clean tree : 0 conviction(s) OK
2. planted „visszaállító jelszavadat”: CONVICTED (isszaállító jelsz) OK
3. same phrase inside a comment : not convicted OK
4. planting removed : 0 conviction(s) OK
hub-copy gate selftest OK — the guard has been watched catching, ignoring and releasing
ss -lnt '( sport = :8007 )' → LISTEN Recv-Q 1025 Send-Q 1024
ls /proc/<proxy>/fd | wc -l → 1024 # == its soft RLIMIT_NOFILE
```
**The control found a defect in its own instrument on the first run.** Step 3 convicted a comment,
because the synthetic source was named `<selftest>` and comment-stripping keys off the `.go`
extension. **The bug was in the guard, and the control is what found it** — which is the entire
argument for insisting a guard be watched working.
`Send-Q` on a listener is the accept backlog; `Recv-Q` is the queue depth. At 1025 against 1024 the
queue had overflowed, because `accept()` was returning `EMFILE` on every call. **1016 of the 1024
descriptors were sockets and 547 connections sat in `CLOSE-WAIT`** — a connection leak, fed by
~85,000 requests/day, that reached the ceiling after 14 days of uptime. Last request served:
2026-08-17 18:15:32 UTC. Offsite DR was therefore down **≈ 9 h 37 m**.
### One list, not two — and the gap is instrumented rather than hidden
**The observation that settled it:** the daemon was wedged from its own loopback too —
`curl https://127.0.0.1:8007/` on ep0 timed out. A listener that cannot serve `127.0.0.1` has no
network left to blame, and that check is cheap enough to make early.
Both lists live in **`scripts/customer_copy_vocab.py`**, the same shared-gate home
(`reuse_refs_check.py`, `instructions_gate.py`) that both repos already consume without copying.
Full record, including everything that was ruled out first (tunnel, nftables, disk, dead daemon):
**`documentation/audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md`**.
**`retrieval_promise_gate.py` has NOT adopted it**, because the end state forbade touching
`felhom-controller` tonight. So rather than ship two copies that drift, **this gate reads the
controller gate's `STEMS` and fails if they disagree** — watched failing:
## 2. What was done
```
DRIFT: the controller gate's STEMS have diverged from customer_copy_vocab.py
controller : ['visszaállíthat', 'visszaszerezhet', 'visszahozhat', 'visszanyit']
shared : ['visszaállíthat', 'visszaszerezhet', 'visszahozhat']
HUB-COPY GATE FAILED: the shared vocabulary is no longer shared.
1. `LimitNOFILE=65536` drop-ins for `proxmox-backup-proxy.service` and `proxmox-backup.service`, each
carrying its reason inline. **The API daemon was not implicated** (15 fds) and its drop-in says so
— a later reader must not mistake it for a second culprit.
2. Restarted both. After: soft limit 65536, fds back to 18, `Recv-Q 0`, loopback `200`.
3. **Verified from the customer side, not only from ep0** — both boxes got `200` in ~0.1 s and
`pvesm status` read `felhom-pbs pbs active`.
4. Re-drove the missed backups **through the product path** — `POST /backup?target=felhom-pbs` on
each agent's local API, issued from inside the guest's controller container with the controller's
own credentials, i.e. the same call the scheduler makes. Not a hand-run `vzdump`.
**Result:** `demo-felhom` → `ct/9201/2026-08-18T03:57:43Z` (4.10 GB, 36.4 s);
`demo-hp` → `ct/9201/2026-08-18T03:58:43Z` (4.29 GB, 41.5 s). Both directories carry a full manifest
on ep0, and both hosts log the re-run vzdump as `OK`. **And the hub agrees** — both boxes' next host
reports carry `felhom-pbs success=true` (04:00:33Z and 04:07:35Z), so the operator view went green on
the evidence rather than on the fact that a restart was performed.
**No data was lost and no backup was skipped** — the daily local
tier was never affected (it completed on both boxes at 05:00 and 05:02), and the PBS tier is weekly,
so the outage window cost exactly one attempt, which was re-driven the same morning.
Evidence copied off ep0 **before** the restart, per standing rule 5:
`documentation/audits/evidence-ep0-fd-2026-08-18/` — pre-restart state, post-fix state, access-log tail.
## 3. What I got wrong, and corrected
- **`proxmox-backup-manager version` prints *available* then *running*.** It read
`4.2.5-1 running version: 4.2.2`, which looks exactly like a daemon left behind by a package
upgrade. It is not: `dpkg -l` shows **4.2.2-1 installed**, 4.2.5-1 merely available in the repo,
and the on-disk binary is dated 2026-06-18. I restarted the API daemon on that mistaken reading;
harmless, and its drop-in is a genuine improvement, but it was not needed.
- **I addressed `demo-hp`'s agent at the island address** because `operations/nodes.md` says that box
is island-migrated. It is not (**R-338**), and the resulting timeout was briefly read as a fault.
## 4. Findings filed — R-336, R-337, R-338
All three are in `documentation/backlog/OPEN-ITEMS.md` with numbers, per the registers-first rule.
- **R-336 — the poll rate is the real defect.** ~1 request/second against a DR endpoint written to
weekly. `LimitNOFILE` raises the ceiling; **it does not fix the leak**, it converts a fortnightly
outage into a multi-year one.
- **R-337 — a status endpoint that trailed its own artifact, then caught up. WATCHING, not a defect.**
`demo-hp`'s `/backup/status` was still serving the superseded 03:27:00Z failure at ~04:03Z while the
snapshot sat on ep0 and the host logged `OK`; `demo-felhom` updated within ~40 s. **I filed this as
a defect and that was premature** — the next host report (04:07:35Z) carried the success and the
skew cleared with no intervention. Rewritten as WATCHING, with the explicit instruction not to open
a fix until someone establishes whether this is just collection cadence. Recorded at all because
during the recovery it read as a second failure, and it was not one.
- **R-338 — `demo-hp` is not on the R-50 island and `nodes.md` says it is.** No `island_bridge` keys,
guest has no `eth1`, `vmbr9` has zero members, and the agent's local API is bound to the customer
LAN — the exposure R-50 existed to remove.
**Not done, deliberately:** PBS 4.2.5-1 was not applied. Upgrading a production offsite endpoint was
outside what this run was authorised to do, and its changelog should be read for the connection-
handling leak first.
## 5. Gates — one pre-existing conviction, and a stated bypass
`python3 scripts/repo_gates.py --fast` → **rc=1, CONVICTED: golden-currency.** Eight of nine gates
pass. The conviction is **R-334, inherited and not caused here**: newest released controller
**0.216.0**, newest golden bake **0.214.0**, so a new install misses two releases. The gate reads
`felhom-controller/CHANGELOG.md` and `documentation/tests/golden-*` — **this session touched
neither**, and its whole diff is documentation. Baking is possible; **vouching is
operator-password-gated, and a baked-but-unvouched golden is worse than none**, so it is not a
one-sided job CC can finish.
**This push therefore used `git push --no-verify`, stated here per `.claude/rules/gates.md`.**
R-334 is updated in the register with the new numbers rather than left reading 0.215.0.
*(Noted for accuracy: the first gate run was piped to `tail`, which returned `rc=0` — `tail`'s exit
code, not the gate's. It was re-run unpiped to read the real `rc=1`. That is standing rule 1's trap
in its smaller form, and the number reported above is the unpiped one.)*
## 6. What to watch
The positive observable is the descriptor count, not the absence of an alert — an empty alert queue
is equally consistent with "healthy" and "wedged again":
```bash
ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); \
ls /proc/$PID/fd | wc -l; ss -lnt "( sport = :8007 )"'
```
…and green again when restored. An **absent** sibling clone is **INCONCLUSIVE (exit 2), never a pass**
— the G-1 lesson. **This is a scaffold, not the destination: R-325** is the few-line felhom-controller
change that makes it import the shared list and delete both its literal and this drift check.
---
## 4. Part 2 — the four scenarios, the clock, and the red-proofs
**Re-established at `file:line`, not taken from the prompt.** The chain end to end: controller
`cmd/controller/main.go:1253` sets `Health.Status = "disabled"` in the final minimal report → hub
`store.go:953-955,968-975` parses `health.status` into `reports.health_status` →
`CustomerSummary.HealthStatus` (`store.go:40`) carries it into `GetCustomers()` →
`web/rollup.go:25` renders `disabled` → **`monitor/staleness.go` ignored it and measured age alone.**
**The halt condition did not fire:** the discriminator is the box's **own last word**, present in the
data this checker already reads. That is not a guess.
### It was TWO doors, not one
Because the state is **recorded** (`StateDisabled`) rather than **deleted** (as the `blocked`
precedent does), `CheckBackupDeadlines` can skip it too. A deleted state returns `""` from `GetState`,
and `""` is not `"down"` — so that check would have gone on e-mailing `expected_backup_missed` every
morning about the same machine. **R-195's shape returning through a second door**, and it is closed
with the first. Recording the state also satisfies the visibility requirement: quiet-on-purpose and
quiet-by-accident no longer look identical.
### The re-enablement judgement
**The clock runs from the report the hub can actually see.** For a box that reports on re-enabling,
that report *is* the re-enablement, so the clock starts there — your recommendation, and the mechanism
already had this shape. Timing from the last report *before* the switch-off would fire an instant
stale/down for a quiet period we asked for: a false alarm produced by fixing false alarms. Leaving
`disabled` re-enters the **same branch as a new customer**, so no `node_recovered` fires for an outage
that never happened. `downtimeStart` is cleared **on entry**, so a later genuine outage cannot compute
its duration from a clock that started before the silence was requested.
**LIMIT, stated rather than hidden:** a box re-enabled that then **fails to report at all** keeps being
suppressed — the hub sees only that final `disabled` report, and its view changes only when a report
arrives. It cannot distinguish that from *still switched off*. **This is exactly why the state is made
visible:** an operator who re-enabled a box and still sees `disabled` is being told it has not come back.
### Scenarios
| | outcome | verified |
|---|---|---|
| **A** deliberately silent for days | no stale, no down, no e-mail; state visibly `disabled` | `TestStaleness_A` — observed healthy **first**, then switched off, then three passes |
| **B** simply stopped reporting | unchanged: stale then down, exactly as today | `TestStaleness_B` — state `down`, events emitted |
| **C** re-enabled, reports promptly | clean transition, **no** recovery event | `TestStaleness_C` — zero events, state `ok` |
| **D** re-enabled then genuinely quiet | stale and down fire normally, timed from re-enablement | `TestStaleness_D` — state `down`, events emitted |
Plus `TestStaleness_DisabledIsAlsoSkippedByTheDeadlineCheck` for the second door.
### Red-proofs — every mutation asserted applied by grep, and reverted after
| # | Mutation | Asserted applied | Outcome |
|---|---|---|---|
| **1** *(the one that matters)* | suppression made **unconditional** (`if true \|\| …`) | `staleness.go:132` grep | **RED — B**: *"a genuinely silent machine is `"disabled"`, want `"down"` — a real alarm was swallowed"*. **A genuinely dead machine was seen NOT alarming.** C and D also red |
| **2** | suppression **removed** (`if false && …`) | `staleness.go:132` grep | **RED — A**: *"a deliberately-disabled machine emitted `[node_down]` — three days quiet BY REQUEST"*. Today's false alarm, reproduced verbatim |
| **3** | state made **sticky** — remembered instead of re-read from the box, so the clock never leaves the pre-disable report | `staleness.go:132` grep | **RED — D**: *"a once-disabled machine was silenced for ever"*. C also red |
**A weakness in my own tests was found by red-proof 2 and fixed.** Scenario A originally seeded the
customer *already* disabled — and the checker's new-customer branch sets the first state without an
event, so **the test passed on its state assertion alone even with the suppression deleted.** It now
observes the machine healthy first, and the same mutation then fails on the **event**. A test that
cannot see the alarm it exists to prevent is not a test.
---
## 5. Part 3 — the real counts, and the number that was wrong
**Yes, a number I have been repeating is wrong.**
```
where felhom stands — 55 claims, verified_on 2026-08-09
walked 23
partial 14 (6 cite evidence, 8 prose only)
built 14 (0 cite evidence, 14 prose only)
missing 4 (0 cite evidence, 4 prose only)
NOT WALKED: 32 of 55
```
**"Nine" is real, and it answers a different question: it is the count of claims carrying
`verdict: downgraded`** — the ones the 2026-08-09 verification pass **lowered**. That is "re-judged",
not "unproven". **Yesterday's session was right to refuse to guess**, and this is what it could not
have found without counting.
The sharper cut is the evidence one: **all 23 walked claims cite an evidence document** (`check_stands.py`
convicts a `walked` claim without one), while of the **32** that are not walked, **only 6 cite evidence
and 26 are prose only**.
---
## 6. The command, and its output
```
$ python3 scripts/unproven.py --summary
where felhom stands — 55 claims, verified_on 2026-08-09
walked 23
partial 14 (6 cite evidence, 8 prose only)
built 14 (0 cite evidence, 14 prose only)
missing 4 (0 cite evidence, 4 prose only)
NOT WALKED: 32 of 55
```
Without `--summary` it prints every not-walked claim with its id, band, verdict and whether it cites
evidence, grouped `partial → built → missing`. It reads the dataset **only** — opens no evidence,
judges nothing, contacts no machine. An unrecognised status prints with a `⚠ status not known to this
script` marker rather than being silently dropped.
**Wired into the end-of-session checklist** in `felhom.eu/CLAUDE.md`, with the instruction to say in
the report if a number moved.
**Its first run found a stale claim** — `claim.code-naming` is still `partial` and its title still
describes the defect R-295 and R-323 have now closed twice over. **I did not move it**, because the
dataset's own header forbids it: *"A status may not be RAISED here — the MAP changes first and this
file follows it."* Filed as **R-327**, with the honest difficulty noted: no customer has typed
„Tulajdonosi jelmondat" yet, so `walked` would be an over-claim.
**The capability map is deliberately NOT restructured** — recorded in R-326 so it does not read as
forgotten.
---
## 7. Versions, manifest, CI
| | |
|---|---|
| **Shipped** | **hub v0.105.0** (`gitea.dooplex.hu/admin/felhom-hub:0.105.0`) |
| **Unchanged** | controller **0.214.0**, agent **0.129.0**, installer **1.28.0**, golden 0.214.0, floor 0.214.0, MinAgent 0.129.0 |
| **Commits** | `b03a105` the work · `bbd59f4` the manifest bump |
| **Manifest** | `manifests/hub.yaml:128` → `0.105.0`, GitOps only — **no `kubectl set image`** |
| **Deploy** | ArgoCD hard-refresh + sync → **Synced / Healthy**; pod `hub-5ff87b556b-bvtvt` **1/1 Running**; deployment image confirmed `0.105.0` |
| **CI, by run ID** | run **336** (`b03a105375`) **success** · run **337** (`bbd59f4a44`) **success** |
| **Gates** | `repo_gates.py --fast` — all **9** OK (the new `hub-copy` included), and again in the pre-push hook on both pushes. **No `--no-verify`** |
| **Green gate** | `hub/`: build rc=0, vet rc=0, **test rc=0 across 18 packages**, run separately from every commit |
**Live verification of the running artifact**, ASCII-only patterns against
`/usr/local/bin/felhom-hub` in the running pod:
```
Tulajdonosi jelmondat 1 ← the field label
tulajdonosi jelmondatodat 2 ← the page lead + the mail
jelszavadat 0 ← the retired possessive is GONE
```
The negative control is the point: the old copy is **absent**, not merely accompanied.
**Method, stated: this proves the deployed artifact carries the new copy — it is not a rendering.**
Rendering the binding page requires minting a self-bind token, which mints a capability URL and sends
an e-mail against a real customer record; with `demo-hp` being re-deployed tonight I would not do that
to any customer row. The *rendering* is covered by `TestSelfBind_ThirdSecretNaming` and
`TestSelfBind_FailureBannerUsesTheSameName`, which drive the real handler through `ServeHTTP`.
**Part 2 has no live observable tonight, and I will not manufacture one.** No machine is in the
disabled state, and producing one would mean switching a real box's reporting off — precisely the kind
of thing that must not happen to a fleet of two on the evening one of them is being re-deployed. The
fix is proven by four scenarios and three red-proofs and is **latent until a box is actually disabled**.
---
## 8. `demo-hp` was not touched
**No `ssh`, no `pct`, no controller call, no agent call, no hub write of any kind** was issued against
`demo-hp` or its records this session. Everything done here was in the `felhom.eu` repo, the hub image
and the hub deployment.
Confirmed read-only, via the hub's own hosts page (`GET /hosts`, HTTP 200):
```
demo-hp-bb76ea ONLINE
demo-felhom-8363b5 ONLINE
drill-r50-0a4f9a DOWN (reverted to `virgin`, powered off — expected)
```
`demo-hp` is **online and reporting normally**, exactly as it was found. **`felhom-agent` and
`felhom-controller` are byte-unchanged** — neither repo was written to.
---
## 9. Register
**Ceiling moved R-322 → R-327** (grepped before minting).
| Row | State |
|---|---|
| **R-323** third near-homograph → „Tulajdonosi jelmondat" | **CLOSED — shipped** |
| **R-324** the hub's customer copy under a guard | **CLOSED — shipped, selftest green** |
| **R-325** controller gate should import the shared vocabulary | **READY (S)** — the drift check is the scaffold |
| **R-326** "what is unproven" made queryable | **CLOSED — shipped** |
| **R-327** the picture still describes a fixed defect | **READY (S)** — map moves first |
| **R-321** a disabled machine alarmed as dead | **CLOSED — both doors** |
| **R-322** the guard had never scanned the hub | **CLOSED by R-324** |
---
## 10. What was dropped, and observations
**Nothing was dropped.** Part 1, Part 1b, Part 2 and Part 3 are all complete. Part 3 was the
designated first drop and was not needed.
### Observations — noticed, not acted on
- **I nearly published a false negative about my own deploy.** The first grep of the running binary
used accented patterns through `kubectl exec → sh -c` and returned **0 for every string, including
ones that are certainly present** — which reads exactly like "the rename did not deploy". The
standing rule (*never let an accented pattern gate a conclusion*) is what caught it; the ASCII-only
re-run gave the real answer. **The rule earns its place again, in a chain it was not written for** —
it was written for `ssh → pct exec`, and `kubectl exec` mangles identically.
- **The `blocked` branch does not clear `downtimeStart`.** The new `disabled` branch does. So a
customer that is blocked, later unblocked and later still goes down will compute its downtime from a
clock that started before the block. Small, pre-existing, and not changed under a row about
something else.
- **`gofmt -l internal/` lists 22 pre-existing unformatted files** in the hub, none of them touched by
this session. Worth one cleanup commit by someone, sometime — it makes `gofmt -l` useless as a
check, which is how a real formatting problem would hide.
- **The empty allowlist in the new gate is load-bearing and fragile in one direction.** It is empty
because the hub genuinely makes no retrieval promise. The moment someone adds one legitimately, they
must register it — and the gate's failure message says so, but nobody reads a failure message until
they hit it.
- **`unproven.py` shows 26 of 32 not-walked claims are prose only.** That is not a defect per claim —
a `missing` claim has nothing to cite — but **14 of 14 `built` claims cite no evidence at all**, and
"built" is the status that most invites being read as "done". Worth a look when the capability-map
session happens.
Healthy is ~20 fds and `Recv-Q 0`. **A count climbing between restarts means R-336's leak is still
live.**