hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes

oobDegraded tested five things and the sixth never arrived.

The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version
that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's
eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with
felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR
KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking.
`operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that
OOB is MEANT to work, not that entry is possible.

Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the
ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now
arriving is how a check stops being read.

SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a
POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was
rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza
without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched
floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this
project has been burned by.

THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so
the alert can never name a different fault from the one that fired. The old form derived it
separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key
is missing. The operator reads this at 07:00.

TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a
test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite
for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran
against a report shape no released agent produces (same family as R-262). Both fixed.

Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the
field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the
silent pass.

G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen
failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument
defects the control caught first: a substring false negative (grep -F healed_at matched
privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded
through an allow-list that already cost offsite_restic (R-122).

The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo:
R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40.

R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264
MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class
could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263,
and C7's test-comment half.

Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue;
what was untrue sat one layer down and the row now records it.

repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
This commit is contained in:
2026-08-08 08:47:02 +02:00
parent 560f0d4451
commit b080ecf411
13 changed files with 836 additions and 58 deletions
+42
View File
@@ -17,6 +17,48 @@
## Standing rulings
**S-38 — A FACT ONE SIDE EMITS AND THE OTHER CANNOT RECEIVE IS A DEFECT, AND A CHECK NOW SAYS SO
(2026-08-08, G-1 / R-260 / R-247).**
`scripts/wire_contract_gate.py`, `--fast`, registered in `repo_gates.py`. For every json tag reachable
from a declared wire ROOT, it asserts that the literal tag occurs somewhere in the receiving repo's
production Go or templates. **A tag occurring nowhere cannot be decoded by any struct, named or
anonymous** — which is why it is a string test and not a struct-to-struct comparison: the hub decodes
one report through several ad-hoc anonymous structs, and shape-pairing false-positived badly when
Campaign 12 tried it.
1. **Adding a field to a wire is a TWO-REPO change**, like the DR-recipe coupling rule. Emitting it is
half the work; the gate refuses the other half being skipped.
2. **A field that genuinely has no receiver goes in the gate's `ALLOWLIST` WITH A REASON.** A quiet
exclusion is a dropped field with paperwork. Two kinds of entry exist and the difference is
deliberate: *redundant* (the hub already decodes an equivalent — the matter is closed) and *R-264*
(no consumer today and one is arguably owed — **allowlisting is not deciding**).
3. **The roots are declared, and an unresolvable root is INCONCLUSIVE, never a pass.** Not covered,
and the gate says so in its own output: the hub's desired-state (served as raw stored
`host.DesiredJSON`, no typed emitter to walk) and the agent's local API (no single root type).
4. **`dr_recipe` is opaque BELOW depth 1, not opaque.** The hub stores each half as `json.RawMessage`
and re-emits nested shapes verbatim, so the leaves are genuinely off the wire — but the TOP-LEVEL
section keys are decoded by `hostHalfShape` / `appHalfShape`, and those are allow-lists that
already silently swallowed `offsite_restic` for months (R-122). The sections are checked.
**Scenario F's choice, and why (the same ruling's second half).** When the receiving side gains a
field, the absence of that field from an older emitter must not read as a value. `operator_key_configured`
decodes as a **pointer**: nil means the agent never said, and is reported **distinctly** from a
reported `false` — never as `ok`. Three options were on the table (unknown = degraded / unknown =
its own state / gate on the agent version); the version gate was rejected **on a measurement, not a
preference**: the field and the `oob` stanza that carries it shipped in the SAME agent version
(v0.72.0, 2026-07-05), so a stanza without the field cannot come from any released agent, the live
fleet is on 0.113.0 and 0.127.0, and the vouched floor is 0.127.0. Building version-gating machinery
the hub does not otherwise have, to handle a case no box can be in, would have been cost without
cover. The unreachable case is handled explicitly anyway — "cannot happen" is the kind of claim this
project has been burned by — and is pinned by `TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd`.
**And the test-shape lesson underneath it.** Every hub OOB test before this built a `HostOOBRow` by
hand. **A test written that way cannot see a field that never decodes**, which is how the defect held
a green suite for five weeks; the pre-existing fixture `oobReport()` also omitted the field, so those
scenarios ran against a report shape no released agent produces. Tests for a wire contract drive the
**decode boundary** with raw JSON.
**DooPlex's own backup, surveyed 2026-08-06 (R-232, read-only).** Five sets nightly, 14/14 in 14
days, single-file restore proven byte-identical. Destination for **every** set is `/dev/sda1`;
`.claude-memory` and the PostgreSQL dumps are the two same-disk cases. **Nothing leaves the box**
+163
View File
@@ -0,0 +1,163 @@
# REPORT — G-1: a gate for the dropped field, then the fields it found (2026-08-08)
*A non-overwritten `REPORT-<topic>.md` sibling, per `CLAUDE.md:82-87` — a parallel session shares this
clone and the shared `REPORT.md` was not touched.*
## 1. The gate's output on today's tree — failing, before anything was fixed
**This is the session.** Captured verbatim in
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`:
```
wire-contract gate — 210 tag(s) checked across 3 declared wire(s); 51 skipped
WIRE-CONTRACT GATE FAILED: 40 emitted field(s) cannot be received.
```
It named every one, with its emit path and its direction, and re-found **`escrow_stale`** (R-247) and
every field R-260 listed. Had it been green, the gate would not work and *that* would have been the
finding — which is not hypothetical: the night before, `deadcode` was rejected for the neighbouring
C6 class for exactly that reason.
**⚠ A count this session's prompt got wrong.** The prompt said *"465 emitted tags, eight
unreachable"*. R-260's wording was "at least eight **decision-bearing** facts", never eight tags in
total. Measured: **40** on the three declared wires. Checked against the repo, not quoted — the
prompt's own rule 6, and the second prompt claim caught that way this week.
## 2. The forty, by disposition
| # | field(s) | direction | decision | what changed |
|---|---|---|---|---|
| 1 | `oob.operator_key_configured` | agent → hub | **receive and act** | decoded (pointer); `oobDegraded` fails on a missing key and the alert names it |
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **receive, message only** | in `HostOOBRow` + the event payload; deliberately NOT in the predicate |
| 3 | `escrow_stale` | hub → controller | **receive and act** | `report.EscrowStatus.Stale`; withheld-hash told apart from hash-less. **R-247** |
| 4 | 12 host/system metric fields | both → hub | **no consumer wanted — redundant** | allowlisted: the hub bands on the `*_percent` figures from the same stanzas |
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **redundant** | sizing is hub-owned intent, not mirrored reality |
| 6 | `storage_targets.smart.model_name` | agent → hub | **redundant** | a display label; `smart.health` + every banded counter ARE decoded |
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **redundant** | wgsync reconciles from its own state |
| 8 | 21 fields (`guest_net`+7, `selfupdate_pending`+1, `healed_recently`, `applied_at`, `mount_parity`/`_inventory`, `config_hash`, `reporting_disabled`, `stacks`, `migrated_to`, `last_db_dump`, `last_integrity_check`) | both → hub | **no consumer today, one arguably owed** | allowlisted **against R-264, OPEN**. Allowlisting is not deciding, and the entries say so |
Full per-field reasons are in the gate's own `ALLOWLIST`, each a claim someone can re-check.
## 3. Scenario F — the choice, and why
**Unknown is reported distinctly and is never `ok`.** `operator_key_configured` decodes as a
**pointer**: nil = the agent never said, which is not a value.
The version gate the prompt thought "probably right" was **rejected on a measurement**: the field and
the `oob` stanza that carries it shipped in the **same** agent version (v0.72.0, 2026-07-05), so a
stanza without the field cannot come from any released agent. The live fleet is 0.113.0 and 0.127.0;
the vouched floor is 0.127.0. Building version-gating machinery the hub does not otherwise have, for a
state no box can be in, is cost without cover. The case is still handled explicitly and pinned by a
test, because "cannot happen" is a claim this project has been burned by.
## 4. R-247 — CLOSED
The field is received, and `reconcileEscrowed` tells a **withheld** hash from a **hash-less** one.
Controller v0.209.0.
**Deliberately not folded in, and said rather than skipped:** the wrong flag on `demo-hp` is an
operator act hub-side (**R-246**, still open), and the customer-facing Hungarian card copy is
unchanged — that is UI work with its own review path.
## 5. The gate's blind spots, and its self-test
Published in the module docstring **and** in the gate's own output, because Campaign 12's C1 guard
turned out blind to one of the three shapes it was written for:
- **generic tag names are not checked** (`name`, `state`, `status`, …) — a repo-wide string test says
nothing about them, so a drop of a generically-named field is **missed**; the gate under-reports
rather than over-reports;
- **reachability of a NAME is not use of a VALUE**;
- **only declared ROOTS are covered** — the hub's desired-state (raw stored JSON, no typed emitter)
and the agent's local API (no single root) are **not**;
- it reads source, not traffic; test files and `testdata/` are excluded on the receiving side
deliberately (a tag present only in a fixture is not decodable — which is R-262 exactly).
`--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction:
**exit 1, planted tag named**; unplanted tree **exit 0**.
**Two instrument defects the CONTROL caught before the gate was trusted:**
1. **A substring false negative**`grep -F healed_at` also matched `privsep_healed_at`. R-260 named
`healed_at`, so its absence from the output was the tell. Now a whole-token regex; 40 not 39.
2. **`dr_recipe` is not wholly opaque** — its top-level section keys ARE decoded, through allow-lists
that already swallowed `offsite_restic` for months (R-122). Now opaque only **below depth 1**.
## 6. What `oobDegraded` says when it fails
```
Host <id>: OPERATOR ACCESS DEGRADED — the operator's authorized_key is NOT installed —
felhom-sshd is up and answering, and nobody can log in through it. The break-glass net
(auto-heal + vaulted root@pam console) is still under the box.
```
and for the unreachable-but-handled unknown:
```
… — the agent reports operator access but is too old to say whether the operator key is
installed (pre-v0.72.0) — treat entry as UNPROVEN, not working. …
```
`oobDegradedReason` is now the single source for both the predicate and the text, so the message can
never name a different fault from the one that fired. The old form derived it separately and had a
vocabulary of two.
## 7. Tests and red-proofs
New: `hub/internal/store/host_oob_decode_test.go` (4 tests, raw JSON at the decode boundary),
`hub/internal/monitor/host_oob_operatorkey_test.go` (6), plus two end-to-end tests in
`host_oob_test.go` driving JSON → store → checker → event.
**Red-proofs — 8 expected outcomes, 0 wrong, each with the mutation asserted applied:**
| mutation | assertion it applied | outcome |
|---|---|---|
| the gate on today's tree | — | **RED, naming all 40** ✔ |
| planted unreachable tag (post-fix) | self-test reports the planted tag by name | **RED on the plant, GREEN unplanted** ✔ |
| drop `operator_key_configured` from the decoder | json-tag occurrences in the decoder 2 → 1 | **RED — the false `ok` returns** ✔ |
| make the check unconditional | `MUTATED unconditional degrade` marker present | **RED — a healthy box alerts** ✔ |
| treat unknown as `ok` | `MUTATED: unknown is silently ok again` marker present | **RED — the silent pass returns** ✔ |
| all three restored | — | **GREEN** ✔ |
**The pre-existing fixture was part of the defect and was fixed too:** `oobReport()` omitted
`operator_key_configured`, so every earlier scenario ran against a report shape **no released agent
produces**. Same family as R-262.
## 8. The capability-map row about operator access
**Checked, and it was NOT claiming something untrue.** `00-capability-map.md:127` claims OOB operator
access is *implemented*, never that it is *monitored*, so no correction was owed. What was untrue sat
one layer down — the hub's own health check could not see the key — and the row now records that,
with the fix and the tests that pin it.
## 9. Gates, and what remains
`python3 scripts/repo_gates.py --fast`**all 8 OK**, including the new `wire-contract` and
`golden-currency`. `go build ./... && go vet ./... && go test ./...` green in **hub** and
**controller** (run separately from every commit). **No `--no-verify` anywhere.**
**The one gate failure that remains is not a failure of this work:** golden **0.208.0** is baked and
byte-verified but **still not vouched**, so fresh installs receive 0.207.0. That is R-242's untouched
half and one operator Save.
## 10. Register
**R-260 CLOSED** (class gated + sharpest instance fixed), **R-247 CLOSED**, **G-1 CLOSED** in
`ROADMAP.md`. **R-264 minted and OPEN** — the twenty-one facts with no consumer, split out so that
gating the class could not be mistaken for deciding them. **Highest ID moved R-263 → R-264.**
Explicitly still open: R-246, R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and **C7's
test-comment half**, which Campaign 12 recorded as *owed, not done*.
## 11. Observations — noticed, NOT acted on
1. **`stacks` is the whole per-stack report object and the hub decodes none of it.** The largest
single unconsumed structure on the controller wire; folded into R-264 rather than sized here.
2. **The hub has no version-gating machinery for report fields at all.** Not needed today (see §3),
but the next additive field whose emitter and stanza do *not* ship together will need it, and
there is no convention to reach for.
3. **`backup.last_db_dump` / `last_integrity_check` are backup-integrity timestamps the hub cannot
see** — the "presence is not success" neighbourhood, and worth ranking first inside R-264 after
guest_net.
4. **The gate cannot cover the hub's desired-state wire** because it is served as raw stored JSON.
That is the one remaining hub→box direction with no contract check of any kind.
+15 -22
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-08 (overnight — Campaign 12).**
**Updated 2026-08-08.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
> part of it in plain words, and **nothing may exist only here**. Not `CONTEXT.md`, which is technical
@@ -46,33 +46,26 @@ it. *(R-252, R-253 — closed 2026-08-08.)*
clean teardown. No secrets involved, but it accumulates with each walk. *(R-244)*
- **Putting restored files back where they belong is still a manual step.** *(R-213)*
## Found overnight — we went looking for siblings, on purpose
## Fixed today — the thrown-away sentence, and a check so there is no next one
Every fault this month came in one of seven shapes; overnight we looked for the others **by shape**
reading and counting only, nothing changed or deployed, no machine touched. Eight new items, none
urgent. Full working: `audits/CAMPAIGN-12-class-sweep-2026-08-08.md`.
We could not tell whether your engineer could get into a machine. The machine says so every few
minutes; **the hub had nowhere to put the sentence and discarded it on arrival**, so a box with the
door open, the lock working and **no key issued** was reported as fine. Not a wrong answer — an answer
to a question nobody was asking. Fixed, and the alert now **names the missing key** instead of saying
"access degraded". *(R-260, R-247 — closed; hub v0.99.0, controller v0.209.0.)*
- **The one that matters.** The machine tells us whether your engineer's emergency key is really
installed — and **we throw that sentence away before reading it**, so a box can read as "we can get
in" when nobody can. Same shape in seven other places. *(R-260)*
- **A green tick meaning only "a backup exists"** — any age counts, and it reddens on a problem
measured across all apps rather than that one. *(R-258)*
- **A disk we failed to measure is drawn as an empty, healthy one** — "0.0 GB / 0.0 GB (0%)" in the
normal colour, while the right pattern sits two files away. *(R-259)*
- **Two comments promise more than the code does** (one guarded by a test that cannot fail on the
thing it names), and **two refusals still stop at "no"**, one in half-English.
*(R-262, R-263, R-256, R-257)*
- **Which shapes can a machine catch from now on?** That mattered more than the list: **one is cheap
and worth doing** (the thrown-away-sentence one, which would have caught every instance on the day
it was written), two need a small decision first, and **two are honestly not automatable**
including the one that looked most automatable, where the standard tool was tested and found blind.
Recorded, not built. *(ROADMAP G-1…G-8)*
**The check was built first and watched failing on 40 facts, before a single one was fixed** — the
night before, an off-the-shelf tool for a neighbouring shape was rejected for failing exactly that
test. Of the 40: three now change what we are told, sixteen are genuinely redundant, and **twenty-one
are recorded as undecided rather than quietly waved through** *(R-264)* — the strongest being
per-guest network health, which we already lost 1 h 15 m to once.
## What we're working on
- **Widening the check** so a fourth secret-in-a-page is caught by a machine, not by someone. *(R-255)*
- **Proving the hub really keeps the old sealed key** when a machine re-seals — needs its own session
and a second deliberate wipe. *(R-198)*
- **Deciding the twenty-one** — for each: give it a reader, or stop sending it. *(R-264)*
- **Proving the hub really keeps the old sealed key** when a machine re-seals. *(R-198)*
- Still open from the overnight sweep, none urgent: *(R-256…R-259, R-261…R-263)*
## Waiting on you
@@ -124,7 +124,7 @@
| **Indítópult megosztás (vendég link)** — capability URL `/s/<token>` serves a standalone read-only guest launcher (no account, no admin session); optional per-share password; QR | controller v0.165.0 | **IMPLEMENTED** | 160-bit `crypto/rand` token, constant-time match (empty stored = disabled = byte-identical to the mux default 404); guest headers `noindex`/`no-referrer`/`no-store`; optional SEPARATE bcrypt share password + its own per-IP attempt map; signed cookie `HMAC(token`\|`passwordHash)` keyed with `session_secret` (rotate-token OR change-password invalidates all cookies); token redacted in logs (`/s/<redacted>`). Groups AG (14 tests) + 3 red-proofs; **§13 endpoint-level live validation on 9201 all-pass** (`felhom-controller/REPORT.md` 2026-07-24). **Design ruling: member accounts SUPERSEDED by this capability-URL model; per-member tile visibility parked under the SSO arc (R-15).** | Full operator browser click-through + a validation doc pending → then PROVEN-LIVE. Accepted residuals: link-preview crawlers fetch once (noindex prevents indexing); reverse-proxy/CF access logs hold the path (ops-tier); the modal link carries the request Host (LAN-IP admin ⇒ LAN-IP link) |
| Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | |
| Multiple household users / per-person accounts | — | **MISSING** | — | Single dashboard password; acceptable for alpha → R-15 |
| WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13 |
| WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13. **CHECKED 2026-08-08 (R-260) and this row was NOT claiming something untrue** — it claims the capability is implemented, never that it is monitored, so no correction was owed. What WAS untrue is narrower and sat one layer down: **the hub's own OOB health check could not see whether the operator's key was installed.** `HostOOBRow` mirrored five of the agent's eight OOB fields, so `operator_key_configured` — emitted every heartbeat since agent v0.72.0, i.e. from this row's own vintage — was discarded by `encoding/json` on arrival, and `oobDegraded` returned `ok` for a box with felhom-sshd active, reachable, a valid config, a configured peer and **no operator key at all**. `operator_peer_configured`, which it did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Fixed hub v0.99.0; the missing key now degrades and the alert NAMES it; a stanza too old to carry the field is reported distinctly and is never a silent ok. Pinned end-to-end from raw report JSON by `TestHostOOB_MissingOperatorKey_EndToEnd` and `TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd` |
| The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent **v0.119.0**, hub **v0.85.0** | **PROVEN-LIVE** (2026-07-31) | `audits/host-addresses-visible-2026-07-31.md` | Before this the LAN IP was **not reportable at all**`HostMetrics` carried no address of any kind — and the WG IP existed only in `/offsite`'s peer table keyed by pubkey (peer→host, never host→peer). New wire field `addresses[]`, one row per (interface, address); `IsGlobalUnicast()` is the whole filter, chosen by MEASURING both demo boxes, and it needs no veth/fwbr denylist because that plumbing carries no IP. Rendered live on both 0.119.0 hosts matching their `ip addr` ground truth exactly. **Two honesty properties carry the risk and are both red-proofed:** WireGuard shows the hub ALLOCATION and whether the box CONFIRMS holding it (allocation alone cannot tell a live tunnel from a peer never applied), and an agent below 0.119.0 renders **UNKNOWN, never "no addresses"** — proven live on `drill-r50-0a4f9a` (0.113.0). **Not covered:** a two-LAN-bridge box and a real WG drift, neither of which exists to observe |
| Break-glass management-plane recovery | agent v0.71, hub v0.84 | **IMPLEMENTED** | `runbooks/break-glass.md` | hub v0.84.0 adds an **operator-SESSION** retrieval path (host page → Console access → Reveal; `POST /hosts/{id}/reveal-recovery-credential`, CSRF-gated, writes a customer-visible `recovery_credential_revealed` event) beside the pre-existing **global-key** one (`GET /api/v1/admin/hosts/{id}/recovery-credential`), which is untouched and stays the route for when the hub UI itself is down. **The credential half is now PROVEN (2026-07-31):** the vaulted `demo-hp-bb76ea` password was verified against the box's own `/etc/shadow` hash AND minted a real PVE ticket — `POST /api2/json/access/ticket`**HTTP 200, `root@pam`, 367-char ticket**, the exact API the login form submits to. Still IMPLEMENTED rather than PROVEN-LIVE overall, because the path has not been exercised on a REAL lockout (SSH was available throughout). Discovered during that check: the card's Copy button shipped `disabled` until a Reveal and silently no-opped, leaving ANOTHER host's password in the clipboard — fixed in hub v0.86.0. The vaulted secret is plaintext at rest → **R-133** |
+59 -2
View File
@@ -177,7 +177,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
| **R-245** | **Should a customer who never decides be auto-abandoned after 30 days? RECORDED, NOT BUILT — and the reasoning against it is recorded with it so the decision can be revisited properly.** **The operator's proposal (2026-08-07):** a box that has been offered recovery for 30 days without the customer deciding is auto-abandoned, entering the 14-day grace, so an undecided box does not sit for ever holding history nobody has claimed. **What was built instead:** the escalating reminders (1/3/7/14 days) and the operator levers `--abandon-extend` / `--abandon-stop`. **The reasoning, as settled with the operator the same day:** (1) **nobody is absent** — a box does not reinstall itself, so whoever rebuilt it was standing there and met the recovery question; the "customer away for months" case does not arise from this situation, because **a reinstall implies a person**. (2) **A customer who cannot find their code will get in touch**, which is the moment to extend or disarm by hand — so the automation would be firing at people we are already talking to, which is why the levers were the thing worth building. (3) **The cost is theirs**: the old history sits in the customer's own storage allowance, and if they are paying to keep something they have not decided about, that is their call and they feel it before we do. (4) **The real harm, if it comes, is QUOTA** — old history blocking new backups — and **that is a condition, not a calendar**. An automatic ending should trigger on the harm, with a dated warning, never on a date alone. **If this is ever built, build it that way.** **✅ RE-FILED 2026-08-08 AS A DECISION TAKEN, not a question pending.** It sat in the operator's queue as `WAITING-ON-OPERATOR` for a day, and **nothing was actually pending** — the operator and the reviewer settled it on 2026-08-07: it is **not built**, the levers were built instead, and the whole reasoning above is the record of why. A settled decision parked in a queue is a queue nobody trusts, and an audit of every `WAITING-ON-OPERATOR` row the same day found this was the ONLY one — so the drift was caught while it was still a single row. **THE CONDITION THAT REOPENS IT, which the reasoning already names: QUOTA — old set-aside history blocking new backups.** Not a calendar. If a customer's retained history ever refuses a new backup, revisit this with a dated warning that triggers on the refusal; until then it stays decided. | **DECIDED 2026-08-07 — not built; reopens on quota** |
| **R-246** | **A leftover staleness flag has silently disabled the new recovery discriminator on `demo-hp` since 2026-08-04, and the flag is WRONG.** Found by a read-only spike, 2026-08-08. **Q1 — traced to an act, to the second:** at `2026-08-04 20:15:49` the hub emitted `offsite_reissued` and `escrow_stale` in the same second — an operator **Re-issue** pressed during the R-201 drill, three minutes after `escrow_blob_served` at 20:12:40/20:12:54. That was `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub v0.95.0 REMOVED the very next day** (R-196 / R-204 item 2) precisely because it marked healthy escrows stale. **Q2 — the flag is wrong, measured on both sides:** the hub's blob seals `restic_pw_sha256 = 8a9e33aa4da6769c…d080a`, and the key the box is actually using hashes to **the identical value**. The blob covers the key. **Q3 — nothing clears it by itself:** the ONLY writer of `stale_at = NULL` is `SaveHostEscrow`'s `ON CONFLICT` — i.e. a fresh escrow ceremony, **which is the one act that would supersede the good blob**. So the only exit from a false alarm is the destructive act the false alarm recommends. **Q5 — the blast radius, enumerated rather than assumed:** (1) `GetEscrowStatusForCustomer` withholds `restic_pw_sha256` from the ACK; (2) a **pending** box can never auto-confirm, so (3) **every off-site run is refused indefinitely** — neither bites `demo-hp`, which was already `escrowed` and is backing up healthily (12 snapshots, last success 2026-08-07T02:15:35Z); (4) the customer is told to create a new code; (5) **NEW — v0.206.0's shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b), so the recovery screen would stay silent even if recovery were needed. **Q6 — a fresh box CANNOT reach this state:** `MarkEscrowStale` has **no production caller anywhere in the tree** (census: only its own definition, two comments and two test references). **The next walk cannot meet it.** **NOT CLEARED, deliberately** — Q2's answer says the fix is to clear this instance, but whether to also stop the column being settable at all is a separate ruling, and the spike was scoped read-only. **✅ THE FLAG IS CLEARED — operator-approved and applied 2026-08-08.** One row, identity-matched on `host_id` and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end, not just in the database:** the hub now serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa4da6769c…d080a` at `11:10:19Z`, and that is **byte-identical to the key it is using** — so shape (c) compares, matches, and correctly stays silent. **The false stale warning is gone, proven with a positive control** rather than an absent line: **0** `escrow-confirm` lines since the restart while **5** scheduler lines in the same window prove the box was logging, and the recorded hash proves an ACK was processed. *(Method note: the hub pod is Alpine with no `sqlite3`; it was installed into the container's ephemeral writable layer — image and node untouched, gone on restart. SQLite's own file locking coordinated the write with the live hub; an earlier attempt failed cleanly on quoting and changed nothing, which is the fail-safe working.)* **STILL OPEN under this ID: the ruling on whether `stale_at` keeps a live setter at all.** It currently has NO production caller, so the column is write-only-by-accident — a field that changes behaviour, that nothing sets and nothing can see (R-248). Either give it an evidential setter or retire it; **do not leave it as a trap that only a database read can spring.** | **READY — flag cleared; the column ruling is still owed** — owner Viktor |
| **R-247** | **The box is being told something false, in its own words, and it recommends the destructive act.** `demo-hp`, live, on controller v0.206.0: *"STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code"*. **Every clause is false.** The hub HAS the hash and is *withholding* it (R-246); there was no supersession (`host_escrow_superseded` has no row for this host); and the bundle **does** cover the password — the hashes match exactly. It raises `EscrowStale`, which renders the customer-facing card *„A letétben lévő helyreállítási csomag nem fedi a jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."* **THE CAUSE IS A FACT ON THE WIRE THAT THE BOX THROWS AWAY.** The hub already sends `escrow_stale` in the ACK (`json:"escrow_stale,omitempty"`), and the controller's `report.EscrowStatus` **has no matching field**, so `encoding/json` drops it silently. The box therefore cannot distinguish *withheld because flagged stale* from *genuinely hash-less*, and guesses the latter. **This is R-241's shape for the third time: the answer is available, and it is discarded at the boundary.** **The fix is small and is NOT made here** (§0 forbids a controller change this session): add the field, and say the true thing — or say nothing, since on an ESCROWED box with matching hashes there is nothing wrong to report. | **READY** — owner Viktor |
| **R-247** | **The box is being told something false, in its own words, and it recommends the destructive act.** `demo-hp`, live, on controller v0.206.0: *"STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code"*. **Every clause is false.** The hub HAS the hash and is *withholding* it (R-246); there was no supersession (`host_escrow_superseded` has no row for this host); and the bundle **does** cover the password — the hashes match exactly. It raises `EscrowStale`, which renders the customer-facing card *„A letétben lévő helyreállítási csomag nem fedi a jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."* **THE CAUSE IS A FACT ON THE WIRE THAT THE BOX THROWS AWAY.** The hub already sends `escrow_stale` in the ACK (`json:"escrow_stale,omitempty"`), and the controller's `report.EscrowStatus` **has no matching field**, so `encoding/json` drops it silently. The box therefore cannot distinguish *withheld because flagged stale* from *genuinely hash-less*, and guesses the latter. **This is R-241's shape for the third time: the answer is available, and it is discarded at the boundary.** **The fix is small and is NOT made here** (§0 forbids a controller change this session): add the field, and say the true thing — or say nothing, since on an ESCROWED box with matching hashes there is nothing wrong to report. | **CLOSED 2026-08-08** — controller v0.209.0. The field is received and the box tells the two conditions apart; see the Campaign-12 follow-through section below. The WRONG FLAG itself is R-246 (operator act, hub-side) and the customer-facing card copy is unchanged — both stated rather than folded in |
| **R-248** | **A flag that changes behaviour is visible to nobody who would look for it.** Q4 of the 2026-08-08 spike, answered plainly. **The customer** sees only a derived card stating a false reason (R-247). **The box** cannot see it at all (R-247's dropped field). **The operator** can see it on exactly ONE page — the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`, `v.EscrowStale = escrow.StaleAt != ""`) — which is the wrong tier for this symptom: an operator investigating an OFF-SITE problem has no reason to open a PBS-DR page. **No alert, no report field, no off-site surface.** The one-shot `escrow_stale` event fired on 2026-08-04 and **was never notified** (a full census of `notification_log` for that customer that day returns 8 rows, none of them this one); it has fired twice ever, both on 4 August. **So the practical answer is: only a database read.** That is a finding in its own right — a flag that silently changes behaviour and cannot be seen is the shape this fortnight has been about (R-241's discarded comparison, R-228's unread field, R-243's unobserved state). **What it needs:** surface `stale_at` on the off-site operator surface and in the host report, or stop using a field nobody can observe to change what a customer is told. | **READY** — owner Viktor |
| **R-249** | **The retrieval passphrase ships in the customer page's HTML, so any headless read puts it in a transcript.** Found 2026-08-07 on the fifth walk **by doing it**: driving the documented rebuild path, the cleartext value reached the session transcript from a `grep` over the saved page — **not** from pressing Reveal. The hub renders the per-customer **Retrieval Password** masked (`••••••••`) behind a Reveal/Copy control, and carries the cleartext in a **`data-secret="…"` attribute** of that control; `hub/internal/web/render_test.go:169` pins exactly this (*"data-secret not populated for the reveal control"*). **The masking is a presentation control, not a containment one.** **Why it is a defect and not a design note:** the same page's own hint reads *"never place it on a command line (the installer reads it at a no-echo prompt)"* and the generated install command deliberately omits it — **the page states the threat model and then violates it in its own markup**. Anything that reads the page rather than looking at it (curl, a scrape, an agent, a saved HAR, a support bundle) gets the secret with no reveal action **and no audit event** — where the break-glass credential, which has the same sensitivity, emits `recovery_credential_revealed` when revealed. **Proposed shape:** serve it from a POST endpoint the button calls — the break-glass credential already works exactly this way (`POST /hosts/{id}/reveal-recovery-credential`) — and render the page with no value in it; that also gets the reveal audited, which it is not today. **Severity MEDIUM:** it is a live per-customer secret that fetches the whole config (`GET /api/v1/config/<id>` with `X-Retrieval-Password`), but the exposure is to someone who can already read the operator page — a defence-in-depth failure, not a boundary crossed. **The walk5 instance is compromised (it is in a transcript) and dies with that customer's teardown, which is owed.** **✅ FIXED — controller v0.207.0, 2026-08-08.** The page now carries only `HasRetrievalPassword`; the value comes from **`POST /settings/retrieval-password/reveal`** — CSRF-covered (POST, not GET, so `CsrfProtect` applies and it is not re-fetchable from history), `Cache-Control: no-store`, **and logged as an act**, which reading it off the markup never was. **Proven live on VM 325 with a positive control in both directions:** with a passphrase stored and the card rendering, v0.206.0's page body contained it **1** time in 50 208 bytes and v0.207.0's **0** times in 51 115 bytes; the reveal endpoint returns it with `no-store`, and the controller log carries the act with **0** occurrences of the value. The tests assert the **RAW RESPONSE BODY** — a red-proof that put the value back into the page reproduced the defect and failed the test, which no test asserting what the customer *sees* could have done. **The census this required found two more instances — R-254, not fixed.** **Rotation:** the `walk5` instance is compromised (it reached a transcript) and dies with that customer's teardown; no other instance is known to have been read, **but nothing records a read, which is itself part of the defect.** | **CLOSED 2026-08-08** |
| **R-250** | **A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time.** Found 2026-08-07 creating the fifth walk's venue. `POST /configs/new` with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — **fail-closed by design** (`offsite.go:111-121`, *"don't serve a descriptor the controller can't verify"*), with `defaultScanBackoff` = 2+4+8+16+30 ≈ **60 s**, sized by its own comment to *"the observed DNS propagation lag"*. **Measured, both halves:** the first create exhausted the ladder — five `no such host`, then `dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` (the name had just begun resolving, **AAAA-first, into a pod with no IPv6 route**) — and the hub logged `[ERROR] offsite provision for walk5`. An **identical second POST succeeded** ~70 s later on its own final rung (`shared already provisioned … (subaccount 285351)`). **Total settle ≈ 100 s against a 60 s budget.** The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, `SSH-2.0-OpenSSH_9.6p1`. **Two distinct things are wrong and should not be merged:** (1) the budget is sized against DNS *existence*, but what actually bit is the **AAAA-before-A** window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) **the operator is told nothing actionable** — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — `one_time_secrets` 1, one sub-account, no double-provision) **but that safety is invisible to the person deciding whether pressing again will double-charge them.** **Severity LOW-MEDIUM:** self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. | **READY** — owner Viktor |
@@ -384,11 +384,68 @@ as "closed individually". **It is not — it is R-247, `READY`.** The live repo
| **R-257** | **C2 — „Az offsite tároló nincs elárvult állapotban." puts an English loanword and an internal state name in front of a Hungarian household customer, and names no route.** `web/offbox_handlers.go:270` (the Go error it mirrors is `backup/offbox.go:343`). „Offsite" is untranslated; „elárvult állapot" is the codebase's own `OffboxOrphaned()` predicate surfacing verbatim. A customer who pressed a button and got this cannot tell whether something failed, whether they did something wrong, or what to do instead. **This is a refusal that is CORRECT and fail-closed and still a dead end** — the same shape R-241 recorded for `--recover-offsite-install`. **Fix shape, not a decision:** say what the customer tried to do, why it does not apply right now, and where to look — or, since this is a state they cannot reach deliberately, do not offer the action at all | **READY** — owner Viktor |
| **R-258** | **C3 — the customer's per-app backup tick is green on the PRESENCE of a restore point, and its only red condition is a GLOBAL one.** `web/handlers.go:1240-1248`: `row.Tier1LastStatus = "ok"` whenever `ListRestorePoints(app)` returns ≥1 point, and `"error"` only when `status.LastDBDump != nil && !status.LastDBDump.Success` — where `LastDBDump` is the box's **single most-recent DB dump** (`backup/backup.go:1064`, `m.lastDBDump`), **not this app's**. Three consequences, in increasing order of how wrong they look to a customer: (a) an app whose own last backup failed shows a **green tick** provided any older restore point exists and some *other* app dumped successfully afterwards; (b) an app with **no database at all** takes the `nil` branch and is green on presence alone; (c) the tick asserts nothing about **recency** — a restore point from three weeks ago is as green as one from last night, while `row.Tier1LastRun` beside it carries the real (old) time. **It is customer-visible:** `templates/backups_apps.html:173-174` renders it as a check or a cross. **This is the project's own "presence is not success" rule as a UI badge** — the artifact's existence is being read as a successful result, which is what the workspace `CLAUDE.md` section of that name forbids. **Found by a COMPLETE sweep, not a sample:** all 9 success-status assignment sites in the controller were read; this is the only new one. **NOT reproduced live** — source reading only | **READY** — owner Viktor |
| **R-259** | **C4 — a disk read that FAILS renders as „0.0 GB / 0.0 GB (0%)" in the nominal colour, on the dashboard's most-looked-at meter.** `system/info_linux.go:259-264`: `readDiskUsage` logs at DEBUG and **returns**, leaving the caller's `TotalGB`/`UsedGB`/`AvailGB`/`Percent` at their zero values. `dashboard.html:57-63` then renders `{{fmtGB .SystemInfo.DiskUsedGB}} / {{fmtGB .SystemInfo.DiskTotalGB}} ({{printf "%.0f" .SystemInfo.DiskPercent}}%)` plus a meter whose fill is `width:0%`, and `usageColor(0)` returns **`"nominal"`** (`web/funcmap.go:189-196`) — so **a failed measurement is drawn as a healthy, empty disk.** There is **no `*Known` companion for the system disk** (grep: 0). **The fix pattern already exists in this codebase, two files away, with the reasoning written out:** R-225's `StatsKnown` guards every derived off-site figure and its own comment says *"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"* (`backups_remote.html:60-62`). The dashboard does not use it. **It also travels:** `report/builder.go:94` puts the same zeroed `TotalGB`/`UsedGB` into the host report, so a failed statfs reaches the hub as a 0-of-0 disk. **Observation attached, not filed separately:** `readDiskUsage` is one of several collectors in that file that return silently on error; only this one was traced to a customer-visible surface. **NOT reproduced live** | **READY** — owner Viktor |
| **R-260** | **C5 — the agent reports at least eight decision-bearing facts the hub models NOWHERE, and the sharpest one blinds the check that answers „can the operator get into this box".** Measured 2026-08-08 by a tag-reachability test over all four wire directions (465 emitted json tags; a tag whose literal string occurs nowhere in the receiving repo cannot be decoded by any struct, named or anonymous). **THE ONE THAT MATTERS: `operator_key_configured`.** The agent emits it every heartbeat (`hub/report.go:178`*"operator authorized_key installed"*); the hub's OOB decoder (`store/host_oob.go:38-45`) mirrors **5 of the agent's 8** OOB fields and has no field for it, nor for `wg_handshake_age_s` or `healed_at`. So `oobDegraded` (`monitor/host_oob.go:60-70`) tests config-invalid and (peer-configured AND not-active-or-not-reachable) — and **a box with felhom-sshd active, reachable, valid config, peer configured and NO OPERATOR KEY INSTALLED is `ok`.** The agent knows and says so; the hub throws it away. The checker's own doc comment claims it *"answers 'can the operator get into this box right now, and if not, why' proactively"* — see R-262's class. **Break-glass is the recovery chain (G1), which is what makes this the top row of the census.** **The rest of the census, hub-side absent (0 occurrences anywhere in `felhom.eu/hub/`):** `guest_net`**the entire per-guest network-health object**, populated every cycle at `hub/collect.go:290`, carrying `dhclient_alive`, `has_route`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`; `selfupdate_pending` + `selfupdate_pending_version` — a staged-but-unapplied agent update is invisible to the fleet view; `healed_recently` (mgmt-plane); `operator_key_configured`, `wg_handshake_age_s`, `healed_at` (OOB); `mount_parity` + `mount_inventory` (restore-test, see R-262); `cpu_temp_c`, `loadavg`, `memory_total_bytes`, `memory_used_bytes`, `uptime_seconds` (host metrics); `model_name` (SMART); `applied_at` (PBS-DR). **Controller→hub, hub-side absent:** `reporting_disabled`, `config_hash`, `last_db_dump`, `last_integrity_check`, `migrated_to`, `catalog_ref`, `storage_bindings`, `container_path`, `subpath`, `temperature_celsius`, `load_avg_{1,5,15}`, `memory_{total,used}_mb`, `uptime_seconds`. **⚠ STATED PRECISELY, because it first looked worse than it is:** the hub stores the whole report as `report_json`, so nothing is lost from the DATABASE — every hub consumer, without exception, re-unmarshals it into a typed struct, so nothing reaches a check, an alarm, a notification or a screen. And a **restore-test mount-parity mismatch is NOT hidden**: it sets `res.Err` and returns before `res.Pass = true`, so `pass:false` travels; what is lost is the *depth* of a pass, not the failure. **The class is what this row is about, not any single field** — this is `escrow_stale` (R-247) for the fourth, fifth and sixth time, and §4 of the campaign report argues it is the cheapest high-value gate available to this project. **Blind spot of the method, recorded:** 29 generically-named tags (`name`, `state`, `status`, …) were excluded by name, so a drop of a generically-named field would be missed | **READY** — owner Viktor |
| **R-260** | **C5 — the agent reports at least eight decision-bearing facts the hub models NOWHERE, and the sharpest one blinds the check that answers „can the operator get into this box".** Measured 2026-08-08 by a tag-reachability test over all four wire directions (465 emitted json tags; a tag whose literal string occurs nowhere in the receiving repo cannot be decoded by any struct, named or anonymous). **THE ONE THAT MATTERS: `operator_key_configured`.** The agent emits it every heartbeat (`hub/report.go:178`*"operator authorized_key installed"*); the hub's OOB decoder (`store/host_oob.go:38-45`) mirrors **5 of the agent's 8** OOB fields and has no field for it, nor for `wg_handshake_age_s` or `healed_at`. So `oobDegraded` (`monitor/host_oob.go:60-70`) tests config-invalid and (peer-configured AND not-active-or-not-reachable) — and **a box with felhom-sshd active, reachable, valid config, peer configured and NO OPERATOR KEY INSTALLED is `ok`.** The agent knows and says so; the hub throws it away. The checker's own doc comment claims it *"answers 'can the operator get into this box right now, and if not, why' proactively"* — see R-262's class. **Break-glass is the recovery chain (G1), which is what makes this the top row of the census.** **The rest of the census, hub-side absent (0 occurrences anywhere in `felhom.eu/hub/`):** `guest_net`**the entire per-guest network-health object**, populated every cycle at `hub/collect.go:290`, carrying `dhclient_alive`, `has_route`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`; `selfupdate_pending` + `selfupdate_pending_version` — a staged-but-unapplied agent update is invisible to the fleet view; `healed_recently` (mgmt-plane); `operator_key_configured`, `wg_handshake_age_s`, `healed_at` (OOB); `mount_parity` + `mount_inventory` (restore-test, see R-262); `cpu_temp_c`, `loadavg`, `memory_total_bytes`, `memory_used_bytes`, `uptime_seconds` (host metrics); `model_name` (SMART); `applied_at` (PBS-DR). **Controller→hub, hub-side absent:** `reporting_disabled`, `config_hash`, `last_db_dump`, `last_integrity_check`, `migrated_to`, `catalog_ref`, `storage_bindings`, `container_path`, `subpath`, `temperature_celsius`, `load_avg_{1,5,15}`, `memory_{total,used}_mb`, `uptime_seconds`. **⚠ STATED PRECISELY, because it first looked worse than it is:** the hub stores the whole report as `report_json`, so nothing is lost from the DATABASE — every hub consumer, without exception, re-unmarshals it into a typed struct, so nothing reaches a check, an alarm, a notification or a screen. And a **restore-test mount-parity mismatch is NOT hidden**: it sets `res.Err` and returns before `res.Pass = true`, so `pass:false` travels; what is lost is the *depth* of a pass, not the failure. **The class is what this row is about, not any single field** — this is `escrow_stale` (R-247) for the fourth, fifth and sixth time, and §4 of the campaign report argues it is the cheapest high-value gate available to this project. **Blind spot of the method, recorded:** 29 generically-named tags (`name`, `state`, `status`, …) were excluded by name, so a drop of a generically-named field would be missed | **CLOSED 2026-08-08** — the class is GATED (G-1, `scripts/wire_contract_gate.py`) and the sharpest instance is fixed (hub v0.99.0). The remaining unconsumed facts are **R-264, OPEN** — allowlisted with reasons, which is not the same as decided. See the follow-through section below |
| **R-261** | **C6 — `CountSelfBindTokens` exists so that callers can assert an invariant, and no production caller asserts it.** `hub/internal/store/selfbind.go:106-111`. Its doc comment: *"it exists so callers can assert the 'after this runs, the only live link is one we just issued — or none' invariant that the auto-mint at customer-create / RESET-completion depends on."* **Census: only its own declaration in production; the two callers are `selfbind_automint_test.go:29` and `customer_delete_test.go:510`.** Tests are not callers (the campaign's rule), so the invariant the auto-mint *depends on* is checked in the test suite and never at the moment it matters. **This is the smallest of the eight rows and is filed at its true size, because the rest of the C6 sweep found INERT dead accessors rather than defects:** `OffboxOrphanedRenamedTo` and `OffboxEscrowState` have no caller but their data reaches the card another way (the template reads the settings field directly, `backups_remote.html:80`) — **R-228 is genuinely closed, and the sweep's first reading that it had regressed was wrong.** **The more consequential C6 result is a method result and is in the report, not here:** `golang.org/x/tools/cmd/deadcode` re-finds **neither** known instance, and a planted probe measured why — it reports an unreachable exported FUNCTION and not an unreachable exported METHOD on a widely-used type, and both known instances are methods | **READY** — owner Viktor |
| **R-262** | **C7 — a comment claims a cross-repo contract is mirrored „field-for-field" and „the key-set tests guard drift"; it is two fields short, AND THE FIXTURE THE TEST READS OMITS THE SAME TWO FIELDS.** `hub/internal/api/handler.go:682-687` covers `hostBackup` **and** `hostRestoreTest`. **It is TRUE of `hostBackup`** (verified field-for-field against `agent/internal/hub/Backup`). **It is FALSE of `hostRestoreTest`:** the agent emits `mount_parity` and `mount_inventory` (`hub/report.go:432-433`, populated in production from `reconcile/restoretest.go:277-283` via `backup/runner.go:517`), and the hub has no field for either — **0 occurrences in the entire hub repo** outside the CHANGELOG. **The guard is blind in exactly the place the drift is:** `TestHostReport_GoldenContract` reads `testdata/host-report.golden.json`, the two copies of which are byte-identical as required — and **neither contains `mount_parity` or `mount_inventory` at all**, so the key sets agree on a shape that is not the shape the agent sends. A test that cannot fail on the drift it names is the R-97b lesson (*prove the consequence, not the mechanism*) landing on a contract test. **Consequence, stated precisely:** the verdict is not lost (a parity mismatch fails the test before `Pass` is set), but the hub cannot distinguish a full-fidelity restore-test pass from a boot-only one, for any agent, ever. **Fix shape, not a decision:** add the two fields and put them in the fixture — or narrow the comment to name `hostBackup` only and say plainly that `hostRestoreTest` is a subset. **Attached observation:** the same fixture carries `cpu_temp_c` and `loadavg`, which no hub struct decodes — a fixture carrying keys the receiver cannot read is the same shape one level down | **READY** — owner Viktor |
| **R-263** | **C7 — „This is the ONLY writer of `StoragePath.BackupTarget`" is false, and nothing pins it.** `settings/settings.go:1317-1319`, on `SetBackupTarget`. **`ClearBackupTarget` (`:1358-1363`) also writes the field**, 17 lines below, in the same file. **The GUARANTEE the comment protects is intact and that is why this is filed small:** the sentence continues *"registration must never set it (E-2 §3: a drive never acquires a role by appearing)"*, and `ClearBackupTarget` only ever writes `false`, so no path other than `SetBackupTarget` **grants** the role. **What is wrong is the claim as written, and the absence of anything holding it:** `backup_target_role_test.go` exercises the behaviour and asserts nothing about writer uniqueness, so if a third writer appeared tomorrow — one that granted — the comment would still read as settled and the suite would still be green. **This is the class's own definition:** an invariant asserted in prose with no test pinning it. **Fix shape:** one word (*"the only writer that GRANTS the role"*) plus a test that fails when a second granting writer appears. **Method honesty:** found in a sample of **60 of 2652** production invariant comments — the "is the only" form only, chosen because a uniqueness claim is the one form a grep can falsify. **~2592 production and all 1440 test invariant comments were NOT examined**, so C7 has the weakest coverage of the seven classes and the task's instruction to check the tests' own claims is **owed, not discharged** | **READY** — owner Viktor |
## CAMPAIGN 12 follow-through — G-1 built, R-260 and R-247 closed, 2026-08-08
**The gate was built BEFORE the fixes and was seen failing on 40 fields**, captured verbatim in
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`. That order was the method, not
bureaucracy: the night before, `deadcode` was rejected for class C6 precisely because it was made to
prove itself first and found neither of the two defects it was meant for.
**⚠ A COUNT THIS SESSION'S OWN PROMPT GOT WRONG, corrected against the repo rather than quoted.** The
prompt said *"465 emitted tags, eight unreachable"*. R-260's wording was "at least eight
DECISION-BEARING facts", never eight tags in total, and its own census already listed more. Measured
on the three declared wires: **210 tags checked, 51 skipped, 40 convicted.** Two prompt claims were
wrong this week and both were caught the same way.
**Two things the gate's CONTROL caught before it was trusted**, each a defect in the instrument:
1. **A substring false negative.** `grep -F healed_at` also matches `privsep_healed_at`, so a
genuinely dropped field read as received. R-260 named `healed_at`, so its absence from the output
was the tell. Now a whole-token regex.
2. **`dr_recipe` is not wholly opaque.** The hub stores each half as `json.RawMessage` and re-emits
nested shapes verbatim — but the TOP-LEVEL section keys are decoded by `hostHalfShape` /
`appHalfShape`, and **those are allow-lists**: a section an emitter adds is silently dropped until
named in both. That already cost `offsite_restic` (R-122). The gate is therefore opaque BELOW
depth 1, so the sections are checked; treating the whole subtree as opaque would have put R-122's
shape back outside its reach.
**THE FORTY, BY DISPOSITION.** Full per-field reasons live in the gate's own `ALLOWLIST`, where each
entry is a claim someone can re-check.
| # | field(s) | direction | decision | what changed |
|---|---|---|---|---|
| 1 | `oob.operator_key_configured` | agent → hub | **RECEIVE AND ACT ON IT** | decoded as a pointer; `oobDegraded` now fails when the key is absent, and the alert NAMES it. Hub v0.99.0 |
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **RECEIVE, for the message only** | decoded into `HostOOBRow` and put in the event payload; deliberately NOT in the predicate — widening the check beyond the fact that is now arriving is how a check stops being read |
| 3 | `escrow_stale` | hub → controller | **RECEIVE AND ACT ON IT** | `report.EscrowStatus.Stale`; the box tells a withheld hash from a hash-less one. Controller v0.209.0. **This is R-247** |
| 4 | `host.{cpu_temp_c,loadavg,memory_total_bytes,memory_used_bytes,uptime_seconds}`, `system.{load_avg_1,5,15, memory_total_mb, memory_used_mb, temperature_celsius, uptime_seconds}` | agent/controller → hub | **NO CONSUMER WANTED — redundant** | allowlisted: the hub decodes `cpu_percent` / `memory_percent` / `disk_percent` from the same stanzas and every threshold is expressed on those |
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: guest sizing is hub-owned INTENT (the manifest), not mirrored reality |
| 6 | `storage_targets.smart.model_name` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: a display label with no threshold on it; `smart.health` and every counter the hub bands on ARE decoded |
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: wgsync reconciles from its own state, and the OOB path's own handshake age is now decoded |
| 8 | `guest_net` + its 7 children, `selfupdate_pending(_version)`, `mgmt_plane.healed_recently`, `pbs_dr.applied_at`, `restore_tests.mount_{parity,inventory}`, `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check` | agent/controller → hub | **NO CONSUMER TODAY, AND ONE IS ARGUABLY OWED** | allowlisted **against R-264, which stays OPEN**. Allowlisting is not deciding, and the entries say so |
**A C7 instance found while doing it, and corrected.** `HostReport.SelfUpdatePending`'s own comment
claimed *"The hub reads an absent field as pending=false, the correct default."* The hub has no field
for it and reads nothing either way. Comment corrected in the agent (no version bump — comment only).
**The hub's OOB test fixture was part of why this survived.** `oobReport()` omitted
`operator_key_configured` entirely, so every pre-existing scenario ran against a report shape **no
released agent produces**. Fixed, and the new tests drive the raw JSON decode boundary — a test that
builds the receiving struct by hand cannot see a field that never decodes, which is the whole class.
| ID | What | State |
|---|---|---|
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change | **READY** — owner Viktor |
**Explicitly still open, untouched by this session:** R-246 (the wrong stale flag on `demo-hp`
clearing it is an operator act hub-side), R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and
**C7's test-comment half**, which Campaign 12 recorded as *owed, not done* (60 of 2652 production
invariant comments sampled; none of the 1440 test comments).
## Why the TOP READY rows rank this way
+1 -1
View File
@@ -137,7 +137,7 @@ by looking a fourth time.**
| Rank | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| **G-1** | **Gate C5 — the cross-repo tag-reachability check.** For every json tag emitted on a real wire (agent→hub, controller→hub, agent→controller, hub→both), assert that the literal tag string occurs somewhere in the receiving repo. A tag occurring nowhere cannot be decoded by any struct, named **or anonymous** — which is what makes this definitive where struct-pair diffing is not. | S (~150 lines Python) | **candidate — recommended first** | **The cheapest high-value gate available to this project.** No network, no container runtime → `--fast`-eligible, so it runs in BOTH the pre-push hook and CI (the constraint that forced `golden_currency_gate.py` to check the bake rather than the vouch). It would have caught `escrow_stale` (R-247), `operator_key_configured` and every other row of R-260's census **on the commit that introduced them**. **Design requirement, not optional:** an allowlist with stated reasons, in the shape `secret_in_markup_gate.py` already uses — the hub deliberately ignores some collections and says so. **Known miss:** generically-named tags (`name`, `state`, `status`, …), where a repo-wide string test is meaningless; 29 were excluded by name in the sweep. → R-260 |
| ~~**G-1**~~ | ~~**Gate C5 — the cross-repo tag-reachability check.**~~ | S | **BUILT AND CLOSED 2026-08-08**`scripts/wire_contract_gate.py`, `--fast`, registered in `repo_gates.py` | Shipped as ranked. **Built BEFORE the fixes and seen failing on 40 fields** (`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`) — the order was the method, because `deadcode` had been rejected for C6 the night before precisely for failing that test. 210 tags checked across 3 declared wires, 51 skipped (generic / opaque / allowlisted, each with a reason). Carries a `--selftest` that plants an unreachable tag on a real root and asserts conviction, and publishes its blind spots in both its docstring and its output. **Two instrument defects the control caught before it was trusted:** a substring false negative (`grep -F healed_at` matched `privsep_healed_at`), and treating `dr_recipe` as wholly opaque when its top-level section keys ARE decoded through an allow-list that already cost `offsite_restic` (R-122) — it is now opaque only BELOW depth 1. **Estimate held:** the size guess was right and the `--fast` judgement was right. **Not covered, and stated in the gate itself:** the hub's desired-state (served as raw stored JSON, no typed emitter) and the agent's local API (no single root). → R-260 CLOSED, R-247 CLOSED, leftover appetite R-264 |
| **G-2** | **Gate C3 — a success verdict may not be set where an incompleteness signal is in scope.** Assert that every literal success-status assignment either has no gap/skip/missing signal available at that point, or consults it. | S | candidate | The whole population in the controller is **9 sites** — Campaign 12 read all of them, which is why this class is the one where "no others exist" is supportable. Small enough to gate by enumeration rather than by inference. **Known miss:** verdicts expressed as booleans, enum constants, or the absence of an error — and the controller does use those elsewhere. Instances: R-240 (open), R-258 (new). |
| **G-3** | **Gate C4 — every rendered count/size/percentage needs a `*Known` companion.** | M | candidate — **needs a convention decision first** | R-225's `StatsKnown` is the house pattern and it is exemplary; a gate can only enforce it once it is *the* house style. Today three-state is also encoded with pointers and with separate error fields, both legitimate, and a name-based gate reads those as unguarded. **The decision owed is "how does this codebase say 'we could not look'", not "should we gate it".** Instance: R-259. |
| **G-4** | **Complete C1's runtime body assertion — 4 of 27 pages today.** | L | candidate — the expensive one, and honestly so | `secret_in_markup_gate.py` covers all 36 templates on the NAME-based check and is **blind to a secret under a neutral page-data key** — verified 2026-08-08 by replaying the three pre-fix templates through it: it convicts 2 of 3 and not the third. The runtime assertion catches all three; extending it means constructing each remaining page's data in a test, which is a per-page cost and is the real reason it has not been done. **Do NOT adopt the Go-side mirror Campaign 12 wrote as a gate on its own** — 27 candidates, 0 findings is bad signal-to-noise in front of every push. → R-255 |
+45
View File
@@ -1,3 +1,48 @@
## v0.99.0 — the hub can finally see whether the operator can get in (2026-08-08, R-260 / G-1)
**`oobDegraded` tested five things and the sixth never arrived.** The agent has emitted
`operator_key_configured` on every heartbeat since v0.72.0 — the same version that introduced the
`oob` stanza carrying it — and `store.HostOOBRow` mirrored five of the agent's eight OOB fields. With
no field for it, `encoding/json` discarded the fact on arrival, so a box with felhom-sshd **active,
reachable, a valid config and a configured peer** reported `ok` **with no operator key installed at
all**. That is not a wrong answer; it is an answer to a question nobody was asking.
`operator_peer_configured`, which the hub did read, is not a substitute: it says the peer IP is in
desired-state — that OOB is MEANT to work — not that entry is possible.
**Decoded now:** `operator_key_configured`, plus `wg_handshake_age_s` and `healed_at`. The last two
are carried for the ALERT TEXT and are deliberately NOT in the predicate — a check that starts
failing for reasons nobody asked for is how a check stops being read.
**`operator_key_configured` decodes as a POINTER.** nil = the agent never said, which is not the same
as saying no. Absence must never read as "the key is installed" — that is the same defect returning
through the version door, and this project has watched an absence read as a fact four times. A stanza
without the field cannot come from any released agent (field and stanza shipped together in v0.72.0,
and the vouched floor is far above it), so it is handled explicitly rather than assumed impossible,
and reported DISTINCTLY from a known-missing key.
**The message names the fault.** `oobDegradedReason` is now the single source for both the predicate
and the alert, so the text can never name a different fault from the one that fired. The old form
derived the reason separately and had a vocabulary of exactly two — "unreachable" or "config
invalid" — with no way to say "the key is missing". The operator reads this at 07:00 and needs to
know which of five things is wrong.
**Tests drive the DECODE BOUNDARY, not a hand-built struct.** Every hub OOB test before this one
constructed a `HostOOBRow` itself and asked what the checker did with it — a test written that way
cannot see a field that never decodes, which is exactly why this survived five weeks under a green
suite. `host_oob_decode_test.go` feeds raw report JSON; `TestHostOOB_MissingOperatorKey_EndToEnd` and
`TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd` drive the whole path from JSON to emitted event.
**The existing fixture was part of the problem and was fixed too:** `oobReport()` omitted
`operator_key_configured` entirely, so every pre-existing scenario ran against a report shape **no
released agent produces**. Same family as R-262.
Red-proofs, each with the mutation asserted applied: dropping the field from the decoder returns the
false `ok`; making the check unconditional alerts a healthy box; treating unknown as ok restores the
silent pass.
**Gate:** `felhom.eu/scripts/wire_contract_gate.py` (G-1), `--fast`, registered in `repo_gates.py`.
## v0.98.0 — drop the retained recovery package when the box says its set-aside history is gone (2026-08-07, R-241)
The hub half of the controller's abandonment countdown, and the **only** reason the hub was touched
+65 -11
View File
@@ -58,18 +58,58 @@ func NewHostOOBChecker(s *store.Store, onEvent EventNotifyFunc, logger *log.Logg
}
// oobDegraded is the degraded predicate: config invalid, OR (OOB meant to work — operator peer
// configured — AND felhom-sshd is not active/reachable).
// configured — AND felhom-sshd is not active/reachable OR the operator's key is not installed).
//
// THE KEY CLAUSE IS NEW (R-260, G-1) AND IT IS THE POINT. Until 2026-08-08 this predicate tested
// five things and the sixth — whether the credential that actually grants entry exists — never
// arrived, because the hub's decoder had no field for it. A box with the service active, reachable,
// a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED. That is not a
// wrong answer; it is an answer to a question nobody was asking.
//
// It is deliberately gated on OperatorPeerConfigured, exactly like the reachability clause: a box
// where OOB was never set up is not "broken", and widening this check beyond the fact that is now
// arriving is how a check stops being read.
func oobDegraded(r store.HostOOBRow) bool {
return oobDegradedReason(r) != ""
}
// oobDegradedReason returns the SPECIFIC reason a host's operator access is degraded, or "" when it
// is not. The reason is separated from the boolean because the operator reads the alert at 07:00 and
// needs to know WHICH of the things this checks is wrong — "out-of-band access degraded" is true and
// useless.
func oobDegradedReason(r store.HostOOBRow) string {
if !r.Present {
return false
return ""
}
if r.ConfigInvalid {
return true
return "felhom-sshd config invalid (sshd -t fails)"
}
if r.OperatorPeerConfigured && (!r.FelhomSshdActive || !r.Reachable) {
return true
if !r.OperatorPeerConfigured {
return ""
}
return false
if !r.FelhomSshdActive {
return "felhom-sshd is not active"
}
if !r.Reachable {
return "felhom-sshd unreachable (local dial to the OOB port fails)"
}
// The operator key, and the two ways it can be wrong. Both are reported distinctly, and NEITHER
// is a silent ok — an absence read as "the key is installed" is precisely the defect this clause
// was added to end, arriving through the version door instead.
if !r.OperatorKeyReported {
// Unreachable for any released agent: `operator_key_configured` and the `oob` stanza that
// carries it shipped together in agent v0.72.0 (2026-07-05), so a stanza without the field
// cannot come from a version anyone runs — the vouched floor is far above it. Handled
// explicitly anyway, because "cannot happen" is the kind of claim this project has been
// burned by, and pinned by TestOOBDegraded_StanzaWithoutKeyField_IsNotOK.
return "the agent reports operator access but is too old to say whether the operator key is " +
"installed (pre-v0.72.0) — treat entry as UNPROVEN, not working"
}
if !r.OperatorKeyConfigured {
return "the operator's authorized_key is NOT installed — felhom-sshd is up and answering, " +
"and nobody can log in through it"
}
return ""
}
// Check evaluates all hosts and emits oob_degraded / oob_recovered on transitions.
@@ -121,22 +161,36 @@ func (c *HostOOBChecker) IsDegraded(hostID string) bool {
func (c *HostOOBChecker) emit(row store.HostOOBRow, eventType, severity string) {
var msg string
if eventType == "oob_degraded" {
reason := "felhom-sshd unreachable"
if row.ConfigInvalid {
reason = "felhom-sshd config invalid (sshd -t fails)"
// The reason comes from the predicate itself, so the message can never name a different
// fault from the one that fired. The old form derived it separately and could only ever say
// "unreachable" or "config invalid" — it had no vocabulary for a missing operator key,
// which is the fault this checker most needs to be able to name (R-260).
reason := oobDegradedReason(row)
if reason == "" {
reason = "operator access degraded"
}
msg = "Host " + row.HostID + ": OPERATOR ACCESS DEGRADED — " + reason +
". The break-glass net (auto-heal + vaulted root@pam console) is still under the box."
} else {
msg = "Host " + row.HostID + ": operator access recovered (felhom-sshd reachable again)."
}
details, _ := json.Marshal(map[string]any{
det := map[string]any{
"host_id": row.HostID,
"felhom_sshd_port": row.FelhomSshdPort,
"active": row.FelhomSshdActive,
"reachable": row.Reachable,
"config_invalid": row.ConfigInvalid,
})
// R-260: carried for context, NOT consulted by the predicate.
"operator_key_configured": row.OperatorKeyConfigured,
"operator_key_reported": row.OperatorKeyReported,
}
if row.WGHandshakeAgeS != nil {
det["wg_handshake_age_s"] = *row.WGHandshakeAgeS
}
if row.HealedAt != "" {
det["healed_at"] = row.HealedAt
}
details, _ := json.Marshal(det)
c.logger.Printf("[%s] Host OOB: %s (%s)", map[string]string{"warning": "WARN", "info": "INFO"}[severity], row.HostID, eventType)
if _, err := c.store.SaveEvent(row.CustomerID, eventType, severity, msg, string(details), "hub"); err != nil {
c.logger.Printf("[WARN] save %s for %s: %v", eventType, row.HostID, err)
@@ -0,0 +1,113 @@
package monitor
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// Scenarios D, E and F of the R-260 / G-1 session, asserted on the CONSEQUENCE (does the check
// report degraded, and does it name the right thing) rather than on the mechanism.
func healthyRow() store.HostOOBRow {
return store.HostOOBRow{
HostID: "h1", CustomerID: "c1", Present: true,
FelhomSshdActive: true, Reachable: true, ConfigInvalid: false,
OperatorPeerConfigured: true,
OperatorKeyConfigured: true, OperatorKeyReported: true,
}
}
// Scenario D — the box this session exists for: everything up, no operator key.
func TestOOBDegraded_NoOperatorKey_IsDegradedAndSaysSo(t *testing.T) {
r := healthyRow()
r.OperatorKeyConfigured = false // reported, and false
if !oobDegraded(r) {
t.Fatal("a box with felhom-sshd active, reachable, a valid config, a configured peer and NO OPERATOR KEY reported ok — that is R-260, the exact question this check exists to answer")
}
reason := oobDegradedReason(r)
// §7.4: the operator reads this at 07:00 and must know WHICH of the five things is wrong.
if !strings.Contains(reason, "authorized_key") {
t.Errorf("the reason must NAME the missing operator key, not just say access is degraded; got %q", reason)
}
for _, wrong := range []string{"unreachable", "config invalid"} {
if strings.Contains(reason, wrong) {
t.Errorf("the reason names %q, which is not what is wrong: %q", wrong, reason)
}
}
}
// Scenario E — a healthy box must be unchanged. No new alert on a box that is fine.
func TestOOBDegraded_HealthyBoxUnchanged(t *testing.T) {
if oobDegraded(healthyRow()) {
t.Fatalf("a fully healthy box alerted: %q", oobDegradedReason(healthyRow()))
}
if got := oobDegradedReason(healthyRow()); got != "" {
t.Errorf("healthy box produced a reason %q", got)
}
}
// Scenario F — an agent too old to report the field. Absence must NOT read as "the key is
// installed". This project has watched an absence read as a fact four times.
func TestOOBDegraded_StanzaWithoutKeyField_IsNotOK(t *testing.T) {
r := healthyRow()
r.OperatorKeyReported = false
r.OperatorKeyConfigured = false // the zero value an absent field leaves behind
if !oobDegraded(r) {
t.Fatal("an agent that never said whether the operator key is installed was reported ok — that is the defect returning through the version door")
}
reason := oobDegradedReason(r)
if !strings.Contains(reason, "too old") || !strings.Contains(reason, "UNPROVEN") {
t.Errorf("the unknown case must be reported DISTINCTLY from a known-missing key, and must not claim the key is absent; got %q", reason)
}
// and it must NOT be worded as the known-missing-key case
if strings.Contains(reason, "nobody can log in") {
t.Errorf("unknown was reported as if it were a known-missing key: %q", reason)
}
}
// The peer gate is preserved: a box where OOB was never set up is not "broken", and this session
// must not widen the check beyond the fact that is now arriving.
func TestOOBDegraded_NoOperatorPeer_StillNotEvaluated(t *testing.T) {
r := healthyRow()
r.OperatorPeerConfigured = false
r.OperatorKeyConfigured = false
r.OperatorKeyReported = true
if oobDegraded(r) {
t.Errorf("a box with no operator peer configured must not alert on a missing key: %q", oobDegradedReason(r))
}
}
// The pre-existing faults must still fire, and still name themselves correctly.
func TestOOBDegraded_ExistingFaultsUnchanged(t *testing.T) {
for _, tc := range []struct {
name string
mutate func(*store.HostOOBRow)
expect string
}{
{"config invalid", func(r *store.HostOOBRow) { r.ConfigInvalid = true }, "config invalid"},
{"not active", func(r *store.HostOOBRow) { r.FelhomSshdActive = false }, "not active"},
{"unreachable", func(r *store.HostOOBRow) { r.Reachable = false }, "unreachable"},
} {
t.Run(tc.name, func(t *testing.T) {
r := healthyRow()
tc.mutate(&r)
if !oobDegraded(r) {
t.Fatalf("%s no longer degrades", tc.name)
}
if got := oobDegradedReason(r); !strings.Contains(got, tc.expect) {
t.Errorf("reason %q does not contain %q", got, tc.expect)
}
})
}
}
// A host with no oob stanza at all is never evaluated — pre-H1 or feature off.
func TestOOBDegraded_AbsentStanzaNeverAlerts(t *testing.T) {
if oobDegraded(store.HostOOBRow{HostID: "h1"}) {
t.Error("a host with no oob stanza alerted")
}
}
+85 -1
View File
@@ -4,13 +4,30 @@ import (
"io"
"log"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
_ "modernc.org/sqlite"
)
// oobReport builds a host report's `oob` stanza AS A RELEASED AGENT ACTUALLY SENDS IT.
//
// ⚠ IT DID NOT, UNTIL 2026-08-08 (R-260), and that mattered. This helper omitted
// `operator_key_configured` entirely, so every scenario below was driven by a report shape NO
// RELEASED AGENT PRODUCES — the field and the stanza that carries it shipped together in agent
// v0.72.0. A fixture that is not the wire cannot detect a field missing from the wire's receiver,
// which is one reason the gap survived five weeks under a green suite. Same family as R-262, where
// the cross-repo golden omits the two fields whose drift its key-set test is supposed to guard.
//
// The key defaults to INSTALLED here so the existing scenarios keep their original meaning
// (they vary the service, not the credential); the key's own scenarios live in
// host_oob_operatorkey_test.go, and the never-reported case is exercised by oobReportNoKeyField.
func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte {
return oobReportWithKey(active, reachable, configInvalid, operatorConfigured, true)
}
func oobReportWithKey(active, reachable, configInvalid, operatorConfigured, keyInstalled bool) []byte {
b := func(v bool) string {
if v {
return "true"
@@ -20,7 +37,16 @@ func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":` + b(active) +
`,"felhom_sshd_port":8822,"reachable":` + b(reachable) +
`,"config_invalid":` + b(configInvalid) +
`,"operator_peer_configured":` + b(operatorConfigured) + `}}`)
`,"operator_peer_configured":` + b(operatorConfigured) +
`,"operator_key_configured":` + b(keyInstalled) + `}}`)
}
// oobReportNoKeyField is the pre-v0.72.0 shape: an `oob` stanza with no operator_key_configured key
// at all. No released agent sends it; it exists so Scenario F is tested against real JSON rather
// than against a hand-set struct field.
func oobReportNoKeyField() []byte {
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":true,"felhom_sshd_port":8822,` +
`"reachable":true,"config_invalid":false,"operator_peer_configured":true}}`)
}
func newOOBStore(t *testing.T) *store.Store {
@@ -101,3 +127,61 @@ func TestHostOOB_NoStanzaIgnored(t *testing.T) {
t.Fatalf("no oob stanza must not alert, got %v", events)
}
}
// TestHostOOB_MissingOperatorKey_EndToEnd drives the WHOLE path — raw report JSON → SaveHostReport
// → GetHostOOBStates → the checker → the emitted event — for the box this session exists for.
//
// The per-predicate tests in host_oob_operatorkey_test.go set the row's fields directly, which is
// fine for the verdict but cannot prove the fact SURVIVES THE DECODE. This one can, and it is the
// test that would have failed before R-260.
func TestHostOOB_MissingOperatorKey_EndToEnd(t *testing.T) {
st := newOOBStore(t)
// healthy, key installed → seeded clean
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
var events []string
var msgs []string
c := NewHostOOBChecker(st, func(_, et, _, msg, _, _ string) {
events = append(events, et)
msgs = append(msgs, msg)
}, log.New(io.Discard, "", 0))
if c.IsDegraded("h1") {
t.Fatal("healthy host with the key installed seeded degraded")
}
// everything still up — only the operator's key is gone
st.SaveHostReport("h1", "c1", oobReportWithKey(true, true, false, true, false), store.HostReportDenorm{})
c.Check()
if len(events) != 1 || events[0] != "oob_degraded" {
t.Fatalf("a box whose operator key vanished must alert exactly once; got %v", events)
}
if !strings.Contains(msgs[0], "authorized_key") {
t.Errorf("the alert must NAME the missing key — the operator reads this at 07:00 and needs to know which of five things is wrong; got %q", msgs[0])
}
// the key comes back → recovered
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
c.Check()
if len(events) != 2 || events[1] != "oob_recovered" {
t.Fatalf("reinstalling the key must recover; got %v", events)
}
}
// Scenario F end to end: an `oob` stanza with no operator_key_configured key at all must NOT be a
// silent ok, and must be reported distinctly from a known-missing key.
func TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd(t *testing.T) {
st := newOOBStore(t)
st.SaveHostReport("h1", "c1", oobReportNoKeyField(), store.HostReportDenorm{})
rows, err := st.GetHostOOBStates()
if err != nil || len(rows) != 1 {
t.Fatalf("GetHostOOBStates: %v rows=%d", err, len(rows))
}
if rows[0].OperatorKeyReported {
t.Fatal("a stanza with no operator_key_configured decoded as though the agent had reported one")
}
if !oobDegraded(rows[0]) {
t.Fatal("an agent too old to report the key was treated as ok — absence read as a fact, which is this defect through the version door")
}
if got := oobDegradedReason(rows[0]); !strings.Contains(got, "too old") {
t.Errorf("the unknown case must say so distinctly; got %q", got)
}
}
+68 -17
View File
@@ -5,6 +5,13 @@ import "encoding/json"
// HostOOBRow is the latest operator-access (OOB) state per host (TASK H1), parsed from the newest
// host_report. Present is false when the agent sent no oob stanza (pre-H1 / feature off) → never
// alerted.
//
// ⚠ THIS STRUCT USED TO MIRROR FIVE OF THE AGENT'S EIGHT OOB FIELDS, and the three it dropped
// included the one that decides the question the OOB checker exists to answer (R-260, G-1). The
// agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the same version
// that introduced the stanza itself — and having no field for it here meant encoding/json discarded
// it on arrival, so `oobDegraded` reported a box with felhom-sshd active, reachable, a valid config,
// a configured peer and NO OPERATOR KEY INSTALLED as `ok`. The agent knew and said so.
type HostOOBRow struct {
HostID string
CustomerID string
@@ -14,6 +21,64 @@ type HostOOBRow struct {
Reachable bool
ConfigInvalid bool
OperatorPeerConfigured bool
// OperatorKeyConfigured — the operator's authorized_key is actually installed on the box.
// `operator_peer_configured` above is NOT a substitute: that one only says the peer IP is in
// desired-state, i.e. that OOB is MEANT to work. This says the credential that actually grants
// entry is there.
OperatorKeyConfigured bool
// OperatorKeyReported distinguishes "the agent said false" from "the agent never said".
// Absence must never read as "the key is installed" — that is this defect returning through the
// version door, and this project has watched an absence read as a fact four times.
//
// It cannot be false while Present is true for any RELEASED agent: the field and the stanza
// shipped together in v0.72.0 (2026-07-05), and the vouched floor is far above it. That is a
// claim, so it is pinned by TestOOBDecode_StanzaWithoutOperatorKey_IsNotSilentlyOK rather than
// left as a comment.
OperatorKeyReported bool
// WGHandshakeAgeS / HealedAt are carried for the ALERT MESSAGE, not for the predicate — the
// operator reads this at 07:00 and needs the context, but a check that starts failing for
// reasons nobody asked for is how a check stops being read. nil / "" when not reported.
WGHandshakeAgeS *int64
HealedAt string
}
// decodeOOBInto parses the `oob` stanza of one raw host report into r.
//
// It is a named function rather than an inline literal SO THAT TESTS CAN DRIVE THE REAL DECODE
// BOUNDARY. A test that constructs HostOOBRow by hand cannot see a field that never decodes, which
// is the entire class of defect this exists because of (R-260): every hub test asked what the
// checker did with a row, none asked whether the row could carry the fact.
func decodeOOBInto(r *HostOOBRow, reportJSON string) error {
var body struct {
OOB *struct {
FelhomSshdActive bool `json:"felhom_sshd_active"`
FelhomSshdPort int `json:"felhom_sshd_port"`
Reachable bool `json:"reachable"`
ConfigInvalid bool `json:"config_invalid"`
OperatorPeerConfigured bool `json:"operator_peer_configured"`
OperatorKeyConfigured *bool `json:"operator_key_configured"`
WGHandshakeAgeS *int64 `json:"wg_handshake_age_s"`
HealedAt string `json:"healed_at"`
} `json:"oob"`
}
err := json.Unmarshal([]byte(reportJSON), &body)
if body.OOB == nil {
return err
}
r.Present = true
r.FelhomSshdActive = body.OOB.FelhomSshdActive
r.FelhomSshdPort = body.OOB.FelhomSshdPort
r.Reachable = body.OOB.Reachable
r.ConfigInvalid = body.OOB.ConfigInvalid
r.OperatorPeerConfigured = body.OOB.OperatorPeerConfigured
// A POINTER, deliberately: nil means the agent never said, which is not the same as saying no.
if body.OOB.OperatorKeyConfigured != nil {
r.OperatorKeyReported = true
r.OperatorKeyConfigured = *body.OOB.OperatorKeyConfigured
}
r.WGHandshakeAgeS = body.OOB.WGHandshakeAgeS
r.HealedAt = body.OOB.HealedAt
return err
}
// GetHostOOBStates returns the latest oob stanza per host (mirrors GetHostMgmtPlaneStates). A report
@@ -35,23 +100,9 @@ func (s *Store) GetHostOOBStates() ([]HostOOBRow, error) {
if err := rows.Scan(&r.HostID, &r.CustomerID, &reportJSON); err != nil {
return nil, err
}
var body struct {
OOB *struct {
FelhomSshdActive bool `json:"felhom_sshd_active"`
FelhomSshdPort int `json:"felhom_sshd_port"`
Reachable bool `json:"reachable"`
ConfigInvalid bool `json:"config_invalid"`
OperatorPeerConfigured bool `json:"operator_peer_configured"`
} `json:"oob"`
}
_ = json.Unmarshal([]byte(reportJSON), &body)
if body.OOB != nil {
r.Present = true
r.FelhomSshdActive = body.OOB.FelhomSshdActive
r.FelhomSshdPort = body.OOB.FelhomSshdPort
r.Reachable = body.OOB.Reachable
r.ConfigInvalid = body.OOB.ConfigInvalid
r.OperatorPeerConfigured = body.OOB.OperatorPeerConfigured
if err := decodeOOBInto(&r, reportJSON); err != nil {
// malformed JSON degrades to zero values, never an error (documented above)
_ = err
}
out = append(out, r)
}
+100
View File
@@ -0,0 +1,100 @@
package store
import "testing"
// These tests drive decodeOOBInto — THE REAL DECODE BOUNDARY — with raw report JSON, not a
// hand-built HostOOBRow.
//
// That distinction is the entire point of R-260 and is not a style preference. Every hub test that
// touched OOB before this file constructed a HostOOBRow itself and asked what the checker did with
// it. A test written that way CANNOT SEE A FIELD THAT NEVER DECODES: it hands the struct the value
// the production path would have discarded, and passes. The defect lived under a green suite for
// five weeks because of exactly that.
const oobHealthyJSON = `{"host_id":"h1","oob":{
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
"config_invalid":false,"operator_peer_configured":true,
"operator_key_configured":true,"wg_handshake_age_s":42,"healed_at":"2026-08-08T01:02:03Z"}}`
// The shape the agent sends for the box this whole session is about: everything up, and the key
// that actually grants entry absent.
const oobNoKeyJSON = `{"host_id":"h1","oob":{
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
"config_invalid":false,"operator_peer_configured":true,
"operator_key_configured":false}}`
// An `oob` stanza with NO operator_key_configured key at all. Unreachable for any released agent —
// the field and the stanza shipped together in v0.72.0 — but absence must be structurally
// distinguishable from a reported false, or this defect returns through the version door.
const oobLegacyNoFieldJSON = `{"host_id":"h1","oob":{
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
"config_invalid":false,"operator_peer_configured":true}}`
func TestOOBDecode_CarriesOperatorKeyConfigured(t *testing.T) {
var r HostOOBRow
if err := decodeOOBInto(&r, oobHealthyJSON); err != nil {
t.Fatalf("decode: %v", err)
}
if !r.Present {
t.Fatal("stanza present but Present=false")
}
// THE ASSERTION THIS FILE EXISTS FOR. Before R-260 the struct had no field, so this line does
// not compile against the old code — which is the red-proof, recorded rather than described.
if !r.OperatorKeyConfigured {
t.Error("operator_key_configured=true on the wire did not reach the row — it is being dropped at the decode boundary, which is R-260")
}
if !r.OperatorKeyReported {
t.Error("the field was on the wire; OperatorKeyReported must be true")
}
if r.WGHandshakeAgeS == nil || *r.WGHandshakeAgeS != 42 {
t.Errorf("wg_handshake_age_s did not decode: %v", r.WGHandshakeAgeS)
}
if r.HealedAt != "2026-08-08T01:02:03Z" {
t.Errorf("healed_at did not decode: %q", r.HealedAt)
}
}
func TestOOBDecode_ReportedFalseIsDistinctFromNotReported(t *testing.T) {
var no HostOOBRow
if err := decodeOOBInto(&no, oobNoKeyJSON); err != nil {
t.Fatalf("decode: %v", err)
}
if no.OperatorKeyConfigured {
t.Error("operator_key_configured=false decoded as true")
}
if !no.OperatorKeyReported {
t.Error("the agent DID say false; that must be distinguishable from never saying")
}
var legacy HostOOBRow
if err := decodeOOBInto(&legacy, oobLegacyNoFieldJSON); err != nil {
t.Fatalf("decode: %v", err)
}
if legacy.OperatorKeyReported {
t.Error("no operator_key_configured key on the wire, yet OperatorKeyReported=true — absence is being read as a statement")
}
// Both are `false`; only the Reported flag tells them apart. If a future change collapses the
// pointer to a plain bool, this is the assertion that fails.
if no.OperatorKeyConfigured != legacy.OperatorKeyConfigured {
t.Fatal("precondition of this test changed")
}
}
func TestOOBDecode_AbsentStanzaIsNotPresent(t *testing.T) {
var r HostOOBRow
_ = decodeOOBInto(&r, `{"host_id":"h1"}`)
if r.Present {
t.Error("no oob stanza, yet Present=true")
}
if r.OperatorKeyReported {
t.Error("no oob stanza, yet OperatorKeyReported=true")
}
}
func TestOOBDecode_MalformedDegradesToZeroNeverPanics(t *testing.T) {
var r HostOOBRow
_ = decodeOOBInto(&r, `{"oob":`) // truncated
if r.Present {
t.Error("malformed JSON must not yield Present=true")
}
}
+79 -3
View File
@@ -94,10 +94,86 @@ GENERIC = {
"created_at", "updated_at", "started_at", "timestamp", "time", "percent", "used_fraction",
}
# A tag that is emitted and deliberately NOT consumed. Every entry is a claim someone must be able
# to re-check, so each carries a reason. A quiet exclusion would be a dropped field with paperwork.
# A tag that is emitted and NOT consumed, with the reason it is acceptable. Every entry is a claim
# someone must be able to re-check later, so none of them is bare. A quiet exclusion would be a
# dropped field with paperwork, which is worse than the defect.
#
# TWO KINDS OF ENTRY, and the difference is deliberate:
# * "redundant" — the hub already decodes something that answers the same question. No consumer
# is wanted; the entry is the end of the matter.
# * "R-264" — a fact with no consumer that ARGUABLY should have one. The entry does NOT
# close the question; it records it against an OPEN register row so that
# allowlisting cannot be mistaken for deciding. R-260 is closed by the gate plus
# the operator-access fix; the leftover appetite is R-264.
#
# Keyed by (wire label, dotted emit path).
ALLOWLIST = {}
_AH = "agent -> hub (POST /host-report)"
_CH = "controller -> hub (POST /report)"
_REDUNDANT_HOST_METRICS = (
"redundant: the hub decodes host.cpu_percent / memory_percent / disk_percent from the same "
"stanza and every host-health threshold is expressed on those. The absolute figure answers no "
"question the hub asks.")
_R264 = (
"no consumer today, and one is arguably owed — recorded against R-264 (OPEN) rather than "
"decided here. Allowlisting is not deciding.")
ALLOWLIST = {
# ---- redundant: the hub already decodes an equivalent ----
(_AH, "host.cpu_temp_c"): _REDUNDANT_HOST_METRICS,
(_AH, "host.loadavg"): _REDUNDANT_HOST_METRICS,
(_AH, "host.memory_total_bytes"): _REDUNDANT_HOST_METRICS,
(_AH, "host.memory_used_bytes"): _REDUNDANT_HOST_METRICS,
(_AH, "host.uptime_seconds"): _REDUNDANT_HOST_METRICS,
(_CH, "system.load_avg_1"): _REDUNDANT_HOST_METRICS,
(_CH, "system.load_avg_5"): _REDUNDANT_HOST_METRICS,
(_CH, "system.load_avg_15"): _REDUNDANT_HOST_METRICS,
(_CH, "system.memory_total_mb"): _REDUNDANT_HOST_METRICS,
(_CH, "system.memory_used_mb"): _REDUNDANT_HOST_METRICS,
(_CH, "system.temperature_celsius"): _REDUNDANT_HOST_METRICS,
(_CH, "system.uptime_seconds"): _REDUNDANT_HOST_METRICS,
(_AH, "guests.spec.disk_bytes"): (
"redundant: guest SIZING is hub-owned intent (the manifest), not box reality. The hub "
"decodes vmid/name/status/controller_version from the guest list and nothing more."),
(_AH, "guests.spec.memory_bytes"): (
"redundant: as guests.spec.disk_bytes — sizing is hub-owned intent, not mirrored reality."),
(_AH, "storage_targets.smart.model_name"): (
"redundant for a verdict: the hub decodes smart.health plus every counter it bands on. The "
"model string is a display label with no threshold attached to it."),
(_AH, "wireguard.last_handshake_age_s"): (
"redundant: hub-side wgsync reconciles peers from its own state, and the OOB path's own "
"wg_handshake_age_s IS now decoded (into HostOOBRow, for the alert text)."),
# ---- no consumer, and one is arguably owed: R-264, OPEN ----
(_AH, "guest_net"): _R264 + " The R-54 guest-network watchdog stanza (whole object).",
(_AH, "guest_net.checked_at"): _R264,
(_AH, "guest_net.guests.has_route"): _R264,
(_AH, "guest_net.guests.dhclient_alive"): _R264,
(_AH, "guest_net.guests.heal_succeeded"): _R264,
(_AH, "guest_net.guests.heals_last_hour"): _R264,
(_AH, "guest_net.guests.last_heal_at"): _R264,
(_AH, "guest_net.guests.damped"): _R264,
(_AH, "selfupdate_pending"): _R264 + (
" NOTE: the agent's own comment beside this field claimed 'the hub reads an absent field as "
"pending=false, the correct default'. The hub had no field at all, so it read nothing "
"either way. The comment was corrected in the same change as this entry."),
(_AH, "selfupdate_pending_version"): _R264,
(_AH, "mgmt_plane.healed_recently"): _R264 + (
" The hub DOES alarm on mgmt_plane.privsep_healed_at, which is the timestamp beside this "
"boolean, so the recurring-clobber signal is not lost — only this flag is."),
(_AH, "pbs_dr.applied_at"): _R264,
(_AH, "restore_tests.mount_parity"): _R264 + (
" R-262: the hub's own comment claims this contract is mirrored field-for-field and that a "
"key-set test guards drift; it is two fields short and the fixture omits the same two."),
(_AH, "restore_tests.mount_inventory"): _R264 + " R-262, as mount_parity.",
(_CH, "config_hash"): _R264,
(_CH, "reporting_disabled"): _R264,
(_CH, "stacks"): _R264 + (
" The whole per-stack report object; the hub's app view is built from app_telemetry."),
(_CH, "storage.migrated_to"): _R264,
(_CH, "backup.last_db_dump"): _R264,
(_CH, "backup.last_integrity_check"): _R264,
}
# A node whose IMMEDIATE CHILDREN are still checked but whose DEEPER descendants are not, because the
# receiver passes the subtree through without decoding it. Keyed by (wire label, dotted path).