Four paper debts and one fact given a reader. Hub-only — nothing to bake.
A4 — the entry about "the tester's machine" named a risk correctly and labelled it
in a way that invited deleting it. Established from the hub's own store: `peti-felhom`
is a REAL machine (482 reports, 2026-02-27 → 2026-07-15, a named person's own box) and
the 3.6 GB with no key and no backup is real. `david` → `tester-1` is a DIFFERENT record
with no host, no escrow and no report, ever — deleted 07:55:49 and re-created 07:56:47
this morning. The prompt's premise conflated the two; the register now says which is which.
A1 — R-312/R-313/R-303 recorded as DECIDED with their re-open triggers, and moved out
of STATUS's "Waiting on you", which is now empty.
A3 — day0-install §C.1 said pushing the installer publishes it. It has not since
R-110. Corrected, with the two manifest pins named and an outside-verification command;
the one copy that repeated it (a dated audit, true when written) carries a superseded note.
A5 — standing rule 5: evidence comes off the machine at the end of the phase that
produced it, before any revert. Earned twice in three days on the same box at the same
point (R-320). Four homes, plus what to do when it is already gone.
R-295 hub half — „Beállító kód" everywhere; „Visszaállító kód" retired. New `reenroll`
mail kind so the mail names the page a REBUILT box actually shows („A szerver
beállítása"), not the „Elfelejtett jelszó" page it has no login screen to reach.
Naming only; the acceptance pin proves the secret is untouched.
R-319 — the hub models `guest_net` after 23 days of receiving and discarding it. The
signal is `heals_last_hour`, not `state`: a guest the watchdog keeps repairing reads
healthy between repairs. `heal_succeeded` decoded too (R-260's lesson). Unknown is never
drawn as healthy — three absences, three sentences. No alarm, deliberately.
Three red-proofs, mutations asserted applied. Wire-gate checked tags 182 → 190.
B1 — the operator's 2026-08-12 dispositions were NOT in the register; they are now.
Third allowlist kind for the five ruled "no reader wanted"; `reporting_disabled`
reclassified redundant. 8 read · 5 deliberately unread · 1 redundant · 6 still owed.
Also filed: R-321 (a deliberately-silent box still alarms stale/down — the checker is
age-only, and decoding the flag would not have fixed it), R-322 (the claim guard has
never scanned the hub; a hand scan returns zero, so it is a scope gap, not a defect).
| WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | `SPIKE-oob-wg-operator-peer-2026-07-05`, `SPIKE-felhom-sshd-2026-07-05` | Mutual-repair desired-state arc not built → R-13. **CHECKED 2026-08-08 (R-260) and this row was NOT claiming something untrue** — it claims the capability is implemented, never that it is monitored, so no correction was owed. What WAS untrue is narrower and sat one layer down: **the hub's own OOB health check could not see whether the operator's key was installed.**`HostOOBRow` mirrored five of the agent's eight OOB fields, so `operator_key_configured` — emitted every heartbeat since agent v0.72.0, i.e. from this row's own vintage — was discarded by `encoding/json` on arrival, and `oobDegraded` returned `ok` for a box with felhom-sshd active, reachable, a valid config, a configured peer and **no operator key at all**. `operator_peer_configured`, which it did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Fixed hub v0.99.0; the missing key now degrades and the alert NAMES it; a stanza too old to carry the field is reported distinctly and is never a silent ok. Pinned end-to-end from raw report JSON by `TestHostOOB_MissingOperatorKey_EndToEnd` and `TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd` |
| The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent **v0.119.0**, hub **v0.85.0** | **PROVEN-LIVE** (2026-07-31) | `audits/host-addresses-visible-2026-07-31.md` | Before this the LAN IP was **not reportable at all** — `HostMetrics` carried no address of any kind — and the WG IP existed only in `/offsite`'s peer table keyed by pubkey (peer→host, never host→peer). New wire field `addresses[]`, one row per (interface, address); `IsGlobalUnicast()` is the whole filter, chosen by MEASURING both demo boxes, and it needs no veth/fwbr denylist because that plumbing carries no IP. Rendered live on both 0.119.0 hosts matching their `ip addr` ground truth exactly. **Two honesty properties carry the risk and are both red-proofed:** WireGuard shows the hub ALLOCATION and whether the box CONFIRMS holding it (allocation alone cannot tell a live tunnel from a peer never applied), and an agent below 0.119.0 renders **UNKNOWN, never "no addresses"** — proven live on `drill-r50-0a4f9a` (0.113.0). **Not covered:** a two-LAN-bridge box and a real WG drift, neither of which exists to observe |
| The operator can see whether a managed host's **guests still have working networking** — and **how often the watchdog had to repair them** | agent **v0.92.0** (emitter, 2026-07-21), hub **v0.104.0** (reader, 2026-08-13) | **IMPLEMENTED** | `backlog/OPEN-ITEMS.md` R-319; `hub/internal/web/hosts_guestnet_test.go` (7 tests, fixtures copied verbatim from `demo-felhom-8363b5`'s live `host_reports` row) | The agent emitted `guest_net` on every heartbeat for **twenty-three days** while the string occurred **nowhere** in `felhom.eu/hub/` — stored as raw text in `report_json`, read by nothing (R-260/R-264, the first of that census's readers to be built). **The fact that carries the risk is `heals_last_hour`, not `state`:** a guest the watchdog keeps repairing is healthy at every instant anyone looks, so rendering the state alone would give it a green tick — the failed-disk-drawn-as-a-healthy-empty-disk shape. `heal_succeeded` is decoded beside it, because six FAILED repairs is a guest that is down while six successful ones is a nuisance. **Unknown is never drawn as healthy:** three absences, three sentences (agent < 0.92.0; a capable agent that sent nothing; a guest whose own state the watchdog did not assert), and a malformed stanza degrades to unknown without a 500. **Three red-proofs, each mutation asserted applied by grep before its run**, including the one that matters — removing the unknown branches and watching a silent machine render as healthy. **Positive control that it is WIRED and not merely written: the wire-contract gate's checked-tag count rose 182 → 190** as the eight `guest_net` allowlist entries were deleted (an allowlisted tag is skipped, so leaving them would have meant these fields were never checked) | **IMPLEMENTED, not PROVEN-LIVE, and the distinction is the honest half.** Every scenario is proven against the real wire in tests, and the healthy case renders correctly for the live fleet — but **no machine has ever been observed with a climbing repair count on this card**, because neither demo box has needed a repair since the watchdog shipped. The signal this card exists for has therefore never been seen firing on hardware. It moves to PROVEN-LIVE the first time a real repair count is watched appearing. **No alarm was added, deliberately** (R-319): the incident behind this was about nobody being able to SEE the condition, and a new email on a fleet of two demo machines is untested noise — revisit when a third machine exists or when a count is seen climbing |
| Break-glass management-plane recovery | agent v0.71, hub v0.84 | **IMPLEMENTED** | `runbooks/break-glass.md` | hub v0.84.0 adds an **operator-SESSION** retrieval path (host page → Console access → Reveal; `POST /hosts/{id}/reveal-recovery-credential`, CSRF-gated, writes a customer-visible `recovery_credential_revealed` event) beside the pre-existing **global-key** one (`GET /api/v1/admin/hosts/{id}/recovery-credential`), which is untouched and stays the route for when the hub UI itself is down. **The credential half is now PROVEN (2026-07-31):** the vaulted `demo-hp-bb76ea` password was verified against the box's own `/etc/shadow` hash AND minted a real PVE ticket — `POST /api2/json/access/ticket` → **HTTP 200, `root@pam`, 367-char ticket**, the exact API the login form submits to. Still IMPLEMENTED rather than PROVEN-LIVE overall, because the path has not been exercised on a REAL lockout (SSH was available throughout). Discovered during that check: the card's Copy button shipped `disabled` until a Reveal and silently no-opped, leaving ANOTHER host's password in the clipboard — fixed in hub v0.86.0. The vaulted secret is plaintext at rest → **R-133** |
| ownership **read** | `:1097` (`_state_get dnsmasq_preexisting`) at uninstall |
| state file **deleted** | `:1268` — *after* the read. **The order was already correct.** |
| Felhom's dnsmasq snippets removed | `:1162`, in the loop just above the ownership decision |
**Why cycle 2 concludes "pre-existing": package presence alone.** The preflight asks dpkg one question
and nothing else — it does **not** consult the absence of a record, and it cannot, because the record
was deleted with the state file. v1.27.0 stopped the unit and left the package, so the answer stayed
`yes` and our own package became "the household's" one cycle later.
**What the first uninstall did and did not remove:** snippets — removed. Unit — stopped + disabled.
State file (and with it the record) — removed. **Package — left.** That last one is the whole defect.
## 3. What v1.28.0 changes
When the record says we installed it, the uninstall **removes the package** as well as stopping the
unit. Order unchanged: read the record → act → delete the state.
**Two packages are recorded, not one.** `dnsmasq` ships the systemd unit; **`dnsmasq-base` ships
`/usr/sbin/dnsmasq`** (`dpkg -S`, measured on the box). Separately installable, so each is recorded at
preflight and taken back only if we added it.
**Guard rails:** ownership **read, never inferred**; the dependency check is an **`apt-get -s purge`
simulation** that proceeds only if the removal set is a subset of ours, else stop+disable **naming the
blocking package**; never interactive; **never fatal** — a wedged apt is recorded and restated in the
closing NOTE; and the success is **re-queried** rather than read off apt's exit code.
## 4. Scenarios and red-proofs
| Scenario | Result |
|---|---|
| **A** three cycles, fixed | install 3 **PASSES**, `:53` free at every uninstall |
| **B** resolver pre-dates Felhom | **untouched** — `install ok installed`, unit active, both packages recorded `yes` |
| **C** something depends on it | **not purged**; log named `household-dns-thing`; stop+disable; `:53` free; dependent survived |
| **D** no ownership record | **untouched**, reason logged, exact command named |
**Red-proofs — mutation asserted applied before each run:**
| Mutation | Outcome |
|---|---|
| remove the purge call | cycle 2 records `yes`; **cycle 3 refuses, exit 1, in those exact words** |
| remove the ownership check | **the household's resolver is PURGED** (`unknown ok not-installed`) |
| infer ownership when no record | **the guess is taken** — a field box loses its own DNS |
**One honest note on red-proof B.** Its first run appeared to pass *for the wrong reason*: apt failed
with `dpkg was interrupted` (a broken state my own earlier manual package juggling left), so the
resolver survived by accident rather than by the guard. I repaired dpkg and re-ran, and it then failed
as required. Worth stating twice over: the mutation looked like a passing guard, and **my own fix's
apt-failure path was incidentally observed doing exactly what it should** — logging, continuing, naming
the command.
## 5. The machines already in the field
**Measured, not reasoned.** A field box is one with the package installed and **no record** — scenario
D. Its uninstall leaves the resolver running with the reason logged, and its next install refuses with
the message quoted in §1.
**Is there an honest durable marker? No, and none can be invented.** The Felhom `/etc/dnsmasq.d/
felhom-*.conf` snippets are deleted by the uninstall's own loop *before* the ownership decision; the
state file carrying the record is deleted at `:1268`; nothing under `/etc/felhom*` survives.
`/var/log/dpkg.log` does record the install and is a **timestamp** — refused by the standing rule as a
heuristic dressed as a fact. → **R-318**
**The preflight message, judged as a customer would:** it states the finding, keeps its two routes and
its written promise not to touch DNS on a host we do not own, and adds the one thing that gets a person
moving — *"THIS LOOKS LIKE OURS"* plus one exact command. It hedges correctly (*looks like*). Its
weakness is that it asks them *"did this host have dnsmasq before Felhom?"* — precisely the question we
can no longer answer for them. **That is a mechanism, not a rule: the command is on the screen at the
moment it is needed.**
## 6. The defect I nearly shipped
The agent decides whether to install dnsmasq with `os.Stat("/usr/sbin/dnsmasq")`
(`felhom-agent/internal/lanresolver/lanresolver.go:105`) — but that path belongs to **`dnsmasq-base`**,
while the unit comes from **`dnsmasq`**. Purging only `dnsmasq` would leave the binary, so the next
install would skip the apt step and then fail to enable a unit that is gone — **a silent resolver where
today there is at least a visible refusal.** That is why ownership is recorded per package.
It remains reachable where `dnsmasq-base` pre-dated Felhom (we correctly keep it). **Pre-existing, not
introduced here**, filed as **R-317**, and the uninstall now says so out loud rather than leaving it to
be found from a resolver that never came up. Fixing it is a one-line agent change, deliberately **not**
made here to keep this session to one repo.
## 7. Publication
`installer-v1.28.0` tagged; **both**`--ref`s in `manifests/webpage.yaml` moved (sidecar 327, init 372);
ArgoCD synced to `823cd29`; live deployment carries the new ref.
```
live URL serves: SCRIPT_VERSION="1.28.0" (was 1.27.0)
_dnsmasq_purge_owned present in the served file: 3
```
**Pushing publishes nothing here** — `/scripts/` follows the tag. The runbook still says otherwise;
**R-309 remains open and is not forgotten**, deliberately out of scope.
## 8. Teardown — four layers
| Layer | State |
|---|---|
| **machine** | `drill-r50`: 0 guests, no `/var/lib/felhom-install`, dnsmasq `not-installed` at the end |
| **host** | `drill-r50`**reverted to snapshot `virgin`, powered off**; `qemu.pid` removed. DooPlex was drill host only |
| **hub** | `drill-r50-0a4f9a`**RETAINED** — pre-existing since 2026-07-25, re-used not duplicated. **No new host or customer row was created this session.** Nothing deleted |
| **off-site** | **Not contacted at all** — no restic, sftp or endpoint call was made. `demo-felhom`: `last_status=ok`, `escrow=escrowed`, **no abandon countdown** |
Both demo boxes untouched and healthy: controller **0.214.0**, agent **0.129.0**, `Up … (healthy)`.
## 9. Evidence, and a repeated miss
Logs at `documentation/audits/evidence-r316-2026-08-13/`: the fixed cycles (`F1`, `F2`, `F3`), all four
scenarios (`B`, `C`, `D`), and every red-proof (`RPA*`, `RPB*`, `RPD*`).
**The Part 1 logs did not survive.** They were on the drill VM's disk and were destroyed by the revert
to `virgin` between Part 1 and Part 2 — **the same mistake as Tuesday, in the same place.** The §1
quotation is verbatim from the live run as read at the time, and `RPA1/RPA2/RPA3.log` are an
independent reproduction of the identical three-cycle failure, retained. Recorded rather than glossed;
the fix is procedural and I have now got it wrong twice.
## 10. Observations — noticed, not acted on
- **`dnsmasq-base` is flagged "automatically installed and no longer required"** after `dnsmasq` goes.
We deliberately do not `autoremove` — that would be a blast radius nobody asked for.
- **`apt-get -s purge` exits 0 even when it prints `E: dpkg was interrupted`.** The simulation output is
the signal, never the exit code — which is why the guard parses `Remv`/`Purg` lines rather than
trusting `$?`. Another entry for the exit-codes-that-lie class.
- The drill VM's `pveam` index is stale on `virgin` and needs `pveam update` before listing templates —
already recorded yesterday, hit again today in passing.
@@ -307,7 +307,7 @@ unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing
| **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — |
| **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:**`a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — |
| **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — |
| **PETI** | **`peti-felhom` deliberately NOT migrated — and the mitigation this row used to name DOES NOT EXIST.** This row said a drive failure there is *"offsite-only recovery"*. Re-read from the hub's own store on 2026-08-10 and again on 2026-08-12, with a control run first (the escrow query returns 1+1 rows for each demo box and 0+0 for `drill-r50`, so it distinguishes the states): **there is no off-site copy, no key, and no local backup either.** Three independent reasons, each a fact rather than an inference: **(1)** the host row was DELETED — `host_deletions` id=1, `peti-felhom-86d37d`, **2026-07-15 08:56:22**, `escrow_acked = 0` — long before host-delete-demotes-escrow-to-retained-custody existed, so nothing was carried over; **(2)** there is **no escrow row of any kind**, current or superseded (only 4 exist hub-wide, all belonging to the two demo boxes), and the off-site restic REPOSITORY password lives in `identity_blob` on that row (`hub/internal/store/store.go:381`) — its only other copy is `<dataDir>/offbox/repo_password` (`controller/internal/backup/offbox.go:395`) **on the very disk whose failure is the scenario**; **(3)** off-site backup **never ran once** — its last report carried `offsite: {escrow_state: "pending", snapshot_count: 0, repo_size_bytes: 0}`, and that is the fork-4 guard working exactly as designed (`controller/internal/settings/settings.go:317-320`: *"no offsite run proceeds until an operator confirms the escrow ceremony"*), not a fault. The local app-data restic repo was also empty (`snapshot_count: 0, integrity_ok: false`), and the whole-guest vzdump shares the failing device. **SO: if that drive fails today, everything on it is lost.****Size, so this is not read as larger than it is:** one lightly-used test box — a single `/` mount, **3.6 GB used of 48.9 GB**, one catalogued app (`rallly`); the dashboard was never claimed (`customer_claims.claimed_at` NULL). **BOUNDARY:** every fact is as of the last report, **2026-07-15 08:39:00 UTC** (controller 0.115.0); the hub has heard nothing since and confirming today's state would mean contacting the machine, which is fenced. **Contact since deletion:** no inbound row of any kind after 2026-07-15 08:39 — the only later rows are the hub's OWN staleness alarms (`source = hub`: `node_stale` 09:09:32, `node_down` 09:39:32) — and no contact attempt, accepted or rejected, in the current hub pod's logs (since 2026-08-09 17:26 UTC; grep proven by 851 `demo-hp` hits against 0 for peti, 0 unauthorized). **The window 2026-07-15 → 2026-08-09 cannot be answered from records**: a report from a deleted host 401s and is not persisted, and those logs are gone. **THE RULING IS LEFT OPEN DELIBERATELY** — whether the machine stays parked is the operator's call and does not need restating here; this row records the FACT, which does not need his opinion to be true | **PARKED — the recorded mitigation is void; ruling OPEN for the operator** | — | **First act of the visit: copy that ~3.6 GB off before anything is reinstalled — it is currently the only copy in existence.** Do not migrate, do not contact | operator |
| **PETI** | **`peti-felhom` deliberately NOT migrated — and the mitigation this row used to name DOES NOT EXIST.** This row said a drive failure there is *"offsite-only recovery"*. Re-read from the hub's own store on 2026-08-10 and again on 2026-08-12, with a control run first (the escrow query returns 1+1 rows for each demo box and 0+0 for `drill-r50`, so it distinguishes the states): **there is no off-site copy, no key, and no local backup either.** Three independent reasons, each a fact rather than an inference: **(1)** the host row was DELETED — `host_deletions` id=1, `peti-felhom-86d37d`, **2026-07-15 08:56:22**, `escrow_acked = 0` — long before host-delete-demotes-escrow-to-retained-custody existed, so nothing was carried over; **(2)** there is **no escrow row of any kind**, current or superseded (only 4 exist hub-wide, all belonging to the two demo boxes), and the off-site restic REPOSITORY password lives in `identity_blob` on that row (`hub/internal/store/store.go:381`) — its only other copy is `<dataDir>/offbox/repo_password` (`controller/internal/backup/offbox.go:395`) **on the very disk whose failure is the scenario**; **(3)** off-site backup **never ran once** — its last report carried `offsite: {escrow_state: "pending", snapshot_count: 0, repo_size_bytes: 0}`, and that is the fork-4 guard working exactly as designed (`controller/internal/settings/settings.go:317-320`: *"no offsite run proceeds until an operator confirms the escrow ceremony"*), not a fault. The local app-data restic repo was also empty (`snapshot_count: 0, integrity_ok: false`), and the whole-guest vzdump shares the failing device. **SO: if that drive fails today, everything on it is lost.****Size, so this is not read as larger than it is:** one lightly-used test box — a single `/` mount, **3.6 GB used of 48.9 GB**, one catalogued app (`rallly`); the dashboard was never claimed (`customer_claims.claimed_at` NULL). **BOUNDARY:** every fact is as of the last report, **2026-07-15 08:39:00 UTC** (controller 0.115.0); the hub has heard nothing since and confirming today's state would mean contacting the machine, which is fenced. **Contact since deletion:** no inbound row of any kind after 2026-07-15 08:39 — the only later rows are the hub's OWN staleness alarms (`source = hub`: `node_stale` 09:09:32, `node_down` 09:39:32) — and no contact attempt, accepted or rejected, in the current hub pod's logs (since 2026-08-09 17:26 UTC; grep proven by 851 `demo-hp` hits against 0 for peti, 0 unauthorized). **The window 2026-07-15 → 2026-08-09 cannot be answered from records**: a report from a deleted host 401s and is not persisted, and those logs are gone. **THE RULING IS LEFT OPEN DELIBERATELY** — whether the machine stays parked is the operator's call and does not need restating here; this row records the FACT, which does not need his opinion to be true. **⚠ WHO THIS IS ABOUT — CORRECTED 2026-08-13, because the row was read as describing a record rather than a machine and that reading nearly deleted a true risk.** Every summary since 2026-08-09 has led with *"the tester's machine"*, and there are now **two different things** that word can mean, one of which carries no risk at all. **This row is about the machine: a physical 80-core Proxmox server belonging to a named person, running Felhom as a BYO guest** (`pilot/PETI-tester-agreement.md`, *"Operator: Viktor. Tester: Peti"*), which **reported to the hub 482 times between 2026-02-27 and 2026-07-15 08:39:00 UTC** (`reports`, counted). It is Tier 2 — protected — *"because there is a real person behind it"* (D-d). **The 3.6 GB and the missing recovery route are therefore real, and this row keeps its rank.****The other thing is `tester-1`, and it is not this:** a customer record created **2026-08-13 07:56:47**, one minute after the record previously called `david` was torn down (`customer_resets` id 16, 07:55:49→07:55:50, every leg `ok`, `hetzner` skipped; `customer_deleted` event 2960). It has **no host row, no escrow row, no host-report and no controller report — measured, all zero** — and `david` before it had the same and was never anything else (its only four events were three hub-side `expected_dbdump_missed` false alarms and its own deletion; that is R-195's subject). **A record with no machine behind it can lose nothing.** So the operator's *"there is no actual tester yet — only a pre-created customer, now renamed"* is true of the **pilot programme and of `tester-1`**, and not of this row: the agreement was drafted, the onboarding runbook stopped at P1, and no pilot ever formally began — **while the hardware and the data have existed the whole time.****Nothing about the risk changed; only the word that names it.** Wherever a document says *"the tester's machine"*, read `peti-felhom` | **PARKED — the recorded mitigation is void; ruling OPEN for the operator** | — | **First act of the visit: copy that ~3.6 GB off before anything is reinstalledLine truncated
| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — |
| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — |
| **R-122** | ~~`AssembleDRRecipe` silently DROPPED `offsite_restic` — the offsite recovery location never reached any recipe~~ | **SHIPPED** (hub v0.83.0, 2026-07-30) | — | **Found 2026-07-30 while scoping R-109's hub half; it had already shipped and nobody knew.** The controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*), the hub stored it for **all three real customers**, and `appHalfShape` never listed the key — so no delivered recipe has ever contained it. No error, no log, green suite, because the fixture `drAppHalf` is hand-written and omits the field. `hostHalfShape`/`appHalfShape` are **ALLOW-LISTS dressed as forward-compat**; `TestAssembleDRRecipe_CarriesEveryEmittedSection` is now the guard, built on halves read verbatim out of the live `dr_recipe` table. `REUSE.md` (both repos) records that a recipe section is a TWO-REPO change | — |
@@ -439,7 +439,7 @@ builds the receiving struct by hand cannot see a field that never decodes, which
| ID | What | State |
|---|---|---|
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.****(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.****(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change | **READY** — owner Viktor |
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.****(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.****(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change. **⚠ DISPOSITIONS RECORDED 2026-08-13 — and the first thing to say is that they were NOT in this register before today.** The rulings were made on 2026-08-12; this row still read **READY — owner Viktor** and the gate's twenty entries still all said *"arguably owed"*, so a session told to *"re-read the dispositions from the register"* would have found none. They are written down now, which is the point of writing them down. **THE COUNT WAS ALSO WRONG:** this row says *twenty-one*; the gate's allowlist held **twenty**, measured. Twenty is the number the dispositions below account for, exactly. **(1) BUILD A READER — four groups, fourteen facts.** (a) guest-network health, (b) the staged-update-pending pair, (d) the restore-test depth pair, (f-part) the two backup-integrity timestamps. **(2) NO READER WANTED — five facts, now recorded as `not consumed, DELIBERATELY` with the ruling and its date** in `scripts/wire_contract_gate.py`, each with its own reason rather than a bare refusal: `mgmt_plane.healed_recently` (the hub already alarms on the timestamp beside it), `pbs_dr.applied_at` (`pbs_dr.state` is the verdict; the timestamp alone is the attempt-read-as-result trap), `config_hash` (the hub authors the config and knows its own generation), `stacks` (the app view is built from the purpose-built `app_telemetry` wire), `storage.migrated_to` (box-local bookkeeping with no hub-side intent to reconcile against). **The emitters are deliberately left alone** — removing one is a coordinated two-repo change and breaks the host-report golden; the honest end here is a recorded decision, not a deletion. The gate grew a THIRD entry kind to carry them, because an undecided fact and a decided one must not read alike. **(3) `reporting_disabled`, decided on its own merits: RECLASSIFIED `redundant`** — `health.status = "disabled"` travels in the same minimal report, is decoded into `reports.health_status`, and IS rendered. **The decision surfaced a real defect the flag would not have fixed: the staleness checker is age-only, so a deliberately-silent box still alarms → R-321.****PROGRESS, 2026-08-13: the first reader is BUILT — guest-network health, R-319.** Its eight allowlist entries are **removed** (an allowlisted tag is skipped, so leaving them would have meant the new reader's fields were never checked); the gate's checked-tag count rose **182 Line truncated
| **R-265** | **A CI run can fail with NO LOG PERSISTED, and the alarm mail then points the operator at a log that does not exist.** Observed 2026-08-08 as run **264** (`650cc8a`, a **documentation-only** commit) sat between two green runs of identical gate code. **Measured rather than assumed — the shape is unmistakable:** every other run in the session took **18–34 s and has a log (HTTP 200)**; 264 took **834 s** (07:12:40 → 07:26:34 UTC) and `GET /actions/jobs/264/logs` returns **HTTP 500 — `actions_log/…/264.log.zst: file does not exist`**. The runner pod never restarted (`act-runner`, 0 restarts, 5 d 17 h uptime), so the runner did not die; the JOB hung and was reaped. **It is NOT a gate finding, and four independent facts say so:** the diff from the green run before it is Markdown only; the same content is green two commits later (run 265, `dd55a3f`, 33 s); the gate code is byte-identical across 263/264/265; and 260–262, which WERE real gate failures, all failed in under 35 s **with** logs. **THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately not guessed at.** DooPlex was doing heavy work in that window (a 139 MB `kubectl cp`, and a `go run` compiling the whole hub module for the live-validation harness), which is a plausible contention story — but the box has 40 cores and sat at load ~5, so it is **not established** and is recorded as a hypothesis, not a cause. **THE FINDING THAT MATTERS IS THE SECOND-ORDER ONE, and it is this workflow's own stated purpose turned against it.**`gates.yml` exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says *"The failing gate names itself in the run log."***Here there is no run log**, so an operator following that sentence finds nothing and cannot tell an infrastructure reap from a real conviction. Worse and **unverified**: the alarm step is `if: failure()`, and whether it even ran for a reaped job is unknown — if it did not, this was a red CI that alarmed nobody, which is exactly the shape the workflow was built to prevent. **Fix shape, not a decision:** (a) make the alarm mail state the run's DURATION and whether a log exists, so a log-less reap is self-identifying; (b) give the job an explicit `timeout-minutes` well under the reap so it fails fast, loudly and with a log; (c) establish whether the alarm fires at all on a reaped job — that is one deliberate test, and until it is run, "CI alarms on failure" is an assumption **SECOND SIGHTING, 2026-08-10 — and I nearly missed it.** felhom.eu CI **run 293** (`67eced8`, the demo-felhom recovery commit) **FAILED**, and its log is **unretrievable**: `/actions/jobs/293/logs` returns **HTTP 500**, while the runs either side of it (292 and 294) both return HTTP 200 with 12 290 bytes. So the failure is real and its reason is **not establishable** — exactly the shape this row was filed for. **What can be said:** the same tree passes the FULL gate set locally (`GITEA_BASE=… python3 scripts/repo_gates.py`, all eight OK), and run **294** on the very next commit passed in CI, so nothing persisted. **What cannot:** why 293 failed. **And the process failure is mine** — the end-of-session checklist says confirm your own push's CI run by ID, and I pushed 67eced8 without doing so; it was found only because the NEXT session happened to list recent runs. A pull check that is skipped is not a check| **CLOSED 2026-08-08 — `timeout-minutes: 5` on the gates job, and the alarm mail now states elapsed seconds and qualifies its own "names itself in the run log" sentence.** ⚠ **The unknown is NOT closed and must not be read as closed:** whether the `if: failure()` alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens inside one | **THIRD SIGHTING 2026-08-12 (controller run 324).** Failed after **11 minutes** where the previous run took 18 s; `/actions/runs/324/logs` and the web log endpoint both **404**. The jobs API *does* answer, and it names the failing step: **"Fetch the pushed commit and the sibling clone it needs"** — i.e. it died at the clone, before any gate ran, while a `felhom.eu` push was in flight. **Diagnosed rather than assumed:** the same sha passed all 11 gates locally on a clean tree, and an API **re-run of the same sha (326) succeeded** with no code change. So the jobs endpoint is a usable substitute for the missing log — it gives the failing STEP, which is most of what the operator needs — and `POST /actions/runs/<id>/rerun` (HTTP 201) is the cheap disambiguator between environmental and real.
@@ -603,19 +603,23 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-300** | **Our own uninstall left the thing that makes our own reinstall refuse.**`--uninstall` removed the dnsmasq snippet and RESTARTED the daemon, leaving it enabled and unconstrained on `0.0.0.0:53`; the next byo install's preflight then hard-refuses with *"a resolver is already bound to :53"* and **the customer goes debugging a home network that was never at fault**. Ownership is now RECORDED at preflight (`dnsmasq_preexisting`) — the only moment it is a fact, since the package is installed by the AGENT (`lanresolver.go:107`), not the installer — and honoured at removal: Felhom's → stop+disable; the owner's → restart only; **no record (every box already in the field) → restart only, fail-safe, with the reason and the exact command logged**. The preflight refusal keeps its two routes and its promise not to touch DNS on a host we do not own, and gains the missing line: when the bound resolver is dnsmasq, it says it looks like ours and names the command. **OBSERVED LIVE 2026-08-12** on `drill-r50`. Chain re-established from `virgin` (dnsmasq **not-installed**, `:53`**free**) → appliance install → the **agent** installed it, snippet-constrained to `10.0.2.15:53`+`127.0.0.1:53` → pre-fix uninstall → `enabled`/`active` on `0.0.0.0:53`+`[::]:53` → pre-fix byo install **refused, exit 1**. Fixed path from `virgin`: recorded `not present before Felhom` → uninstall logged `stopping + disabling it` → `:53 FREE` → byo preflight **PASS**. Owner's side also proven: with dnsmasq pre-installed the record read `yes` and the uninstall left it running. **But see R-305 — the cleanup fires only once per machine** | **CLOSED — observed live + PUBLISHED as `installer-v1.27.0` (both refs bumped)** | — | — | CC |
| **R-301** | **The abandon countdown banner makes the retired promise a third time, and as a flat statement.**`controller/internal/web/templates/layout.html:143`: *"A korábbi távoli mentéseidet N nap múlva véglegesen töröljük, a kérésed szerint. **Addig még visszaszerezheted őket a helyreállítási kóddal.**"* Unlike the orphan card this one is not hedged at all. **It is probably TRUE where it renders** — it is gated on `backupMgr.AbandonStatus().Active`, i.e. the customer chose to abandon a recovery offer that exists — which is why it was NOT changed (this session was fenced to the orphan card). **But it renders in `layout.html`, i.e. on EVERY page, and a rebuilt box can have an active abandon countdown WHILE its remote store is orphaned**, so a customer can read *"we cannot determine whether these can be opened"* on the backups page and *"until then you can still retrieve them with your recovery code"* in the banner above it, about copies they have no way to tell apart. **Not established:** whether the two sentences refer to the same bytes. The spec anticipated this in its §4 note | **CLOSED — premise CONFIRMED and fixed in controller v0.213.0 (R-302)** | R-299 | Establish whether the two can co-render for one customer and whether they name the same copies; if so, make the banner name WHICH copy | operator + CC |
| **R-302** | **The abandon banner promised retrieval it could not see was still true — fixed by PINNING a fingerprint at the decision.** The countdown strip said *„Addig még visszaszerezheted őket a helyreállítási kóddal"* unconditionally, on every page, and R-301 established it can co-render with the orphan card correctly saying we cannot tell. **THE OBVIOUS CONDITION WAS REJECTED AND HERE IS WHY, so nobody re-proposes it:** the proxy *"does the hub hold a key different from the one I use?"* asks about the WRONG key — the set-aside copies were written under an OLDER key the box no longer has, which is why they were set aside — so on a twice-rebuilt box it answers "promise it" about copies no key on file can open. **Demonstrated, not argued:** under the proxy both Scenario B (package replaced) and Scenario D (legacy countdown) flip back to promising. Instead `startAbandonCountdown` pins `AbandonPinnedEscrowKeySHA256` — the hub's escrow key fingerprint AS CACHED AT THE DECISION — written once, never refreshed; the banner then asks only *"is the hub still holding that same package?"*. **⚠ THE PIN IS A RECORDED ASSUMPTION:** nothing on the box records which key wrote the set-aside copies, so it presumes the package held at the decision is that one — true in the ordinary rebuilt-box story, NOT provable, wrong on a twice-rebuilt box. Recorded in the field comment so it can be narrowed rather than hardening into a fact. Empty is not a match on either side; a countdown started before v0.213.0 carries no pin and takes the cautious branch (deliberately NOT backfilled). **A template sweep found a FOURTH instance** (the backups-page abandon block — same condition applied, since fixing the strip and not the page would leave them contradicting) **and a FIFTH** (the abandon confirmation screen, deliberately LEFT: it renders at the moment of the decision, where the promise is true by construction because that is the package about to be pinned) | **CLOSED — controller v0.213.0** | R-301 | — | CC |
| **R-303** | **`markOrphaned` has no guard against an active abandon countdown — the co-render is made HARMLESS, not IMPOSSIBLE.** `ensureOffboxRepo` calls `markOrphaned()` for a claimed box (`offbox.go:804`) with no check on `AbandonAt`, so a later run finding the FRESH store unopenable re-raises the orphan card while the countdown runs. R-302 ensures the two surfaces no longer contradict each other in that state, but the state itself is still reachable and is arguably incoherent: a box counting down to deleting its old history while simultaneously reporting its NEW history is unopenable is in trouble in two ways at once and says so in two separate cards. **Ranked LOW deliberately** — it is a coherence question, not a correctness one, and the wrong fix (suppressing the orphan card during a countdown) would hide a real second fault |**READY (S) — NEW 2026-08-12** | R-302 | Decide whether a countdown should suppress, defer or annotate the orphan card — and beware hiding a genuine second failure | operator + CC |
| **R-303** | **`markOrphaned` has no guard against an active abandon countdown — the co-render is made HARMLESS, not IMPOSSIBLE.** `ensureOffboxRepo` calls `markOrphaned()` for a claimed box (`offbox.go:804`) with no check on `AbandonAt`, so a later run finding the FRESH store unopenable re-raises the orphan card while the countdown runs. R-302 ensures the two surfaces no longer contradict each other in that state, but the state itself is still reachable and is arguably incoherent: a box counting down to deleting its old history while simultaneously reporting its NEW history is unopenable is in trouble in two ways at once and says so in two separate cards. **Ranked LOW deliberately** — it is a coherence question, not a correctness one, and the wrong fix (suppressing the orphan card during a countdown) would hide a real second fault.**RULED 2026-08-13 (operator): LEAVE IT AS IT IS.** Two true cards side by side, on the stated ground that **the real-world likelihood of the combined state is unknown** — it has only ever been reached in a constructed test — and the tidier fix risks hiding a genuine second failure. **THE TRIGGER, recorded because a decision without one becomes a permanent silence: an observation of the combined state occurring OUTSIDE a constructed test.** One sighting on a real machine reopens this; nothing else needs to | **DECIDED 2026-08-13 — left as it is; reopens on a real-world sighting** | R-302 | — |
| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own **CLOSED 2026-08-13 by R-316** — reproduced on `drill-r50` from `virgin` on the published v1.27.0 (cycle 2 recorded `yes`, cycle 3 refused exit 1), then fixed by removing the package the record says we installed, and watched passing on cycle 3. | **CLOSED — superseded by R-316** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC. **RULED 2026-08-12 (operator): KEEP the set-aside history — cancel the countdown.** Done the same evening through the product's own operator path (`--abandon-stop`, which refuses rather than silently no-opping), with the container stopped first so the running controller could not overwrite `settings.json` from memory. **Proved, not trusted to the exit code:**`abandon_started_at` and `abandon_at` are GONE (`AbandonStatus` returns `Active=false` when `AbandonAt` is empty — `offbox_abandon.go:111-113`, so no countdown renders); `abandon_repo_path` deliberately REMAINS as the pointer to the preserved store; the store still holds **36 snapshot objects** and its full `config/data/index/keys/locks/snapshots` structure; both repositories still present on the endpoint; **nothing deleted anywhere**. See R-313 — what was preserved cannot currently be opened by anyone | **CLOSED — countdown cancelled 2026-08-12 on the operator's ruling** | R-301, R-302, R-303, R-313 | — | CC |
| **R-308** | **~~The stored controller password no longer opens `demo-felhom`~~ — WITHDRAWN 2026-08-12, this was MY BUG, not a defect.** The original diagnosis (`Hibás jelszó`) came from stripping only DOUBLE quotes off the `~/.config/credentials` value; the values are wrapped in **SINGLE** quotes, so a literal `'` was sent as part of the password. Correctly unquoted the value is 13 characters and logs in first try — **HTTP 302 with a `felhom_session` cookie**, measured. **This is the third time this project has produced a wrong 'the credential is stale' verdict from that exact trap**, and the standing memory about it is what caught it here — but only after the same mistake had also made the R-311 live test read as a FAILURE (HTTP 400) for twenty minutes. **The lesson is not 'strip quotes' but 'never let a shell decide what a secret is'**: parse the credentials file in Python, strip a matched pair of EITHER quote, and verify the length before use | **WITHDRAWN — not a defect (my error)** | — | Consider a tiny `read_credential.py` helper so no session hand-rolls the parse again | CC |
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.**`documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two**`--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not |**READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.**`documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two**`--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not.**FIXED 2026-08-13.** §C.1 now opens by naming its own former error, then states the three acts that actually publish: bump `SCRIPT_VERSION` and push (**publishes nothing**), cut `installer-v<version>`, and move **BOTH**`--ref` pins in `manifests/webpage.yaml` — the sidecar AND the init container (lines 327 and 372), **with the one-pin trap named**, because the running pod keeps serving until it restarts and a fresh pod seeded by a stale init container then serves the OLD script with no error anywhere. **And it now says how to check from OUTSIDE**, since a push, a green sync and a correct-looking manifest are each consistent with nothing having been published: `curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION'`. **Measured while writing it: served `1.28.0`, `main` `1.28.0`, both pins `installer-v1.28.0` — the three agreeing is the observation; any one alone is not.****The claim was copied elsewhere and the copy was hunted:**`audits/SPIKE-universal-iso-3-2026-07-31.md:184` said the same thing and **cited `day0-install.md` as its source**, which is how it spread. It was **true on the day it was written** (R-110 shipped 2026-08-03), so the dated finding is kept verbatim and carries a SUPERSEDED note rather than being rewritten — falsifying a dated record to tidy it is its own defect. Two other hits are correct in context: `hostinstall_gates.py:198` states the consequence of the manifest LOSING its tag, and the 2026-08-12 drill record already names the sentence as false | **CLOSED — corrected, and the one copy annotated** | R-110 | — | CC |
| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |
| **R-311** | **A correct recovery code for a retained package stopped being reported as wrong.** The drill of 2026-08-12 proved the retained key WORKS (it opened a set-aside store and restored planted files byte-identical) while the product answered *"the recovery code did not open the sealed bundle"*. **What was already right:** the customer-facing screen did not bluntly accuse — R-222/R-226 hedged, naming both causes and the kept package. **What was wrong was the hedge's reason:***„innen nem tudjuk megkülönböztetni őket"* was true only because NOTHING EVER LOOKED. **Shipped, three repos.** Hub v0.103.0: `GET /hosts/<id>/escrow/retained` — the first production caller `ListSupersededEscrow` has ever had — self-scoped, same recovery-mode gate, same audit event before the bytes leave, rows with a NULL `identity_blob` WITHHELD and returned as `unopenable_count` (they can never open anything, and serving them would let the screen promise recovery on exactly the boxes the original defect hurt), capped at 16. Agent v0.129.0: `FetchRetained` consulted ONLY after the current package refuses, `ErrCodeOpensRetained` + the supersession date, local API answers **422**, bounded at 6 attempts (~1 s of scrypt each), and fail-safe in every direction — a nil fetcher, a 404 from an older hub, a transport failure or a malformed package all leave the original refusal standing. Controller v0.214.0: class `RecoveryCodeOpensRetained` gated on `FeatureRetainedRecoveryClass` (MinAgent 0.129.0, SEPARATE from the R-224 gate because a box can sit between 0.126.0 and 0.129.0), and a message that says the code is correct, names the date, says the earlier package is kept, says the CURRENT backups are unaffected, and **deliberately promises no restore** — it routes to support, which can do it by hand. **The lie's true home was the agent:** removing the CONTROLLER case yields the neutral message (R-224's safe default catches it); removing the AGENT's retained lookup brings back the fail-closed wrong-code error in those exact words — both demonstrated, mutations asserted applied **DELIVERED FLEET-WIDE 2026-08-12 night.** Operator vouched golden 0.214.0 / agent 0.129.0 / min agent 0.129.0 and raised the floor to 0.214.0; both artifact shas in `hub_settings` match the bake and the release byte-for-byte (`3a40379c…`, `53a54f06…`). **The Part D / R-216 hold was OBSERVED FIRING for the first time** — with the floor at 0.214.0 and `demo-hp` still on agent 0.128.0 the hub logged *"managed floor HELD for demo-hp: agent 0.128.0 < MinAgent 0.129.0 (controller floor withheld)"*, and the box took the update **6 seconds** after its agent was brought to 0.129.0 (`auto-floor`, 19:03:18→19:03:24Z). That guard exists because a box once landed on a controller newer than its agent and its customer was told a correct recovery code was wrong; **the MinAgent declaration in the CHANGELOG header is what made it hold**, which is the whole argument for the three-part coupling rule. Both demo boxes now run controller 0.214.0 + agent 0.129.0, verified ON the boxes. | **CLOSED — shipped + delivered: hub v0.103.0 + agent v0.129.0 + controller v0.214.0** | R-199, R-222, R-224, R-226, R-304 | — | CC |
| **R-312** | **There is no in-product route from the recovery screen to a set-aside store, and building one is not wiring — it is new surface.** Established read-only before any code was written (the session's §4 spike). Every restore entry point resolves the repository from `m.settings.GetOffboxTarget()` and the password from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`), `offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **There is no repo-path parameter anywhere in the chain** — a grep for one returns nothing. The only existing seam that installs a recovered password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file, i.e. **ADOPTS** the set-aside store as the machine's current target. So the two options are (a) thread an alternative (repo, password) through three functions plus the UI, or (b) adopt — and adoption is a different product decision. **The session HALTED here by its own rule** and shipped R-311 alone. What the drill did to read the set-aside store was `restic` by hand with `-r <alt repo>` and an overridden `RESTIC_PASSWORD_FILE`; **that distance is exactly what (b)-to-(c) costs** |**READY (L) — NEW 2026-08-12** | R-304, R-311 | Decide (a) vs (b). If (a): the parameter is small and contained; the UI question ("which package?" when a customer has several) is the real design work. **Do not choose (b) without deciding what happens to the machine's current off-site history** | operator + CC |
| **R-313** | **`demo-felhom`'s set-aside store is UNRECOVERABLE — 36 snapshots whose key we destroyed ourselves.** `/home/felhom-repo.orphaned-20260810` holds **36 snapshot objects and exactly one key slot**, and it does NOT open with the box's current password (`Fatal: wrong password or no key found`, exit 1 — measured). Its password is the one hashed `48741892f0ef4d59…`, which is retained row id 4 — **`identity_blob` NULL**, a pre-v0.93.0 row. So the material was dropped by the R-198 defect during its two-month window, and no recovery code in existence opens that store. **This is the concrete, still-present cost of R-198**, sitting on the endpoint rather than in a post-mortem. It also means the operator's ruling to KEEP it (R-307, countdown cancelled — see below) preserves bytes nobody can read: correct as a decision, and worth knowing as a fact |**READY (S) — NEW 2026-08-12** | R-198, R-307 | Decide whether to keep paying storage for 36 unreadable snapshots, or delete them deliberately and record why. **Not urgent and not automatic — but it should be a decision rather than an accumulation** | operator |
| **R-312** | **There is no in-product route from the recovery screen to a set-aside store, and building one is not wiring — it is new surface.** Established read-only before any code was written (the session's §4 spike). Every restore entry point resolves the repository from `m.settings.GetOffboxTarget()` and the password from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`), `offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **There is no repo-path parameter anywhere in the chain** — a grep for one returns nothing. The only existing seam that installs a recovered password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file, i.e. **ADOPTS** the set-aside store as the machine's current target. So the two options are (a) thread an alternative (repo, password) through three functions plus the UI, or (b) adopt — and adoption is a different product decision. **The session HALTED here by its own rule** and shipped R-311 alone. What the drill did to read the set-aside store was `restic` by hand with `-r <alt repo>` and an overridden `RESTIC_PASSWORD_FILE`; **that distance is exactly what (b)-to-(c) costs**.**RULED 2026-08-13 (operator): NOT BUILT, DELIBERATELY — recovering an old backup stays a phone call.** Neither (a) nor (b) is taken. A customer in this position is a support conversation, and the capability genuinely exists on that path: the 2026-08-12 drill opened a set-aside store by hand and restored planted files byte-identical, so the promise R-311 makes (*"your code is correct, write to us"*) is one we can keep. **THE TRIGGER: a real need appearing — one request from a customer who is not us.** Until then the honest position stated in R-304 stands unchanged: retention is an **operator-only** capability, and nothing anywhere may promise the customer can perform it themselves | **DECIDED 2026-08-13 — deliberately not built; re-evaluate on a real customer request** | R-304, R-311 | — |
| **R-313** | **`demo-felhom`'s set-aside store is UNRECOVERABLE — 36 snapshots whose key we destroyed ourselves.** `/home/felhom-repo.orphaned-20260810` holds **36 snapshot objects and exactly one key slot**, and it does NOT open with the box's current password (`Fatal: wrong password or no key found`, exit 1 — measured). Its password is the one hashed `48741892f0ef4d59…`, which is retained row id 4 — **`identity_blob` NULL**, a pre-v0.93.0 row. So the material was dropped by the R-198 defect during its two-month window, and no recovery code in existence opens that store. **This is the concrete, still-present cost of R-198**, sitting on the endpoint rather than in a post-mortem. It also means the operator's ruling to KEEP it (R-307, countdown cancelled — see below) preserves bytes nobody can read: correct as a decision, and worth knowing as a fact.**RULED 2026-08-13 (operator): KEPT — as a TEST FIXTURE, and that is the whole of the advantage claimed for it.** The operator asked what keeping it actually buys and accepted the specific answer rather than a sentimental one: **it is the only state in existence where a set-aside store is PRESENT and CANNOT BE OPENED**, which is precisely the case any future handling of lost backups must face honestly — and a case that cannot be manufactured, because manufacturing it would mean destroying another key on purpose. Storage cost is pennies; nothing depends on it; no customer sees it. **THE TRIGGER TO DELETE IT, recorded so this stops being an accumulation the moment it stops being a fixture: when the work it is a fixture FOR ships, or is abandoned** — i.e. when R-312 is built, or when R-312's ruling above is made permanent. On either event, delete it deliberately and record why | **DECIDED 2026-08-13 — kept as a test fixture; delete when R-312 ships or is abandoned** | R-198, R-307, R-312 | — |
| **R-314** | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route | **READY (S) — NEW 2026-08-12, RANK 3** | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
| **R-315** | **The wire-contract gate's positive control FAILS on the new wire: it checks name-presence, not decodability.** R-311 declared `hub -> agent (GET /escrow/retained)` as a fourth ROOT, and the gate's tag count rose 174 → 182, so the fields ARE inspected. But renaming the agent-side `superseded_at` json tag to `superseded_at_RENAMED`**still passed** — because the string `superseded_at` also occurs as a map key in the agent's local-API response, and the check is a repo-wide name search. The gate documents this ("name-reachability is not use"), so it is a known limit rather than a regression — but it means **declaring this wire bought documentation, not enforcement**, and a report that claimed coverage would have been wrong. The mutation was asserted to have applied before the run | **READY (M) — NEW 2026-08-12, RANK 3** | R-311 | Make the check resolve the RECEIVER'S mirror type and compare field-by-field, or state per-root which kind of check it got. **A gate whose positive control fails is an instrument nobody has calibrated** | CC |
| **R-316** | **The removal now genuinely reverses the installation — R-305's once-per-machine defect closed.** v1.27.0 stopped and disabled the unit but left the PACKAGE, and the preflight's ownership question is `dpkg-query … "install ok installed"` and **nothing else — package presence alone** (`felhom-host-install.sh:1718`), so cycle 2 recorded `yes`, the second uninstall treated our own package as the household's, and cycle 3 refused. **Measured on `drill-r50` from `virgin` on the PUBLISHED v1.27.0 before anything was changed:** cycle 1 `no` → `:53` FREE; cycle 2 **`yes`** → left running on `0.0.0.0:53`; cycle 3 **PRE-FLIGHT FAIL exit 1**. **v1.28.0:** when the record says we installed it, the uninstall removes the package too. Order unchanged and load-bearing — read the record, act, THEN delete the state file (`:1268`) that holds it. **TWO packages are now recorded** (`dnsmasq` ships the unit, **`dnsmasq-base` ships `/usr/sbin/dnsmasq`**) and each is taken back only if we added it. **The dependency check is a SIMULATION** (`apt-get -s purge`, proceed only if the removal set is a subset of ours) rather than a guess; it falls back to stop+disable and NAMES the blocking package. Never interactive, never fatal — a wedged apt is recorded and restated in the closing NOTE. The success is re-queried rather than read off apt's exit code. **Watched:** three fixed cycles → install 3 PASSES; household resolver → untouched; dependent package → not purged and named; no record → untouched with the command named. **Red-proofs, mutations asserted applied:** remove the purge → cycle 3 refuses in those exact words; remove the ownership check → a household resolver is PURGED; infer ownership when absent → the guess is taken. **Published as `installer-v1.28.0`** | **CLOSED — shipped + published, observed on the cycle that actually fails** | R-300, R-305 | — | CC |
| **R-317** | **The agent decides whether to install dnsmasq by stat-ing a file the OTHER package owns.**`EnsureDnsmasq` (`felhom-agent/internal/lanresolver/lanresolver.go:105`) does `os.Stat("/usr/sbin/dnsmasq")` and skips the apt install when it exists — but that path is shipped by **`dnsmasq-base`**, while the systemd unit comes from **`dnsmasq`** (confirmed on the box: `dpkg -S /usr/sbin/dnsmasq` → `dnsmasq-base`; `dpkg -S /usr/lib/systemd/system/dnsmasq.service` → `dnsmasq`). So on any host carrying `dnsmasq-base` without `dnsmasq`, the agent skips the install and then runs `systemctl enable --now dnsmasq` against a unit that is not there: the resolver never comes up and the failure is a retried WARN in the journal rather than anything a customer or the install sees. **Pre-existing, NOT introduced by R-316** — but R-316 makes the shape reachable, because a host whose `dnsmasq-base` pre-dated Felhom now keeps it while `dnsmasq` is removed. R-316's uninstall says so explicitly instead of leaving it to be found from a silent resolver. **Ranked 2 (costs time), not 1:** the box installs fine, only LAN name resolution is missing | **READY (S) — NEW 2026-08-13** | R-316 | Probe what is actually needed — the unit or the `dnsmasq` package — rather than a path a sibling package owns. One-line change in the agent; deliberately NOT made here to keep this session to one repo | CC |
| **R-318** | **No honest marker exists that says Felhom installed dnsmasq on a machine already in the field, and none can be invented.** Established from source and on the box: the Felhom `/etc/dnsmasq.d/felhom-*.conf` snippets are deleted by the uninstall's own loop (`:1162`) BEFORE the ownership decision and do not survive it; the install state file that carries the record is deleted at `:1268`; nothing under `/etc/felhom*` remains. `/var/log/dpkg.log` does record the install — and is a **timestamp**, which the standing rule refuses as a heuristic dressed as a fact. **So for every box installed before v1.27.0 the answer is the preflight message, and that is a mechanism rather than a rule:** it names the finding, keeps its two routes and its promise not to touch DNS on a host we do not own, and adds *"THIS LOOKS LIKE OURS … systemctl disable --now dnsmasq"*. **Judged as a customer would:** it is honest, it hedges correctly (*looks like*), and it names one exact command — the one thing that gets that person moving. Its weakness is that it asks them to answer *"did this host have dnsmasq before Felhom?"*, which is precisely the question we can no longer answer for them | **CLOSED — established, no action possible for existing boxes** | R-300, R-316 | — | CC |
| **R-319** | **The guest-network watchdog finally has a reader — the first of R-264's twenty-one, and it is the repair COUNT that matters, not the state.** The agent has reported `guest_net` on every heartbeat since **v0.92.0** (R-54, 2026-07-21) and the string `guest_net` occurred **nowhere** in `felhom.eu/hub/` — stored as raw text inside `report_json`, read by nothing. **Established before anything was built:** the facts DO arrive and ARE persisted — `demo-felhom-8363b5`'s newest row carries `guest_net.checked_at` plus per-guest `vmid/state/mode/ip/has_route/dhclient_alive/checked_at/message`, and the agent's wire type (`felhom-agent/internal/hub/report.go:139-160`) additionally carries `healed`, `heal_succeeded`, `last_heal_at`, `heals_last_hour`, `damped`, absent from the live rows only because they are `omitempty` on a box that has never needed a repair. **So this was hub-only: no wire change, no agent change, no controller change, nothing to bake.****What was built** (`hub/internal/web/hosts.go``parseGuestNet`/`guestNet` + the host-detail *Guest network* card): the hub MODELS the stanza instead of storing it as text, renders it where a person looks at a machine, and **surfaces `heals_last_hour` beside the state rather than behind it** — because a guest the watchdog keeps repairing is healthy at every instant anyone looks and is nevertheless failing, which is the exact shape of the failed-disk-drawn-as-a-healthy-empty-disk defect. `heal_succeeded` is decoded too, and deliberately: **R-260 is this project's warning that a decoder short of three fields dropped the one that decided the question**, and six FAILED repairs is a guest that is down while six successful ones is a nuisance. **An unknown is never drawn as healthy**, following the August companion-flag convention — three absences kept apart with three different sentences (agent older than 0.92.0; a capable agent that sent nothing; a guest whose own `state` the watchdog did not assert), and a malformed stanza degrades to unknown without a 500. **ALARM JUDGEMENT: no email, deliberately.** The incident behind this (`INCIDENT-guest-dhclient-killed-2026-07-20.md`, a killed `dhclient` that took a tunnel down for 1 h 15 m) was about nobody being able to SEE the condition, not about nobody being paged; and a new alarm on a fleet of two demo machines is untested noise on a dispatcher whose severity contract is exact-match. **Revisit when a third machine exists or when a repair count is seen climbing on real hardware** — the visible count is what will supply that evidence. **Red-proofs, mutations asserted applied by grep before each run:** (1) the unknown branches replaced by the healthy badge → **both** C sub-cases go red, a silent machine seen rendering as healthy; (2) `RepairCount: 0` in the decoder → B goes red on *"a guest repaired 6 times in an hour is reported as fine"*; (3) `Degraded()` forced true → A goes red on *"a machine that is fine is being alarmed on"*, proving the guard is reachable in both directions. **Positive control that the wiring is real, not just written: the wire-contract gate's checked-tag count rose 182 → 190 and its skipped count fell 88 → 80** as the eight `guest_net` allowlist entries were REMOVED — an allowlisted tag is skipped, so leaving them would have meant the new fields were never checked at all | **CLOSED — shipped hub-side 2026-08-13** | R-54, R-260, R-264 | — | CC |
| **R-320** | **Evidence has been destroyed twice in three days, in the same place, by the same act.** 2026-08-12, the retained-key drill: the Phase A logs lived on `drill-r50`'s disk and were destroyed by the revert to `virgin` between Phase A and Phase B (`audits/DRILL-retained-key-2026-08-12.md` §11.5). 2026-08-13, R-316: the Part 1 logs, same disk, same revert, between Part 1 and Part 2 (`audits/REPORT-r316-installer-v1.28.0-2026-08-13.md` §9 — *"the same mistake as Tuesday, in the same place"*). **That report was copied into `audits/` by this session on purpose:** it lived in `REPORT.md`, which every session overwrites — so writing tonight’s report would have destroyed the record of a destroyed record. **Both times the conclusions survived on luck** — the quotations had been read live, and an independent reproduction happened to exist. **Both times the golden-bake runbook's existing "scp the log OUT first" was applied to the FINAL teardown and not the intermediate one**, which is the whole finding: the middle revert is the one that gets forgotten. **The rule, now standing rule 5 in `workspace-CLAUDE.md` (so it loads in every session) and repeated where a session actually meets it — `runbooks/target-selection.md`, `RUNBOOK-rehearsal-v3.md`, and the `PROMPT-TEMPLATE.md` report section: evidence is copied off the machine at the end of the phase that produced it, before any revert, snapshot restore or teardown — not at the end of the session.****A rule without a mechanism is a wish, so the mechanism is named: the pull is the LAST ACT OF THE PHASE**, not a step to remember later. **And the already-gone case is documented rather than improvised: say so plainly in the report and reproduce the finding independently** — which is what both sessions did, and it should be the expectation rather than a good instinct under pressure | **CLOSED — rule written, four homes** | — | — | CC |
| **R-321** | **A box on which reporting is deliberately switched off still alarms as stale, then down.** Found while deciding `reporting_disabled` on its own merits (R-264). **The product supports the state:** with hub reporting off the controller sends ONE minimal report carrying `reporting_disabled: true` and `health.status: "disabled"` (`felhom-controller/controller/cmd/controller/main.go:1246-1260`) and then goes quiet by design. **The state IS visible** — `health_status` is decoded and stored, and `hub/internal/web/rollup.go:25` renders that customer as `disabled`, which is why the `reporting_disabled` FLAG itself is reclassified *redundant* rather than owed a reader: it is a second spelling of a fact already read. **But the alarm does not consult it.**`StalenessChecker.Check` (`hub/internal/monitor/staleness.go:88+`) computes its verdict from report AGE alone; the only skip is `IsCustomerBlocked`. So a deliberately-silent box goes `node_stale` at 30 minutes and `node_down` at 60, and the operator is paged about a machine that is fine — **the classic false alarm that teaches people to ignore the channel**, and a sibling of R-195, where the guard that should have covered a customer was keyed off the wrong fact. **NOT FIXED HERE, deliberately: adding a decoder for the flag would have felt like progress and left the alarm firing.** The fix belongs in the checker, which already holds the health status it needs. **Bounded honestly: no machine is in this state today** (all three known hosts report normally), so this is a latent defect found by reading, not an observed outage | **READY (S) — NEW 2026-08-13, RANK 3** | R-195, R-264 | Skip (or downgrade) the staleness verdict for a customer whose last report declared `health.status = "disabled"`, and **log the deferral with its own counter** — the v0.73.0 precedent: a quiet check must not look like a check that did not run. Fail OPEN on an unreadable status: an unreadable state must never SUPPRESS a real alarm | CC |
| **R-322** | **The claim guard has never scanned the hub, and the hub sends the customer's first sentence.**`retrieval_promise_gate.py` lives in `felhom-controller/controller/scripts/` and its declared surfaces are that repo's `internal/web/templates` plus ONE Go handler file — extended to Go strings only on 2026-08-12 (R-311), on the ground that *"the highest-stakes customer copy in the product… none of it had ever been scanned"*. **The same sentence is true one repo over.** The hub composes and sends every customer-facing e-mail (`internal/notify/templates.go`) and renders the self-bind pages, i.e. the copy a customer reads BEFORE they ever see a box screen, and no gate in either repo looks at it. **Scanned by hand while shipping R-295's hub half: `felhom.eu/hub/` returns ZERO occurrences of all four stems** (`visszaállíthat`, `visszaszerezhet`, `visszahozhat`, `visszanyit`) across every non-test `.go` and `.html` under `internal/`. **So nothing was hiding, and this is a SCOPE gap rather than a live defect** — which is exactly the moment to record it, before the first hub-side retrieval promise is written by someone who assumes the guard has them covered. **Recommended shape, and the reason it is not one line:** the gate is invoked by `controller_gates.py`, so pointing it at a sibling repo makes a controller gate fail on a felhom.eu edit — the cross-repo lesson from G-1 (a gate needing a sibling passes locally and exits INCONCLUSIVE in CI, and must never SKIP when the sibling is absent). The cheaper honest option is a hub-side sibling registered in `repo_gates.py` that shares the stem list, with ONE list rather than two drifting copies | **READY (S) — NEW 2026-08-13, RANK 3** | R-294, R-299, R-302, R-311 | Scan the hub's customer-facing surfaces with the same stems, from `repo_gates.py`. **Share the stem list — two copies of a word list is how the plural got past the singular (R-299).** Do NOT make a controller gate depend on a felhom.eu clone | CC |
@@ -25,6 +25,25 @@ Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixtur
demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
prohibition as covering the act it names and nothing more.
## Before you revert it — take the evidence off first
> **A phase's evidence is copied off the machine at the end of THAT phase, before any revert, snapshot
> restore or teardown. Not at the end of the session.** (Standing rule 5, R-320.)
>
> **The intermediate revert is the one that gets forgotten.** Both losses this project has recorded
> were the *middle* teardown, never the final one — the Phase A logs of the 2026-08-12 retained-key
> drill and the Part 1 logs of the 2026-08-13 R-316 run, both on `drill-r50`, both destroyed by a
> revert to `virgin` between phases, three days apart. Both times the conclusions survived only
> because the quotations had been read live and an independent reproduction happened to exist. **That
> is luck.**
>
> **The mechanism:** make the pull the last act of the phase, not a step to remember later —
> `pct pull` / `scp` into `documentation/audits/evidence-<topic>-<date>/` on DooPlex, then revert.
> Tier 0 machines are *disposable*, which is precisely why nothing you need may be left on one.
>
> **If you notice it is already gone:** say so plainly in the report and **reproduce the finding
> independently**. That is the documented expectation, not an improvisation.
| Tier | Meaning | Machines |
|---|---|---|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
@@ -56,6 +56,15 @@ roles. **A file being open in the editor is NOT an instruction. If no task is st
"working" and "stopped entirely".
4. **A recommendation that is not followed gets one line saying why.** Silence reads as agreement and
the disagreement is lost.
5. **Evidence is copied off the machine at the end of the phase that produced it — before any revert,
snapshot restore or teardown. Not at the end of the session.** The intermediate revert is the one
that gets forgotten; both losses were the *middle* teardown, never the final one. **The mechanism,
because a rule without one is a wish:** the last act of a phase that ran on a machine is `scp`/`pct
pull` its logs to the evidence directory on DooPlex — the same act that ends the phase, not a
separate step to remember later. **And when a session notices the evidence is already gone: say so
plainly in the report and REPRODUCE it independently.** That is the documented expectation, not an
improvisation to be invented under pressure — it is what both sessions did, and it is the only
reason two sets of conclusions survived.
<!--
R-96 incident record (committed 2026-07-27) — rationale, not directives.
@@ -72,6 +81,16 @@ R-96 incident record (committed 2026-07-27) — rationale, not directives.
4. Twice in the R-88/R-97 arc a review point was absorbed rather than argued: R-84 was folded into
R-82 without a word, and R-97a's operator-only guard was dropped while the claim it was meant to
enforce got committed as a comment — which is how a false guarantee shipped and survived a release.
5. Twice in three days, on the SAME machine and at the SAME point. 2026-08-12, the retained-key drill:
the Phase A logs lived on `drill-r50`'s disk and were destroyed by the revert to `virgin` between
Phase A and Phase B (`audits/DRILL-retained-key-2026-08-12.md` §11.5). 2026-08-13, R-316: the Part 1
logs, same disk, same revert, between Part 1 and Part 2 (`audits/REPORT-r316-installer-v1.28.0-2026-08-13.md`
§9 — "the same mistake as Tuesday, in the same place"; preserved out of `REPORT.md`, which is
overwritten every session). Both times the golden-bake runbook's existing "scp the log OUT first"
was applied to the FINAL teardown and not the intermediate one. Both times the conclusions survived
only because the quotations had been read live and an independent reproduction happened to exist —
that is luck, and the second occurrence is what makes it procedural rather than another apology.
Filed as R-320.
-->
## Shared conventions
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.