# OPEN-ITEMS — the single source of truth for open work

**Rebuilt 2026-07-27 by read-only triage.** `ROADMAP.md` keeps the full history and reasoning; this
page keeps only what is **open**, and it is the file to read first. Root `REPORT.md` is per-session
and **overwritten** — nothing durable may live only there; a session that must not clobber it writes
a non-overwritten `REPORT-<topic>.md` sibling instead (`CLAUDE.md:82-87`), of which 14 now exist.

State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row has an owner.

## Operator rulings — 2026-08-04

Recorded here because a ruling that lives only in a conversation binds nobody (the R-96 standing rule).

1. **Run the recovery drill, after R-198.** R-198 shipped in hub v0.93.0; **the drill is the next
   session.** Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10 — demo-hp, ~3–4 h, a recovery
   code created and KEPT, a sentinel file, wipe, reinstall, recover, and **pass = a byte-identical
   sha256, not "the repository opened"**. → R-201
2. **Delete the orphaned ciphertext** (~1.2 GB across the two demo boxes, in set-aside restic stores
   nothing prunes). **STILL OWED — not done in v0.93.0.** It is a destructive act on a protected
   endpoint and belongs to a session that is scoped for it, not to a release that ships a schema
   change. → R-193
3. **Accept the risk on R-193 candidate (c)** — no repository password retained on the Proxmox host.
   **This is what makes R-198 load-bearing rather than tidy:** with no host-retained copy, the
   customer-present recovery path is the ONLY way back from a rebuild, and that path runs entirely
   through the retained identity blob. → R-193, R-199, R-200, R-201

~~**Still open and untouched by v0.93.0:** R-199, R-200, R-201.~~ **UPDATED 2026-08-04 (evening), after
hub v0.94.0 + agent v0.125.0 + controller v0.195.0:**

- **R-199 — CLOSED, proven on hardware.** Chain links 6–8 are assembled and walked. The offsite
  repository password came back out of the sealed bundle **byte-identical** to the one on disk
  (`c60c8bc737a6…` from three independent sources: the box's file, the recovered bundle, and the hash
  the hub already stored).
- **R-200 — the plumbing half shipped**, the customer-facing form did not, deliberately.
- **R-201 — PASSED 2026-08-04 (night run).** A customer's file survived a machine rebuild and came
  back byte-identical, through the customer's own restore flow. **It passed only because a person was
  there:** four manual interventions stood between the recovered key and the restored file, none of
  them in any design document → R-204.
- **R-202 — untouched.** The orphan card still promises recoverability unconditionally.
- **The orphaned ciphertext deletion (~1.2 GB) is STILL OWED** — ruling 2, above.

**UPDATED 2026-08-05, after controller v0.198.0 + hub v0.95.0 (R-204):**

- **R-204 items 1–3 — CLOSED.** The reset code works without a restart; a re-issue no longer marks a
  healthy escrow stale (→ **R-196 CLOSED**); a unit restore states what it did NOT restore.
- **R-204 item 4 — OPEN and unstarted:** a rebuilt box cannot obtain an off-site credential unaided.
  It needs an operator ruling on the one-shot credential design → **R-193**.
- **Still open and untouched by this session, stated so nothing is presumed closed by association:**
  **R-202** (the orphan card's unconditional promise), **the ~1.2 GB orphaned-ciphertext deletion**
  (ruling 2 — still owed, still needs its own scoped session), and **R-198's retention, which remains
  UNIT-PROVEN ONLY** — nothing has superseded a key in production, and proving it needs a SECOND
  deliberate wipe. That retention drill is the next item, and it is not this session's.

v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. v0.197.0 got the file into
the snapshot and the drill got it out again. **v0.198.0/v0.95.0 remove three of the four crutches the
drill needed — the fourth is R-193, and until it goes the recovery is still operator-assisted.**

## R-201 — THE RE-WALK, 2026-08-06 (attended)

**The question was asked a second time, on the fixed build, on a brand-new appliance built from the
published ISO. The answer is still no — but it is a nearer no.**

| half | verdict |
|---|---|
| **the data** | **PASS** — all three sentinels byte-identical, including a 12 MB binary and an accented Hungarian filename **whose NAME BYTES are also byte-identical** (verified as hex, not as rendered text). Restored in **23 s** out of the pre-destruction snapshot `a7bc23bd`, through the customer's own two-step full-restore flow |
| **the journey** | **FAIL — two dead ends**, against Phase 1's four. One needed a command line **inside the guest**, one a **Proxmox-host** action |

**RTO: the unaided figure is STILL UNDEFINED**, because the unaided journey still does not complete.
Attended: login 11:42:22 → key placed **+45 s** → tier up **+24 m 12 s** (after intervention 1) →
all three sentinels restored and verified **+30 m 13 s** (after intervention 2). **The 30 m figure must
not be quoted as the customer number.** The only segment that reflects the product working alone is
**23 seconds** to pull 12.8 MB back once everything was in place.

**The two dead ends:** **R-218's consume half** (row corrected above) and **R-220** (drives
unenrollable after a rebuild — reproduced and red-proved again; without it no app can be redeployed,
and without a redeployed app the restore page is empty, which is R-213's territory and follows from
R-220 rather than being separate).

**What PASSED and is worth keeping:** the recovery screen **appeared without being sought**
(`/` → `/launcher` → `/recovery`); it answered all three of its questions and its seal date matched the
hub's `created_at` exactly; the emailed reset code worked **first try**; the unlock took **1.528 s** —
a real unseal — and placed the key; and **R-225's fix was seen working in the wild** (the store read
„a pillanatképek száma még ismeretlen" rather than a false zero).

**R-216 part 4 reproduced live:** the reinstall **downgraded the agent 0.126.0 → 0.125.0**, back to the
vouched version — an operator's hand-fix undone by the very event that makes recovery necessary.

⚠ **THE DELIVERY GAP, and it is owed.** A fresh install landed on controller **0.201.0** / agent
**0.125.0** — the vouched versions, **neither carrying the fixes**. They were installed **by hand**.
Fleet delivery needs a golden carrying 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched;
that is the operator's act.** **This re-walk proves the JOURNEY on the fixed build; it does NOT prove a
customer would receive that build.**

Evidence: `tests/rewalk-r201-2026-08-06/journal.md`.

## CAMPAIGN 11 — the recovery journey, 2026-08-05

**The whole journey was walked end to end for the first time, on a throwaway appliance built from the
published ISO. The data came back byte-identical; the journey did not exist.** **Ten findings from
Phases 1 and 3, R-214 … R-223** (seven fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**;
**three deliberately still open**, each blocking a real flow), **plus five from Phase 2's injected
faults, R-224 … R-228.** Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` (Phases 0/1/3)
and `journal-phase24.md` (Phases 2/4). Campaign document:
`audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`.

> **Phase 2's verdict in one line.** The cryptography, the retention and the transport all work and
> are now proven live. **What fails is being told the truth:** a mistyped code, a hub outage, a
> stopped agent and a correct code for a retained earlier package all produce **one** message, and
> three of the four are wrong.

| ID | What | State |
|---|---|---|
| **R-216** | **A correct recovery code was reported to the customer as wrong.** The unseal needs agent v0.125.0; on an older agent the route 404s and the unlock was attempted anyway, producing *„…nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót pontosan…"* in 0.134 s. **The default state** — the Day-0 manifest vouches 0.120.0, and a reinstall actively DOWNGRADES a hand-fixed box back to it. The hub's own guard could not catch it: `ResolveManagedFloor` compared against the GOLDEN's MinAgent while serving a FLOOR that pointed elsewhere (the 8th entry in `CLAUDE.md`'s comment-vs-code table, and the first where the false invariant was a guard) | **SHIPPED** (controller v0.201.0 + hub v0.97.0/0.97.1) — **but see R-223**: the feature does not work on a NEW box until the manifest vouches agent 0.125.0. Until then such a box is correctly HELD, not lied to |
| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **CLOSED 2026-08-06 — controller v0.203.0, proven live.** *(State corrected 2026-08-06: this field read REOPENED while the body below already recorded the fix shipped and proven. The history of the over-claim is kept deliberately — it is why the row is worded as it is.)* **The over-claim, as it stood: the fix covered the DECLARATION half only.** Measured on the R-201 re-walk: the box declared, and **`offsiteheal` re-staged the secret at 11:44:57** saying *"the box re-consumes on its next cycle"* — **the next cycle came and went** (`host-report` 11:55:46, `Received report` 11:55:54, a full cycle **with a positive control that it ran**) **and the credential was still not consumed.** 23 minutes after the re-stage the box's last off-site-apply attempt was still the pre-re-stage one. A census of the customer-reachable actions on `/backups/remote` (`config`, `reset`, `run`, `toggle`) found **none that fetches a staged credential**, and the only lever is `systemctl restart felhom-controller-bootstrap.service` **inside the guest** — which worked in **18 s** (Campaign 11 measured 17), confirming nothing was wrong with the credential, the target or the key: **the only thing missing is anything at all to trigger a retry.** **This is the FIRST of the two dead ends that keep the recovery journey failing** | **CLOSED 2026-08-06 — shipped in controller v0.203.0 and PROVEN LIVE on a genuinely rebuilt box.** `Bridge.RetryIfDeclared` re-runs the SAME reconcile on a 5-minute tick for exactly as long as the box's own published declaration (`OffboxReportStatus().State`) says it needs a credential — the very statement the hub acts on, so the two can never disagree. Measured on the Part 4 venue (`documentation/tests/part4-rewalk-2026-08-06/journal.md`): after the rebuild the retry ticked and logged its reason honestly, the hub's `offsiteheal` re-staged at **13:24:57Z**, and the box **collected it on a tick and configured the tier** — `[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo` — **with no guest command line and no operator action**. On a healthy box the job registers, ticks, completes in 0 s and says nothing, which is asserted. |
| **R-219** | **The listing the screen promises could never render on the shape it exists for.** Listing needs a target; a target cannot exist without a repository password; shape (a) is defined by having none. And placing the key flipped the offer false, so the unlock response was the only chance and was guaranteed not to contain it | **SHIPPED** (controller v0.201.0) — the unlock now places the key, brings the tier up, then lists |
| **R-217** | **An unreadable store reported as "opened, with unattributable content".** The failure path passed `backup.OffsiteInventory{}`, whose `Empty=false` the template read as `InvUntagged`. The field built to prevent exactly this names the hazard in its own doc comment | **SHIPPED** (controller v0.201.0) — opened / empty / unreadable are three distinguishable states |
| **R-222** | **Reaching for a RETAINED earlier package read as a wrong code.** The engine is right (it fails closed against the current package); the message was not. Proven live: the correct code for the orphaned history got *„check your ten words"* | **SHIPPED** (controller v0.201.0 + hub v0.97.0) — the ACK carries `superseded_present`/`superseded_at` and the screen names the situation. **It states what the hub knows and promises nothing** — the read path is still unbuilt (R-199's inventory) |
| **R-215** | **`GET /recovery` rendered the recovery story on a box that never had off-site backups.** The predicate was right and the page never asked it; the POST sibling and the backups-area template both gated the same sentence correctly | **SHIPPED** (controller v0.201.0) |
| **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** |
| **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/<name>` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **CLOSED 2026-08-06 — shipped in agent v0.127.0 and PROVEN LIVE on a genuinely rebuilt box.** The fix is **corroborated, not a widened prefix**: a mountpoint outside `/mnt/felhom-drives` is forgiven only when the SAME device is also mounted under the managed path — a pairing only Felhom's own enrolment produces, so a disk another system is using at `/srv/data` or even `/mnt/someone-elses-disk` is still refused (own test + red-proof). Read from `/proc/mounts` deliberately: the lsblk invocation is pinned verbatim in the sudoers file, so switching to plural `MOUNTPOINTS` would have shipped a sudoers change with the binary. Fail-safe: an unreadable mount table corroborates nothing. **Measured on the Part 4 venue after a real guest purge, with both raw mounts still present on the surviving host:** `/disks/candidates` returned both drives in `attach` and `initialize` (before the fix: two empty lists), and both **re-attached through the customer endpoint** (`registered: true`). The customer-facing refusal was corrected in controller v0.203.0. |
| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** |
| **R-223** | **The Day-0 manifest vouched agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-broke the recovery path — the one event that makes the feature necessary. **⚠ AND IT WAS NOT A DROPDOWN.** The first vouch attempt was REFUSED by R-120's gate (`configs.go:1162`): *"golden 0.192.0 is older than the newest controller the fleet reports (0.201.0)"*. **The artifacts form saves as a unit, so the agent could not be vouched while the golden was stale — and the golden had been stale since controller 0.193.0, meaning the Day-0 manifest had been effectively UNVOUCHABLE for days and nobody had cause to notice.** The real remedy was a golden rebake | **CLOSED 2026-08-05.** Golden **0.201.0** baked in the drill VM (658 165 766 B, sha `e730d7cab343eb35…f007654`, **round-trip verified from Gitea**), then manifest set in one save: `agent=0.125.0 golden=0.201.0 min_agent=0.125.0`. A fresh install now lands on current agent AND current controller |

### Phase 2 — the injected faults, 2026-08-05/06 (unattended)

> **ALL FIVE CLOSED 2026-08-06** in controller **v0.202.0** + agent **v0.126.0**. The rule they now
> enforce, stated so it outlives them: **on the unlock path the customer is blamed only after a real
> attempt REFUSED their code; every other outcome, including an unclassifiable one, says something
> else.**
>
> **STILL OPEN AND DELIBERATELY UNTOUCHED BY THAT WORK — said explicitly rather than left to
> inference:** **R-214** (the console never stops showing a stale pairing code), **R-220** (a rebuilt
> box's drives cannot be re-enrolled — *currently worked around BY HAND on the campaign venue*, which
> is the only reason an app could be deployed there at all), **R-221** (a rebuilt box cannot run the
> escrow ceremony), **R-213** (putting files back), **R-202** (the orphan card's unconditional
> promise — now the **last** place on that surface still promising recoverability, two doors from
> where R-228 removed the same promise).

Eleven faults, each judged on **the message, not the outcome**, each with a positive control proving
the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/journal-phase24.md`.

| ID | What | State |
|---|---|---|
| **R-224** | **Every non-code failure on the unlock path is reported to the customer as a statement about their code.** Two faults, one defect. **F3** — the hub REJECTed (control: `302` → `exit 7`) and the **correct current** code returned M4 (*„Ez a kód nem nyitja meg azt a csomagot…"*) in **0.0556 s**. **F4** — `felhom-agent` stopped (control: `:8443` gone) and the same correct code returned M4 in **0.0299 s**. Neither elapsed time is compatible with a real unseal (~1.0 s of scrypt; F1's wrong-code attempts took 1.194/1.004/1.014 s), so **no unseal was attempted in either case** — the failure was upstream and was attributed to the code anyway. **The discriminator exists and is thrown away**: the agent's own `err` reads `fetching the sealed bundle: hub: transport error: … no route to host` (F3) versus `the recovery code did not unwrap the identity escrow` (F1), but both return **HTTP 400** under one merged sentence (*"…or the bundle could not be fetched"*), and the controller's failure path has **no branch for "could not ask / could not reach"**. **⚠ The vocabulary already exists and is wired to the wrong condition** — „A gép házon belüli kapcsolata most nem elérhető" fires only when `s.agentClient()` fails to *construct*, never when the RPC fails. **And R-216's gate cannot catch it**: the box's own ring reads `recovery capability gate: offsite_key_recovery=yes (source=version)` — the gate discriminates the agent's **age**, not its **reachability**, so a dead agent of the right version sails through the guard whose own comment says *"An attempt that cannot succeed must never be made, because its failure is attributed to the code."* **This is Phase 1's headline finding relocated from the version channel to the transport.** Breaches **I6** | **CLOSED 2026-08-06 — controller v0.202.0 + agent v0.126.0.** The discriminator is now a VALUE: `escrow.ErrBundleFetch` → **HTTP 502** at the agent, `agentapi.RecoveryRefusal` carrying the status at the controller, and `ClassifyRecoveryFailure` mapping it to one of five classes **from the value, never the text**. **PROVEN LIVE on the venue**, same wrong code, only the hub's reachability changed: `hub up → 400 "…did not open the sealed bundle"` · `hub REJECTed → 502 "…could not be fetched — the recovery code was NOT used"` · `hub restored → 400`. Red-proof: deleting the agent case reproduces `got 400, want 502` with the wrong-code sentence. **Coupled `MinAgent 0.126.0`** — an older agent answers 400 for both causes, so the reading is withheld and the 400 degrades to NEUTRAL; the gate blocks nothing. **The customer-facing messages were NOT re-driven end-to-end**: `/recovery` correctly redirects since F7 set the old data aside, and restoring that state is the reconfiguration §11 forbids — they are covered by handler tests + red-proofs |
| **R-225** | **The remote store reports `0 pillanatkép · 0 / 50 GB` when the box cannot read it — directly above a card stating the store holds backups.** `/backups/remote` renders both at once. **Ground truth, measured over SFTP with the box's own credential (read-only, no decryption):** `/home/felhom-repo/snapshots` holds **`f3d9cd67…`** — the Phase 0 snapshot with all three sentinels, the customer's only surviving copy — and `du -s` returns **12 535 KB**, matching the journal's `repo_size_bytes 12 611 522`. **Mechanism, from the box's own state:** after the rebuild the `offbox` block carries **no `snapshot_count` and no `repo_size_bytes` key at all** — the values are *unknown* and unknown renders as the zero value. **This is R-217's defect class in a second location**, and `OffsiteInventory.Empty` exists precisely because *"len(Apps)==0 is also what a failed read looks like"*. **I5 checked and NOT breached** — the hub's `0.0 GB` is 12.5 MB rounded, and the pool total is consistent; only the box's snapshot **count**, an integer, is false. Breaches **I6** | **CLOSED 2026-08-06 — controller v0.202.0.** `StatsKnown` is a **named** state (the `OffsiteInventory.Empty` pattern), because zero is what an unread store and an empty one both look like and `omitempty` makes "absent" and "0" the same bytes. The fill bar renders only when the fill is known — a 0 %-wide bar is a picture of emptiness. **PROVEN LIVE both ways**: before a run the venue read „a pillanatképek száma még ismeretlen"; after one, „2 pillanatkép … / 50 GB". A measured zero still says zero |
| **R-226** | **M1 — the only message that tells a customer to check their typing — is unreachable on any box that has re-escrowed.** The failure path tests M4's condition *before* M1's (`if present, at := s.recoverySuperseded(); present { …M4…; return }`), so on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4. **Measured**: F1's three wrong-code attempts each returned M4 after a real ~1 s unseal. M4 is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but it offers no hint to re-check the ten words and routes the customer to support about *older* backups. **The population that has re-escrowed is exactly the population that has just been handed a new recovery code and is most likely to be typing one.** R-222's fix removed one conflation and introduced another on the same branch | **CLOSED 2026-08-06 — controller v0.202.0.** The retained-package message now names **both** possibilities and restores the ten-words prompt, because the two are indistinguishable at the engine and saying so is the honest thing. It still does not promise the earlier package can be opened. Red-proof: removing the clause makes the prompt unreachable again |
| **R-228** | **After „I do not want the old data", the set-aside history becomes invisible — the box records where it is and shows it to nobody.** The move-aside itself is **correct and was verified byte-for-byte**: `/home/felhom-repo` → `/home/felhom-repo.orphaned-20260805` with its mtime, its `du -s` (**12 535 KB**) and snapshot **`f3d9cd67`** all unchanged, and a fresh empty repo initialised beside it. **Nothing was deleted.** But `settings.json` then carries `"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"` and a census returns **zero** references to `OrphanedRenamedTo` in any template or web handler — the field is written and read by nobody. `/backups/remote` after the set-aside contains no occurrence of the path, „félretéve", „régi előzmény" or any equivalent (instrument controls: `felhom-repo` → 2, „letétbe helyezve" → 1). **12.5 MB of the customer's deliberately retained data sits at a path the box knows and never shows**; its only mention is a flash message on the redirect, gone on the next click. Meanwhile `GET /recovery` → 302 and `POST /recovery/unlock` → 302 with no message, so a customer who changes their mind gets **no explanation at all** (correctly, not a typing accusation — but not an explanation either). **The project's own "seam built but never wired" pattern**, landing on the one promise the set-aside screen makes | **CLOSED 2026-08-06 — controller v0.202.0.** `OrphanedRenamedTo` is surfaced as two facts and stops. **It does not promise the history can be reopened** — it cannot be, by anyone, today (R-199's inventory is unbuilt) — and the set-aside **confirmation copy was corrected** for the same reason: *"a helyreállítási kód nélkül többé nem lesznek megnyithatók"* implied that WITH the code they could be. The field's own comment said "recovery-code-recoverable", the same over-promise in the code. **PROVEN LIVE**: the notice renders on the venue |
| **R-227** | **A controller restart mid-unlock returns a raw English `Bad Gateway`.** F8 restarted the container at T+0.7 s, inside the unseal window (control: `StartedAt` moved). The customer got **HTTP 502 / „Bad Gateway"** from traefik — a raw upstream error, in English, naming no reason and saying nothing about whether the key was installed. **The state half is clean**: the four `/data/offbox` files stayed byte-identical with mtimes frozen, and the controller returned healthy in 40 s. Breaches **I3** | **CLOSED 2026-08-06 — controller v0.202.0, partially and stated as such.** **The layer that answers is traefik**, whose config this repo generates — but traefik v3 serves no static files, so a branded proxy page needs a **new always-up container** for every 502 on the box: **scoped, not built**. Shipped: the unlock posts via `fetch` and answers a gateway failure in Hungarian in-page. **Progressive enhancement — with no JS the plain POST still shows the proxy's error** |

## Instruction files — deferred half, 2026-08-06

| ID | What | State |
|---|---|---|
| **R-229** | **The instruction-file rightsizing landed for `felhom-controller` and the workspace root; three pieces were deliberately deferred.** Done 2026-08-06: controller split into a 92-effective-line core plus four `paths:`-scoped `.claude/rules/*.md`; workspace root 208→142 effective lines with its versioned copy kept byte-identical; surgical corrections to `felhom-agent` and `felhom.eu` (expired TEMPORARY block, every version literal, the Legacy-Windows copies, the duplicated health-check rule); five contradictions resolved — including a drill-VM claim **measured live** (`qm list` on demo-hp shows VM 300 `drill-r50`; `felhom-agent` was right, `felhom-controller` was wrong); new shared `felhom.eu/scripts/instructions_gate.py` registered in `controller_gates.py` and `agent_gates.py`, 20 fixture tests + red-proof. **Leg (a) CLOSED 2026-08-06 (part 2):** `felhom.eu/CLAUDE.md` **227 → 115 effective lines**, split into a core plus `.claude/rules/{hub,website,manifests,docs}.md`; `instructions_gate` **registered in `scripts/repo_gates.py`** (six gates, all OK) in the required order — trim first, register second, because a registered-but-failing gate refuses every push. Scoping proven from the `InstructionsLoaded` hook log in two fresh sessions, not from frontmatter. **Still deferred:** (b) **CLOSED 2026-08-06 (close-out)** — `felhom-agent/CLAUDE.md` **175 → 99 effective lines** (measured 175, not 173: the CI correction added two), split into a core plus `.claude/rules/{proxmox,localapi,backup,storage}.md` beside the existing `health-checks.md`. The release section now points at the `felhom-build-deploy` skill instead of restating a table that drifts from the script. **Every `CLAUDE.md` in the workspace is now ≤120 effective lines except the workspace root at 142, which is deliberate — it is the only file re-injected after `/compact`.** (c) **CLOSED 2026-08-06 (part 2)** — all 44 orphans resolved with **zero deletions** (file count 158 before and after): 4 durable `reference`-type files indexed, 40 dated episode records moved to `.claude-memory/archive/`. `MEMORY.md` 145 → **150 lines / 17,977 bytes**, and `instructions_gate` check 6 now watches it (over-limit FAILS, orphan WARNS, absent store PASSES *printing its reason*). (d) **The spec-as-failing-test pilot** — moved to R-230. Full accounting: `audits/LEDGER-instruction-trim-2026-08-06.md` + `audits/LEDGER-instruction-trim-part2-2026-08-06.md` | **READY** — owner Viktor |
| **R-230** | **Three instruction/memory follow-ups deliberately left by the part-2 session (2026-08-06), each needing a decision rather than an implementation.** (a) **A ruling is owed on auto-written staleness.** The hand-written `CLAUDE.md` files are now clean of version literals and expired blocks — the gate enforces it — but `MEMORY.md`, which Claude writes and which is the LARGER half of what loads (8.4k tokens vs the root file's 6.6k), carries **21 lines with component version literals**, **5 with bare host addresses**, and an entry still reading *"demo boxes REMOTE till ~08-02"* — the same expired-TEMPORARY class the gate was built to kill, now surviving in the one file the gate's content rules do not cover. **Partly actioned 2026-08-06 (close-out), and the ruling is STILL OWED:** the **three statements that were actively false** were corrected — `R-193 decision open` (closed 2026-08-05), `demo boxes REMOTE till ~08-02` (the box answers on the home LAN), `OPEN R-25b` (shipped 2026-07-21) — and gate check 6 now **WARNs** on version literals, host addresses, expired statements and stale-open citations in the index. WARN, never FAIL: Claude writes that file between sessions, so a hard failure would refuse a human's push over a line no human typed, and the warning is read by the model that will next edit it. **The remaining 32 version literals and 4 host addresses were deliberately left** for that loop. What is still owed is the bulk-correction ruling. **Correcting the premise:** the earlier report's "three expired statements" were all FALSE POSITIVES — each matched an ISO date inside a markdown link target, i.e. a filename — while the one real expired claim carried no ISO date at all. (b) **CLOSED 2026-08-06 (close-out)** — the workspace-root `CLAUDE.md` **is now a relative symlink** to the versioned copy, so the divergence class is gone rather than policed. Check 5 learned two shapes: for a link it asserts the target resolves to a real file (**a dangling link is worse than a diverged copy — the instructions load NOTHING and there is no content left to notice is wrong**), for two files byte-identity as before, so a clone elsewhere is unaffected. **Proven, not assumed:** three fresh sessions logged `session_start` for the link path, and a fourth **with no tools at all** quoted standing rule 1 verbatim — the content reaches the model, not just the path. (c) **The spec-as-failing-test pilot**, approved in principle and not started (was R-229(d)). | **READY** — owner Viktor |
| **R-232** | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **READY** — owner Viktor |
| **R-231** | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **READY** — owner Viktor |
| **R-233** | **The golden bake's acceptance checks were a list of strings the script does not print** — found 2026-08-06 while baking golden 0.203.0 by following `runbooks/RUNBOOK-manual-build.md` §4.1 verbatim. Two of the three named pass markers **cannot ever match**: `overlay2 OK` is not in `build-golden.sh` at all (the line it means is `  docker OK (overlay2; data-root /var/lib/docker)`), and `including mount point … mp1` refers to a volume that stopped existing in `build-golden.sh` **v3.0.0**, when R-165 collapsed the two data volumes into one. The 404 pre-gate's URL was also wrong — the published filename is `golden.tar.zst`, not `felhom-golden-<VER>.tar.zst`, so the pre-gate would 404 for the wrong reason and pass **even when the version already existed**. This is the *"an instrument that can silently drop results is not a measurement"* class landing on the bake's own acceptance check: a grep for an impossible string reads `0` forever, and `0` is indistinguishable from failure. **The bake was never actually unguarded** — the script's own `[ "$drv" = "overlay2" ] || { echo FATAL; exit 1; }` is fail-closed and the run exited 0 with no `FATAL`. The **document** was the broken part, which is why nothing had ever gone wrong and nobody had noticed. **FIXED in the same session:** all markers re-captured from the real log rather than paraphrased, the corrected pre-gate URL, the token handling moved off the command line into an in-VM runner script (the old `--setenv=GITEA_TOKEN=$GT` form put the value where `systemctl show` prints it), a required **positive control** on the token-leak grep, and the vouch step rewritten as the three-field change it actually is. **The general lesson:** a runbook's pass markers must be **copied from a captured log, never written from memory** — §4.0 of that same file already learned this for the qemu launch line and says so; §4.1 had not. | **CLOSED 2026-08-06** — fixed in `RUNBOOK-manual-build.md` |

| **R-234** | **An off-site run reports success while silently omitting an app the customer just switched on.** Found 2026-08-06 on the Part 4 venue (`part4`, VM 323), and found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. Sequence, measured: a run with no app selected produced **1 snapshot**; `POST /backup/offbox/toggle` enabled `calibre-web` (HTTP 302, and the „Nincs távoli mentésre jelölt alkalmazás" warning disappeared, so the selection HAD landed); the next run finished in 30 s and reported **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one snapshot. The app restore then refused: **„offbox: nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A THIRD run took the snapshot count to 2 and the same restore then succeeded. So a run that the customer sees as a green success did not carry the app they had just enabled, and **nothing in the card distinguishes that from a run that did**. The customer's belief ("my app is off-site") and the truth diverge silently, and they would discover it only at restore — the worst possible moment. The snapshot count is on the same card, which is what makes the omission detectable in hindsight and invisible in the moment. ****ROOT-CAUSED 2026-08-06, and it was NOT the hypothesis in this row.** Two things were measured on demo-hp rather than assumed. (1) **The filed mechanism cannot happen for a deployed app:** the off-site run's own pre-dump phase calls `captureAllRecoveryUnits` for every DEPLOYED stack, through `admitApp`, before the push — a recovery unit moved aside was RECREATED by the run, which then reported `ok`. So "the toggle leaves the app without a bundle and the first run skips it" is not a state that survives a run. (2) **The actual cause was the single-flight:** the manual run was dropped because an earlier run was still going, `runOffboxBackup` returned **nil**, the handler had already answered „A távoli mentés elindult”, and the card then showed the PREVIOUS run's „✓ Rendben” — read as covering the app just selected. **Fixed in controller v0.205.0** on both fronts: the single-flight decision is taken SYNCHRONOUSLY in the handler and a dropped request says so („Már fut egy távoli mentés — ez a kérés nem indított újat. A most látható eredmény még a korábbi futásé…”, proven live on demo-hp), and — independently, because it is a real gap in its own right — **a run that skipped an app the customer selected is now `incomplete`**, not `ok`. Which skips count: selected+deployed with no unit YES; selected but NOT deployed no (named, with what to do — a box left amber by an app somebody removed is a status nobody reads); disconnected/decommissioned drive no (own signal); nothing selected no. The nightly path still returns nil on purpose. §7.3 measured: a recovery unit is a few KB of compose+manifest, only ENUMERATES dumps, does not stop the app, and already runs inside the off-site run — so there was no wait to remove and nothing was built. **This is the exact class the project already has a rule for** — "presence is not success": the run's timestamp and tick record that a run HAPPENED, not that it carried what the customer asked for. | **CLOSED 2026-08-06** — controller v0.205.0 |
| **R-235** | **The appliance console keeps telling an already-paired box to go and pair itself.** Measured 2026-08-06 on VM 323: **25 minutes after** the operator bind, with the guest provisioned, the controller reporting 0.203.0 and the agent ONLINE, the physical console still displayed „Felhom — a doboz készen áll, és a **párosításra vár**" together with the now-spent pairing code `US3-6GP` — and, in the same panel, the promise **„Ez a képernyő magától frissül — nincs teendő a doboznál"**. It does not refresh. A customer looking at their screen is told the setup has not happened, and is told the screen would have updated if it had. Cosmetic in mechanism, not in effect: it invites the customer to re-pair a working box, or to call for help about a box that is already fine. Same family as R-234 — a surface asserting a state that stopped being true. | **READY** — owner Viktor |

| **R-236** | ~~**After a guest rebuild the hub never re-stages the off-site credential.**~~ **WITHDRAWN 2026-08-06 — THIS WAS WRONG, and the mechanism works.** Measured 2026-08-06 on the Part 4 venue. The unlock screen promises „A gép még várja a házon kívüli tárhely kapcsolódási adatait — amint megvannak, a mentéseid listája megjelenik… **Nincs teendőd**". What actually happens: the rebuilt box declares `needs_credential`, R-218's retry job ticks every 5 min exactly as designed and logs the honest reason — `[offsite-apply] credential retry: consume one-time password: no unconsumed offsite password (already consumed or none provisioned) (the box still declares a need; retrying)` — **forever**, because the one-time password was consumed by the guest that no longer exists and nothing mints a new one. The fix is a single operator action that already exists (`/configs/<id>/offsite-reissue`), and pressing it resolved the stall within one tick. **R-218's consume half is not at fault — it is the half that works**; the gap is upstream, in who re-stages after a rebuild. **What the hub log actually shows** (`kubectl logs`, 2026-08-06): `offsite-delivery` refused to self-heal once a minute, correctly, naming `internal/offsiteheal` as the owner of the remediation and warning that a second mechanism minting there would double-issue; then at **15:24:57 local (13:24:57Z)** — `offsiteheal: re-staged the stored one-time offsite secret for customer part4 (declared needs_credential across 2 reports) — no provider credential was minted`. **The reconciler fired on schedule and succeeded.** The operator Re-issue at **13:26:39Z** came **102 seconds LATER**, was redundant, and **minted an unnecessary provider credential** (`re-issued shared credentials for part4 (subaccount 284735)`) — exactly the double-issue the guard exists to prevent. **The error was mine: I did not wait for the documented two-report debounce** (~16 minutes after a rebuild, since reports are ~15 min apart and the reconciler ticks every 5), and attributed the recovery to my own button press. „Nincs teendőd" is TRUE, and the wait is far inside the „egy napon belül" the card promises. **Operational lesson, not a product defect:** after a rebuild, wait for the debounce — the hub says once a minute, in its own log, that it owns this remediation. | **CLOSED 2026-08-06 — not a defect** |
| **R-237** | **After a successful recovery the customer is shown no backups at all, because the restore surface is keyed on apps that are currently installed and currently marked for future remote backup.** Measured 2026-08-06. Post-rebuild, with the key recovered, the tier configured and the escrow re-sealed, `/backups/restore` said „**Nincs telepített alkalmazás**" and „Nincs távoli mentésre jelölt alkalmazás — a kijelölés a Távoli mentés oldalon történik" — pointing at a page which itself said there were no installed applications. A **circular dead end**: to restore an app you must select it; to select it, it must be installed; to know what to install, you must see the backup you cannot see. Reaching the restore actually required three undocumented steps in order — re-attach the drives, redeploy the app, and toggle it on for **future** remote backups (`/backups/restore/app` refuses with „Ez az alkalmazás nincs távoli mentésre kijelölve", i.e. a *forward-looking* setting gates a *backward-looking* action). None of this is hinted at by the recovery screen, which says „Nincs teendőd". A household that has just lost its box does not know which apps it used to run. | **CLOSED 2026-08-06** — controller v0.204.0: the list is now built from `OffsiteInventoryList` (the repository's own snapshot tags). Installed-ness became a property OF a row, never a filter; an unreadable store renders as UNKNOWN **and keeps the action offered**; `felhom-offbox` and `_shares` are excluded. 7 new tests incl. a rendered-page test for the rebuilt shape, and a red-proof that keys the list back on installed-and-toggled apps. |
| **R-238** | ~~**„Teljes visszaállítás előkészítése" accepts the click and does nothing.**~~ **RECLASSIFIED 2026-08-06 — a HARNESS ARTIFACT, with a real residue that IS fixed.** Measured 2026-08-06, repeatedly. `POST /backup/offbox/restore` with `mode=full` (the exact form the page renders) returns **302**, and then: no job is ever recorded (`/api/backup/restore-status` `last` stays `null` across ~9 minutes of polling), the wizard stays on step 1 „Előkészítés" with the same two forms, no error is shown to the customer, and **the controller's own debug ring contains no line for it at all** — zero `offbox`, `prepare` or `snapshot` entries. `mode=unit` on the identical form works and reports properly, so the plumbing and the session are fine. This is the terminal dead end of the recovery journey: everything upstream succeeded — code accepted, key recovered, tier configured, escrow re-sealed, drives re-attached, app redeployed — and the customer still cannot get their files back. **PASS for the R-201 re-walk was defined as a sentinel's sha256 byte-identical after recovery; this is why that criterion was NOT met.** **The diagnosis** (`restore_wizard.go`, `offbox_handlers.go:314`): `mode=full` without `confirm=1` is **step 1 of a deliberate two-step** — it computes size + headroom, **starts no job by design**, and redirects carrying `&full_prep=<app>&full_size=<size>`. `deriveWizardStep` reveals the commit **only** when `FullPrepApp == App`. The endpoint driver posted step 1 and re-fetched the wizard **without** that parameter, so the pure function correctly returned the intent step, and `restore-status.last == null` was correct too. **The operator drove the same restore to completion in a browser.** The wizard was NOT re-keyed. **The residue was real and is fixed in controller v0.204.0:** neither branch of step 1 wrote anything to the log — `offboxRedirectTo` only flashes to the page — so a refusal, **including by the headroom gate**, left no trace on the box. Both branches now log, as does the concurrent-op refusal; pinned by a handler-level test with a red-proof. | **CLOSED 2026-08-06** — controller v0.204.0 |

| **R-239** | **The fixes are written, tested, pushed — and a machine installed tonight gets none of them.** Measured on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`): a fresh install from the published ISO landed on agent **0.127.0** (vouched, **no hand upgrade** — that half works) and controller **0.203.0**, while the newest released controller is **0.205.0**. The vouched golden still bakes 0.203.0, so a new box has neither **R-237** (v0.204.0 — the restore list driven by the store) nor **R-234** (v0.205.0 — the skipped-app verdict and the single-flight message). **Proven from the customer's side on that box, not inferred:** T3 — with the future-backup toggle switched off, the restore page showed **0** entries for an app whose snapshot demonstrably exists, and the wizard redirected away; T2 — two off-site runs back to back both answered „A távoli mentés elindult", the second as a SUCCESS flash, which is exactly the sequence that made a customer read a stale green tick as covering an app they had just selected. **Not a regression and not a code defect — a delivery gap.** The remedy is already documented: bake a golden carrying the current controller and vouch it as a three-field change (`golden_version` + `agent_version` + `min_agent`), per `runbooks/RUNBOOK-manual-build.md` §4.1. **Until that happens, every closure claimed for R-234 and R-237 is true of the code and false of the fleet** — which is the distinction this register exists to keep. **✅ CLOSED 2026-08-07 — golden `0.205.0` baked, published, round-trip verified and VOUCHED (operator-approved this session).** Evidence: `tests/golden-0.205.0-2026-08-07/`. sha256 `8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8`, 656,937,561 bytes. **Verified by round trip rather than trusted:** the published bytes were fetched back, re-hashed (match), `zstd -t`'d clean, and `./etc/felhom-controller-image` was read **out of the downloaded archive** → `felhom-controller:0.205.0` — the step that matters, because `GOLDEN_VERSION` is derived from the tag argument and could have been right over stale content. A **third independent witness**: the hub's own Day-0 dropdown lists `0.205.0` with `data-sha="8f49b2e8…"`. **In the event it was a ONE-field change, not three:** `golden_version` 0.203.0→0.205.0 (plus its derived sha); `agent_version` and `min_agent` both stayed **0.127.0**, because the new golden's MinAgent is also 0.127.0 — read from live `hub_settings` before and after, not assumed. `wrapper_sha256` was carried through explicitly: the handler reads it from the form and **clears it when omitted**, so a headless POST that forgets it silently drops the PBS-DR wrapper hash. **The R-120 gate passed exactly** — the newest controller the fleet reports is 0.205.0 (demo-hp), so a 0.204.0 golden would have been REFUSED. Vouching is reversible by re-selecting the previous values; a bake never deletes an older golden's package. **The systemic half is NOT closed** — nothing enforces that a release reaches a golden → **R-242**. | **CLOSED 2026-08-07** |
| **R-240** | **A backup that covered nothing calls itself „Sikeres".** On a configured box with no app selected for off-site backup, a run reports status `ok` with the warning „Sikeres — nincs mentésre jelölt alkalmazás" — *successful* immediately beside *nothing is selected*. Measured as T4 on the final walk, 2026-08-07; flagged once before (2026-08-06) and deliberately not touched then, because the task that noticed it forbade changing that path. **It is the same rhetorical shape the project has spent a fortnight removing** — R-203's *a warning beside a success is read as a success*, R-234's *„✓ Rendben" over an app that was skipped*, R-225's *unknown rendered as zero* — one notch weaker each time, and this is the weakest and last of them. The state itself is honest and must stay `ok`: an unconfigured box reporting `incomplete` forever is its own defect, pinned by a test. **The defect is the word „Sikeres", not the verdict.** Wording such as „Nincs mentésre jelölt alkalmazás — ez a futás semmit nem mentett" says the same thing without congratulating the customer on it. | **READY** — owner Viktor |

| **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shape (c)** — needs no hub change and no new protocol field — **plus a `decided` latch**, because `ResetOrphanedRepo` clears `RepoState` without running a ceremony, so an `H`-mismatch discriminator alone would re-offer the screen forever to a customer who explicitly declined the old data. **Four operator decisions are stated and left unanswered in §"THE OPERATOR'S DECISION".** **✅ FIXED 2026-08-07 — controller v0.206.0 + hub v0.98.0, and the ruling above is what the fix follows.** **(1) It stops minting:** the guard is a CONJUNCTION (a package held AND no key present), so a first-time box mints exactly as before; the refusal is a HOLDING state, not a failure — the transport is still written so the recovery screen can bring the tier up the instant the key arrives (R-219), and returning an error instead would have left the hub re-staging a consumed credential for ever. New declared state `offsite.state=awaiting_recovery_key`, shown INERT to every existing hub reader from their code rather than assumed. **(2) The discriminator is persisted and drives the offer as shape (c).** §7.2 resolved deliberately: **a known difference offers however old the reading** (age is NOT gated on — gating would make a box offline from the hub silently stop offering, the very failure this removes), and **a hash never learned falls back to (a)/(b)**, because an empty hash is the hub positively saying its package seals no key rather than an unknown. **(3) Abandoning ends the question** — a 14-day countdown, visible and reversible, whose terminal step removes the set-aside store AND the sealed package together, after which shape (c) has nothing to compare and the offer falls silent **because the state is right, not because something remembers it once was not**. The two halves cannot be atomic across two machines, so it is a two-phase commit whose confirmation rides the SAME ACK that carries the request. **(4) The surface:** the full page appears **once per ENTRY into the offered state, not once ever** (an epoch — a box rebuilt months later is a new situation); three dismissal levers with three scopes, and **none removes the entry point on the backups page**. **Q7's trap does not survive:** while a recovery is outstanding „Helyreállítási kód létrehozása" is **unavailable**, not merely captioned. **§2.4 honoured:** the abandon confirmation no longer promises *„félretesszük — nem töröljük"* — it states the deletion date. **TWO REAL BUGS WERE CAUGHT BY TESTS RATHER THAN REVIEW, and both are recorded because the shape matters:** `OffboxAwaitingRecoveryKey` omitted `t.Enabled`, so a customer who had switched off-site OFF would have declared a holding state (caught by the EXISTING `TestOffsiteDeclare_DisabledTargetIsNotStranded`); and `recoveryInterrupts` returned early when the offer was false, so the FALLING edge was never recorded and the full page never came back — **the exact defect the epoch exists to fix, reintroduced inside the fix**. **Nine red-proofs, each with the mutation confirmed present in the file before its result was trusted**, including the ships-inert one (unwiring `RecordEscrowKeyHash`, which leaves everything compiling and every test passing while shape (c) reads an empty hash for ever). **NOTHING WAS DELETED ANYWHERE** — the terminal step has only ever run against injected fakes and an injected clock (§7.4). **Still open and NOT built by this:** R-242 (the release-to-golden gate) and R-245 (the automatic 30-day ending). | **FIXED 2026-08-07 — v0.206.0 / hub v0.98.0** |

| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real.  **⚠ THE GATE FIRED FOR REAL, 2026-08-08 — and it was right.** Controller **v0.207.0** (R-249/R-252/R-253) is released, tested and pushed, and **no golden carries it** — the newest bake is 0.206.0 — so `golden_currency_gate.py` FAILED, saying exactly the true thing: *a machine installed right now would receive v0.206.0*. **The `felhom.eu` push therefore used `git push --no-verify`, declared here, in the commit message and in the session report.** A bypass and NOT a waiver, deliberately: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.207.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change). **This row's own remaining half is unchanged — nothing gates the VOUCH itself.**  **✅ THE OWED BAKE IS DONE, SAME DAY — golden 0.207.0 baked, published, round-trip verified and VOUCHED (2026-08-08).** The gate went from red to **green**, and the `--no-verify` bypass declared above is now historical rather than standing. **Round trip is the evidence, not the build log:** the published bytes were downloaded back — 656 879 192 B, sha256 `20ec9602…22995`, both identical to what the bake reported — and **`./etc/felhom-controller-image` read OUT of the downloaded archive says `felhom-controller:0.207.0`**, which is the delivered artifact naming the controller it will start. **The vouch was a three-field change with all three checked deliberately** (`MinAgent 0.127.0` read from the golden's controller CHANGELOG header, not assumed; `agent_version` already ≥ it; `min_agent` not above `agent_version`, so not the R-216 shape) and verified by **re-reading the manifest rather than trusting the flash**. **This row's remaining half is UNCHANGED and is the whole of what is still open: nothing gates the VOUCH itself** — the currency gate's own docstring says it checks the bake, so a baked-but-unvouched golden still passes it silently. Evidence: `tests/golden-0.207.0-2026-08-08/`.  **⚠ RED AGAIN, 2026-08-08 (second time in two days) — controller v0.208.0 (R-254) is released and the vouched golden is 0.207.0.** `golden_currency_gate.py` FAILS, correctly: a machine installed right now receives 0.207.0 and none of today's fixes. **The `felhom.eu` push used `git push --no-verify`, declared in the commit, the CHANGELOG and the session report** — **a bypass, not a waiver**, on the same reasoning as yesterday: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.208.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; three-field change, `MinAgent 0.127.0` unchanged). **Note the cadence this is establishing: two releases, two bakes owed within 24 h.** That is the argument for this row's OTHER half — nothing gates the vouch, so the only thing standing between a release and an undelivered fleet is somebody remembering. | **READY — the vouch half only** — owner Viktor |
| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** **⚠ UPDATED 2026-08-07 (v0.206.0) — the STATE this row describes can no longer be entered, but the ALARM GAP is untouched and the row stays open.** R-241's mint guard means a box no longer mints a key over a sealed package, so it no longer arrives in the "escrow stuck pending against a self-minted key" state by itself. **What replaces it is a state that is VISIBLE rather than silent:** the box declares `offsite.state=awaiting_recovery_key` and the customer is offered the recovery screen. **But the hub still raises nothing for it**, and for the same three reasons: `isStale` needs `escrowed`, the delivery checker skips the `applied` shape, and `backup_failed` needs a run that never happens. **So a box whose customer never acts still stops backing up off-site with no operator signal** — the difference is that the customer can now see it and act, where before nobody could. **The remaining work is an operator-side signal for a box held in `awaiting_recovery_key` past some age**, and it is deliberately not bundled into R-241's fix.  **⚠ MEASURED ON A REBUILD, 2026-08-07 (fifth walk) — the gap is real for the state this row describes, and NOT for the state a rebuild produces.** 88 seconds after the walk5 guest was destroyed and rebuilt, the hub emitted `offsite_delivery_stuck` (**warning**) and wrote an **operator-channel** `notification_log` row recording `offsite_credential_restaged` / status **REFUSED** with an accurate reason — *"the credential was applied and worked; the target was lost afterwards … a guest rebuild does, R-193"*. So on the **regressed-apply** shape the operator IS told, promptly and correctly, and this row's *"skips the applied shape"* does not apply. The gap stands for a box that reaches the held state **without** a prior working tier in its report history. **Recorded so the row is not read wider than it measures.** | **READY** — owner Viktor |

| **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. **⚠ STILL OWED, AND NOW MEASURED RATHER THAN ESTIMATED (2026-08-08 census, read-only, no truncation).** `app_log_issues` holds **1309 rows**; **71 reference a torn-down venue** (`finalwalk`, `c11`, `rewalk`, `part4`); of those **44 are ORPHANS** — they name only torn-down customers and are safely deletable — and **27 are SHARED with a live customer** (`demo-felhom`, `peti-felhom`, …) and **must be de-referenced, never deleted**. 1238 rows are untouched. **The 27 are exactly why the leg was never written**, and why a `DELETE … WHERE customer LIKE` would destroy a live customer's issue history. **What it needs, precisely:** a cascade leg that (a) removes the customer id from `affected_customers` / `context_customer`, and (b) deletes only rows whose `affected_customers` becomes empty; plus a one-off sweep for the four venues already gone. **Why it was NOT done on 2026-08-08:** the fix is hub code, and that session's scope forbade a hub version bump; a hand-run SQL mutation over 71 rows — 27 of them needing surgical de-referencing — with no tested code path and no red-proof is precisely the shape that goes wrong on a live database. **It accumulates one venue at a time, so the next walk adds to it**; the numbers above mean the next session starts from data rather than a guess.  **⚠ IT GREW AGAIN, AS PREDICTED — walk5 teardown, 2026-08-08.** The fifth walk's venue was torn down with a full-schema census taken **before and after**: **168 rows → 67**. Of the 67, **37 are by design** (`events` 21, `notification_log` 14, `host_deletions` 1, `customer_resets` 1) and **30 are `app_log_issues`** — this row's gap, and the count was **predicted in the pre-run enumeration rather than discovered afterwards**, which is the difference from the ledger that once recorded *"0 occurrences"* from a narrower query. **The running total across torn-down venues therefore rises from 71 to ~101 rows** (`finalwalk`, `c11`, `rewalk`, `part4`, now `walk5`) — the shared-with-a-live-customer subset must still be **de-referenced, never deleted**. **It accumulates one venue at a time and it did so again.** Evidence: `tests/walk5-r201-2026-08-07/teardown-walk5-2026-08-08.md`. | **READY** — owner Viktor |

| **R-245** | **Should a customer who never decides be auto-abandoned after 30 days? RECORDED, NOT BUILT — and the reasoning against it is recorded with it so the decision can be revisited properly.** **The operator's proposal (2026-08-07):** a box that has been offered recovery for 30 days without the customer deciding is auto-abandoned, entering the 14-day grace, so an undecided box does not sit for ever holding history nobody has claimed. **What was built instead:** the escalating reminders (1/3/7/14 days) and the operator levers `--abandon-extend` / `--abandon-stop`. **The reasoning, as settled with the operator the same day:** (1) **nobody is absent** — a box does not reinstall itself, so whoever rebuilt it was standing there and met the recovery question; the "customer away for months" case does not arise from this situation, because **a reinstall implies a person**. (2) **A customer who cannot find their code will get in touch**, which is the moment to extend or disarm by hand — so the automation would be firing at people we are already talking to, which is why the levers were the thing worth building. (3) **The cost is theirs**: the old history sits in the customer's own storage allowance, and if they are paying to keep something they have not decided about, that is their call and they feel it before we do. (4) **The real harm, if it comes, is QUOTA** — old history blocking new backups — and **that is a condition, not a calendar**. An automatic ending should trigger on the harm, with a dated warning, never on a date alone. **If this is ever built, build it that way.** **✅ RE-FILED 2026-08-08 AS A DECISION TAKEN, not a question pending.** It sat in the operator's queue as `WAITING-ON-OPERATOR` for a day, and **nothing was actually pending** — the operator and the reviewer settled it on 2026-08-07: it is **not built**, the levers were built instead, and the whole reasoning above is the record of why. A settled decision parked in a queue is a queue nobody trusts, and an audit of every `WAITING-ON-OPERATOR` row the same day found this was the ONLY one — so the drift was caught while it was still a single row. **THE CONDITION THAT REOPENS IT, which the reasoning already names: QUOTA — old set-aside history blocking new backups.** Not a calendar. If a customer's retained history ever refuses a new backup, revisit this with a dated warning that triggers on the refusal; until then it stays decided. | **DECIDED 2026-08-07 — not built; reopens on quota** |

| **R-246** | **A leftover staleness flag has silently disabled the new recovery discriminator on `demo-hp` since 2026-08-04, and the flag is WRONG.** Found by a read-only spike, 2026-08-08. **Q1 — traced to an act, to the second:** at `2026-08-04 20:15:49` the hub emitted `offsite_reissued` and `escrow_stale` in the same second — an operator **Re-issue** pressed during the R-201 drill, three minutes after `escrow_blob_served` at 20:12:40/20:12:54. That was `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub v0.95.0 REMOVED the very next day** (R-196 / R-204 item 2) precisely because it marked healthy escrows stale. **Q2 — the flag is wrong, measured on both sides:** the hub's blob seals `restic_pw_sha256 = 8a9e33aa4da6769c…d080a`, and the key the box is actually using hashes to **the identical value**. The blob covers the key. **Q3 — nothing clears it by itself:** the ONLY writer of `stale_at = NULL` is `SaveHostEscrow`'s `ON CONFLICT` — i.e. a fresh escrow ceremony, **which is the one act that would supersede the good blob**. So the only exit from a false alarm is the destructive act the false alarm recommends. **Q5 — the blast radius, enumerated rather than assumed:** (1) `GetEscrowStatusForCustomer` withholds `restic_pw_sha256` from the ACK; (2) a **pending** box can never auto-confirm, so (3) **every off-site run is refused indefinitely** — neither bites `demo-hp`, which was already `escrowed` and is backing up healthily (12 snapshots, last success 2026-08-07T02:15:35Z); (4) the customer is told to create a new code; (5) **NEW — v0.206.0's shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b), so the recovery screen would stay silent even if recovery were needed. **Q6 — a fresh box CANNOT reach this state:** `MarkEscrowStale` has **no production caller anywhere in the tree** (census: only its own definition, two comments and two test references). **The next walk cannot meet it.** **NOT CLEARED, deliberately** — Q2's answer says the fix is to clear this instance, but whether to also stop the column being settable at all is a separate ruling, and the spike was scoped read-only. **✅ THE FLAG IS CLEARED — operator-approved and applied 2026-08-08.** One row, identity-matched on `host_id` and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end, not just in the database:** the hub now serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa4da6769c…d080a` at `11:10:19Z`, and that is **byte-identical to the key it is using** — so shape (c) compares, matches, and correctly stays silent. **The false stale warning is gone, proven with a positive control** rather than an absent line: **0** `escrow-confirm` lines since the restart while **5** scheduler lines in the same window prove the box was logging, and the recorded hash proves an ACK was processed. *(Method note: the hub pod is Alpine with no `sqlite3`; it was installed into the container's ephemeral writable layer — image and node untouched, gone on restart. SQLite's own file locking coordinated the write with the live hub; an earlier attempt failed cleanly on quoting and changed nothing, which is the fail-safe working.)* **STILL OPEN under this ID: the ruling on whether `stale_at` keeps a live setter at all.** It currently has NO production caller, so the column is write-only-by-accident — a field that changes behaviour, that nothing sets and nothing can see (R-248). Either give it an evidential setter or retire it; **do not leave it as a trap that only a database read can spring.** | **READY — flag cleared; the column ruling is still owed** — owner Viktor |
| **R-247** | **The box is being told something false, in its own words, and it recommends the destructive act.** `demo-hp`, live, on controller v0.206.0: *"STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code"*. **Every clause is false.** The hub HAS the hash and is *withholding* it (R-246); there was no supersession (`host_escrow_superseded` has no row for this host); and the bundle **does** cover the password — the hashes match exactly. It raises `EscrowStale`, which renders the customer-facing card *„A letétben lévő helyreállítási csomag nem fedi a jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."* **THE CAUSE IS A FACT ON THE WIRE THAT THE BOX THROWS AWAY.** The hub already sends `escrow_stale` in the ACK (`json:"escrow_stale,omitempty"`), and the controller's `report.EscrowStatus` **has no matching field**, so `encoding/json` drops it silently. The box therefore cannot distinguish *withheld because flagged stale* from *genuinely hash-less*, and guesses the latter. **This is R-241's shape for the third time: the answer is available, and it is discarded at the boundary.** **The fix is small and is NOT made here** (§0 forbids a controller change this session): add the field, and say the true thing — or say nothing, since on an ESCROWED box with matching hashes there is nothing wrong to report. | **CLOSED 2026-08-08** — controller v0.209.0. The field is received and the box tells the two conditions apart; see the Campaign-12 follow-through section below. The WRONG FLAG itself is R-246 (operator act, hub-side) and the customer-facing card copy is unchanged — both stated rather than folded in |
| **R-248** | **A flag that changes behaviour is visible to nobody who would look for it.** Q4 of the 2026-08-08 spike, answered plainly. **The customer** sees only a derived card stating a false reason (R-247). **The box** cannot see it at all (R-247's dropped field). **The operator** can see it on exactly ONE page — the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`, `v.EscrowStale = escrow.StaleAt != ""`) — which is the wrong tier for this symptom: an operator investigating an OFF-SITE problem has no reason to open a PBS-DR page. **No alert, no report field, no off-site surface.** The one-shot `escrow_stale` event fired on 2026-08-04 and **was never notified** (a full census of `notification_log` for that customer that day returns 8 rows, none of them this one); it has fired twice ever, both on 4 August. **So the practical answer is: only a database read.** That is a finding in its own right — a flag that silently changes behaviour and cannot be seen is the shape this fortnight has been about (R-241's discarded comparison, R-228's unread field, R-243's unobserved state). **What it needs:** surface `stale_at` on the off-site operator surface and in the host report, or stop using a field nobody can observe to change what a customer is told. | **READY** — owner Viktor |
| **R-249** | **The retrieval passphrase ships in the customer page's HTML, so any headless read puts it in a transcript.** Found 2026-08-07 on the fifth walk **by doing it**: driving the documented rebuild path, the cleartext value reached the session transcript from a `grep` over the saved page — **not** from pressing Reveal. The hub renders the per-customer **Retrieval Password** masked (`••••••••`) behind a Reveal/Copy control, and carries the cleartext in a **`data-secret="…"` attribute** of that control; `hub/internal/web/render_test.go:169` pins exactly this (*"data-secret not populated for the reveal control"*). **The masking is a presentation control, not a containment one.** **Why it is a defect and not a design note:** the same page's own hint reads *"never place it on a command line (the installer reads it at a no-echo prompt)"* and the generated install command deliberately omits it — **the page states the threat model and then violates it in its own markup**. Anything that reads the page rather than looking at it (curl, a scrape, an agent, a saved HAR, a support bundle) gets the secret with no reveal action **and no audit event** — where the break-glass credential, which has the same sensitivity, emits `recovery_credential_revealed` when revealed. **Proposed shape:** serve it from a POST endpoint the button calls — the break-glass credential already works exactly this way (`POST /hosts/{id}/reveal-recovery-credential`) — and render the page with no value in it; that also gets the reveal audited, which it is not today. **Severity MEDIUM:** it is a live per-customer secret that fetches the whole config (`GET /api/v1/config/<id>` with `X-Retrieval-Password`), but the exposure is to someone who can already read the operator page — a defence-in-depth failure, not a boundary crossed. **The walk5 instance is compromised (it is in a transcript) and dies with that customer's teardown, which is owed.**  **✅ FIXED — controller v0.207.0, 2026-08-08.** The page now carries only `HasRetrievalPassword`; the value comes from **`POST /settings/retrieval-password/reveal`** — CSRF-covered (POST, not GET, so `CsrfProtect` applies and it is not re-fetchable from history), `Cache-Control: no-store`, **and logged as an act**, which reading it off the markup never was. **Proven live on VM 325 with a positive control in both directions:** with a passphrase stored and the card rendering, v0.206.0's page body contained it **1** time in 50 208 bytes and v0.207.0's **0** times in 51 115 bytes; the reveal endpoint returns it with `no-store`, and the controller log carries the act with **0** occurrences of the value. The tests assert the **RAW RESPONSE BODY** — a red-proof that put the value back into the page reproduced the defect and failed the test, which no test asserting what the customer *sees* could have done. **The census this required found two more instances — R-254, not fixed.** **Rotation:** the `walk5` instance is compromised (it reached a transcript) and dies with that customer's teardown; no other instance is known to have been read, **but nothing records a read, which is itself part of the defect.** | **CLOSED 2026-08-08** |
| **R-250** | **A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time.** Found 2026-08-07 creating the fifth walk's venue. `POST /configs/new` with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — **fail-closed by design** (`offsite.go:111-121`, *"don't serve a descriptor the controller can't verify"*), with `defaultScanBackoff` = 2+4+8+16+30 ≈ **60 s**, sized by its own comment to *"the observed DNS propagation lag"*. **Measured, both halves:** the first create exhausted the ladder — five `no such host`, then `dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` (the name had just begun resolving, **AAAA-first, into a pod with no IPv6 route**) — and the hub logged `[ERROR] offsite provision for walk5`. An **identical second POST succeeded** ~70 s later on its own final rung (`shared already provisioned … (subaccount 285351)`). **Total settle ≈ 100 s against a 60 s budget.** The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, `SSH-2.0-OpenSSH_9.6p1`. **Two distinct things are wrong and should not be merged:** (1) the budget is sized against DNS *existence*, but what actually bit is the **AAAA-before-A** window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) **the operator is told nothing actionable** — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — `one_time_secrets` 1, one sub-account, no double-provision) **but that safety is invisible to the person deciding whether pressing again will double-charge them.** **Severity LOW-MEDIUM:** self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. | **READY** — owner Viktor |
| **R-251** | **The recovery listing renders one row per restic TAG, so the customer is shown an "app" they never installed and their data counted twice.** Measured on the fifth walk, 2026-08-07, on the screen the customer reaches after entering R. The snapshot carries tags `felhom-offbox,calibre-web`; the listing renders **two rows** — `calibre-web · 2026-08-07 14:57 · 12.8 MB` and `felhom-offbox · 2026-08-07 14:57 · 12.8 MB`. `felhom-offbox` is the tier's own marker tag, not an application. **The screen's whole job is to let the customer check that what is in the store is what they expect** (*"Nézd át, hogy tényleg azt találod-e itt, amire számítasz"*), and it shows them a stranger's name beside their own data and a total that is double the truth. **Cosmetic, not a data defect** — the restore page correctly offers only `calibre-web`. **Fix:** filter the marker tag out of the listing, or key the rows on the app tag. | **READY** — owner Viktor |
| **R-252** | **After a rebuild the restore refuses because the data drives are not registered, and nothing on the recovery path says so.** Measured on the fifth walk, 2026-08-07, at the last step of a successful recovery. The customer enters R, sees the listing, presses through to the restore — and gets **„nincs elérhető adatmeghajtó a visszaállításhoz"**. The drives physically survived (the raw mounts are the R-220 condition and were deliberately left in place); what did not survive is their **registration**, which lived in the destroyed guest's settings. **It is recoverable without a shell** — `GET /api/disks/candidates` offers both disks (`mountable: true`, `data_bearing: true`) and Tárhely → Meghajtók → „Meglévő meghajtó csatolása" re-registers them — **but nothing tells the customer that, and the recovery screen's own hand-off ("Tovább a visszaállításhoz") walks them straight into it.** **This is why the walk's journey half passes on the letter and not the spirit:** no guest command line was needed, and a customer who did not already know the product would stop here. **Fix:** detect the unregistered-drive state on the recovery/restore path and say what to do, or offer the re-attach inline. **Related:** R-253, which is the very next step and worse.  **✅ FIXED — controller v0.207.0, 2026-08-08.** The restore page now states the precondition BEFORE the customer presses anything — „Előbb csatold vissza az adatmeghajtót”, saying the backups and the drives are both still there, and linking to `/storage` — and the refusal string names the same route. The page asks through the backup manager's own `HasRestoreDestination()`, which reads the **same** `GetSchedulableStoragePaths()` the scratch resolver reads, so page and handler cannot drift apart. Conditional: a healthy box renders exactly as before, pinned by a test that fails if the notice becomes unconditional. **Note for whoever re-tests it live:** the state is **rebuild-only** — the customer API refuses to deregister the last usable drive, so it cannot be reached by removing drives from the dashboard, and the live check on VM 325 therefore exercised the healthy branch (Scenario E) while the notice itself is pinned at the template + predicate level. | **CLOSED 2026-08-08** |
| **R-253** | **The restore page promises it will reinstall the app, and the restore then refuses because the app is not installed — in the customer's own language, three lines apart.** Measured on the fifth walk, 2026-08-07. `/backups/restore` lists the app with the note **„Nincs telepítve — a visszaállítás előbb újratelepíti."** (*not installed — the restore will reinstall it first*). Pressing through to the full restore returns **„A teljes visszaállítás sikertelen: a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"** (*…not installed — restore the application first, then the data*). **Both sentences are on the recovery path, both are addressed to the same customer, and they say opposite things about the same fact.** The customer clears it by redeploying from the catalog (customer-facing, ~90 s) and re-running the restore — which then works — but they must work out for themselves that the first sentence was wrong. **This is the R-203/R-234/R-240 class one level up:** not a warning misread as a success, but a promise the next screen contradicts. **Fix:** either make the reconstitute path deploy the app when it is absent (which is what the list already claims), or change the list's note to say the app must be deployed first. **Do not leave the two sentences both shipped.**  **✅ FIXED — controller v0.207.0, 2026-08-08.** **The promise was the wrong half, and the reason is structural:** reconstitution writes to the app's own `GetStackHDDPath`, which exists only once the CUSTOMER has chosen a drive at deploy time — an automatic reinstall would mean the product making that choice for them, which is the one decision this whole recovery path exists to leave with them. So the copy now reads „Nincs telepítve — előbb telepítsd újra, utána hozhatod vissza az adatait” and links to `/stacks/<app>/deploy`; the reconstitute refusal was reworded to match, so the two cannot say opposite things again. **A pre-existing test pinned the OLD sentence** — its intent (state the consequence, do not hide it) is unchanged, and it now asserts the consequence AND the route. | **CLOSED 2026-08-08** |
| **R-254** | **The same render-then-hide pattern R-249 fixed is live in two more places, and one of them carries a real per-install secret.** Found by the §7.1 census that R-249's fix required — *a pattern found once is worth a census*, and it was. **(1) `app_info.html:185` — the serious one.** An app's auto-generated first-login password is rendered into `<span id="initcred-pw-val" hidden>{{.InitialCreds.Password}}</span>` beside a „Megjelenítés" button. `hidden` is the same class of control as R-249's `display:none`: it stops a browser drawing the value and leaves it in the response body, so a fetch of the app page returns it. **The value is a REAL per-install secret** — `ReadInitialCredentials` reads it live out of the deployed container (`internal/stacks/initialcreds.go`), it is not the catalog's published `default_creds`. **(2) `deploy.html:482` — the weaker one.** An auto-generated `type: secret` deploy field renders into a `<input type="password" … value="{{$val}}">` with a „Megjelenítés" toggle. On the PRE-deploy form this is close to unavoidable — the form must post the value, and it does, in a sibling hidden input — but on an **already-deployed** app's page (`$isDeployed`) the hidden input is correctly omitted while the readonly input still carries the value, and there the exposure is gratuitous. **Not fixed here, deliberately:** this session's scope was R-249/R-252/R-253, and each of these needs its own reveal endpoint and its own body-asserting test rather than a shared quick edit. **The fix shape already exists** — `POST /settings/retrieval-password/reveal` (v0.207.0) and `escrow_handlers.go`'s rule that a secret is revealed by an XHR and never templated server-side into HTML. **Severity MEDIUM for (1)** — a real credential in a page any logged-in customer opens, with no audit event, reaching caches, history and screen-shares; **LOW for (2)**. **Recommended next**, because R-249 proved the pattern is not theoretical: it was found by the value landing in a session transcript.  **✅ BOTH SITES CLOSED — controller v0.208.0, 2026-08-08, and they turned out to be two different problems.**
| **R-255** | **The check that would catch a fourth secret-in-the-body covers 4 of 27 pages, and the cheap gate that covers all 36 templates is blind to the shape that actually shipped.** Filed 2026-08-08 while closing R-254, **because a partial guard reported as complete is worse than no guard — it stops the next person looking.** **Two nets, both measured.** **(1) `scripts/secret_in_markup_gate.py`** reads all 36 templates and convicts any `{{ … }}` naming a secret unless allowlisted with a reason. It catches `{{.RetrievalPassword}}` and `{{.InitialCreds.Password}}`, **and it catches a launder through a local variable** because the assignment itself names the secret (`{{$v := .InitialCreds.Password}}` is convicted — verified). **It is blind to a secret arriving under a NEUTRAL PAGE-DATA KEY** — `data["Tagline"] = creds.Password` then `{{.AppInfo.Tagline}}` passes it cleanly, also verified. **That is exactly the shape of R-254 site two** (`value="{{$val}}"` inside an `{{if eq .Type "secret"}}` branch), so the gate **would not have caught one of the three instances it was written for.** **(2) The runtime body assertion** — render the page and grep the response for a sentinel — catches every shape, including that one (demonstrated on the same planted leak the gate missed). But it needs each page's data to be constructible in a test, and **only 4 of 27 page templates have that today**: `settings_security`, `app_info`, `deploy`, `backups_restore` — the four that were touched by R-249/R-252/R-253/R-254 and therefore got their own tests. **The other 23 pages have no runtime coverage at all.** **What closing this needs, so the cost is not re-estimated:** a per-page data fixture for the remaining 23 (most need a wired `Server` — `stackMgr`, `backupMgr`, agent seams), then one table-driven test that renders each with a sentinel substituted for every string in its data and asserts the sentinel is absent. **That is real scaffolding, which is why it was NOT built inside R-254's session** rather than half-built and declared done. | **READY** — owner Viktor |

**SITE ONE — the same defect, fixed the same way.** `app_info.html` no longer renders the value; the page carries the username, the note and a boolean, and the password comes from **`POST /apps/<slug>/initial-credentials/reveal`**, which **re-reads the running container** rather than serving a cached copy (caching it in the handler would have put it straight back into the body one layer in). `no-store`, CSRF-covered, and **logged as an act**. Both controls — Megjelenítés *and* Másolás — go through it, and a reveal that cannot read the value **says why** instead of returning an empty string that would render as a blank password.

**SITE TWO — NOT the defect the row described, and the difference is the finding.** The hidden input is **deliberate and was left alone**: it fires only on the PRE-DEPLOY form, and `README §318` documents why the value must round-trip — the customer is shown the generated secrets so they can note them down, and submitting them back is what makes the saved value the same one they saw (*"no silent re-generation on submit"*). **A form must carry what it submits.** What *was* indefensible is the neighbouring **readonly display input**: on an ALREADY-DEPLOYED app the hidden input is correctly omitted — nothing is being submitted — yet the secret was still rendered into a page the customer merely opens. Fixed by **`POST /stacks/<name>/auto-field/reveal`**, authorised by requiring the field to be a `type: secret` auto-generated field *of that stack's catalog metadata* — that check is what stops it becoming "read me any value out of any app". Both directions are pinned: the deployed page must not carry the value, **and the pre-deploy form must still submit it**.

**⚠ THE PREMISE THAT THIS BROKE A REPO RULE DOES NOT HOLD, and it is recorded rather than quietly dropped.** The rule cited was *"Password fields require explicit user input or generation (no silent auto-fill)"*. **No such line exists anywhere in the repo.** What exists is `CONTEXT.md:2070` — *"Password fields require explicit input | Prevents accidental empty-password deployments"* — which is about **emptiness**, not auto-fill, and which the hidden input does not contradict.

**§7.3 — HOW MUCH WAS ACTUALLY EXPOSED, measured on the fleet rather than assumed.** **Site one: nothing.** `crafty-controller` is the ONLY catalog app declaring `initial_credentials`, and it is **deployed nowhere** — the card renders only when `found.Deployed && found.Meta.InitialCreds != nil`, so that code path has never run in production. **Site two: nothing measurable either.** 26 catalog apps declare a generated `type: secret` field, but `demo-hp` has exactly **three** apps deployed — `calibre-web`, `opengist`, `privatebin` — and **none of the three declares one**. **THE HONEST LIMIT: this is a CURRENT-STATE measurement.** An app deployed and later removed would not appear in it, and **nothing anywhere recorded a read** — which is itself part of the defect being fixed. So: **no evidence of exposure, and no mechanism that could have produced evidence either way.** **Rotation is therefore not indicated by anything measured** — the decision is the operator's, and this note is the input to it.

**Gate:** `scripts/secret_in_markup_gate.py`, registered in `controller_gates.py`. **Its blind spot is measured and in its docstring** — see R-255. | **CLOSED 2026-08-08** |

**Recorded against existing rows by Phase 2:**

- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
  The box's own `/settings` renders „Minimális verzió (üzemeltető) **0.200.0**" (`GetFloor()`, whose
  only writer is the report-ACK handler; **both hold branches serve `Floor=""`**, pinned by
  `managed_floor_test.go:94`), and a **cold-started** controller logs
  `settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)` against the same line reading
  `floor still unknown after 1m30s` while the hold was in force. The hub's HELD lines ran every
  15 min to 22:12:06 and stopped, with a liveness control proving the hub kept logging. **The floor
  is served.**
- **⚠ A correction to how that positive was to be taken.** `SetFloor`'s line is `u.dbg(...)`, gated on
  `cfg.Logging.Level == "debug"` and written to the **logger** — it can **never** reach the logx debug
  ring, so it cannot appear in `/api/debug/logs` at any level. A controller restart alone would not
  have produced it. Confirmed with a level census on the ring first (1196 DEBUG / 2802 INFO / 2 WARN),
  so the absence was known to be structural rather than evidential.
- **R-218 — the live half is STILL NOT MEASURED, deliberately.** The fix is present and correct
  (`needsOffsiteCredential` now retires on the **target**, not the key), but the venue **has** a
  target, so the box correctly does not declare; declaring here would be the bug. The state that
  exercises it is shape (a), which the venue no longer holds. **Recorded as not measured rather than
  inferred from the unit test.**
- **R-217 — its fix HELD under exactly its fault** (F5): with the store blocked after a successful
  unlock, the page rendered M3 and **no listing block at all**; the three false-claim strings are
  absent, verified in UTF-8 with accented positive controls present.
- **R-215 — its fix is present** and the `GET /recovery` gate consults the same predicate as the POST
  sibling.
- **R-199's back-pointer in `architecture/00-capability-map.md` was already added** — the brief lists
  it as owed; it is present on the escrow-recovery row, explicitly labelled as the omitted
  back-pointer. **No action taken; the brief's assumption was stale.**

**Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting
files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's
unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing cost of).



| ID | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|
| **R-213** | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | Design the live-vs-backup comparison, then the put-back flow on top of it. Do NOT fold it into the recovery screen | Operator + CC |
| **R-212** | **The orphaned-ciphertext deletion HALTED: the stores on the storage box do not match this register's record.** Ruling 2 (2026-08-04) says *"delete the orphaned ciphertext, ~1.2 GB across the two demo boxes, in set-aside stores nothing prunes"*. **Measured 2026-08-05 over SFTP with each box's own credential, read-only:** demo-felhom (`u629488-sub1`) holds `felhom-repo` **1.2 G**, `felhom-repo.orphaned-20260717` **1.4 G**, `felhom-repo.orphaned-20260718` **3.0 M**; demo-hp (`u629488-sub3`) holds `felhom-repo` **582 K**, `felhom-repo.orphaned-20260804` **43 M**. So there are **THREE** set-aside stores totalling **~1.45 GB**, not two totalling ~1.2 GB — and **the figure that DOES match ~1.2 GB is demo-felhom's LIVE `felhom-repo`**, which is the configured `repo_path` and must not be deleted (it is unopenable by the box, which is R-193's point, but it is not a set-aside store) | **CLOSED 2026-08-05 — all three deleted after the operator confirmed the corrected list** | an exact path list, confirmed by the operator | Confirm which of the three `*.orphaned-*` paths to delete. **Nothing was deleted**; the listing above is the complete measured state | **DONE 2026-08-05.** The operator was shown the measured listing, confirmed *delete all three*, and all three `*.orphaned-*` stores were removed: demo-felhom `felhom-repo.orphaned-20260717` (1.4 G) + `felhom-repo.orphaned-20260718` (3.0 M), demo-hp `felhom-repo.orphaned-20260804` (43 M). **~1.45 GB freed.** **Both LIVE `felhom-repo` stores are untouched** (demo-felhom 1.2 G, demo-hp 582 K) — confirmed by a full listing before AND after on each account. **Proof nothing live was caught:** a REAL off-site run on demo-hp immediately afterwards returned `status: ok`, `orphaned: false`, `last_error: ""`, `last_run 2026-08-05T09:14:02Z`, 6 snapshots. **METHOD NOTE for the next session:** the Hetzner storage box has a RESTRICTED shell — no shell operators, no `test`, no GNU long flags. `rm -r <path>` as ONE simple command works; `test -d X && rm -rf -- X` returns *"Command not found"* and silently does nothing, which is how the first attempt failed CLOSED (verified by an unchanged listing). |
| **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — |
| **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — |
| **E-2d** | **Prove E-2 on a fresh VM** — a real `felhom-host-install.sh` 1.22.0 run, Case B naturally, a claimable customer, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **CLOSED — PARTIALLY PROVEN** (2026-07-29) | — | **C1, C2 proven** (`audits/E2D-fresh-vm-2026-07-29.md`); **C3, C4 proven live** (`audits/SESSION-C-2026-07-29.md`); **C5 FAILED → R-116** — the gate fires and an alarm reaches the hub, but it is the generic event, so the alarm and its recovery cannot be paired. **R-116 is the single named open leg**; per the Session-C runbook §9, decided in advance, a failed claim closes the item as partially proven rather than triggering a re-run. Both audits carry the full record — the `local-lvm` fence, the ISO/PAIRING derivation, the Phase 0 answers, the per-claim observables and the teardown evidence — and are the place to read it, not this cell. **The arc's actual definition of done is R-106 + R-109, R-108 and D5**, none of which this detour touched | CC |
| **R-94** | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | **CLOSED — SHIPPED** (hub v0.87.0, 2026-08-02) | — | **All three legs closed.** **(a) closed by DELETION, not derivation** — deriving is not achievable honestly: the Setup command fetches `felhom-host-install.sh` at RUN TIME from a website that git-syncs `main` every 30 s (R-110), so no build-time value in the hub can be true, and a number that is wrong carries a version number's authority while being a guess. The const, the `pageData.ScriptVersion` field, its assignment and the rendered label are gone; a NOTE stands where the const was so it is not helpfully re-added. **(b)** `hostinstall_gates.py` gate 1 INVERTED — it now asserts the hub carries **no** host-install version literal, in six code shapes across every `.go`/`.html` under `hub/`; and the gate is now invoked, by `scripts/repo_gates.py` and the pre-push hook (→ R-29). **(c)** the tautological `render_test.go:219` assertion is deleted, not replaced — there is no version to assert. It was demonstrated PASSING with the const at `9.9.9` while the script was 1.22.0. The label had been wrong for 19 days (since 2026-07-14) | — |
| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly. **OPERATOR RULING 2026-08-03 — option (b) chosen: the publish channel moves from `main`-tracking to a TAG.** Publishing becomes *moving the tag*, and rollback becomes *moving it back* — the property `main`-tracking cannot have at any price. Recorded here, **not built this session, by instruction**. **The ruling carries a condition that decides whether the fix works at all: it must cover BOTH channels.** (i) the nginx-served `/scripts/` git-sync (`manifests/webpage.yaml`, `--branch=main`, 30 s period) that every `felhom-bootstrap.sh` fetch and every operator day-0 command reads, AND (ii) **the nine files `felhom-host-install.sh` fetches from `raw/branch/main`** (`:2072`–`:2206`), of which the hub vouches a sha for exactly ONE. Fixing only (i) leaves a tagged installer pulling nine untagged files from `main` at run time — a staging story that is false in the place it matters most, since one of those nine (`felhom-backup-target-apply`) is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. **Exposure is still zero** (no boxes installing), which is exactly why it stays cheap. **CLOSED 2026-08-03 — installer v1.23.0, and the both-channels condition was HONOURED, but not in the shape the ruling assumed.** **The spec's mechanism for channel 2 rested on a factual error, found by reading the code:** the run-time fetches are **sixteen, not nine**, and they come from the **`felhom-agent`** repo, not from `felhom.eu` — so no tag on this repo could ever have covered them, and `§8.1`'s *"derive the tag from `SCRIPT_VERSION`"* was unimplementable for them. Operator ruled on the alternative: pin them to **the agent version being installed**, which the installer already resolves from the hub manifest and already sha-verifies. `fetch_raw` now fetches `raw/tag/v$ART_AGENT_VER/`, with **no fallback to a branch** — a vouched version whose tag is missing fails loudly, because a silent fallback is the appearance of control with none of it. That also fixed a latent skew → **R-183**. **Channel 1:** `manifests/webpage.yaml` split into TWO git-syncs — the website still tracks `main` at 30 s (a copy edit must never need a release), `/scripts/` tracks **`installer-v1.23.0`**. Both trees are seeded by init containers, so a fresh pod is not Ready until the tag is checked out and there is no 404 window; `maxUnavailable` rounds to 0 on one replica, so a failed scripts-init leaves the OLD pod serving — the failure direction is *no update*, never *no /scripts/*. **Channel 3 needed no change, and that is recorded rather than left as a silence:** `https://felhom.eu/scripts/felhom-host-install.sh` never carried a ref — the ref lives in the manifest — so `felhom-bootstrap.sh` and the hub's day-0 command follow the tag with **no edit and no hub version bump**, which is why §1's no-bump rule was never in tension. **PROVEN LIVE, both scenarios, by HTTP against the real URL.** *P-A (measured before the manifest was touched):* git-sync v4.4.0 follows a tag **and notices a MOVED one** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period. *Scenario A:* a real push to `main` without moving the tag — the website tree advanced to the new commit while the scripts tree stayed put, the served sha stayed **byte-identical (`2f859555…`)** and a marker comment deliberately planted in that commit was **absent** from the served URL. *Scenario B:* moving the tag published it in ~40 s (sha → `ea2b4aa9…`, marker present), and moving it back rolled it back to **exactly** the pre-publish sha with the marker gone; `https://felhom.eu/` returned 200 throughout. **Gate 6 in `hostinstall_gates.py`** pins all of it structurally with no network, so it stays in `--fast` and runs in CI: no `raw/branch/` ref in the installer, `fetch_raw` still pinned, the manifest still splitting tag-vs-main. **It deliberately does NOT assert that a tag exists for the current `SCRIPT_VERSION`** — that would go red on the very push that bumps the version, before publishing, and publishing being a separate act is the whole ruling; the same reasoning §8.4 applies to the agent gate | — |
| **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC |
| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, 2026-08-03) | — | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design. **THIRD INSTANCE, 2026-08-03 — and it was found by a runbook that had been told there was nothing left to do.** Agent **v0.120.0** — the agent half of the R-165 merge — was built, committed at `cd6e267`, and deployed to BOTH demo hosts, and was **never published**: `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200), and the hub manifest accordingly vouched **0.119.0**. The consequence is the sharpest yet, because installer step 5's idempotent skip requires `installed == vouched` EXACTLY: a documented-path reinstall would have **downgraded both boxes** from the merge-aware 0.120.0 to the pre-merge 0.119.0 — silently, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) never fires and the install would have *succeeded* while proving a stack nobody ships. **R-178's own row asserted `agent v0.120.0 is live on BOTH hosts` and `no code left to write`; both were true and both were beside the point** — the gap was publication, which no one checks. Fixed in-session on the operator's ruling: `scripts/publish-agent.sh 0.120.0` (sha `a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`, round-trip GET verified) then vouched, and both reinstalls then fetched and sha-verified it from Gitea. **This is the third instance of a row that has been WAITING-ON-OPERATOR since 2026-07-29; option (b) — a gate that refuses to deploy or vouch an unpublished version — would have caught all three.** **OPERATOR RULING 2026-08-03 — mechanism (b), build-side: a gate that REFUSES to deploy or vouch a version that is not published.** It is the strongest of the four options and the only one that fails closed; (c) and (d) were reminders, and R-29's whole finding is that reminders do not hold. Recorded here, **not built this session, by instruction**; it is now CC's to build. **The third instance is the argument for the ruling and belongs inside it:** agent **v0.120.0** (the agent half of the R-165 merge) was built, committed and deployed to BOTH demo hosts while `GET …/generic/felhom-agent/0.120.0/felhom-agent` returned **HTTP 404**, so the hub vouched 0.119.0. Because installer step 5's idempotent skip requires `installed == vouched` EXACTLY, a documented-path reinstall would have **silently downgraded both boxes** to the pre-merge agent — and would have *succeeded* while doing it, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's fatal `mp1` resize never fires. A gate at deploy/vouch time is the only one of the four standing between that and the operator. **CLOSED 2026-08-03 — both halves, no version bump (no Go code changed).** **(1) `scripts/release-agent.sh` is now THE release path**: build → **tag** → publish → **verify by an INDEPENDENT download**. It calls the existing `publish-agent.sh` rather than reimplementing it, refuses a dirty or unpushed tree, refuses to re-release an existing version (one version name must never mean two binaries), and **deliberately does not vouch** — vouching points machines at a version and stays the operator's act. `CLAUDE.md`'s raw `go build` line is replaced by it, so the documented way to release cannot complete without publishing. It also **tags**, because R-183 made the tag part of the released artifact. **(2) `scripts/check-published-versions.py`**, registered in `agent_gates.py` as **not `--fast`** — it needs network, and a push must not fail because Gitea blinked. **The CI workflow now runs the FULL set instead of `--fast`**, without which the gate would have been registered and never run: the built-but-never-wired failure this project has shipped four times. **THE INVARIANT IS NOT THE ONE THE TASK SPECIFIED, and the reason was measured (P-C), not argued.** §8.4 asked for *"the version the hub tells machines to install must be downloadable"* — the better invariant, and **CI cannot see it**: the hub's artifact manifest is **401** without a per-customer passphrase and Gitea's package **listing** api is **401** without a token, while the package **download** url and the **tags** api are anonymous. Adding an operator credential to CI is the operator's call, not a gate author's. The implemented invariant — **every `v<semver>` tag must have a downloadable package and a tag tree serving the agent's configs** — needs no credential and **catches all three recorded instances**, because the release script creates the tag and publishes in one act. **What it does NOT catch is stated rather than assumed away: the hub vouching a version that was never released at all → R-184.** **Red-proof F, MEASURED ON REAL CI and not inferred:** runs **69** and **70** are on the *same commit* `0db7766` — **success** before a tagged-but-unpublished `v9.9.9` existed, **failure** after pushing it. Same code, same workflow, one variable. Locally: the gate exits 1 naming the 404; **deregistered from the entry point** the same bad state reports `all agent gates OK` **rc=0**; restored → `CONVICTED: published` rc=1. `v9.9.9` deleted afterwards (`git ls-remote --tags` → only `v0.120.0`). **One deliberate CI failure e-mail reached the operator at ~12:5x CEST — that was this proof, not an incident** | — |
| **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — |
| **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — |
| **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC |
| **R-121** | **A BOX's installed agent can sit releases behind the vouched one and nothing notices — the R-120 gate does not cover it.** demo-hp ran agent **0.113.0** while the hub vouched **0.116.0**, through the whole R-116/R-117 arc, and no signal existed on any channel | **READY (S) — NEW 2026-07-30** | — | **Fourth instance of the drift family** (R-111 golden's agent 17 releases behind, R-115 built+deployed but never published, R-120 golden a controller behind — and now installed-vs-vouched on a live box). **Confirmed at source that R-120's gate cannot catch it:** `hub/internal/web/configs.go:1165-1169` compares `goldenVer` against `store.NewestReportedControllerVersion()` — it is a **golden-artifact vs fleet-CONTROLLER** check and says nothing about the agent installed on a box. **`MinAgent` does not cover it either:** it is used to HOLD the controller floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`) — protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor comparison was satisfied. **The cost, measured:** R-117's whole subject is the R-113 conjunction, which landed in **0.114.0** — so the designated drill host could not exercise the code under investigation at all, and the R-117 spike had to route every predicate result through an out-of-repo probe built from `main` instead of the installed agent (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §1, §2.3). Discovered because the R-117 task made bringing the box current an explicit prerequisite. **Fix shape (not implemented):** the hub already receives `AgentVersion` on every host report, and already has semver comparison in Go — the missing piece is a checker comparing reported agent vs the **vouched** agent and surfacing it, operator-tier. Note the honest tension: a box legitimately lags between publish and deploy, so this wants a staleness window rather than an instant alarm | CC |
| **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC |
| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |
| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |
| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |
| **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **CLOSED — both halves shipped** (2026-08-02) | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either **UPDATE 2026-08-02 — leg (a) CLOSED** (`felhom-controller` `c432f70`, its own reviewed diff as specified): `appexport/estimate.go`'s `-v` is a NAMED VOLUME mounted read-only into a throwaway container, no host path, structurally identical to the allowlisted `backup/backup.go` entry — allowlisted with its why; `realVolumeSize` untouched. **Leg (b) HALF-SHIPPED:** the 'decide where gates run' ruling is now made and half-implemented — **every repo has ONE entry point** (`felhom.eu/scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`, `felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py`), each mandated in its `CLAUDE.md` and each wired to `.githooks/pre-push` via `--fast`. **THE CENSUS, which is the finding:** thirteen gate scripts across four repos; **every gate a `CLAUDE.md` names was GREEN, and two of the four nobody names were RED** — `hostinstall_gates.py` (red since 2026-07-14) and `reuse_refs_check.py` (red on all four repos); a third, `docker_run_volume_path_gate.py`, was named only in `REUSE.md:284` and was also red. Correlation with 'named in a CLAUDE.md' was exact. **STAYS OPEN for the automatic half** — a hook is per-clone and `--no-verify` skips it; the unbypassable half is CI → **R-168** **CLOSED 2026-08-02, on the demonstrated ALARM and not on a green run.** Leg (b)'s automatic half is now live: a Gitea Actions runner re-runs every repo's entry point on every push, independent of who pushed and of what they typed (→ R-168). The class this row opened — *a gate that exists, asserts something true, is red, and is invoked by nothing* — is answered at both ends: the pre-push hook refuses locally, and CI catches a `--no-verify` bypass and **emails the operator**, proven with a real red run and a provider accepted-id. What remains is not this row's finding but a working-style choice — CI reports rather than blocks because there is no merge to gate (→ R-169) | CC |
| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC |
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.121.0**, hub **v0.91.0**) | — | **The rule that shipped:** *let A be the newest archive on a tier that has settled ≥24 h; the tier is DUE when A exists and A has not already been proven.* Daily tier → proved daily on yesterday's archive; weekly tier → weekly on its own; newborn → UNKNOWN. The daemon-start ticker survives only as the **evaluation interval**. **THE TRAP, recorded because it is the version a reasonable person writes:** the row's own wording implemented literally — *"due when the newest archive is ≥24 h old"* — is NEVER true on a **daily** tier, because a new archive resets the newest-archive age to zero long before it reaches the lag; it would have silently switched restore-testing OFF for the tier that matters most. Red-proved at **0 runs over 5 simulated days**. **The state now records WHICH archive was proven**, not when a tier last passed — a time cannot answer *have we proven this archive*. A pre-R-86 state file keeps its time (rotation ordering survives) and yields no proven archive, so each tier is due exactly once after the upgrade: the safe direction. **Two knobs replace one and the old one is not silently repurposed:** `restore_test_eval_interval_seconds` (6 h) and `restore_test_settle_seconds` (24 h); the deprecated `restore_test_cadence_seconds` keeps its DISABLE meaning verbatim, now seeds the settle lag, and the daemon WARNs once at start-up naming both. **6 h is bounded from both ends, not picked:** MEASURED cost of one evaluation on demo-felhom — local dir storage **18 ms**, PBS tier over the WAN to ep0 **392 ms**, both **430 ms** — so cost is irrelevant; the CEILING is that a FAILING tier stays due, making the evaluation interval its retry interval for a multi-GB restore. **Part 2 shipped WITH it and was not optional** — see the hub half in this row's sibling text and `07-backup-architecture.md` §3: `restoreProvenStaleAfter` was a flat 7 days derived from the very cadence this removed, and a healthy weekly tier's proof age reaches **exactly** 168 h against a 168 h window — it sat ON the line, so any ordinary delay tipped it into a nightly alarm about a working system. The window is now per tier from that tier's observed archive interval, ×4 generations, floored at the old 7 days and capped at 12 days (strictly inside the two-week offsite retention), falling back to the tier's DECLARED rhythm (26 h host / 8 d offsite — the backup-freshness checker's own thresholds) when history is too short to observe one. **A hollow test caught by its own red-proof:** the first Scenario-G fixture had no jitter and PASSED under the flat-window mutation, because a perfectly regular weekly tier sits exactly ON the line rather than over it. The jitter is what makes it a test. **Also fixed in passing:** the candidate picker now skips archives failing `archivePlausiblyComplete` (under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof and leave the tier due at EVERY evaluation), and the due-check runs BEFORE the heavy-operation gate is taken (a frequent poll must not be able to make a starting backup record a failure — F-A1). **Live proof:** see `felhom-agent/REPORT.md` | CC |
| **R-87** | The restic tier is never restore-tested | **READY — RE-RANKED UP 2026-08-03 (R-86 closed)** | — | Design a controller-side test (no scratch-guest analogue transfers). **Most of what this row needed now exists.** R-86 built the piece that was missing: a tier is proved **per archive generation**, on its own rhythm, with the proof recorded as *which archive* — which is exactly the shape a weekly-ish restic tier needs, and the reason this row could not simply reuse the whole-guest scheduler before. What remains is genuinely restic-specific and is NOT a scheduling problem: there is no scratch-guest analogue, so the test has to be a controller-side restore of a bounded sample into a throwaway path, with its own definition of "proved". **Two things to carry over rather than re-derive:** the proof must record the SNAPSHOT it proved (not a timestamp), and the hub's staleness window must learn this tier's rhythm the way `restoreProvenWindow` now does — a restic tier on a weekly cadence lands on the same false-alarm line the flat 7 days did. **And R-95 still applies:** that credential can delete, so a restic restore-test must never be able to write to the repo | CC |
| **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.123.0**, installer **1.24.0**) | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began **BOTH HALVES FIXED, AND THE ROOT CAUSE WAS NOT WHERE IT LOOKED.** **(1) The silence.** `Client.Permissions` asks `GET /access/permissions?path=/storage/<target>` **as the agent's own token**, and one `capability.Status` per configured tier reports it — composed around the sudo prober the way the pool-read check already is, with the `Status` wire shape untouched so the hub's existing critical-degraded alert applies with **no hub change**. **MEASURED BEFORE IT WAS DESIGNED AGAINST, and the obvious reading is wrong:** an ungranted path answers neither empty nor 403 but with the privileges INHERITED from the box-wide `/` grant — `felhom-pbs` → `{Datastore.Allocate:1, Datastore.AllocateSpace:1}` vs `felhom-backup` → `{Sys.Audit:1, SDN.Use:1, Datastore.Audit:1}`. A probe asking *"is the path present?"* or *"does it have Datastore.Audit?"* would report the blinded storage HEALTHY; the probe tests `Datastore.AllocateSpace` and a red-proof pins that. The probed set comes from the box's own `BackupTiers()`, never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical (the hub alerts only on critical), **except** the `local` fallback target, which is reported but does not page: turning host-install's own documented DEGRADED configuration into an alert is how a signal becomes something an operator archives unread. It never looks at content, so it cannot alarm on a newborn tier by construction; it never reports ok when it could not ask. **(2) The installer.** The row and the task both assumed `PVE_STORAGES` was the culprit. It is not: `configure_backup_target`'s **Case A** arm creates the storage and grants in the same breath, so a box that builds its own target was always fine. The **Scenario-F** arm — *"the target already exists, leave it exactly as it is"* — returned **without granting**, which is why a box whose `felhom-backup` pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token could not read. That arm now ensures the ACL through the same guarded wrapper; the storage DEFINITION is still untouched, so Scenario F is unviolated, and `pveum acl modify` is idempotent. `$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES` — that list is granted in step 4/5, before the target is resolved in step 6, and `--acl-storages` entries are preflight-checked for existence. A **gate** now asserts every arm that resolves the target also grants on it (red-proved: *resolves in 2 place(s) but grants in only 1*). **LIVE, in order:** the three-way observation reproduced unchanged (root 3 archives / token `{"data":[]}` / `local` lists through the same token) → agent v0.123.0 deployed → **`capability DEGRADED … pve:store-grant:felhom-backup … lacks Datastore.AllocateSpace … INVISIBLE to the agent and it is never restore-tested`, critical=true**, and the hub raised `agent_capability_degraded` **and emailed the operator** → grant applied (user AND token) → the token lists **3 archives** → `ok=70 total=70 degraded=0` and the hub logged `degraded → ok (agent_capability_recovered)` → **the host tier is a due-check candidate for the first time on that box**, correctly picking the 08-02 archive because the 08-03 one has not settled 24 h. **demo-hp CARRIES THE SAME DRIFT AND WAS FIXED TOO** — `local_backup_target=felhom-backup`, no ACL row, now granted and listing 4 archives. §8.6 assumed one box was affected; the same one-line additive idempotent fix applies to the other, and leaving a known-blind backup tier on a Tier-0 box after finding it would be this row happening twice. The tester's box was NOT touched (Tier 2) — see the note in this row's outcome. **CORRECTION + WIDER IMPACT, measured 2026-08-03 during the R-185 close-out run.** This row (and the runbook annotation written with it) said the drift was invisible because writes went through a root path. **That is wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST — `POST …/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)` — and the hub raised `whole_guest_backup_failed` at the first (*"retrying with backoff"*), edge-triggering suppressing the rest. So the impact was not only an unreadable tier: **the agent's own whole-guest backups to it were failing**, and the operator was told once. The failures stop at the grant. **AND THE GRANT WAS LOST, NOT MERELY NEVER ISSUED — on this box.** A vzdump by `felhom-agent@pve!agent` with `--storage felhom-backup` **succeeded at 04:44:50 CEST the same day**, and the first 403 is 09:24:56. Ruled out: a host reinstall (uptime 12 days), any logged `pveum`/ACL activity (none in syslog 04:00–10:00, none in the cluster log). Correlated but NOT established as causal: `host_leaf_changed` 09:15 and `controller_started` 09:19, i.e. guest 9201 was reprovisioned nine minutes before the first failure. **How the grant disappeared is unexplained → R-190.** | CC |
| **R-186** | **A released agent binary's sha256 cannot be reproduced from its tag.** `release-agent.sh` builds at step 3 and tags at step 4, so Go's VCS stamp records a PSEUDO-version (`v0.120.1-0.20260803130452-4d825910…`) in the published bytes, while any rebuild after the tag exists stamps `v0.121.0` — a different binary. Measured 2026-08-03 on v0.121.0: published `b2128f3c…` (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source, identical toolchain, 4 096 bytes apart | **CLOSED — SHIPPED + MEASURED 2026-08-03** (agent **v0.122.0**) | — | **Why it matters:** the sha the operator vouches is the one thing tying a machine to a binary, and today nobody can independently rebuild it to check. **The build order is deliberate** (the script's own comment: a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it), so the fix is not to swap the steps blind. Candidates: `-buildvcs=false` or `-trimpath` for a version-stable stamp, or tag-then-build with the tag deleted on a failed publish. **Mitigation used this session:** the DEPLOYED binary is the PUBLISHED artifact, downloaded from Gitea — not a local rebuild — so the running bytes are the vouchable ones **FIXED by removing the cause, not by sequencing around it.** `-buildvcs=false` drops the VCS stamp (nothing in the repo reads it — no `ReadBuildInfo` caller) and `-trimpath` removes absolute build paths, so the same source produces the same bytes whether or not the tag exists yet and regardless of checkout directory. **Measured before the change**, same source, same toolchain, one commit: default flags with no tag `18f4a495…` 14 085 464 B (mod `v0.121.2-0.2026…`) vs default flags tagged `4a38f394…` 14 085 440 B (mod `v0.121.99`) — different bytes; with the new flags, `7ffcdf1d…` 14 064 574 B **identical both ways**. **Measured after the change on the real release:** published `d5f294e5…` 14 076 649 B = rebuild-at-tag #1 = rebuild-at-tag #2, **all three identical**. **A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and therefore produced a binary **74 KB smaller** (13 990 236 B vs 14 064 574 B) than the release path built for the same version — one version name, two binaries, decided by which entry point ran. Both paths now build identically, with a comment in each naming the other. **The operator can now check a published sha without being there:** the exact command is in `felhom-agent/CLAUDE.md`. | CC |
| **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC |
| **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v<ver>` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. **THE REORDER MOVED THE RACE RATHER THAN REMOVING IT — observed 2026-08-04, and it is narrower but real.** Releasing v0.124.0 produced CI run **28 (id 115) FAILED** and run **29 (id 116) SUCCESS** on the same sha `257c4d85`. Timing: the tag was created locally at **07:38:36Z**; run 28 started **07:38:31Z** (the COMMIT push, five seconds earlier) and run 29 at **07:38:41Z** (the tag push). The old race was *tag visible before its package*; the new one is *package published before its tag is pushed* — the exact window the converse probe was added to detect, and it detected it. **NOT CONFIRMED from the run log** (Gitea's run-log endpoint 404s for this runner), so this is the timing-consistent explanation rather than a proven one; the gate passes locally with all 7 released versions installable and none untagged. **What it means in practice:** a release can still produce one red run and one operator e-mail, now from the opposite side. The window is seconds rather than the previous coin-flip, and it is self-clearing. **If it is worth closing:** have the workflow skip runs whose ref is a branch push occurring inside a release, or have the converse probe tolerate a package younger than ~2 minutes — but note that BOTH are the 'make the gate blind for a window' shape this row explicitly refused, so the honest options are to accept the narrow window or to make releases quiet to CI another way. | CC |
| **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC |
| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **MITIGATION SHIPPED 2026-08-04** (agent **v0.124.0 → v0.124.1**) — **MECHANISM STILL OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/<vmid>` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/<id>` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had **THE ROW NOW REFLECTS THE MITIGATION, NOT THE CAUSE — stated plainly because the two are different things.** The box is resilient; the loss is still unexplained. **Mitigation:** when the store-grant probe finds the grant absent, the agent runs the EXISTING root wrapper (`felhom-backup-target-apply grant <id>`) and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including its restraint. **No new privileged surface:** the sudoers vector `grant *` already covers any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants BOTH user and token. The verb existed, was permitted, and had only ever been called at storage CREATION — the *built but never wired* shape in a verb rather than a seam, this project's seventh instance. Bounded at one attempt per tier per hour (a storage can be unreadable for reasons an ACL cannot fix; re-granting every cycle is a repair loop wearing a fix's clothes). **THE RECORD IS THE HALF THIS ROW IS ABOUT, and v0.124.0 got it wrong in production while every unit test passed.** It reported degraded for *one cycle* — meaning the probe call that repaired. But `probeAll` is invoked INDEPENDENTLY by the self-check log and by the collector building a host-report: on the box the repairing call was the log's (`09:39:34`, journal shows the repair and `degraded=1`) and the report three seconds later found the grant present and sent **`ok`**. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence this row exists for, re-created inside its own mitigation. **v0.124.1** replaces it with a latch on TIME (20 min > the 900 s report interval), so at least one report must carry it. **PROVEN LIVE, twice, on demo-felhom** (grant deleted by hand, both rows): agent logs `store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read`; the ACL rows return; and on v0.124.1 the **host-report at 08:00:30Z carried `status=degraded`** with the explanation and the hub raised `agent_capability_degraded` and **e-mailed the operator** at 08:00:40. Nothing new was built to carry it — the hub's existing ok→degraded→ok edge is the channel, and the text rides `Feature` because that is the field the hub interpolates into the e-mail (`Reason` does not travel). **PART 3 — the single bounded pass at the mechanism, with the negatives named.** The lead §8.6 nominated is real as a CLASS and is documented in our own installer: *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"*. **It does NOT fit this box.** A rotation purges ALL of the token's ACLs and mints a NEW secret; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it retained its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at the time). Previously ruled out and unchanged: a host reinstall, any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry. **Ruled out on THIS box; NOT ruled out fleet-wide** — any installer run still purges and re-grants only the hardcoded `PVE_STORAGES` set, though installer 1.24.0's reuse-arm fix now re-grants the backup target on that path. **A NEW OBSERVATION FROM THE LIVE RUNS, relevant to the timeline:** PVE **caches permissions** — after deleting both ACL rows the probe still read the privilege as present for ~40 s in one run and ~16 min in another. Detection is only as prompt as that cache, and a cache expiry could equally explain why a box kept working for hours after a grant was removed. → **R-194**. **The alert pair CLOSED on its own at 10:30:40** (`degraded → ok`, `agent_capability_recovered`) once the 20-minute latch expired — one lost grant, one e-mail, one recovery, nothing further. | CC |
| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC |
| **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
| **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — |
| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **CLOSED 2026-08-05 — hub v0.95.0** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 **FIVE instances, not three** — the spec expected three and named two; a census found five: `offsite/offsite.go` (the `MarkEscrowStale` justification), `api/handler.go` (the F3 re-enroll comment) and **three in `store/store.go`** (the `stale_at` ALTER comment, the `MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). All five now state what the code does, name the correction, and cite the recon; the staleness mark is documented as **precautionary** (the box's re-apply may mint a fresh repository password — the guest-rebuild shape) rather than evidential, with R-197's measured signal named as the evidential one. **THE BEHAVIOUR IS UNCHANGED AND THIS ROW STAYS OPEN:** on the ordinary re-issue shape — a box that still holds its `repo_password` file — the password does not change and the hub still marks a healthy escrow stale and asks the customer for an unnecessary ceremony. That is a behaviour change and must not ride a comment-correction release; it is also now **more** consequential than when filed, because under R-198 an unnecessary ceremony is no longer harmless bookkeeping — it supersedes a blob. Fix shape unchanged: mark stale on the evidence that it IS stale (R-197's hash comparison), not on a re-issue. **MEASURED LIVE 2026-08-04 night, in the flow where it does real damage.** After the Re-issue a rebuilt box needs to configure its off-site tier, `stale_at` was set at `2026-08-04 20:15:49` **while `restic_pw_sha256` was unchanged** — the escrow perfectly covered the box's current (recovered) key. The consequence is not cosmetic: a stale escrow makes the hub withhold the hash from the ACK, `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` refuses every off-site run. **The only documented way out is a fresh ceremony, which supersedes the identity blob and destroys the key being recovered.** This row's fix — mark the escrow stale on the evidence that it IS stale (a changed `restic_pw_sha256`, R-197's comparison), not on a Re-issue — is now on the critical path for R-201/R-204, not a tidy-up. Full chain: `audits/DRILL-r201-night-run-2026-08-04.md` §3(b) | CC **CLOSED (hub v0.95.0, 2026-08-05).** The pessimistic `MarkEscrowStale` and its `escrow_stale` event are GONE from `ReissueCredentials`; `offsite_reissued` is untouched. The removal was not a softening — nothing now guesses at re-issue time. **The case the mark guessed at is measured in two places, and one of them the mark was BLINDING:** a stale flag empties `restic_pw_sha256` from the report ACK, which is the exact value the controller's Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo password every cycle — so on the guest-rebuild shape the box could only ever report the hash-LESS reason, which is false. R-197's `offsite_repo_key_changed` covers the supersession edge. Red-proofs run: restoring the mark fails `TestReissue_DoesNotMarkAHealthyEscrowStale`; disabling the detector fails `TestEscrowPut_ChangedRepoKey_RaisesSignal`. **`MarkEscrowStale` is deliberately KEPT with no caller** — the flag is still correct and a future EVIDENTIAL writer is the right way to set it; `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps it from decaying meanwhile. **Disagreement recorded (R-96 rule):** the task asked that a real key change ALSO mark the escrow stale; it must not — the hub learns of a real change at the instant a supersession seals the NEW password, when the escrow is freshest, so the correct consequence there is the operator alarm, not a demand to redo the ceremony that just ran.|
| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) **SHIPPED.** `SaveHostEscrow` returns the hash it replaced; `handleHostEscrowPut` raises **`offsite_repo_key_changed`** when both hashes are known and differ. **Edge-triggered** (once per supersession, never per report — the dispatcher owns cooldown), **operator-only** (registered in `notify.operatorOnlyEvents` in the same commit that mints the type, because a missing `customerMessages` entry is NOT a block — the v0.78.0 defect), and **no hash value travels** in the message or the details. The in-between shapes (a first-ever hash, a hash-less supersession) are LOGGED rather than dropped, so *"we chose not to alarm"* and *"the check did not run"* never look identical. **Severity = warning, chosen for the world v0.93.0 creates:** before R-198 a changed key meant the previous history was unopenable by anyone ever, which would have argued for `error`; from v0.93.0 the superseding ceremony retains the old identity blob, so the fact is *"this customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss. `warning` also routes (the dispatcher treats `info` as an intentional non-notify). Driven through the real endpoint in test, not by calling the emitter. **Red-proof observed:** removing the comparison from the escrow PUT → the changed-key scenario fails with *"the repository key demonstrably changed and NO signal was raised"* while the unchanged-key scenario still passes. | CC |
| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 **SHIPPED.** `host_escrow_superseded` gains `identity_blob` (CREATE + additive `ALTER TABLE`) and `demoteCurrentEscrowTx` carries it, so **both** callers — re-escrow and host-delete demotion — are fixed by one change to the routine its own comment calls *"THE ONE escrow row-copy routine"*. `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob is reachable from Go at all. The table comment now records that the ruling stated above it was not met and what that cost. **Tests assert the CONSEQUENCE, which is why the existing one stayed green:** `TestSaveHostEscrow_RetainsSuperseded` asserted that a retained row exists carrying the old K-blob and passed throughout; `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still yield a repository password, and pins the load-bearing ordering (the identity blob is written by `SaveHostDRBundle` AFTER `SaveHostEscrow`, so the demote sees the PREVIOUS generation — if that inverts, the retained bytes would be the new blob filed under the old hash, recoverable-looking and wrong). `TestDeleteHost_DemotesIdentityBlob` proves the shared routine through its other caller. **Red-proofs, both observed failing:** dropping `identity_blob` from the copy (production behaviour ≤ v0.92.0) fails BOTH scenarios; fixing only the re-escrow caller fails the delete scenario while the re-escrow one passes — the §8.2 mistake, demonstrated rather than asserted. **Nothing was backfillable and it was CHECKED, not deduced:** rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; the live database holds exactly 2 retained rows, both from 2026-08-04, both `identity_blob` NULL. **Who the fix protects, measured live:** 2 of 2 hosts with a current escrow carry an identity blob (`demo-felhom-8363b5`, `demo-hp-bb76ea`) — their NEXT ceremony now retains a recoverable off-site key instead of destroying one. **STILL UNIT-PROVEN ONLY after the 2026-08-04 night drill.** Part 2 (wipe again, do NOT recover, let a ceremony seal a DIFFERENT password, then inspect the superseded row's `identity_blob`) was gated on the first drill passing and **did not run** — the verdict was not reached, and a second wipe would have destroyed the state that makes the first one finishable in five minutes. **Nothing has yet superseded a key in production**, so the retention's live behaviour is unobserved. That check remains the cheapest way to prove or disprove it. | CC |
| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts/<id>/escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC |
| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC |
| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **BOTH HALVES PASSED — DATA 2026-08-04, JOURNEY 2026-08-07** — the customer file came back byte-identical (four times now), and on the fifth walk the customer's own journey completed with **zero guest command lines**. **It does NOT claim:** that the journey is smooth (R-252/R-253 are two unsignposted stops on it), nor that the fingerprint discriminator's POSITIVE half is proven (the mint guard fired, so there was no divergent key for shape (c) to catch — only its negative half was measured) | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **UNBLOCKED 2026-08-04 by R-203 (controller v0.197.0).** The sentinel now lands in the off-site snapshot and is listed there by name and size — the thing whose absence halted the drill. **Everything the resumed run needs is already in place on demo-hp:** agent v0.125.0, controller v0.197.0, the recovery code held by the operator (`R_DEMO-HP`), a working off-site repository (3 snapshots), `calibre-web` deployed with a mandatory userdata path, and the sentinel at sha256 `643166269103a25c…` — **verified byte-identical after the R-203 migration moved it to the corrected directory**. **What remains is exactly steps 4–11 of the drill**: re-verify the snapshot listing, take the deliberate rollback archive (P4), the §7 STOP, then wipe, reinstall, recover, install, restore and compare. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **NIGHT RUN 2026-08-04 (`audits/DRILL-r201-night-run-2026-08-04.md`). THE HEADLINE: after a real rebuild — controller data volume destroyed, sentinel deleted from disk — the customer's recovery code produced `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, BYTE-IDENTICAL to the pre-wipe on-disk key and to the hub's independent record. The off-site backup key is recoverable after a machine is rebuilt, and that had never been shown.** **Step 7's assertion PASSED:** `identity_blob` 572 B and `restic_pw_sha256` unchanged across the wipe (`updated_at` still 11:11:37) — nothing re-escrowed itself. **Step 9a passed:** the key installed cleanly on a bare box (the "installed" branch's first real run); **9b:** the apply kept it. **THE VERDICT WAS NOT REACHED** — step 10 never ran, so there is no post-restore sha256 and no snapshot count. **That is not a FAIL** (nothing came back wrong and no fresh history was started); it is a wall, and the wall is **R-204**. **The wipe was faithful to the incident, deliberately:** the 2026-08-03 rebuild R-193 is filed against was NOT a guest reprovision — the journal shows guest 9201 running continuously with no `pct destroy`/`pct restore`/`--selftest=provision` — so a controller-data-volume wipe reproduces it, and an unrehearsed provisioning chain improvised unattended is what §8.10 exists to prevent. **A precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the sentinel because the afternoon's experiment produced a later same-day snapshot and `forget --keep-daily 7 --group-by host,tags` pruned the good one — **a good snapshot is not durable against a later bad run on the same day.** **TO FINISH (~5 min, operator present):** re-claim the box, confirm the escrow (**NOT a new ceremony** — it would supersede the identity blob and destroy the key under test), run a backup, restore the sentinel and compare to `643166269103a25c…`. Rollback available: the verified archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` **THE DRILL PASSED (`audits/DRILL-r201-night-run-2026-08-04.md`).** demo-hp's controller data volume was destroyed and the sentinel deleted from disk; the customer's recovery code recovered the key (`8a9e33aa4da6…`, byte-identical to the pre-wipe on-disk key AND to the hub's independent record); it installed on the bare box; the **existing repository OPENED** (`repo_state: null`, **3 snapshots — NOT 1**, 42 026 B = the pre-wipe size exactly); and the customer restore flow returned **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` — byte-identical to the pre-wipe sentinel.** **The Felhom backup story is proved end to end for the first time.** **Step 7's assertion held:** `identity_blob` 572 B and `restic_pw_sha256` unchanged across the wipe — nothing re-escrowed itself, and no ceremony was run at any point (superseded rows still 2). **The wipe was faithful to the incident:** the 2026-08-03 rebuild R-193 is filed against was a controller-DATA-VOLUME loss, not a guest reprovision — the journal shows guest 9201 up throughout with no `pct destroy`/`pct restore`/`--selftest=provision`. **IT TOOK FOUR UNDOCUMENTED STEPS (→ R-204):** an operator Re-issue; a re-claim (whose escape hatch needs a controller restart to work at all); a manual escrow confirm; and `mode=full` on the restore, because the default `mode=unit` returns the app definition and NOT the customer's files. **A customer hitting this alone today would not get their data back.** Box left healthy, claimed, re-armed, sentinel restored to its live location. **⚠ WALKED TO COMPLETION 2026-08-06/07 — DATA PASS, JOURNEY FAIL, and the row stays open.** The full walk ran overnight on a new venue: built from the published ISO, fixtured, soaked through a real scheduled cycle, destroyed, rebuilt, and finished in the morning with the operator's emailed code. **DATA: PASS** — all three sentinels byte-identical out of snapshot `f5c53b03`, accented filename bytes included. **JOURNEY: FAIL** — the customer had **no route at all** to enter the code they hold: the recovery screen had retired itself, and the remote page offered to CREATE a new code instead. Root cause **R-241** (the credential self-heal writes a fresh repository key and moves the box out of the recovery-offer's pristine case, while orphan detection is unreachable behind `escrow_state: pending`). Recovery needed three guest command lines. **Also established:** the credential chain runs end to end unaided on an unclaimed box (first live sighting of its success line), and **R-239** — a fresh install lands on controller 0.203.0 while 0.205.0 is released, so R-234 and R-237 are undelivered. Full record: `tests/finalwalk-r201-2026-08-07/journal.md`  **✅ CLOSED 2026-08-07 — BOTH HALVES PASS, on the fifth walk** (`documentation/tests/walk5-r201-2026-08-07/journal.md`). A fresh appliance was installed from the published ISO on `demo-hp` (VM 325), given three sentinels, escrowed, backed up off-site, then **destroyed on purpose** — guest and data volumes — and rebuilt through the documented day-0 path. **THE DATA: PASS** — all three sentinels byte-identical out of snapshot `5b0f20f7`, including the accented filename's *bytes*, read back with `os.listdir` on a bytes path so no decode round trip could launder a `U+FFFD`. **THE JOURNEY: PASS — zero guest command lines were needed to progress**, against three on the previous walk; the reset-code hatch was used once, in Phase A only, where §3 permits it. **RTO 71.7 s** from login to an open store (12.44 s of it the unseal itself; ~22 s a harness retry of mine). **The recovery screen appeared without being sought** (`/` → `/launcher` → `/recovery`) and answered all three questions, with a sealed-at timestamp matching `host_escrow.created_at` exactly. **What made the difference is R-241's mint guard, exercised live for the first time:** at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport and **refused to mint a repository password** over the sealed package — where the previous walk minted one and lost the journey silently. **Two obstacles remain and are filed rather than absorbed into this close — R-252 and R-253** — neither needing a shell, both cleared from the dashboard, and neither signposted; the journey succeeds and is not yet smooth. **This close does NOT claim** that shape (c) fired positively (it did not — with no local key the offer comes from shape (a); shape (c) was measured in its negative half in Phase A), nor that a customer would clear R-252/R-253 unaided. | **CLOSED 2026-08-07** |
| **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator |
| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `<hdd>/…` and `<hdd>/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/media/books` → `/mnt/sys_drive/felhom-data/userdata/media/books`, the capture log went `0 mandatory path(s)` → **`1 mandatory path(s)`**, and **the sentinel is in the snapshot's own file listing** — `-rw-r--r-- 1000 1000 181 … /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt` — not a green status. **Red-proofs:** the bare-path call makes the two paths differ; inverting the drive-kind comparison breaks every enrolled row; leaving the export site bare emits the short path; and the verdict fails under both an unreachable gap-recording and an unconditional `ok`. **One red-proof PASSED and the test was wrong, not the code** — the first Scenario-C test only reached `offboxCaptureSet` while the mutation lives in `runOffboxInternal`; a run-level test replaced it | CC |
| **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ALL FOUR ITEMS CLOSED 2026-08-05** (items 1–3 controller v0.198.0 + hub v0.95.0; item 4 controller v0.199.0 + hub v0.96.0) | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the superseded code returned „Hibás vagy lejárt kód" and the current one was accepted first time.** **Item 2 (a re-issue marks a healthy escrow stale) — CLOSED, → R-196.** Test-proven; deliberately NOT fired live on demo-hp. **Item 3 (the restore's default silently returns the wrong thing) — CLOSED, proven live.** A `mode=unit` restore now names what came back, what did not and the step that gets it; the wizard's intent card states its scope BEFORE the choice; the full-restore size gate is untouched and pinned as unchanged. **The default stays `unit`** — all three wizard forms set `mode` explicitly, so changing it would alter nothing the customer sees while silently changing a mode-less POST. **ITEM 4 REMAINS AND IS THE WHOLE OF WHAT IS LEFT HERE: a rebuilt box cannot obtain an off-site credential unaided**, because the one-time password was spent by its predecessor, so an operator Re-issue is still required. **Its dependency is the one-shot credential design decision — it needs an operator ruling and belongs to → R-193.** Not begun in this session, deliberately. **ITEM 4 CLOSED 2026-08-05 — controller v0.199.0 + hub v0.96.0.** The box now DECLARES that it needs a credential (`offsite.state=needs_credential`) instead of reporting an absence the hub cannot interpret; the hub's new `internal/offsiteheal` answers it. **Operator ruling, recorded because a ruling that lives only in a conversation binds nobody (R-96): the trigger is a state the BOX DECLARES, not an inference.** An absent off-site object has FOUR meanings — never configured, mid-restart, a transient config read failure, rebuilt-and-stranded — and the hub cannot tell them apart; the box can, from two local facts (a fresh data area AND a hub-held recovery package). **Both halves are required:** freshness alone would make every un-configured box in the fleet ask for a credential, which is what `TestOffsiteDeclare_NeverHadOffsiteSaysNothing` exists to catch. The reconciler mirrors `pbsdrheal`: declared states only, a two-DISTINCT-REPORT debounce (derived from the ~15-min report cadence), **restage before mint**, an event per remediation, and a healthy box is a pure no-op. **PROVEN LIVE:** demo-felhom 9201 arranged (reversibly) into the stranded shape produced report id=16743 carrying `{enabled:false, state:needs_credential, quota_gb:0, repo_size_bytes:0}`, and the single declaration was **absorbed by the debounce** — no self-heal event fired — with the box restored the same minute. **What is deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. **Credential automatic, key customer-present.** **Second ruling recorded: the dashboard-password exposure on the recovery preview is METADATA (backup dates, app names), not content, and is ACCEPTED.**|
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **CLOSED 2026-08-05 — the guard's scoping half closed BY REPLACEMENT** (hub v0.96.0) | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. **RECON 2026-08-04 adds two inputs and changes no verdict** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(1) The guard's own conclusion — that a fix here protects the plumbing and not the data — is now stronger, not weaker:** even a perfect restage leaves the rebuilt box minting a fresh repo password, and per **R-198** the old one is destroyed by the re-ceremony the box is pushed into. Whatever shape the message takes must say that in the same breath, or it will read as an all-clear. **(2) The recency-bounded discriminator this row asks for has a ready anchor the hub already receives:** the report ACK's `escrow{identity_blob_present, restic_pw_sha256}` moves when a box re-keys, so "evidence since the newest re-key" is computable from data already stored — the same observation R-197 makes, aimed at this guard's time window. **BOTH HONESTY HALVES SHIPPED 2026-08-04 (hub v0.93.0); THE GUARD'S LOGIC IS DELIBERATELY UNTOUCHED.** **(a) The message now describes what was measured.** The one stuck state is reported as the two situations it actually covers — **burned** (`OffsiteReportsSinceConsume == 0`) and **regressed** (> 0, the demo-hp shape) — each stating its own measurement and carrying its own recommendation; the regressed text explicitly WITHDRAWS Re-issue and points at what removes an offbox target (a guest rebuild, R-193). `offsite_reports_since_consume` rides the details for the first time. **(b) Every refusal to self-heal leaves a record** — a `notification_log` row on the operator channel, status `refused`, with its reason (the R-182 suppressed-e-mail precedent), riding the stuck event's 24 h cadence so it sits beside the e-mail it explains rather than accumulating per tick. The two conditions were split into separate branches solely so each can name its own reason; **the set of situations in which the heal fires is byte-for-byte what it was**. **(c) Not in the spec and done anyway, narrowing only:** `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither was ever registered and neither has a `customerMessages` entry — which is not a block — so a customer with a configured recipient was in line for an English e-mail about one-time passwords being *"likely burned"*. Measured live: `notification_log` holds operator rows for demo-hp and no customer rows, which is NOT evidence the leg was blocked (equally consistent with no configured recipient), so the register makes it structural. **WHAT STAYS OPEN, and it is this row now:** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT 500` — the OLDEST 500 reports after the consume — so the counts describe the start of the window, not the present. Its correct shape (recency-bounded, rebuild-aware) depends on the recovery chain that is not yet assembled (R-199/R-200/R-201), so it was NOT fixed here. **The window is named inside the alert text** so the limitation travels with the number instead of being laundered into a confident sentence. **Red-proofs observed:** restoring the single hardcoded sentence fails both message scenarios (the first mutation attempt left the default branch in place and only the burned scenario failed — recorded because a mutation that does not remove every guard is not a red-proof); replacing the regressed branch with a bare `return` fails the refusal record and its cadence test. | CC **CLOSED 2026-08-05 (hub v0.96.0), by REPLACEMENT rather than repair.** The guard inferred the shape by counting how many of the **OLDEST 500** reports after a consume carried an offbox target — on demo-hp all 500 predated the rebuild, so the checker reported the *regressed* shape with confidence and declined to heal for 108 reports while the box sat stranded. A DECLARED state needs no window, no count and no inference, so `shapeDeclared` now **outranks both inferred shapes**, the stuck e-mail states what the box declared instead of prescribing Re-issue, and `maybeHeal` **stands down with a record** because `internal/offsiteheal` owns the remediation — two mechanisms healing one customer would double-issue, and this one can only mint, so it would also skip the cheap restage path. The counting fields remain for boxes on a pre-v0.199.0 controller, where there is still nothing better to go on.|
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CLOSED 2026-08-05 — controller v0.200.0** (credential half v0.199.0/v0.96.0; the recovery SCREEN v0.200.0) | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `<DataDir>/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** **RECON 2026-08-04 — the chain was traced link by link, and the picture is worse than the spike's** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(A) THE SPIKE'S CANDIDATE (b) IS OVERTURNED IN PART.** "Recoverable from escrow" is not blocked by zero-knowledge — the *hub* cannot open the blob, the *customer* can, with R. What is genuinely impossible is an **unattended** recovery. A **customer-present** one is a real design, and the operator has now ruled on its shape (below). **(B) THE CHAIN IS NOT ASSEMBLED — eleven links, and the automation stops at four.** Mint → stage → seal → store on the hub are PROVEN-LIVE. Then: the hub's blob-serving endpoints have **no client anywhere** (**R-199**); unsealing's only production caller is a `--selftest` mode reading R from an env var; nothing extracts `restic_repo_password` from the recovered bundle; the injection seam **has no form** (**R-200**); and reopening an existing repo with a recovered password has **never happened** (**R-201**). **(C) THE LOAD-BEARING NEW FACT, and it retires this row's own "hub v0.60.0 retains superseded escrow" premise: the retention does NOT retain the restic repo password** — `host_escrow_superseded` has no `identity_blob` column (**R-198**). Both demo boxes' old passwords were destroyed by the 2026-08-04 re-ceremonies, so the orphaned snapshots are unrecoverable for a **second, independent** reason; keeping R would not have helped. **(D) A FAIL-CLOSED REFUSAL IS IMPLEMENTABLE — this is the most useful thing settled.** The hub already tells every box, on every report ACK, `escrow{identity_blob_present, restic_pw_sha256, created_at}` (`hub/internal/api/handler.go:504-510`) — and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it (the `ClaimSync` set-only pattern, `report/claim_sync.go:39-53`) and refusing to mint when a blob covers a password we do not have needs **no new hub API and no new secret**. **(E) OPERATOR RULING, 2026-08-04, recorded verbatim:** *"If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding."* Priced row by row in the recon §9: fresh-install signal **exists** (the mint branch's own `os.Stat`); hub-has-a-blob **exists on the wire, S to persist**; the yell **S**; an **R** form **does not exist** (the UI has only ever *emitted* R) **S**; unsealing must cross agent→controller because the controller image ships no `age` — **M**, one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts/<id>/escrow`; the **preview is cheap and read-only** — `restic snapshots --json` + `stats --mode raw-data --json` are already how the box counts snapshots (`offbox.go:1234-1265`), so count, dates, sizes, app tags and paths are all knowable before committing **S**. **Security question put to the operator, not answered:** the form sits behind the dashboard password (bcrypt + CSRF + 5/min lockout); the preview exposes backup cadence and app names; the form is an **oracle** for a stolen R and must fail as generically as `UnwrapIdentity` already does; and R transits the agent, which is the same trade as option (c) in a smaller, time-bounded form. **(F) THE DRILL IS DESIGNED AND NOT RUN** (recon §10): demo-hp, ~3–4 h, R kept deliberately, sentinel file sha256 before and after, **pass = byte-identical sha256, NOT "the repository opened"**, fail = a snapshot count of 1. **Run R-198's fix first.** **(G) Q3 STILL UNMEASURED:** neither box has run since (`last_run` absent on both, 2026-08-04 09:49/09:56 reports) — the decisive run remains **2026-08-05 ~02:15 UTC**. **OPERATOR RULINGS 2026-08-04, and one of them changes what the other items are for.** (1) **Candidate (c) is REFUSED — the risk is accepted:** no repository password is retained on the Proxmox host. **That makes the customer-present recovery path the ONLY way back from a rebuild**, which is why R-198 was shipped the same day as a load-bearing fix rather than a tidy-up: with no host-retained copy, everything runs through the retained identity blob, and until hub v0.93.0 the ceremony destroyed it. (2) **Run the drill, after R-198** — R-198 has shipped, so the drill is the next session (design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10; tracked as R-201). (3) **Delete the orphaned ciphertext** — ~1.2 GB across the two demo boxes; **STILL OWED**, deliberately not done in v0.93.0 (a destructive act on a protected endpoint does not ride a schema-change release). **What v0.93.0 delivers against this row:** the key now SURVIVES a supersession (R-198) and a changed key is now REPORTED on the day (R-197). **What it does NOT:** the chain that hands the key back is still unassembled at three links — R-199 (no client for the hub's blob-serving endpoints), R-200 (no form for the injection seam), R-201 (never exercised end to end). This row stays open until the drill returns a byte-identical sentinel file. | CC + operator **THE CREDENTIAL HALF IS CLOSED (2026-08-05).** A rebuilt box no longer needs an operator Re-issue: it declares, and `internal/offsiteheal` re-arms the stored one-time secret (or mints if there is nothing to re-arm). See R-204 item 4. **STILL OPEN under this ID:** (a) the customer-facing recovery SCREEN — a freshly installed box that finds a sealed package waiting should say so and offer a recovery-code box with a preview; the operator ruling on its dashboard-password exposure is recorded (metadata, not content — accepted), but nothing is built; (b) ~~the orphaned-ciphertext deletion~~ — **DONE 2026-08-05**, ~1.45 GB across three set-aside stores, after the operator confirmed a CORRECTED list (the register's "two stores, ~1.2 GB" was wrong in both numbers, and the 1.2 GB figure matched a LIVE repo). See R-212. **THE RECOVERY SCREEN SHIPPED 2026-08-05 (controller v0.200.0) — R-193 IS CLOSED.** A full page (not a banner) takes over the landing pages while the hub holds a sealed package this box cannot open; afterwards it stays reachable from the backups area for as long as the situation lasts. **It UNLOCKS AND ONLY UNLOCKS** (operator ruling): it explains, takes the recovery code, opens the repository and lists what is in there — apps, dates, sizes — and restores NOTHING. **Three ways out, none a dismiss button:** recover; „most nem" (the full page stops interrupting, the entry point stays PERMANENTLY — bound to the offer, never to the postpone flag); and „nem kérem vissza a korábbi adatokat", confirmed TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes. **ONE CORE, TWO CALLERS:** `RecoverInstallCore` is split out of the CLI, whose exit codes and printed lines are byte-identical (every pre-existing CLI test passes unchanged); both directions asserted from source by AST. **A SHAPE THE TASK DID NOT NAME had to be added:** `WriteOffboxSecrets` auto-generates a repository password on re-apply — R-193's own orphaning mechanism — and since hub v0.96.0 the re-apply happens by itself within ~15–30 min, so the literal "no repository password" trigger would have opened a window that closes on its own before the customer next logs in. The predicate is therefore *the hub holds a package AND (no repository password OR the tier is orphaned)*. **Two defects the tests caught before shipping:** an UNCLAIMED (legacy-open) box would have been shown the page, and the inventory nil-dereferenced in exactly the pristine rebuilt shape. **Still owed under other IDs:** putting files back in place (→ R-213), and one end-to-end re-walk of the whole journey.|
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
| **R-91** | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
| — | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC |
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — |
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
| **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success` **held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — |
| **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — |
| **C9-F1** | ~~Tier-2 „Fájlok visszaállítása" is offered for apps whose copy has no restorable file leg; stops the app, restores 0 files, reports „Nincs hiányzó fájl — minden fájl megvan a helyén."~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | **Phase 0 sized it: 43 of 53 catalog apps read NOTHING, 9 read file legs but never their DB/volumes, 1 stateless.** Honesty half shipped: `Tier2RestoreCoverage` refuses UP FRONT without stopping the app and NAMES the working action; a run that proceeds claims only what it **examined** and discloses that the database and volumes are not covered. Live on demo-felhom: bookstack refused, uptime stayed „Up About an hour" (was „Up 25 seconds"); paperless A1 re-run still byte-identical, 16/16 docs clean | — |
| **C9-F2** | ~~An app in a Docker crash loop never alarms on any channel; `StateRestarting` is in no down-set~~ | **SHIPPED + PROVEN-LIVE** (controller v0.183.0, 2026-07-28) | — | `StateRestarting` deliberately NOT added to `IsDownState` (that alarms on every deploy fleet-wide); a SUSTAINED run becomes down after `crashLoopAfter`=5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. Dashboard counter uses the same predicate so it no longer contradicts the alarm. Red-proof that matters: the naive `IsDownState` change fails the brief-restart test | — |
| **C9-F3** → **R-104** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC |
| **C9-F1b** → **R-103** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC |
| **C9-F4** → **R-102** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC |
| **R-108** | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT~~ | **SHIPPED + PROVEN-LIVE** (controller v0.187.0, 2026-07-30) | — | **D5 IS UNBLOCKED — that is this item's whole point.** `RefuseAsAppNamespace` (`internal/settings/settings.go`) is one fail-closed predicate consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. **The fix inverted the obvious one:** the bind was NOT narrowed, because it cannot be — the `:rslave` share-ROOT bind is load-bearing (a 2026-07-22 probe proved an in-container access through it wakes the idle automount trigger), there is no `userdata/` layer to scope to (apps on a share store at `<share>/<app>`), and creating one would write Felhom convention onto a customer's own NAS, which R-67 forbids. Operator ruling: refuse the placement, keep the browse bind. **FIVE surfaces, not the four this row named** — `handleStorageDecommission` mode=migrate guarded only its SOURCE, so a whole namespace could be decommissioned ONTO a NAS; and the *boundary* is the deploy **POST**, not the dropdown (it takes any caller-supplied `HDD_PATH`, validated only by `os.Stat`). **Nothing stranded:** zero apps on network storage across all six customers incl. Peti. R-67's bind verified byte-identical before/after. Supersedes the controller README's "NAS backup locality — decision A". `audits/R108-network-app-namespace-2026-07-30.md` | — |
| **D5** | ~~**Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest~~ | **SHIPPED + PROVEN-LIVE** (controller v0.188.0, 2026-07-30) | — | **CLOSED — the arc's architectural centrepiece is done, and Tier-1/2 no longer depend on the whole-guest tier.** A customer now needs **the drive and nothing else**. **Part 0 overturned the brief's own recommendation, on evidence gathered before any code** — that is the substantive part of this row. It proposed that only `data_key`-flagged secrets travel; two findings killed that: (1) the flag is **unreliable** — only 5 fields across 4 apps carry it, yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` carry the SAME labels as flagged `adventurelog/SECRET_KEY` and are unflagged (→ **R-127**), so data-keys-only would omit real data keys and the fail-closed gate would not fire for them; (2) a **DB password is not resettable in practice** — proven on a throwaway `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is **ignored** (initdb skipped), so a regenerated value fails over the compose network (`FATAL: password authentication failed`) while the old one still works AND the dump replay still SUCCEEDS via the container's local trust socket — a restore that reports success onto data the app cannot reach. 18 DB/root-password fields affected; MariaDB fails louder (`getMariaDBPassword` reads the new value against a datadir holding the old hash → Access denied). **Operator ruling 2026-07-30: `type: secret` travels (45 fields), `type: password` NEVER (7) plus a code register (`vaultwarden/ADMIN_TOKEN`); plaintext.** The exclusion is what LICENSES the plaintext — coupled, not independent. `stacks.PortableSecretEnvVars` is the single boundary; the register is **code, not a catalog flag** (a boundary a catalog push can move is not a boundary — R-97a). **Precedence: the UNIT WINS** over the guest, because the unit's secrets were captured in the same run as the dumps beside them and therefore match the data being restored; pinned both directions. **Fail-closed data-key gate UNCHANGED.** Manifest → schema 2 + `portable_secret_env_vars` (names only); schema-1 units still restore from the guest. **Live proof** on a scratch drill guest through the real endpoints: AdventureLog restored with the guest `app.yaml` moved aside → `secrets recovered=2/2`, 27.6 s, then **the app read the seeded row over TCP with its own credential** (the observable that matters), pre-backup row back / post-backup row gone, **no `.sql` dump** so the DB came from the volume tar. Withheld half proven with Grafana: sentinel live in the container, `ENC:` in the guest, **0 files** under the whole backup namespace. 4 red-proofs, each verified to land. `audits/D5-drive-alone-restore-2026-07-30.md` Flips `07` §3, §7.1, §7.3, §7.4 (new), §8 rows 3/3c/13, §10.1; new capability-map row. **Consequence recorded, not changed:** the unit already travels to Tier-2 (another customer drive, plaintext, same reasoning) and offsite via restic (encrypted at rest under the customer-owned repo password) — no tier code touched | — |
| **R-127** | **The catalog's `data_key: true` flag is UNRELIABLE — at least four data-encrypting keys the catalog itself labels as encryption keys are unflagged; and the O4 restore path can regenerate a DB password that then does not match the restored data directory** | **READY (S/M)** | — | **Found by D5's Part 0, and it is why D5's boundary is `type: secret` rather than `data_key`.** Two separable legs. **(a) The misclassification.** Only 5 fields across 4 apps set `data_key: true` (`adventurelog/SECRET_KEY`, `homebox/HBOX_AUTH_API_KEY_PEPPER`, `papra/AUTH_SECRET`, `sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}`), yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the catalog's own Hungarian labels contradict the flag. **D5 makes this non-urgent but not harmless:** everything `type: secret` now travels, so the keys DO reach the drive; what stays wrong is the **fail-closed gate**, which only refuses for `data_key` names — so if one of these is missing from both sources the restore proceeds onto data it cannot decrypt instead of refusing. Fix = flag them (`app-catalog-felhom.eu`, a catalog-only change) + a gate/test that the flag set and the label set agree. **(b) The regenerated-DB-password trap.** `internal/backup/restore_unit.go` O4 generates a replacement for any missing non-data-key secret. Proven on `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is ignored (initdb skipped), so the app fails over the compose network while the dump replay still succeeds through the container's local **trust** socket — success reported, data unreachable. v0.188.0 corrected the WARN's false claim that *"stored data is unaffected"* and scoped it, but did **not** add a guard: D5 shrinks this to the rare case (the secret was empty at capture AND absent from the guest). Real fix = either treat a DB password as fail-closed like a data key, or `ALTER USER` to the regenerated value after the volume restore. 18 DB/root-password fields are in scope; MariaDB fails loudly instead (Access denied), which is the safer half | CC |
| **R-126** | **A `.fab` bundle — plaintext secrets, OPTIONAL password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | READY (S) | — | Split out of R-108, which closed without it: this is an explicit customer-chosen **export destination**, not a browsing surface reaching a backup tree, so it was never part of D5's precondition (`07` §7.3 records that reasoning). Was recorded inside R-108's row as its "second effect, independent of D5"; promoted to its own row so it does not vanish with R-108's closure. Fix = filter network paths out of the export destination list, or require the bundle password when the destination is a share | CC |
| **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:<user>@<host>:<path>`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — |
| **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — |
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
| **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/<vmid>` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — |
| **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — |
| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **CLOSED — PARTIALLY PROVEN** (Session C, 2026-07-29) | — | **CLOSED by `audits/SESSION-C-2026-07-29.md`.** C1/C2 proven in E-2d; **C3 and C4 PROVEN LIVE** this session (R-114, R-112); **C5 FAILED** — the gate fires and an alarm reaches the hub, but it is the generic event, not `backup_target_absent` (→ **R-116**, the one named open leg). Per the runbook's §9, decided in advance: a failed claim closes E-2 as partially proven with a named leg rather than re-running. The arc's stated definition of done is R-106+R-109, R-108 and D5 — none of which this detour touched. Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is ~~installer-logic-tested, not install-tested~~ — **INSTALL-TESTED 2026-07-29** on a fresh nested box via the real ISO/PAIRING route, rc=0 (`audits/E2D-fresh-vm-2026-07-29.md` §3). **Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level** (decline path, `restart_required:true`, E-2a wrapper, healthy-renders-nothing). **Still NOT proven, and now known to be BROKEN rather than merely untested:** the banner/offer never reach a customer (**R-112**) and `backup_target_absent` cannot fire on device loss (**R-113**), with the absent-state message itself wrong (**R-114**) | CC |
| **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — |
| **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — |
| **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — |
| **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator |
| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — |
| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — |
| **R-122** | ~~`AssembleDRRecipe` silently DROPPED `offsite_restic` — the offsite recovery location never reached any recipe~~ | **SHIPPED** (hub v0.83.0, 2026-07-30) | — | **Found 2026-07-30 while scoping R-109's hub half; it had already shipped and nobody knew.** The controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*), the hub stored it for **all three real customers**, and `appHalfShape` never listed the key — so no delivered recipe has ever contained it. No error, no log, green suite, because the fixture `drAppHalf` is hand-written and omits the field. `hostHalfShape`/`appHalfShape` are **ALLOW-LISTS dressed as forward-compat**; `TestAssembleDRRecipe_CarriesEveryEmittedSection` is now the guard, built on halves read verbatim out of the live `dr_recipe` table. `REUSE.md` (both repos) records that a recipe section is a TWO-REPO change | — |
| **R-123** | **R-105 and R-106 were `READY` in `ROADMAP.md` with no row on THIS page** — each referenced only inside R-109's prose, which is precisely the thread-loss the register exists to prevent | **PARTLY CLOSED** (2026-07-30) | — | R-106 registered above (and shipped). **R-105 still needs a row** — it is M-sized, is about three hub-held DR records being `{}`, and is NOT part of the recipe-completeness set that shipped today. **The process gap is the real item:** nothing checks that a `READY` ROADMAP row has an OPEN-ITEMS row. A grep-level gate would catch it | CC |
| **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC |
| **R-125** | **A "test through the production path" is only true up to the seam it injects at.** agent v0.118.0 shipped an INERT R-106: two tests ran the real `Collector.Collect()`, but both injected a `fakeObserver`, and the break was one layer below in `mergeConfig` — which dropped the pbs namespace, so the recipe still said `"root"`, now with `namespace_state: resolved` beside it (confident AND wrong, worse than the original defect). Caught by LIVE VALIDATION, not by the green suite | **FIXED** (agent v0.118.1) — filed for the DOCTRINE point | — | The fix is shipped and red-proofed. What is open is the practice: a production-path claim should NAME the seam it injects at, so the untested span is visible. Candidate for the `felhom-testing` skill; the state field is also what made the wrong value legible, which argues for the three-state shape generally. `audits/R106-R109-recipe-completeness-2026-07-30.md` §6 | CC |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
| **R-128** | ~~`build-felhom-iso.sh:44` comments that `ISO_VERSION` "aligns with felhom-host-install SCRIPT_VERSION" — a claim nothing evaluated~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | **Closed by correcting the claim, NOT by making the assertion real — and that distinction is the durable part.** The coupling it asserted does not exist: the ISO is a frozen artifact, while `felhom-host-install.sh` is fetched at RUN TIME from the website's git-sync of `main` (→ **R-94**, **R-110**), so whatever version an ISO carries, the script a box runs is always current. Making the build read `SCRIPT_VERSION` and refuse a mismatch — the fix the old row proposed — would have **invented a constraint** and coupled two things that are deliberately decoupled. `build-felhom-iso.sh:45-52` now states the independence in place of the false claim. **Do not 'restore' the assertion on the strength of seeing a disabled one** | — |
| **R-154** | ~~`[first-boot]` is automated-install-only and nothing in the Felhom tree said so~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | A property of the PVE installer, not a Felhom defect, but one that silently delivered nothing on the path we now ship. Measured with a same-image control in `audits/SPIKE-universal-iso-3-2026-07-31.md` §2: on an interactive install the hook never runs and the `proxmox-first-boot` **package is not even installed** (`Config.pm:118` defaults `first_boot.enabled=0`, `set_first_boot_opt` is never called, `Install.pm:746` returns early, `:1360` skips the package, and `proxinstall` contains **zero** occurrences of `first-boot`). Now recorded in pushed source at `scripts/iso/pkg/build-deb.sh:6-11` — the header of the mechanism that replaced it — and cited in `runbooks/iso-release-gate.md` G7 | — |
| **R-155** | ~~`iso-repack.sh` refuses any ISO without `auto-installer-mode.toml`, blocking the no-`answer.toml` posture~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | **NARROWED, not deleted — and the difference matters.** The guard protects a real promise: in single-entry appliance mode the menu shows one button labelled "Felhom telepítés" that boots the AUTOMATED installer, and without that file the same button would drop the user into the manual disk-picker. That promise is unchanged and the guard still enforces it for `FELHOM_MENU=single` (`iso-repack.sh:121-128`). It simply does not apply to `release`, where the file's absence **is** release-gate criterion **G1**. **Do not remove the guard wholesale on the strength of "R-155 closed it"** — deleting it would put an unattended installer behind a button promising one | — |
| **R-129** | **Every doc says demo-hp has "no baked SSH key"** and needs the G1 break-glass password — but `ssh -o BatchMode=yes demo-hp` authenticated **by key**, first try, 2026-07-31 | READY (XS) | — | Stale in the expensive direction: a session that believes it sends itself to the hub vault for a credential it does not need. Verify who owns the key and when it landed, then correct `CLAUDE.md`, `runbooks/target-selection.md:41-42`, `runbooks/workspace-CLAUDE.md` and `felhom-agent/CLAUDE.md` together — or remove the key if it was not deliberate | CC |
| **R-130** | **A "hard min" that only warns.** A fresh box's `local-lvm` was ~75 GiB against `HARD_MIN_LVM_GIB=120` (`scripts/felhom-host-install.sh`); the installer logged `[WARN] local-lvm free ~75 GiB < hard min 120 GiB` and went on to a **fully successful** install | READY (S) | — | Either the minimum is not hard (rename it and state the real floor) or it is wrong (and 120 GiB is not what a working appliance needs). Leaving it is the R-29 shape: a check that reads as coverage while providing none. Evidence: same audit §8 | CC |
| **R-131** | **`sess-f` is a fourth orphaned scratch customer** on the hub ("R-120 golden 0.186.0 proof", DOWN), left by the 2026-07-30 session | READY (XS) | — | After `drill-r50`, `sess-c`, `sess-d` — the accumulation `runbooks/target-selection.md:86-87` and `PROMPT-TEMPLATE.md` §13 both warn about, now on its fourth instance. Delete it (see the recorded command in `audits/tester-gate-golden-0.188.0-2026-07-31.md` §7.1); the recurrence itself argues for a periodic scratch-customer sweep rather than another reminder | CC |
| **R-132** | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it | **ACTION: rotate `HUB_PW`** | — | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor |
| **R-133** | **The hub enforces uniqueness on `customer_id` only** — `domain` is `TEXT NOT NULL DEFAULT ''` with no UNIQUE/CHECK (`hub/internal/store/store.go:114`) and the create path only rejects a duplicate id (`hub/internal/web/configs.go:644`), so two customers can be given the identical domain silently | READY (XS) | — | Harmless while every customer owns their own zone; a real footgun the moment customers share one (the subdomain-onboarding plan). Fix = reject a duplicate domain on create/edit, or warn. Evidence: `audits/RECON-subdomain-onboarding-2026-07-31.md` §2.2 | CC |
| **R-134** | **Two zone-resolvers disagree on depth.** The controller strips labels progressively (`controller/internal/cloudflare/zone.go:18`); the hub's `resolveZone` tries the exact name then `parentDomain`, which strips exactly ONE label (`hub/internal/cloudflare/unblock.go:115,136`) | READY (XS) | — | For a one-label Felhom-issued subdomain both work; for anything deeper the hub silently fails to find the zone while the controller succeeds — the geo-unblock would then no-op with a "no active zone found" error. One concept, two implementations. Same audit §2.6 | CC |
| **R-135** | **`validateCSRF` returns TRUE when there is no session cookie** (`hub/internal/web/server.go:678-683`) — measured live: `POST` with Basic auth and no cookie goes straight past the CSRF gate (404, not 403), while the same POST with a cookie and no token is 403 | READY (S) — **security** | — | Browsers cache HTTP Basic credentials per origin and resend them automatically on cross-origin requests, and `SameSite` does not govern the `Authorization` header. So if the operator has ever Basic-authed to the hub in a browser, any attacker page can POST to every mutating route. Latent on the condition, not guaranteed absent. Fix = require the token whenever the request is not provably programmatic, or drop browser-usable Basic auth. Same audit §4.3 | CC |
| **R-136** | **Rename `hub_session` → `__Host-hub_session`** — makes cookie tossing structurally impossible | READY (XS, one line) | — | Verified on the live production response that all three prefix preconditions already hold: `Path=/`, `Secure`, no `Domain`. **Caveat for the ticket:** browsers reject a `__Host-` cookie without `Secure`, and `isSecure` is conditional on `r.TLS`/`X-Forwarded-Proto`, so plain-HTTP *browser* access to the hub would stop working (non-browser access uses Basic auth, unaffected). Tested consequence: `r.Cookie` returns the FIRST match and never tries the others, so a tossed cookie wins outright. Same audit §4.1-4.2 | CC |
| **R-137** | **Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults.** `globalRuleDesc = "[felhom-geo] Global"` (`waf.go:18`) is one literal description per ZONE; `appRuleDescPrefix` keys by app name with no customer (`waf.go:21`); `BuildGlobalExpression` has no positive hostname scoping (`waf.go:241`); `applyDiff` deletes every `[felhom-geo]` rule not in THIS box's desired set (`geosync.go:320`) | READY (M) — **blocks shared-zone onboarding** | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's `RemoveGeoRules`) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by `customer_id` + add `http.host ends_with "<domain>"` to both expressions — a TWO-REPO change (controller + hub `RemoveGeoRules`). Same audit §5.1 | CC |
| **R-138** | **A shared-zone `cf_api_token` is a zone-wide DNS-write capability on a customer's box** — written 0600 to `/opt/docker/stacks/traefik/.env` (`controller/internal/infra/infra.go:123`) | READY (S) | — | Today each box holds a token for a zone nobody else uses, so the blast radius is one customer. Under a shared customer zone, one compromised tester box could repoint every other tester's DNS. The ACME path is already switchable — an empty token selects HTTP-01 (`traefik.yml.tmpl`) — so the fix is policy plus a guard that refuses to hand a shared-zone customer a zone-scoped token. Same audit §5.2 | CC |
| **R-133** | **The vaulted break-glass console credential is PLAINTEXT AT REST — every hub DB backup is a fleet-wide console-credential dump.** `host_recovery.secret` holds each managed box's `root@pam` password verbatim, so any copy of the SQLite DB (Longhorn snapshot, PBS backup of the hub PVC, a hand-taken copy during a diagnosis) carries root console access to every Felhom host in one file | **READY (M) — NEW 2026-07-31** | — | **The deferred leg of hub v0.84.0** (Console access card), filed separately because v0.84.0 changed only WHO can retrieve the secret, never how it is stored. v0.84.0 makes it more worth doing, not more broken: retrieval now rides the hub SESSION, so the DB and the login password are jointly the whole protection (ruling **S-4**, `CONTEXT.md`). Fix shape: **envelope-encrypt the `host_recovery.secret` column under a KEK held outside the DB** — the hub already proves it can hold something it cannot itself read (escrow blobs), and that contrast is the argument. Two constraints the design must respect: the credential must stay retrievable **when the box is unreachable** (that is the whole point of break-glass), so the KEK cannot live on the box or depend on the agent; and the global-key API path must keep working with the hub UI down. Would flip the capability-map row **"Break-glass management-plane recovery"**, which today reads IMPLEMENTED with this as its caveat | CC |
| **R-156** | **An app's data is neither persisted nor backed up, and it reports healthy.** A template mounts a volume at a path the application never writes, so the data sits in the container's **writable layer**: lost on redeploy, and tarred nightly as an empty directory while the healthcheck stays green. **papra** (Campaign 10) and **gramps-web** + **wishlist** (the 53-template sweep) all convicted. | **CLOSED — all three apps fixed** (papra template, 2026-08-03) | — | **The gate SHIPPED**: `app-catalog-felhom.eu/scripts/check-volume-persistence.py` (runtime probe; `docker diff` + mount-occupancy + writability, canary self-test, fails closed). It convicts papra `/app/data`[vol,EMPTY] → `db.sqlite` in the writable layer. **gramps-web and wishlist were FIXED in the sweep; papra was NOT — it is referred**, because the fix needs either the app to use `/app/data` or the template to mount `/app/app-data`. **THE REFERRAL IS RESOLVED, 2026-08-02 — papra is deployed NOWHERE, so the template fix strands nothing and can be applied.** The referral existed because changing where the volume mounts moves live data: an installed papra writes `db.sqlite` into the container's writable layer, and a remount relocates the path out from under it. With no instance deployed there is no live data to move, so the cheaper leg — **the template mounts `/app/app-data`** — is takeable directly, without waiting on upstream to adopt `/app/data`. **Provenance, stated because it decides the row:** the observation is `docker ps -a` on demo-hp's **guest 9201** returning empty, supplied with the 2026-08-02 task; **this session did not re-measure** (documentation-only, every box fenced). **Scope of that evidence, honestly:** it covers guest 9201 — the guest papra was convicted on in Campaign 10 — and **no other customer's guest was enumerated**, so a re-check belongs in the task that edits the template, before it edits it. **Next action: apply the template fix in `app-catalog-felhom.eu` and re-run `scripts/catalog_gates.py`** (deliberately not done here — that repo was out of scope for this task). See R-161 (nothing runs the gate automatically) and R-159/R-160. **CLOSED 2026-08-03 — papra fixed, and the precondition was CHECKED rather than inherited.** The 2026-08-02 evidence covered only guest 9201 and both demo boxes have been wiped since, so it was re-measured three ways: `docker ps -a` (INCLUDING stopped containers) on **both** demo guests → no papra; and the hub's `/hosts` fleet view → exactly **two** enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra references. Deployed nowhere ⇒ the template fix strands nothing. **The fix decided from the IMAGE, not the README or the upstream docs:** `docker inspect` of `ghcr.io/papra-hq/papra:26.6.1-rootless` gives `WORKDIR=/app` and **all three** data paths under `./app-data` — `DATABASE_URL=file:./app-data/db/db.sqlite`, `DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and **`/app/data` does not exist in the image at all**, so the old mount pointed at a path nothing could ever write. **Departure from the task's stated preference order, recorded because it was deliberate.** Option (1) — reconfigure the app to write where the template already mounted — WAS available (all three paths are env-settable). It was not taken: it enumerates data paths, so a fourth one added upstream would silently escape to the writable layer again, which is R-156's exact failure mode re-armed and invisible. Mounting the app's own data ROOT (`papra_data:/app/app-data`) captures every current AND future path by construction. One line instead of three env vars with an ongoing coupling to upstream. **The runtime gate is the arbiter and it was run, in BOTH directions.** `check-volume-persistence.py papra` → **CLEAN**, and its mandatory self-test passed on that run (*'prober flags the R-156 signature and clears a correct template — trustworthy'*), so the verdict carries a live proof that the instrument discriminates. **Red-proof on the real template, not just the canary:** reverting the mount to `/app/data` and re-running → **BROKEN**, with the exact R-156 evidence — *'mount /app/data is NOT writable by the app's own uid=999'*, *'DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. [db.sqlite])'*, *'declared volume /app/data is EMPTY'*. Restored, re-run, CLEAN. **Two operational notes for the next person to run this gate:** it needs **root** (it reads volume contents under `/var/lib/docker/volumes`, mode `drwx--x---`; as a normal user its own canary self-test fails UNDETERMINED and it correctly refuses to report), and it hardcodes a scratch path `/srv/felhom-gate`. Run it **scoped to the app you touched** — unscoped it deploys all 53 templates and takes far longer than a session allows | — |
| **R-157** | ~~**`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (B: controller v0.189.0; A: v0.190.0, 2026-08-02) | — | **Both mechanisms closed. (B)** the container-count signal → recorded intent (R-166). **(A)** the sweep looked ONCE at T+5 s, deriving candidates from a fleet docker was still restoring — 3 of 6 hard resets. Now a **settle-then-sweep window**: sample the fleet every 5 s, settled after 3 identical samples, sweep ONCE at the end; ends on settled OR a 50 s budget, and the log says which. **The budget is 50 s because a test rejected 60 s**: settle+budget+one 30 s retry must stay under the 90 s `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s. A window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps — the grace was NOT widened to hide it (§8.3). **A defect in the fix, found by live validation not review:** `GetStacks()` is the Manager's cache, refreshed by the scheduler every 10 s, so sampling it every 5 s without refreshing let "settled" mean "the cache did not update" — observed missing a container removed 5 s before the window closed. `sampleBootFleet` now refreshes first. **Live: 6/6 hard resets on the shipped build, every app back every time** (settle times 10/40/10/10/15/15 s — i.e. the window routinely waited 2–8× longer than the old fixed 5 s), plus a before/after on ONE app on ONE box: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35, the fixed one found and recovered it at 18:18:50 | — |
| **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — |
| **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — |
| **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — |
| **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — |
| **R-175** | ~~**`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.**~~ | **CLOSED — FIXED 2026-08-03** (same pass as R-165) | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits **FIXED.** §7.5 gained a **7.5.1** which (a) states plainly that the bound is a FUNCTION of `mp1` and applies only to a box still on the split layout, naming all three real shapes, and (b) records that the ceiling itself has been removed by R-165 for boxes built from golden ≥ 0.192.0. Fixed in the same pass as the merge rather than filed and forgotten, because the section would otherwise have been wrong in two ways at once | CC |
| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `<mp1>/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC |
| **R-183** | **A fresh install fetched the vouched agent BINARY and its sixteen CONFIG files from two different refs, and nothing compared them.** `felhom-host-install.sh` resolved the agent version from the hub manifest and sha-verified the binary — then took `felhom-agent.service`, `felhom-agent.sudoers` and fourteen more from `raw/branch/main`, i.e. whatever the agent repo's tip happened to hold at that second. One install, two refs, no comparison | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Found while implementing R-110, by reading `fetch_raw`'s call sites rather than the spec's description of them** — the task said nine files from `felhom.eu`; they are **sixteen** and they come from **`felhom-agent`**. **Why it is a defect and not only untidiness:** these files are the agent's own operating surface — its systemd unit, its sudoers, its guarded wrappers — and `configs/felhom-backup-target-apply` is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. A config newer than the binary is a root-executed artifact the vouched version was never tested against. **Not hypothetical in shape:** the fleet has shipped exactly this class before, where a config and the code that reads it moved independently. **Fixed by pinning to the agent version the install is already committed to**, on the operator's ruling: `raw/tag/v$ART_AGENT_VER/`, resolved from the hub manifest that the binary's sha is already checked against — so binary and configs now come from ONE ref by construction. **No fallback to a branch**, deliberately: a missing tag dies loudly rather than quietly serving `main`. Pinned by `hostinstall_gates.py` gate 6 (no `raw/branch/` anywhere; the `$ART_AGENT_VER` pin still present), red-proofed by reverting one of the sixteen and by removing both assertions. `felhom-agent` now carries `v<semver>` tags (`v0.120.0` created retroactively at `cd6e267`, the commit the published binary was built from; `configs/` is byte-identical there and at `main`, so nothing depended on the choice) and `release-agent.sh` creates them as part of releasing | — |
| **R-184** | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/<customer>` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC |
| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **CLOSED — SHIPPED** (controller v0.194.0 + hub v0.90.0/.1, 2026-08-03) | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records. **CLOSED 2026-08-03 — controller v0.194.0 (`88897a2`) + hub v0.90.0/.1 (`dd40f85`, `f21e7ca`), proven live on demo-hp.** **The operator ruled against the obvious fix** (putting `app` in the cooldown key): on a full disk that is one e-mail per app, the volume problem wearing the correctness problem's clothes. What shipped instead is **one digest per run** plus **a record for every failure**. **(1) The silent drop is gone.** `processOperator`'s cooldown no longer returns bare — a suppressed operator event writes a `suppressed` row carrying the message and the key that suppressed it, **for every operator event type, not only the one that exposed this**. The cooldown's duration and semantics are deliberately unchanged: this makes the drop visible, not absent. **(2) `backup_run_failures`** — one operator mail at the end of a run listing every failed app, its leg and its reason, with the counts and the free space; in `allowedEventTypes` AND `operatorOnlyEvents`, with a test demonstrating a customer who has enabled it receiving nothing. **(3) `recordOnlyEvents`** — a third routing class. `recovery_unit_capture_failed` is now stored + logged every time and never mailed: **the record and the notification are different things**, which is the durable half. A register rather than a severity downgrade, which would have relabelled a genuine failure as informational everywhere it is queried. **(4) `cooldownRunSuffix`** — a sibling of `cooldownTierSuffix`, so `tier` keeps byte-identical semantics and R-97a is untouched. A unique `run_id` per real run means the cooldown can never collapse a manual run into the nightly one (the operator ruled on that explicitly). **A GAP THE SPEC DID NOT ANTICIPATE, AND ITS FIX.** `recovery_unit_capture_failed` also fires from `GetFullStatus`'s periodic sweep, outside any run. Making it record-only would have left those failures recorded and **never notified** — a NEW silence created while closing one. So the sweep emits a digest too, deliberately with **no `run_id`**, so it stays under the ordinary hourly cooldown exactly as before while the mail now lists every failing app instead of whichever was first. **A refusal is noted ONCE, where the verdict is taken**, not at the three legs that consult it — R-181's contract is one verdict per app per run, and noting it per leg listed a single refused app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not in review.** **LIVE PROOF, demo-hp, the same method that produced the original measurement** — a real 64.6 GiB `fallocate` to 241 MB free (thin pool held at **30.78 → 30.78**, instrument re-proven). The hub's `notification_log` for the run reads: `recovery_unit_capture_failed | operator | **recorded**` ×2 (opengist, privatebin), `backup_run_failures | operator | **sent**` ×1 listing BOTH, and `backup_run_failures | customer | **skipped** | operator_only` — i.e. two failures, one mail, both named, and the customer refused. Against yesterday's two failures → one mail naming one app and one vanishing. **Scenario D proved itself on an event I did not plan:** `disk_critical | operator | **suppressed** | operator cooldown 1h, key=demo-hp:disk_critical` — fillwatch alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now VISIBLE where yesterday it would have left nothing at all. **Scenario E:** a second run in the same hour produced a second digest. **Scenario B:** after freeing the space, a run completed with 2 volume dumps and **no digest at all**. **§3's safety property confirmed rather than assumed:** the hub's deadline check raises `expected_backup_missed`/`expected_dbdump_missed` (`monitor/deadline.go:396,417`) from report freshness and stored events, independently of any mail the box sends — so a digest's silence still means "the run finished and found nothing wrong". **v0.90.1 followed from reading the first real digest:** every app row repeated the same filesystem clause the mail already prints once, which down a list of a dozen apps pushes the part that differs off a phone screen | — |
| **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **CLOSED — SHIPPED** (controller v0.193.0 + v0.193.1, 2026-08-03) | unblocks **R-165** | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware. **CLOSED 2026-08-03 — controller v0.193.0 (`fef07c3`) + v0.193.1 (`6c43bf6`), proven live on demo-hp.** **The fix is ONE admission verdict per app per run** (`internal/backup/admission.go`), taken before that app's FIRST write and consulted by all three legs — the three write under one per-app root (`appbackup.RecoveryUnitPath`), which is exactly why one verdict can honestly cover them. **Decided LAZILY at the app's first write, never once at run start**: app A's dump can put app B under the reserve, so a run-start verdict would read a disk that no longer exists — the same class of mistake one level up. **Never re-decided between an app's own legs** (that IS the split this closes) and **reset per run**. Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act, so a refused app is never bounced; placed AFTER the volume-less check, which has no write to gate. Exactly ONE operator alert per refused app per run. Leg order unchanged. **The floor is now SIZE-AWARE**, which is the second half of the defect: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — the term whose absence admitted an app at 96% and then let it write 2 GB. Estimate = the app's previous `.sql`+`.tar` on disk; **no history → headroom-only** deliberately, or the first backup becomes the one that can never happen, and the alert says so. **A container-based `du` was MEASURED and rejected, not assumed**: 66 timed runs on demo-hp guest 9201, **median ~355 ms/volume (341–404)** on volumes holding tens of KB — the cost is container start-up, not the walk. Decisive on top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure the reserve exists to handle; and the previous-dump estimate measures the ARTIFACT that will be written rather than the live volume. **THE MESSAGE WAS NOT WEAKENED — the behaviour moved so the wording became true**, and it is checked by sha256 tree fingerprint, not by reading the log line (which is what lied). **LIVE PROOF, demo-hp guest 9201, the same method that found it.** The `fallocate` instrument was RE-PROVEN on the rebuilt box before use (guest `df` 1.2G→6.2G on a 5 GiB step while thin-pool `data_percent` stayed **36.83 → 36.83**: zero blocks allocated), because a real fill of a 70 G volume would exhaust the 53.93 GiB pool. **Headroom term @ 08:59:46** — 906 MB free / 99%: both apps refused, **`TREE_SHA` 111d1760c18d3440f700634ab325f8b8 IDENTICAL before and after** (10 files, incl. opengist's tar still at 182,272 B — R-181's own 'before' figure), **no `Stopping <app> for safe volume dump` line at all** (it is present in the 08:58 baseline, which is what makes its absence evidence), 0 volume dumps, one `recovery_unit_capture_failed` per app HTTP 200. **Freed and re-run @ 09:01:33** — both captured normally. **SIZE term proven separately @ 09:03:00**, reproducing the original sequence with a real 2 GiB file in opengist's volume (its previous tar then **2,147,666,432 B**, the exact live figure) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"* — while **privatebin was ADMITTED and dumped normally**, proving the term is per-app and not a global halt. **Teardown complete**: fill removed, planted file removed, `pct fstrim 9201` returned 67.5 GiB, pool **29.43%** (below the 36.83% baseline), tree byte-identical to the pre-test fingerprint. **v0.193.1 shipped in the same session**, found by this very proof run: the estimate was rendered fixed to 2-decimal GiB, so opengist's real **178 KB** printed as `estimated 0.00 GiB write` — which reads as *no estimate was available* and is the opposite of what happened. Rendering moved to `humanizeBytes`; arithmetic unchanged. Re-verified live: `estimated 178.0 KB write`. **11 new tests + 4 red-proofs**, each demonstrated failing then restored: both dump-leg gates removed (= v0.192.0) → Scenario A red with the tree shown changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F red; the floor moved above the warning band → Scenario G red. **Recorded honestly: the specified Scenario-F mutation (remove the reserve entirely) did NOT turn F red** — removing it makes every app write, which overwrites and adds but deletes nothing, so a deletion-watching test correctly stays green; the prune mutation is the one that proves the assertion. The DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne` — `strings.Contains` is insufficient, a commented-out call still contains the string. **§3's correction CONFIRMED in passing and not chased**: `restore_points.go:57-59` takes the manifest mtime then `newestArtifact` over `.sql` and `.tar`, so the newest of the three wins — the restore point does NOT show a stale timestamp. **New finding from the live run → R-182.** | — |
| **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC |
| **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-<share>.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC |
| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **CLOSED — BOTH BOXES REINSTALLED AND PROVEN (2026-08-03)** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits. **CLOSED 2026-08-03 — both boxes reinstalled from the merged golden, by two DIFFERENT supply paths, and proven end to end** (`REPORT.md`). **demo-hp — the layout proof**, installed with `--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst` (installer v1.22.0, sha `ed02acb2…`, byte-identical to the repo copy): `mp0 …mp=/var/lib/felhom,backup=1,size=70G`, **no `mp1`**; `/var/lib/docker` → `…disk--1[/docker]` and `/mnt/sys_drive` → `…disk--1[/sys_drive]`, both real mounts, both writable, both in `/etc/fstab`; ONE `df` figure (69G/65G) and `stat -c %d` = `64519` on all three paths; **reboots 3/3** (08:13:33 / 08:13:59 / 08:14:19, controller healthy in 12s/7s/7s, all three still mountpoints after each). **demo-felhom — the pipeline proof**, installed with `--force-gitea-golden` and NO local golden used (preflight logged *"golden: none local — will fetch + verify from Gitea in step 7/8"*, bypassing the 06:58 bake artifact sitting on the same box): **`verified sha256 54e2a4c431daf580… matches the hub manifest`** for the golden and **`verified sha256 a7763d31b55b5ce7…`** for the agent — the observable this second box exists to produce; `mp0 …size=250G`, `grep -c '^mp1:'` → 0, one `df` figure (246G/233G), reboots 3/3 (09:18:55 / 09:19:12 / 09:19:30). **Journey proven on BOTH**, endpoint-level (no browser on DooPlex — the exact endpoints the dashboard's own JS calls): claim (`POST /claim` with the pre-auth HMAC CSRF + `felhom_claim_csrf` cookie; gate discriminator flipped `dashboard not yet claimed` → `authentication required`) → deploy (`POST /api/stacks/<n>/deploy`) → capture (`POST /api/debug/backup/dbdump`, which runs the production `RunDBDumps`) → **restore** (`POST /backup/restore`): a planted marker deleted from the live volume came back with an **identical sha256** on each box (`ac1faae6…ae861` privatebin/demo-hp in 9.2s; `bc550798…b59e939` opengist/demo-felhom in 9.4s), recovery units on the single volume in both cases. **Ceiling gone, measured:** 65 GiB (demo-hp) and 233 GiB (demo-felhom) available to a recovery unit, against the 19 GiB and 45 GiB their pre-wipe `mp1` slices offered. **Two deviations, both the operator's call and both recorded:** the golden was ALREADY vouched when the session opened (hub log `2026/08/03 07:23:26 Artifact manifest set: agent=0.119.0 golden=0.192.0`, ~10 min before the first read of this session — so §7's prove-then-vouch order was already spent and the operator elected to accept it); and agent **0.120.0 had never been published**, so the vouched agent was 0.119.0 — published + vouched before the reinstalls (→ **R-115** third instance). **Three new findings: R-179, R-180, R-181** | — |
| **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC |
| **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC |
| **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — |
| **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
| **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does **UPDATE 2026-08-02:** `catalog_gates.py` gained `--fast` (gate 1 only — the network and runtime gates are deliberately NOT in a hook: a push that pulls images and starts containers gets bypassed within a week and the bypass becomes the habit) and `.githooks/pre-push` now runs it. **The automatic half now has a designated successor row: R-168** (Gitea Actions runner). This row stays open at its reduced scope — the runtime gate remains a deliberate periodic run **UPDATE 2026-08-02 (second):** the automatic half now EXISTS — R-168's runner executes `catalog_gates.py --fast` on every push to this repo (measured: run #1, `image-pin gate OK — 53 templates`, with the two runtime gates announced as skipped and their own output absent from the log). This row's *original* scope — the RUNTIME volume-persistence gate — is deliberately still NOT automatic and should stay that way: CI that pulls 53 images on every push gets disabled. It remains a periodic run | operator |
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
| **R-163** | ~~**`mp1` is RETENTION, not staging — and it is sized as if it were neither.**~~ | **CLOSED by R-165 — the ceiling it describes no longer exists** (golden v3.0.0, 2026-08-03) | — (the sizing question is answered; the work is **R-165**) | **Closed, not merely re-framed.** This row was the record of a constraint that was to stay open *"until the merge lands"*. It has landed: the golden ships ONE data volume, so there is no separate 20 G area for a driveless app's recovery unit to outgrow, and the free space an app can use is the box's actual free space. **What replaced the constraint is recorded on R-165**: the bulkhead the partition also provided is now B2's explicit capture floor (controller v0.192.0), and the measured 2× DB-app unit size this row documented is what justifies the floor's reserve being a reserve rather than a working budget. **Caveat carried forward, deliberately:** no box has been reinstalled from the merged golden yet (R-178), so every box in the field still has the split layout and this row's consequences remain live ON THOSE BOXES until they are reinstalled. Original finding unchanged below | CC |
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. IMPLEMENTED — the LAYOUT is proven live on both boxes (R-178, 2026-08-03); the BULKHEAD'S REPLACEMENT IS NOT (→ R-181)** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178**. **STATUS SETTLED 2026-08-03, operator ruling: IMPLEMENTED, not PROVEN-LIVE, and the reason is the substantive part.** R-178 proved the *layout* on both boxes past any doubt — one volume, no `mp1`, both binds real mounts, one `df` figure, 3/3 reboots each, claim→deploy→backup→restore, and the ceiling's removal measured at 65 GiB / 233 GiB against the old 19 GiB / 45 GiB slices. **But B2, which this row records as the bulkhead's deliberate replacement, does not guard the leg that fills the volume** — proven live on demo-hp at 06:40:03 and filed as **R-181**: the floor is consulted ONLY in `captureAllRecoveryUnits` (`recovery_unit.go:328`), while `runVolumeDumps` (`backup.go:535`) writes the bulk with no floor check at all, and its refusal message's claim *"the previous unit is untouched"* was measured FALSE. **This row's own framing is what makes that gate the status:** it says the partition's bulkhead "is now B2's explicit capture floor". Until R-181 closes, the merge has removed a bulkhead and its stated replacement covers the cheap leg only — and post-merge the unguarded leg can reach Docker's data-root, which pre-merge it could not (it could only fill the dedicated 20 G `mp1`). PROVEN-LIVE when R-181 closes and a fill is re-run | CC |
| **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — |
| **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — |
| **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — |
| **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator |
| **R-205** | ~~**`RootFsPressureDespiteHousekeeping` can never fire** — a bare PromQL `and` across mismatched label sets~~ | **CLOSED — SHIPPED + RED-PROVEN LIVE** (`homelab-manifests` `6808a4b`, 2026-08-05) | — | **`and` → `and on(instance)`.** Left vector carries `{device,fstype,instance,job,mountpoint}`; the right is a node_exporter *textfile* metric carrying `{instance,job}`. Nothing ever matched, so the alert returned the empty vector unconditionally — `health=ok`, `inactive`, no error, **dead from the day it was written**. It stayed silent through the entire 2026-07 build-cache incident that took `/` to 86% used and put Longhorn's default disk into `Schedulable=False (DiskPressure)`. **Red-proof against the live Prometheus API, both sides individually true** (threshold substituted `0.25`→`0.5` because P3's prune had already cleared the incident; the defect is in label matching, not the threshold): bare `and` → **`result_count 0`**, `and on(instance)` → **`result_count 1`**. **The annotation was also wrong and is rewritten** — it sent the reader to Longhorn replicas/snapshots (~10% of the growth) instead of the unbounded build cache (157 GB, ~95%); it now names `docker buildx du` first and records the 30 GB cap so a reverted cap is diagnosable from the alert text. ArgoCD `monitoring` Synced/Healthy at `6808a4b`; rule live and `health=ok`. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | — |
| **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** The `node_housekeeping` role must also carry the prune, which today it is forbidden to run | **READY (M) — NEW 2026-08-05** | — | **The spike validated the recipe; this row builds it.** Three parts. **(a) Template `/etc/docker/daemon.json`** with the **`policy` array** form — **the flat form (`{"gc":{"reservedSpace":…}}`) is SILENTLY IGNORED**, measured: the daemon starts, logs nothing, and `docker buildx inspect` still reports the built-in defaults. **The oracle is `docker buildx inspect`, never `dockerd --validate`** — the validator returned `configuration OK` for a bogus key AND for the config that then **crashed the daemon** (`filter` takes one value per policy entry, not an array; `error initializing buildkit: filters expect only one value`). **(b) Narrow the role's Docker ban** (`node-housekeeping.sh.j2:10-14`) to permit exactly `docker builder prune -af` and nothing else — the ban's stated premise ("Docker here runs only unrelated jarr-* dev containers") is obsolete: the growth is Felhom Go build cache. **The measured prune is SYNCHRONOUS** (150.35 GB back at t+0, two consecutive polls <1 MB apart within 60 s) — **unlike containerd's image GC, so it needs no `settle_imagefs` equivalent**, but it MUST measure the filesystem rather than trust the command: `prune` claimed **156.9 GB** and the filesystem returned **150.35 GB**, the 6.5 GB gap being layers still shared with images. **(c) A restart-safety note in the role:** a bad `daemon.json` takes the daemon down AND leaves the `unless-stopped` dev containers stopped — they needed a manual `docker start` — so the role must restart-and-verify, not validate-and-assume. Recipe + every measurement: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | CC |
| **R-207** | **`DRY_RUN=1` on `node-housekeeping.sh` is NOT non-mutating — it destroys the metric history it is supposed to let you inspect** | **READY (S) — NEW 2026-08-05** | — | `write_metrics()` (`node-housekeeping.sh.j2:119-150`) has **no `DRY_RUN` guard at all** — `DRY_RUN` appears in it only inside a log line — and it is called from an **unconditional `EXIT` trap** (`:150`). A dry run therefore atomically renames over the live node_exporter textfile, overwriting `node_housekeeping_last_success_timestamp_seconds` with *now* and `reclaimed_bytes` with ~0 — **resetting the staleness clock `HousekeepingStale` watches and erasing the 8-week reclaim history**. Confirmed by reading the source in both the 2026-08-05 audit and this spike; **neither run executed it**, so the history survives. Fix: guard `write_metrics` on `DRY_RUN`, or have the trap skip it. Pairs naturally with R-206 (same file, same role) | CC |
| **R-208** | **Every Felhom Go build re-downloads its modules because `ARG VERSION` sits ABOVE the module-download layer — ~440 MB of dead cache per build, 90.5 GB of the 157 GB** | **READY (S) — NEW 2026-08-05, ROOT CAUSE PROVEN** | — | **Measured, not inferred.** All **208** retained `go mod download` records carried **`Usage count: 1`** — not one was ever reused in ~a month of builds. The mechanism was isolated by four controlled builds: an unchanged tree rebuilt with the **same** `--build-arg VERSION` → `RUN go mod download` **CACHED**; the **same** tree with a **new** `VERSION` → **executed**. `COPY go.mod ./` stays CACHED either way, which is the tell: a `COPY`'s key is content-based, while a `RUN`'s key includes the stage **environment**, and `ARG VERSION`/`ARG GIT_COMMIT` are declared *before* the download in `felhom-controller/controller/Dockerfile`. Since every real build passes a fresh version, the layer is invalidated **every single time**. **`felhom.eu/hub/Dockerfile` has the identical defect** (`ARG VERSION`/`ARG BUILD_TIME` above `COPY go.mod go.sum*` → `RUN go mod download`) — and because both Dockerfiles produce byte-identical `buildx du` description strings, the 208 records are a COMBINED count and must not be attributed to one project. **Fix shape (one line each, not applied here):** move the `ARG VERSION`/`ARG GIT_COMMIT`/`ARG BUILD_TIME` declarations down to just above the final `go build`. **Worth more than the cap and the move combined** — the cap bounds the symptom, this removes the source. `build.sh`'s `rm -rf` + `cp -a` and its host-side `go mod tidy` were **ruled out by fingerprinting**: the tree is byte-identical across runs and `tidy` is a no-op | CC |
| **R-209** | ~~**Should the containerd store move to SSD2 at all?**~~ | **EXECUTED 2026-08-05 on operator ruling — reboot validation DEFERRED → R-209a** | — | **Operator ruled "proceed" having read the pre-analysis; CC's `storageReserved` condition was applied with it.** Moved with **zero loss, verified on four independent observables BEFORE the original was touched** (550,891 entries = 550,891; **448 = 448 `trusted.overlay` xattrs**; 37,243 = 37,243 hardlinks; byte-identical `meta.db` sha256) and again after (identical image/tag/volume ID **sets**, cache 2.782 GB/38 records, pg 4 DBs / 31 tables / 175,135,767 B, redis 2437). **`-X` is load-bearing** — overlayfs stacking rides `trusted.overlay.*`. End-to-end proof was a **real build** on the relocated store, `rc=0`. **k3s was never at risk and this was established before stopping anything:** it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up; only the two `jarr-*` dev containers were affected. `storageReserved` on SSD2 **0 → 80 GB**, still `Schedulable=True` at 76.34%. **A TRAP was found while proving the guard, and it is the reusable part: `RequiresMountsFor` on a path with NO mount unit is a SILENT NO-OP** — containerd started normally against an absent-but-unmounted path, so a typo'd guard buys nothing and says nothing (the *built-but-never-wired* shape again). The guard was therefore verified positively at the unit level (`Requires=` **and** `After=mnt-ssd_2.mount` on both units), and refusal was then proven with a genuinely absent **device** — via a temporary synthetic `.mount` unit, because `/mnt/ssd_2` hosts 12 live Longhorn replicas and must never be unmounted, and editing `fstab` on a production host risks emergency mode at boot: `Job containerd.service/start failed with result 'dependency'`, `is-active: inactive`. **Rollback is one documented sequence** (audit §11.8); the pre-move tree is **moved aside, not deleted**. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §11 | — |
| **R-209a** | **The SSD2 move has NOT survived a reboot, so by this project's own standard it is not fully validated** | **WATCHING — NEW 2026-08-05** | the next DooPlex reboot | **Operator ruled explicitly: do NOT reboot DooPlex.** Uptime verified unbroken (7 weeks 6 days, since 2026-06-10). **The distinction is stated rather than glossed: the MECHANISM is proven** — the guard is wired into both units and containerd refuses to start when a required mount's device is absent — **but the CONSEQUENCE is not**: that a real boot mounts `/mnt/ssd_2` before containerd starts, in this host's actual ordering. Mount-ordering reasoning is precisely the class this project has been burned by (`RequiresMountsFor` RE-MOUNTS rather than refusing — the ep0 lesson), and `CLAUDE.md` prefers a consequence assertion over a mechanism one. **Two deliberate consequences: (1)** the rollback copy `/var/lib/containerd.pre-move-2026-08-05` (**34.3 GB on `/`**) **STAYS** until a reboot validates — which is why `/` sits at 54% and not lower; deleting it now would trade a cheap 34 GB for the only cheap way back. **(2)** validation is **automatic and needs no one to remember it**: `felhom-store-postboot-check.service` (oneshot, enabled, dry-run PASS at install) runs at **every** boot and writes `RESULT: PASS`/`FAIL` to `/var/log/felhom-store-postboot-check.log`, asserting positively that `/mnt/ssd_2` is mounted, that containerd's root is on it, that **`/var/lib/containerd` does NOT exist** (the empty-store trap), that ≥100 images are visible and that both dev containers run. **Next action: after the next reboot — planned or not — read that file; on PASS, `rm -rf /var/lib/containerd.pre-move-2026-08-05` returns ~34 GB to `/`** | operator + CC | **P3's prune already removed the urgency: `/` went 86% → 53% used and SSD1's Longhorn disk went `Schedulable=False (DiskPressure)` → `Schedulable=True` (18.85% → 50.32% available).** The move was ruled "cap then move"; the cap is in and the pressure is gone, so this is now a deliberate choice rather than a rescue. **The numbers, measured (`Crucial-SSD-240G`, `/mnt/ssd_2/data/longhorn`, `storageMaximum` 235,148,750,848):** available today **214,958,080,000 (91.41%)**; 25% floor **58,787,187,712**. Moving the whole containerd tree at steady state (~31.5 GB images + ≤30 GB cache ≈ 65 GB) leaves **63.77%, i.e. +38.8 pp above the floor — comfortably safe as measured.** **But `storageScheduled` on SSD2 is 139,586,437,120 while `df` says only 20,094,939,136 is actually used** — Longhorn has overcommitted 6.9× — and if those volumes ever inflate to their scheduled size, the same disk lands at **13.00%, i.e. 12 pp BELOW the floor → `Schedulable=False`**, which is exactly the failure that just took SSD1 out. **Recommendation: do the move only together with setting `storageReserved` on SSD2 to cover the containerd tree (~80 GB); SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right.** Mechanism, if it goes ahead: **containerd's `root` in `/etc/containerd/config.toml`** (the key is present but commented out) — **not** Docker's `data-root`, which would move only 0.62 GB. Guard: `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`, remembering that **`RequiresMountsFor` RE-MOUNTS rather than refusing** ([[ep0-datastore-volume-move-2026-07-27]]) — so it must be tested with a genuinely absent device, and the move is not validated until it has survived a **reboot**. Full pre-analysis: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §P6 | operator |
| **R-210** | **Which of the 345 local images may be deleted — 131 controller tags and 62 hub tags exist ONLY on this box and are not recoverable by `docker pull`** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **Nothing was deleted; this is a list, not an action.** The registry was queried directly: `felhom-controller` has **76** tags in Gitea vs **207** locally, `felhom-hub` **45** vs **107**. The **131 + 62 local-only tags are all OLD** — controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`, hub `0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0` — while everything from controller `0.136.0` and hub `0.58.0` upward IS in the registry and therefore re-pullable. **Size the prize honestly before spending a decision on it:** per-tag sizes sum to 139.29 GB, but that double-counts shared layers — `docker system df` puts the **real** dedup'd image footprint at **31.02 GB with 27.02 GB reclaimable**, i.e. an order of magnitude less than the build cache P3 already returned. `docker image prune -a` would remove 343 of 345 (only `redis:7-alpine` and `postgres:16-alpine` are held by running containers). **CC's view: not worth doing for the space** — it buys ~27 GB against 199 GB now free, and its only real benefit is dropping unrecoverable clutter | operator |
| **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** | **READY (S) — NEW 2026-08-05** | — | Found while verifying R-205 rather than by looking for it. The `mon-system/prometheus` Deployment runs **one** container (`prom/prometheus:v3.12.0`) with **no `configmap-reload`/`prometheus-config-reloader` sidecar**. After the ArgoCD sync the updated `node-housekeeping-alerts.yml` was present **inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus **rules API still served the old expression** — for **4+ minutes**, with no error anywhere. It only took effect after an explicit `POST /-/reload`. **The consequence is general, not specific to R-205: every rule edit in this repo since the stack was built has silently not applied until something happened to restart the pod** — so "committed and synced" has never meant "in force", and ArgoCD reporting `Synced/Healthy` is true and beside the point. `--web.enable-lifecycle` IS already set, so the fix is small: add a reloader sidecar watching the ConfigMap, or a `checksum/config` pod annotation so a rules change rolls the pod. **Same class as the four *built-but-never-wired* seams** — the control exists, nothing walks it | CC |

## CAMPAIGN 12 — the class sweep, 2026-08-08 (unattended)

Eight rows, **grouped by class so the classes are visible as classes**. Full method, controls, blind
spots and the Part-4 gating ranking: `audits/CAMPAIGN-12-class-sweep-2026-08-08.md`. Gating candidates
are in `ROADMAP.md`, not here. **C1 produced no new instance** and has no row, deliberately.

**⚠ A correction the campaign owed to its own brief:** the task described C5's `escrow_stale` instance
as "closed individually". **It is not — it is R-247, `READY`.** The live repo is the source.

| ID | What | State |
|---|---|---|
| **R-256** | **C2 — „A mentéskezelő nem elérhető." names no route at all.** `web/offbox_handlers.go:47` and `:181` flash this to the customer on the off-site backup surface. It states an internal component's unavailability in the operator's vocabulary („mentéskezelő" = the backup Manager object), gives no reason the customer can act on, and names no next step — not "try again in a few minutes", not "contact support", not a page to go to. **Contrast, in the same subsystem and shipped the same week:** R-252's fix reads *„Meghajtók", „Meglévő meghajtó csatolása". Utána gyere vissza ide.* — a route. **Severity is low and stated so it is not over-ranked:** the condition is a nil backup manager, which on a healthy box does not occur; this is about the copy, not a broken path. Found by the C2 sample (19 refusals on the recovery/restore/offbox surface; **~202 of the repo's 221 refusal strings were NOT examined**) | **READY** — owner Viktor |
| **R-257** | **C2 — „Az offsite tároló nincs elárvult állapotban." puts an English loanword and an internal state name in front of a Hungarian household customer, and names no route.** `web/offbox_handlers.go:270` (the Go error it mirrors is `backup/offbox.go:343`). „Offsite" is untranslated; „elárvult állapot" is the codebase's own `OffboxOrphaned()` predicate surfacing verbatim. A customer who pressed a button and got this cannot tell whether something failed, whether they did something wrong, or what to do instead. **This is a refusal that is CORRECT and fail-closed and still a dead end** — the same shape R-241 recorded for `--recover-offsite-install`. **Fix shape, not a decision:** say what the customer tried to do, why it does not apply right now, and where to look — or, since this is a state they cannot reach deliberately, do not offer the action at all | **READY** — owner Viktor |
| **R-258** | **C3 — the customer's per-app backup tick is green on the PRESENCE of a restore point, and its only red condition is a GLOBAL one.** `web/handlers.go:1240-1248`: `row.Tier1LastStatus = "ok"` whenever `ListRestorePoints(app)` returns ≥1 point, and `"error"` only when `status.LastDBDump != nil && !status.LastDBDump.Success` — where `LastDBDump` is the box's **single most-recent DB dump** (`backup/backup.go:1064`, `m.lastDBDump`), **not this app's**. Three consequences, in increasing order of how wrong they look to a customer: (a) an app whose own last backup failed shows a **green tick** provided any older restore point exists and some *other* app dumped successfully afterwards; (b) an app with **no database at all** takes the `nil` branch and is green on presence alone; (c) the tick asserts nothing about **recency** — a restore point from three weeks ago is as green as one from last night, while `row.Tier1LastRun` beside it carries the real (old) time. **It is customer-visible:** `templates/backups_apps.html:173-174` renders it as a check or a cross. **This is the project's own "presence is not success" rule as a UI badge** — the artifact's existence is being read as a successful result, which is what the workspace `CLAUDE.md` section of that name forbids. **Found by a COMPLETE sweep, not a sample:** all 9 success-status assignment sites in the controller were read; this is the only new one. **NOT reproduced live** — source reading only | **READY** — owner Viktor |
| **R-259** | **C4 — a disk read that FAILS renders as „0.0 GB / 0.0 GB (0%)" in the nominal colour, on the dashboard's most-looked-at meter.** `system/info_linux.go:259-264`: `readDiskUsage` logs at DEBUG and **returns**, leaving the caller's `TotalGB`/`UsedGB`/`AvailGB`/`Percent` at their zero values. `dashboard.html:57-63` then renders `{{fmtGB .SystemInfo.DiskUsedGB}} / {{fmtGB .SystemInfo.DiskTotalGB}} ({{printf "%.0f" .SystemInfo.DiskPercent}}%)` plus a meter whose fill is `width:0%`, and `usageColor(0)` returns **`"nominal"`** (`web/funcmap.go:189-196`) — so **a failed measurement is drawn as a healthy, empty disk.** There is **no `*Known` companion for the system disk** (grep: 0). **The fix pattern already exists in this codebase, two files away, with the reasoning written out:** R-225's `StatsKnown` guards every derived off-site figure and its own comment says *"a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a claim"* (`backups_remote.html:60-62`). The dashboard does not use it. **It also travels:** `report/builder.go:94` puts the same zeroed `TotalGB`/`UsedGB` into the host report, so a failed statfs reaches the hub as a 0-of-0 disk. **Observation attached, not filed separately:** `readDiskUsage` is one of several collectors in that file that return silently on error; only this one was traced to a customer-visible surface. **NOT reproduced live** | **READY** — owner Viktor |
| **R-260** | **C5 — the agent reports at least eight decision-bearing facts the hub models NOWHERE, and the sharpest one blinds the check that answers „can the operator get into this box".** Measured 2026-08-08 by a tag-reachability test over all four wire directions (465 emitted json tags; a tag whose literal string occurs nowhere in the receiving repo cannot be decoded by any struct, named or anonymous). **THE ONE THAT MATTERS: `operator_key_configured`.** The agent emits it every heartbeat (`hub/report.go:178` — *"operator authorized_key installed"*); the hub's OOB decoder (`store/host_oob.go:38-45`) mirrors **5 of the agent's 8** OOB fields and has no field for it, nor for `wg_handshake_age_s` or `healed_at`. So `oobDegraded` (`monitor/host_oob.go:60-70`) tests config-invalid and (peer-configured AND not-active-or-not-reachable) — and **a box with felhom-sshd active, reachable, valid config, peer configured and NO OPERATOR KEY INSTALLED is `ok`.** The agent knows and says so; the hub throws it away. The checker's own doc comment claims it *"answers 'can the operator get into this box right now, and if not, why' proactively"* — see R-262's class. **Break-glass is the recovery chain (G1), which is what makes this the top row of the census.** **The rest of the census, hub-side absent (0 occurrences anywhere in `felhom.eu/hub/`):** `guest_net` — **the entire per-guest network-health object**, populated every cycle at `hub/collect.go:290`, carrying `dhclient_alive`, `has_route`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`; `selfupdate_pending` + `selfupdate_pending_version` — a staged-but-unapplied agent update is invisible to the fleet view; `healed_recently` (mgmt-plane); `operator_key_configured`, `wg_handshake_age_s`, `healed_at` (OOB); `mount_parity` + `mount_inventory` (restore-test, see R-262); `cpu_temp_c`, `loadavg`, `memory_total_bytes`, `memory_used_bytes`, `uptime_seconds` (host metrics); `model_name` (SMART); `applied_at` (PBS-DR). **Controller→hub, hub-side absent:** `reporting_disabled`, `config_hash`, `last_db_dump`, `last_integrity_check`, `migrated_to`, `catalog_ref`, `storage_bindings`, `container_path`, `subpath`, `temperature_celsius`, `load_avg_{1,5,15}`, `memory_{total,used}_mb`, `uptime_seconds`. **⚠ STATED PRECISELY, because it first looked worse than it is:** the hub stores the whole report as `report_json`, so nothing is lost from the DATABASE — every hub consumer, without exception, re-unmarshals it into a typed struct, so nothing reaches a check, an alarm, a notification or a screen. And a **restore-test mount-parity mismatch is NOT hidden**: it sets `res.Err` and returns before `res.Pass = true`, so `pass:false` travels; what is lost is the *depth* of a pass, not the failure. **The class is what this row is about, not any single field** — this is `escrow_stale` (R-247) for the fourth, fifth and sixth time, and §4 of the campaign report argues it is the cheapest high-value gate available to this project. **Blind spot of the method, recorded:** 29 generically-named tags (`name`, `state`, `status`, …) were excluded by name, so a drop of a generically-named field would be missed | **CLOSED 2026-08-08** — the class is GATED (G-1, `scripts/wire_contract_gate.py`) and the sharpest instance is fixed (hub v0.99.0). The remaining unconsumed facts are **R-264, OPEN** — allowlisted with reasons, which is not the same as decided. See the follow-through section below |
| **R-261** | **C6 — `CountSelfBindTokens` exists so that callers can assert an invariant, and no production caller asserts it.** `hub/internal/store/selfbind.go:106-111`. Its doc comment: *"it exists so callers can assert the 'after this runs, the only live link is one we just issued — or none' invariant that the auto-mint at customer-create / RESET-completion depends on."* **Census: only its own declaration in production; the two callers are `selfbind_automint_test.go:29` and `customer_delete_test.go:510`.** Tests are not callers (the campaign's rule), so the invariant the auto-mint *depends on* is checked in the test suite and never at the moment it matters. **This is the smallest of the eight rows and is filed at its true size, because the rest of the C6 sweep found INERT dead accessors rather than defects:** `OffboxOrphanedRenamedTo` and `OffboxEscrowState` have no caller but their data reaches the card another way (the template reads the settings field directly, `backups_remote.html:80`) — **R-228 is genuinely closed, and the sweep's first reading that it had regressed was wrong.** **The more consequential C6 result is a method result and is in the report, not here:** `golang.org/x/tools/cmd/deadcode` re-finds **neither** known instance, and a planted probe measured why — it reports an unreachable exported FUNCTION and not an unreachable exported METHOD on a widely-used type, and both known instances are methods | **READY** — owner Viktor |
| **R-262** | **C7 — a comment claims a cross-repo contract is mirrored „field-for-field" and „the key-set tests guard drift"; it is two fields short, AND THE FIXTURE THE TEST READS OMITS THE SAME TWO FIELDS.** `hub/internal/api/handler.go:682-687` covers `hostBackup` **and** `hostRestoreTest`. **It is TRUE of `hostBackup`** (verified field-for-field against `agent/internal/hub/Backup`). **It is FALSE of `hostRestoreTest`:** the agent emits `mount_parity` and `mount_inventory` (`hub/report.go:432-433`, populated in production from `reconcile/restoretest.go:277-283` via `backup/runner.go:517`), and the hub has no field for either — **0 occurrences in the entire hub repo** outside the CHANGELOG. **The guard is blind in exactly the place the drift is:** `TestHostReport_GoldenContract` reads `testdata/host-report.golden.json`, the two copies of which are byte-identical as required — and **neither contains `mount_parity` or `mount_inventory` at all**, so the key sets agree on a shape that is not the shape the agent sends. A test that cannot fail on the drift it names is the R-97b lesson (*prove the consequence, not the mechanism*) landing on a contract test. **Consequence, stated precisely:** the verdict is not lost (a parity mismatch fails the test before `Pass` is set), but the hub cannot distinguish a full-fidelity restore-test pass from a boot-only one, for any agent, ever. **Fix shape, not a decision:** add the two fields and put them in the fixture — or narrow the comment to name `hostBackup` only and say plainly that `hostRestoreTest` is a subset. **Attached observation:** the same fixture carries `cpu_temp_c` and `loadavg`, which no hub struct decodes — a fixture carrying keys the receiver cannot read is the same shape one level down | **READY** — owner Viktor |
| **R-263** | **C7 — „This is the ONLY writer of `StoragePath.BackupTarget`" is false, and nothing pins it.** `settings/settings.go:1317-1319`, on `SetBackupTarget`. **`ClearBackupTarget` (`:1358-1363`) also writes the field**, 17 lines below, in the same file. **The GUARANTEE the comment protects is intact and that is why this is filed small:** the sentence continues *"registration must never set it (E-2 §3: a drive never acquires a role by appearing)"*, and `ClearBackupTarget` only ever writes `false`, so no path other than `SetBackupTarget` **grants** the role. **What is wrong is the claim as written, and the absence of anything holding it:** `backup_target_role_test.go` exercises the behaviour and asserts nothing about writer uniqueness, so if a third writer appeared tomorrow — one that granted — the comment would still read as settled and the suite would still be green. **This is the class's own definition:** an invariant asserted in prose with no test pinning it. **Fix shape:** one word (*"the only writer that GRANTS the role"*) plus a test that fails when a second granting writer appears. **Method honesty:** found in a sample of **60 of 2652** production invariant comments — the "is the only" form only, chosen because a uniqueness claim is the one form a grep can falsify. **~2592 production and all 1440 test invariant comments were NOT examined**, so C7 has the weakest coverage of the seven classes and the task's instruction to check the tests' own claims is **owed, not discharged** | **READY** — owner Viktor |

## CAMPAIGN 12 follow-through — G-1 built, R-260 and R-247 closed, 2026-08-08

**The gate was built BEFORE the fixes and was seen failing on 40 fields**, captured verbatim in
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`. That order was the method, not
bureaucracy: the night before, `deadcode` was rejected for class C6 precisely because it was made to
prove itself first and found neither of the two defects it was meant for.

**⚠ A COUNT THIS SESSION'S OWN PROMPT GOT WRONG, corrected against the repo rather than quoted.** The
prompt said *"465 emitted tags, eight unreachable"*. R-260's wording was "at least eight
DECISION-BEARING facts", never eight tags in total, and its own census already listed more. Measured
on the three declared wires: **210 tags checked, 51 skipped, 40 convicted.** Two prompt claims were
wrong this week and both were caught the same way.

**Two things the gate's CONTROL caught before it was trusted**, each a defect in the instrument:

1. **A substring false negative.** `grep -F healed_at` also matches `privsep_healed_at`, so a
   genuinely dropped field read as received. R-260 named `healed_at`, so its absence from the output
   was the tell. Now a whole-token regex.
2. **`dr_recipe` is not wholly opaque.** The hub stores each half as `json.RawMessage` and re-emits
   nested shapes verbatim — but the TOP-LEVEL section keys are decoded by `hostHalfShape` /
   `appHalfShape`, and **those are allow-lists**: a section an emitter adds is silently dropped until
   named in both. That already cost `offsite_restic` (R-122). The gate is therefore opaque BELOW
   depth 1, so the sections are checked; treating the whole subtree as opaque would have put R-122's
   shape back outside its reach.

**THE FORTY, BY DISPOSITION.** Full per-field reasons live in the gate's own `ALLOWLIST`, where each
entry is a claim someone can re-check.

| # | field(s) | direction | decision | what changed |
|---|---|---|---|---|
| 1 | `oob.operator_key_configured` | agent → hub | **RECEIVE AND ACT ON IT** | decoded as a pointer; `oobDegraded` now fails when the key is absent, and the alert NAMES it. Hub v0.99.0 |
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **RECEIVE, for the message only** | decoded into `HostOOBRow` and put in the event payload; deliberately NOT in the predicate — widening the check beyond the fact that is now arriving is how a check stops being read |
| 3 | `escrow_stale` | hub → controller | **RECEIVE AND ACT ON IT** | `report.EscrowStatus.Stale`; the box tells a withheld hash from a hash-less one. Controller v0.209.0. **This is R-247** |
| 4 | `host.{cpu_temp_c,loadavg,memory_total_bytes,memory_used_bytes,uptime_seconds}`, `system.{load_avg_1,5,15, memory_total_mb, memory_used_mb, temperature_celsius, uptime_seconds}` | agent/controller → hub | **NO CONSUMER WANTED — redundant** | allowlisted: the hub decodes `cpu_percent` / `memory_percent` / `disk_percent` from the same stanzas and every threshold is expressed on those |
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: guest sizing is hub-owned INTENT (the manifest), not mirrored reality |
| 6 | `storage_targets.smart.model_name` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: a display label with no threshold on it; `smart.health` and every counter the hub bands on ARE decoded |
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **NO CONSUMER WANTED — redundant** | allowlisted: wgsync reconciles from its own state, and the OOB path's own handshake age is now decoded |
| 8 | `guest_net` + its 7 children, `selfupdate_pending(_version)`, `mgmt_plane.healed_recently`, `pbs_dr.applied_at`, `restore_tests.mount_{parity,inventory}`, `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check` | agent/controller → hub | **NO CONSUMER TODAY, AND ONE IS ARGUABLY OWED** | allowlisted **against R-264, which stays OPEN**. Allowlisting is not deciding, and the entries say so |

**A C7 instance found while doing it, and corrected.** `HostReport.SelfUpdatePending`'s own comment
claimed *"The hub reads an absent field as pending=false, the correct default."* The hub has no field
for it and reads nothing either way. Comment corrected in the agent (no version bump — comment only).

**The hub's OOB test fixture was part of why this survived.** `oobReport()` omitted
`operator_key_configured` entirely, so every pre-existing scenario ran against a report shape **no
released agent produces**. Fixed, and the new tests drive the raw JSON decode boundary — a test that
builds the receiving struct by hand cannot see a field that never decodes, which is the whole class.

| ID | What | State |
|---|---|---|
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change | **READY** — owner Viktor |

**Explicitly still open, untouched by this session:** R-246 (the wrong stale flag on `demo-hp` —
clearing it is an operator act hub-side), R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and
**C7's test-comment half**, which Campaign 12 recorded as *owed, not done* (60 of 2652 production
invariant comments sampled; none of the 1440 test comments).

## Why the TOP READY rows rank this way


This covers the next few only — it is deliberately **not** a full ordering of the table above, so that
there is one ranking to maintain rather than two.

1. **R-95** — the largest *data* exposure: the tier holding the customer's documents and photos is the
   one whose credential can delete. The snapshot mitigation is now **armed** (daily 00:00, keep 7),
   but it has taken zero snapshots so far and it does not touch the root cause — the box can still
   `forget --prune` its own repo.
2. **R-94** — **de-ranked 2026-07-29.** The prior rationale ("until it moves every hub-driven
   install gets the pre-R-82 default") was false: the constant selects no script and every install
   already fetches 1.22.0. What remains is a wrong label plus two pieces of dead safety equipment —
   a drift gate nobody runs and a test that compares a constant to itself. Cheap and worth doing;
   not high-consequence, and it blocks nothing.
3. ~~**R-86**~~ — **CLOSED 2026-08-03**, agent v0.121.0 + hub v0.91.0, proven live on demo-felhom.
4. **R-87** — **re-ranked UP**: R-86 built most of what it was waiting for (per-archive due-ness, a
   proof that names its archive, and a staleness window that learns a tier's rhythm). What is left is
   restic-specific — there is no scratch-guest analogue — so it still needs its own design, but it is
   no longer waiting on a scheduling model that did not exist.
5. ~~**R-185**~~ — **CLOSED 2026-08-03**, agent v0.123.0 + installer 1.24.0, proven live on both demo
   boxes. The silence was fixed as well as the grant: the box now asks whether it may READ each tier
   it depends on, because an empty listing cannot distinguish forbidden from newborn.
6. ~~**R-189**~~ — **CLOSED 2026-08-03** with **R-188** and **R-186**, agent v0.122.0. The three
   reporting/release signals that misreported their own work are fixed; **R-185 is the one that
   remains open from that group** and is untouched by this — it is a missing storage ACL on
   demo-felhom, not a reporting defect.
7. **R-110** — last **because it is not a READY row**: the ruling is the operator's, not CC's, and
   there is nothing for CC to build until it lands. Ranked here rather than omitted because it is the
   only item on this page about the *publish channel* of the most privileged artifact Felhom ships,
   and today's exposure is zero — which makes now the cheapest moment it will ever be to decide.

### The 2026-08-02 intake (R-156 … R-164), ranked

Filed in one pass from Campaign 10, its two spikes, and the 53-template catalog persistence sweep.
**R-156 and R-157 had lived only in audit documents** — the identical *"minted in a spike doc and
never carried across"* failure the register already records for R-153/R-154/R-155, caught by the
sweep's own §8.0 while it was happening. R-158 was minted by a second session on the same day for an
unrelated finding, which is why the sweep's proposals were renumbered to R-159…R-162 at filing time.

1. **R-157** — highest: a `deployed: true` app can stay **down indefinitely** after a power cut or
   hard reset, and in mechanism **B** nothing reports it on any channel (`0 currently down`). It is the
   only row here where the customer loses service and has no signal at all.
2. **R-156** — the class is now detectable and two of three apps are fixed; what remains is **papra's
   referral**, one app, well understood. *(Promoted 2026-08-02: R-161 was ranked here because nothing
   ran the gate; it now has a mandated entry point, so R-156's residue is the larger remaining item.)*
4. **R-163** — a real ceiling that silently caps local backup once an app outgrows `mp1`, and it gates
   Tier-2 and Tier-3 as well. Ranked below the above only because **overflow itself is safe today** —
   it refuses per app and preserves the last good unit byte-identical. **RE-FRAMED 2026-08-02:** no
   longer waiting on a ratio — decision **D-a** merges `mp1` away, so the row is now the record of the
   constraint and the work moves to **R-165** (with **R-167** shipping in the same step). R-165 inherits
   this rank; it is the highest-ranked item that must land **before any external install**.
5. **R-158** — the gap that makes R-163 dangerous: cross the size line and **one page** tells you. On
   its own it is a notification gap, not a silent failure, which is why it sits here and not higher.
6. **R-164** — blocked on a predicate, no customer impact today; it only becomes urgent if the unit
   size in R-163 is judged unacceptable, since the tar-drop is the cheapest way to halve it.
7. **R-161** — **de-ranked 2026-08-02, ruled and shipped at reduced scope.** The gate now has one
   mandated entry point (`catalog_gates.py`), which is the shape that actually gets run here. What is
   left is the automatic half, and that is sufficient while **one** person touches templates — so it
   ranks low by design, not by neglect. Revisit when a second does.
8. **R-162** — `WATCHING` only. A limitation that fails closed; revisit if a non-overlay driver ships.

**R-159 and R-160 are SHIPPED** and are not ranked; they are filed to record the class, and R-159's
class (an image `VOLUME` at an unmounted path) is still live — `immich-server` has one today.
