felhom.eu/documentation/audits/R106-R109-recipe-completeness-2026-07-30.md

# AUDIT — R-106 + R-109 (+ R-122): the DR recipe stops guessing (2026-07-30)

Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
(`felhom.eu` `acfc2b7`). Live on **both** demo boxes.

---

## 1. Baselines, established separately at the start of the session

| component | value | how |
|---|---|---|
| agent — demo-felhom | **0.115.0** | `ssh felhom-pve "felhom-agent --version"` |
| agent — demo-hp | **0.117.0** | `ssh demo-hp "felhom-agent --version"` — R-117's §5.1 step HAD landed here |
| agent — source HEAD | v0.117.0 | top of `felhom-agent/CHANGELOG.md` |
| controller (live/golden) | 0.186.0 vouched as golden | `hub_settings.artifact_golden_version` |
| hub | **0.82.0** | `style.css?v=` + `deploy/hub` image tag |
| `felhom.eu` HEAD | **`3d504d5`** | as the task expected |
| R-121 | not moved this session | — |

Fleet was split three ways (0.115.0 / 0.116.0 / 0.117.0) before this session; both demo boxes are now
0.118.1. `sess-f` (0.116.0) and `drill-r50` (0.113.0) were deliberately **not** touched.

---

## 2. Part 0 — the set and the owner

### 2.1 The real open set, and two items missing from the register

`OPEN-ITEMS.md:1` declares itself "the single source of truth for open work"; `ROADMAP.md` keeps
"the full history and reasoning".

| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|---|---|---|---|
| R-105 | `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
| R-106 | `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
| R-108 | `:111` | `:50` | registered |
| R-109 | `:112` | `:61` | registered |

R-109's own cell — *"third recipe-completeness defect beside R-105/R-106"* — was the **only** mention of
either in the register. That is precisely the thread-loss the register exists to prevent → filed **R-123**.

**The set closed here is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
**R-105 is NOT in it and was not worked**: it is M-sized and about three *hub-held* DR records being `{}`
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third already closed by the
2026-07-28 target move). Different fields, different owner, different size. It still needs an OPEN-ITEMS row.

### 2.2 Where the recipe is generated — THREE producers, not two

| half | repo | function |
|---|---|---|
| host — guests/pbs/drives/pve_storage | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
| app — customer/apps/offsite_restic | `felhom-controller` | `controller/internal/report/dr_recipe.go:35-42` |
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`; route `internal/web/server.go:439` |

R-109's "host-half" is the **agent** — and the field must ALSO pass the **hub's** allow-list (§4).

### 2.3 What the namespace field contained — the eleven-session-old brief was RIGHT

Live pre-fix, both boxes: `"namespace": "root"`. Against `/etc/pve/storage.cfg` on the same boxes:
`namespace demo-felhom` / `namespace demo-hp`.

Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises
empty → `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.

**Authority taken:** storage.cfg's `namespace` on the pbs storage — the same field `vzdump --storage <pbs>`
makes PVE read, and the source of the agent's own verify client (`cmd/felhom-agent/main.go:1164`).

---

## 3. R-109's ambiguity, in the boxes' own pre-fix recipe

```json
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
{ "name": "local",         "type": "local",     "content": "backup,import,vztmpl,iso" }
```

No `backup_target` key. **And the ambiguity is not theoretical** — on disk, same day:

| box | `felhom-backup` newest archive | `local` newest archive |
|---|---|---|
| demo-felhom | `2026-07-30 04:36` (6.1 GB) | **`2026-07-28 17:58`** (5.9 GB) |
| demo-hp | `2026-07-30 04:36` (2.3 GB) | **`2026-07-28 17:44`** (1.7 GB) |

`local` froze at the target-move date on both boxes and has never been refreshed since. A restorer picking
it gets a guest that restores cleanly and is silently stale — two days at the time of writing, unbounded
thereafter. That is the worst shape a backup defect can take, and it is what the recipe left to a guess.

---

## 4. R-122 — a FOURTH defect, found here, already shipped

`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists dressed as forward-compat**. The
controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*);
`appHalfShape` never listed the key. Verified both ways:

- **stored** — `dr_recipe.app_half_json` carries it for all three real customers (`peti-felhom`,
  `demo-felhom`, `demo-hp`).
- **delivered** — the pre-fix download's top-level keys were
  `recipe_version, customer, guests, pbs, drives, pve_storage, apps`. **No `offsite_restic`.**

So a restorer had **no offsite location at all**, for the feature's whole life, with a green suite
throughout — because the fixture `drAppHalf` is hand-written and omits the field. R-109's new
`backup_target` would have been swallowed identically, making the fix read as shipped while changing
nothing an operator could see.

**Deviation from the task's §7.10 ("findings filed, none fixed"), stated rather than absorbed:** I fixed
it. Part 0 authorises working the real set; it is the same symptom the task is named for and the worst
instance of it; and it lives in the same two structs R-109 forced me to edit. Filed as R-122, SHIPPED.

---

## 5. Before / after, quoted, same boxes

### demo-felhom

```
BEFORE  agent 0.115.0, hub 0.82.0
  pbs            {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
  backup_target  (absent)
  offsite_restic (absent)

AFTER   agent 0.118.1, hub 0.83.0
  pbs            {"repo_id":"felhom-pbs","namespace":"demo-felhom","namespace_state":"resolved","latest_snapshot_id":"9201"}
  backup_target  {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/hdd_1"}
  offsite_restic {"host":"u629488-sub1.your-storagebox.de","user":"u629488-sub1","port":23,"repo_path":"/home/felhom-repo"}
```

### demo-hp

```
BEFORE  agent 0.117.0, hub 0.82.0
  pbs            {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
  backup_target  (absent)
  offsite_restic (absent)

AFTER   agent 0.118.1, hub 0.83.0
  pbs            {"repo_id":"felhom-pbs","namespace":"demo-hp","namespace_state":"resolved","latest_snapshot_id":"9201"}
  backup_target  {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/nvme-1tb"}
  offsite_restic {"host":"u629488-sub3.your-storagebox.de","user":"u629488-sub3","port":23,"repo_path":"/home/felhom-repo"}
```

**The two boxes disagreeing is the point** — different namespaces, different mountpoints, different restic
sub-accounts. Nothing is hardcoded; each value is resolved from that box's own configuration. Method:
`GET /customers/<id>/dr-recipe.json` against the hub ClusterIP with operator Basic auth — the exact
endpoint the operator UI's download link invokes (no browser on DooPlex).

---

## 6. v0.118.0 shipped an INERT half of R-106, and live validation is what caught it

Deployed 0.118.0 to demo-felhom, the recipe read:

```json
"pbs": { "namespace": "root", "namespace_state": "resolved", ... }
```

`resolved` **and wrong** — strictly worse than the original defect, because it asserts confidence in a
wrong answer. The new state field is what made it legible at a glance; without it this would have been
indistinguishable from the pre-fix output and would have shipped as "done".

**Cause:** `mergeConfig` (`internal/storage/observe.go:457`) overlays the CLUSTER storage config onto the
NODE entry through a hand-listed field set — `Type, Server, Export, Share, Datastore, Fingerprint, VGName,
ThinPool, Path, Content` — and `Namespace` was not on it. `NodeStorage` does not return the namespace at
all, so the merged entry's was always empty. The recipe logic was correct and fed a wrong input.

**Why the tests missed it:** every v0.118.0 test constructs `StorageTarget` values directly — *including
the two that run `Collector.Collect()`*, because they inject a `fakeObserver`. What I had called "the
production generation path" started one layer below the break. Fixed with
`TestObserve_CarriesPBSNamespaceThroughMerge` (real `Observe`, PVE's actual split returns reproduced) and
`TestMergeConfig_CarriesPBSNamespace` (the merge tabled, fill-if-empty vs never-clobber).

**The transferable lesson:** "a test through the production path" is only true up to the seam you inject
at. Naming the injected seam — here `fakeObserver` — makes the untested span visible; not naming it is how
a fix ships inert. `Username` is the one remaining unmerged field; the comment now says to add it here the
moment anything consumes it.

---

## 7. Tests

**Agent (11 new).** Consequence-level: a box with two `content=backup` storages names the LIVE one and not
the frozen one; `assertBackupCandidateAmbiguity` **fails the test if the fixture ever stops posing that
problem**, so it cannot pass hollow. Fixtures are the storage set demo-felhom really had, provenance
recorded in-file (the box's own pre-fix recipe + its `storage.cfg`). Wrong cases: unresolvable → explicit
unknown with no `storage_id`/`mount_path` key at all and no leaked candidate value; missing storage → the
other reason with the configured id recorded; no pbs storage → namespace unknown, never defaulted to root.

**Hub (3 new).** Built on halves read **verbatim out of the live `dr_recipe` table**.
`TestAssembleDRRecipe_CarriesEveryEmittedSection` enumerates every section both emitters produce and fails
on any dropped — the guard the allow-list needed and never had.

**Red-proofs: 7**, each mutation asserted to have landed *before* the run:

| # | mutation | caught by |
|---|---|---|
| 1 | namespace from the snapshot again | the live `"root"` symptom reproduces, unit + production-path |
| 2 | unknown target guesses a `content=backup` storage | 5 assertions |
| 3 | `backup_target` never populated | 3 tests + the cross-repo contract guard |
| 4 | decoy loses `content=backup` (hollow fixture) | the ambiguity guard |
| 5 | `offsite_restic` off the allow-list (R-122 restored) | 2 tests |
| 6 | `backup_target` off the allow-list | 4 tests, naming the section |
| 7 | `Namespace` off `mergeConfig` (the §6 gap) | both new observer tests |

Suites run **separately from every commit**: agent rc=0 / 29 packages / 0 FAIL; hub `vet` rc=0, suite
rc=0 / 17 packages / 0 FAIL.

---

## 8. Publish observables — treated as a gate

| # | observable | result |
|---|---|---|
| 1 | published at the new version | `GET .../generic/felhom-agent/0.118.1/felhom-agent` → **HTTP 200**, 14043066 bytes, sha256 `9a74c87d…`; the **downloaded** binary reports `felhom-agent 0.118.1`. Anonymous fetch, independent of the publish script |
| 2 | the manifest vouches it | `hub_settings.artifact_agent_version = 0.118.1`, `artifact_agent_sha256 = 9a74c87d…` — **the hub re-derived the sha from Gitea itself**, matching the build exactly. `golden 0.186.0`, `min_agent 0.113.0`, `wrapper_sha 104db0a4…` all carried through unchanged (the handler rewrites every field; omitting one clears it) |
| 3 | the box reports it running | hub-side `hosts.agent_version`: `demo-felhom-8363b5 = 0.118.1`, `demo-hp-bb76ea = 0.118.1` — asked of the hub, not of the box |

The vouch normally sits with the operator (the `felhom-build-deploy` skill calls it an operator
follow-up); **it was performed here on the operator's explicit in-session approval**, since the task makes
the three observables a gate. R-120's golden gate was checked before submitting and passes
(golden `0.186.0` == newest fleet-reported controller `0.186.0`).

Hub deploy: ArgoCD app `felhom` hard-refreshed → `OutOfSync` at `acfc2b7` → synced → **Synced/Healthy**,
rollout complete, live image `felhom-hub:0.83.0`, clean startup log.

---

## 9. Findings

| id | finding | disposition |
|---|---|---|
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life | **FIXED** (hub 0.83.0) — deviation stated in §4 |
| **R-123** | R-105 and R-106 were `READY` in ROADMAP with no OPEN-ITEMS row | R-106 registered + shipped; **R-105 still needs a row**, and nothing checks the invariant |
| **R-124** | the recipe spells PBS's root namespace `"root"`, but the API spells it `""` and no namespace is literally named `root` — `pct restore --ns root` would fail | **OPEN (XS)**, deliberately not changed mid-fix; affects only a box with no `namespace` line, i.e. no real customer today |
| **R-125** | v0.118.0 shipped an inert R-106 because "the production path" test injected `fakeObserver` one layer below the break (§6) | **the fix is shipped**; filed for the doctrine point — a production-path claim must name the seam it injects at |

---

## 10. What did NOT happen, and why

- **R-105, R-108, D5** — out of scope per the task's §6. R-108 blocks D5; starting either would leave both half-done.
- **The backup machinery** — untouched. This corrects the RECORD, not the doing. And the record was never
  acted on programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`, *"the
  destructive restore is a separate, operator-present step"*), so the blast radius was exactly one
  operator reading a wrong value. `plan.PBS` inherits the fix for free.
- **`sess-f` (agent 0.116.0) and `drill-r50` (0.113.0) not upgraded** — neither was named as a venue and
  `drill-r50` is fenced by §6.
- **No golden re-bake, no guest restart, no destructive operation** — none was needed.
- **`R-124` not fixed** — changing the wire's spelling of the root namespace mid-R-106 would have shifted
  the field's meaning during the very fix meant to make it trustworthy.