felhom.eu/documentation/audits/R106-R109-recipe-completeness-2026-07-30.md

AUDIT — R-106 + R-109 (+ R-122): the DR recipe stops guessing (2026-07-30)

Shipped: agent v0.118.0 → v0.118.1 (felhom-agent 1c8a67e, 6b5dade) + hub v0.83.0 (felhom.eu acfc2b7). Live on both demo boxes.

---

1. Baselines, established separately at the start of the session

component	value	how
agent — demo-felhom	0.115.0	ssh felhom-pve "felhom-agent --version"
agent — demo-hp	0.117.0	ssh demo-hp "felhom-agent --version" — R-117's §5.1 step HAD landed here
agent — source HEAD	v0.117.0	top of felhom-agent/CHANGELOG.md
controller (live/golden)	0.186.0 vouched as golden	hub_settings.artifact_golden_version
hub	0.82.0	style.css?v= + deploy/hub image tag
felhom.eu HEAD	3d504d5	as the task expected
R-121	not moved this session	—

Fleet was split three ways (0.115.0 / 0.116.0 / 0.117.0) before this session; both demo boxes are now 0.118.1. sess-f (0.116.0) and drill-r50 (0.113.0) were deliberately not touched.

---

2. Part 0 — the set and the owner

2.1 The real open set, and two items missing from the register

OPEN-ITEMS.md:1 declares itself "the single source of truth for open work"; ROADMAP.md keeps "the full history and reasoning".

item	ROADMAP.md	OPEN-ITEMS.md	verdict
R-105	READY — 2026-07-28 (:108)	absent	open but UNREGISTERED
R-106	READY — 2026-07-28 (:109)	absent	open but UNREGISTERED
R-108	:111	:50	registered
R-109	:112	:61	registered

R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the only mention of either in the register. That is precisely the thread-loss the register exists to prevent → filed R-123.

The set closed here is R-106 + R-109, matching the arc's stated definition of done (OPEN-ITEMS.md:14). R-105 is NOT in it and was not worked: it is M-sized and about three hub-held DR records being {} (hosts.dr_record_json, host_escrow.directive_json, and the drives third already closed by the 2026-07-28 target move). Different fields, different owner, different size. It still needs an OPEN-ITEMS row.

2.2 Where the recipe is generated — THREE producers, not two

half	repo	function
host — guests/pbs/drives/pve_storage	felhom-agent	BuildDRRecipeHostHalf, internal/hub/dr_recipe.go:86
app — customer/apps/offsite_restic	felhom-controller	controller/internal/report/dr_recipe.go:35-42
assembly + delivery	felhom.eu/hub	AssembleDRRecipe, internal/store/dr_recipe.go:104; served by handleDRRecipeDownload, internal/web/dr_recipe.go:14; route internal/web/server.go:439

R-109's "host-half" is the agent — and the field must ALSO pass the hub's allow-list (§4).

2.3 What the namespace field contained — the eleven-session-old brief was RIGHT

Live pre-fix, both boxes: "namespace": "root". Against /etc/pve/storage.cfg on the same boxes: namespace demo-felhom / namespace demo-hp.

Traced to source: Snapshot.Namespace decodes ns (internal/pbs/client.go:97), which PBS does not echo per item once the list is namespace-scoped via ?ns= (:118-120) → always empty → ToHub normalises empty → "root" (internal/pbs/report.go:22-25) → latestPBSCoord writes it in.

Authority taken: storage.cfg's namespace on the pbs storage — the same field vzdump --storage <pbs> makes PVE read, and the source of the agent's own verify client (cmd/felhom-agent/main.go:1164).

---

3. R-109's ambiguity, in the boxes' own pre-fix recipe

{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
{ "name": "local",         "type": "local",     "content": "backup,import,vztmpl,iso" }

No backup_target key. And the ambiguity is not theoretical — on disk, same day:

box	felhom-backup newest archive	local newest archive
demo-felhom	2026-07-30 04:36 (6.1 GB)	2026-07-28 17:58 (5.9 GB)
demo-hp	2026-07-30 04:36 (2.3 GB)	2026-07-28 17:44 (1.7 GB)

local froze at the target-move date on both boxes and has never been refreshed since. A restorer picking it gets a guest that restores cleanly and is silently stale — two days at the time of writing, unbounded thereafter. That is the worst shape a backup defect can take, and it is what the recipe left to a guess.

---

4. R-122 — a FOURTH defect, found here, already shipped

AssembleDRRecipe's hostHalfShape/appHalfShape are allow-lists dressed as forward-compat. The controller has emitted offsite_restic since fork-4 ("so DR knows WHERE to recover from"); appHalfShape never listed the key. Verified both ways:

• stored — dr_recipe.app_half_json carries it for all three real customers (peti-felhom, demo-felhom, demo-hp).
• delivered — the pre-fix download's top-level keys were recipe_version, customer, guests, pbs, drives, pve_storage, apps. No offsite_restic.

So a restorer had no offsite location at all, for the feature's whole life, with a green suite throughout — because the fixture drAppHalf is hand-written and omits the field. R-109's new backup_target would have been swallowed identically, making the fix read as shipped while changing nothing an operator could see.

Deviation from the task's §7.10 ("findings filed, none fixed"), stated rather than absorbed: I fixed it. Part 0 authorises working the real set; it is the same symptom the task is named for and the worst instance of it; and it lives in the same two structs R-109 forced me to edit. Filed as R-122, SHIPPED.

---

5. Before / after, quoted, same boxes

demo-felhom

BEFORE  agent 0.115.0, hub 0.82.0
  pbs            {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
  backup_target  (absent)
  offsite_restic (absent)

AFTER   agent 0.118.1, hub 0.83.0
  pbs            {"repo_id":"felhom-pbs","namespace":"demo-felhom","namespace_state":"resolved","latest_snapshot_id":"9201"}
  backup_target  {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/hdd_1"}
  offsite_restic {"host":"u629488-sub1.your-storagebox.de","user":"u629488-sub1","port":23,"repo_path":"/home/felhom-repo"}

demo-hp

BEFORE  agent 0.117.0, hub 0.82.0
  pbs            {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
  backup_target  (absent)
  offsite_restic (absent)

AFTER   agent 0.118.1, hub 0.83.0
  pbs            {"repo_id":"felhom-pbs","namespace":"demo-hp","namespace_state":"resolved","latest_snapshot_id":"9201"}
  backup_target  {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/nvme-1tb"}
  offsite_restic {"host":"u629488-sub3.your-storagebox.de","user":"u629488-sub3","port":23,"repo_path":"/home/felhom-repo"}

The two boxes disagreeing is the point — different namespaces, different mountpoints, different restic sub-accounts. Nothing is hardcoded; each value is resolved from that box's own configuration. Method: GET /customers/<id>/dr-recipe.json against the hub ClusterIP with operator Basic auth — the exact endpoint the operator UI's download link invokes (no browser on DooPlex).

---

6. v0.118.0 shipped an INERT half of R-106, and live validation is what caught it

Deployed 0.118.0 to demo-felhom, the recipe read:

"pbs": { "namespace": "root", "namespace_state": "resolved", ... }

resolved and wrong — strictly worse than the original defect, because it asserts confidence in a wrong answer. The new state field is what made it legible at a glance; without it this would have been indistinguishable from the pre-fix output and would have shipped as "done".

Cause: mergeConfig (internal/storage/observe.go:457) overlays the CLUSTER storage config onto the NODE entry through a hand-listed field set — Type, Server, Export, Share, Datastore, Fingerprint, VGName, ThinPool, Path, Content — and Namespace was not on it. NodeStorage does not return the namespace at all, so the merged entry's was always empty. The recipe logic was correct and fed a wrong input.

Why the tests missed it: every v0.118.0 test constructs StorageTarget values directly — including the two that run Collector.Collect(), because they inject a fakeObserver. What I had called "the production generation path" started one layer below the break. Fixed with TestObserve_CarriesPBSNamespaceThroughMerge (real Observe, PVE's actual split returns reproduced) and TestMergeConfig_CarriesPBSNamespace (the merge tabled, fill-if-empty vs never-clobber).

The transferable lesson: "a test through the production path" is only true up to the seam you inject at. Naming the injected seam — here fakeObserver — makes the untested span visible; not naming it is how a fix ships inert. Username is the one remaining unmerged field; the comment now says to add it here the moment anything consumes it.

---

7. Tests

Agent (11 new). Consequence-level: a box with two content=backup storages names the LIVE one and not the frozen one; assertBackupCandidateAmbiguity fails the test if the fixture ever stops posing that problem, so it cannot pass hollow. Fixtures are the storage set demo-felhom really had, provenance recorded in-file (the box's own pre-fix recipe + its storage.cfg). Wrong cases: unresolvable → explicit unknown with no storage_id/mount_path key at all and no leaked candidate value; missing storage → the other reason with the configured id recorded; no pbs storage → namespace unknown, never defaulted to root.

Hub (3 new). Built on halves read verbatim out of the live dr_recipe table. TestAssembleDRRecipe_CarriesEveryEmittedSection enumerates every section both emitters produce and fails on any dropped — the guard the allow-list needed and never had.

Red-proofs: 7, each mutation asserted to have landed before the run:

#	mutation	caught by
1	namespace from the snapshot again	the live "root" symptom reproduces, unit + production-path
2	unknown target guesses a content=backup storage	5 assertions
3	backup_target never populated	3 tests + the cross-repo contract guard
4	decoy loses content=backup (hollow fixture)	the ambiguity guard
5	offsite_restic off the allow-list (R-122 restored)	2 tests
6	backup_target off the allow-list	4 tests, naming the section
7	Namespace off mergeConfig (the §6 gap)	both new observer tests

Suites run separately from every commit: agent rc=0 / 29 packages / 0 FAIL; hub vet rc=0, suite rc=0 / 17 packages / 0 FAIL.

---

8. Publish observables — treated as a gate

#	observable	result
1	published at the new version	GET .../generic/felhom-agent/0.118.1/felhom-agent → HTTP 200, 14043066 bytes, sha256 9a74c87d…; the downloaded binary reports felhom-agent 0.118.1. Anonymous fetch, independent of the publish script
2	the manifest vouches it	hub_settings.artifact_agent_version = 0.118.1, artifact_agent_sha256 = 9a74c87d… — the hub re-derived the sha from Gitea itself, matching the build exactly. golden 0.186.0, min_agent 0.113.0, wrapper_sha 104db0a4… all carried through unchanged (the handler rewrites every field; omitting one clears it)
3	the box reports it running	hub-side hosts.agent_version: demo-felhom-8363b5 = 0.118.1, demo-hp-bb76ea = 0.118.1 — asked of the hub, not of the box

The vouch normally sits with the operator (the felhom-build-deploy skill calls it an operator follow-up); it was performed here on the operator's explicit in-session approval, since the task makes the three observables a gate. R-120's golden gate was checked before submitting and passes (golden 0.186.0 == newest fleet-reported controller 0.186.0).

Hub deploy: ArgoCD app felhom hard-refreshed → OutOfSync at acfc2b7 → synced → Synced/Healthy, rollout complete, live image felhom-hub:0.83.0, clean startup log.

---

9. Findings

id	finding	disposition
R-122	AssembleDRRecipe allow-list dropped offsite_restic for the feature's whole life	FIXED (hub 0.83.0) — deviation stated in §4
R-123	R-105 and R-106 were READY in ROADMAP with no OPEN-ITEMS row	R-106 registered + shipped; R-105 still needs a row, and nothing checks the invariant
R-124	the recipe spells PBS's root namespace "root", but the API spells it "" and no namespace is literally named root — pct restore --ns root would fail	OPEN (XS), deliberately not changed mid-fix; affects only a box with no namespace line, i.e. no real customer today
R-125	v0.118.0 shipped an inert R-106 because "the production path" test injected fakeObserver one layer below the break (§6)	the fix is shipped; filed for the doctrine point — a production-path claim must name the seam it injects at

---

10. What did NOT happen, and why

• R-105, R-108, D5 — out of scope per the task's §6. R-108 blocks D5; starting either would leave both half-done.
• The backup machinery — untouched. This corrects the RECORD, not the doing. And the record was never acted on programmatically: the host_loss plan is surface-only (internal/dr/plan.go:125, "the destructive restore is a separate, operator-present step"), so the blast radius was exactly one operator reading a wrong value. plan.PBS inherits the fix for free.
• sess-f (agent 0.116.0) and drill-r50 (0.113.0) not upgraded — neither was named as a venue and drill-r50 is fenced by §6.
• No golden re-bake, no guest restart, no destructive operation — none was needed.
• R-124 not fixed — changing the wire's spelling of the root namespace mid-R-106 would have shifted the field's meaning during the very fix meant to make it trustworthy.