From 70f84941d4d89ff9cbbf0d09121e7d7aee21bf1b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 30 Jul 2026 13:28:10 +0200 Subject: [PATCH] R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes, the on-disk proof that `local` froze at the 2026-07-28 target move while felhom-backup kept running, all seven red-proofs, and the three publish observables. R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real Collector.Collect() but both injected a fakeObserver, and the break was one layer below in mergeConfig, which dropped the pbs namespace. The recipe still said "root" — now with namespace_state "resolved" beside it, confident and wrong. Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for the doctrine point that a production-path claim must name the seam it injects at. --- REPORT-r106-r109-recipe-completeness.md | 29 +- ...106-R109-recipe-completeness-2026-07-30.md | 250 ++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 5 +- documentation/backlog/ROADMAP.md | 4 +- 4 files changed, 280 insertions(+), 8 deletions(-) create mode 100644 documentation/audits/R106-R109-recipe-completeness-2026-07-30.md diff --git a/REPORT-r106-r109-recipe-completeness.md b/REPORT-r106-r109-recipe-completeness.md index d29b277..eda86ad 100644 --- a/REPORT-r106-r109-recipe-completeness.md +++ b/REPORT-r106-r109-recipe-completeness.md @@ -2,7 +2,12 @@ Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched. -Shipped: **agent v0.118.0** (`felhom-agent` `1c8a67e`) + **hub v0.83.0**. Neither half is useful alone. +Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0** +(`felhom.eu` `acfc2b7`). Neither half is useful alone. + +**Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is +what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full +account in the audit §6, filed as **R-125**. ## Part 0 — the answers, before the fix @@ -96,9 +101,23 @@ symptom the task is named for (the recipe is incomplete), and the worst instance (c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition. -## 3. The before/after recipe — same box, quoted +## 3. The before/after recipe — both boxes, quoted -Full text in `documentation/audits/R106-R109-recipe-completeness-2026-07-30.md` §5. +``` +demo-felhom BEFORE "namespace":"root" backup_target absent offsite_restic absent + AFTER "namespace":"demo-felhom" backup_target {resolved, felhom-backup, /mnt/hdd_1} + offsite_restic {u629488-sub1…} +demo-hp BEFORE "namespace":"root" backup_target absent offsite_restic absent + AFTER "namespace":"demo-hp" backup_target {resolved, felhom-backup, /mnt/nvme-1tb} + offsite_restic {u629488-sub3…} +``` + +The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical: +on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at +**07-28 17:5x**, frozen at the target-move date. The recipe now names the live one. + +Full evidence, all seven red-proofs and the publish observables: +`documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`. ## 4. Findings filed (none of them fixed except R-122, see §2) @@ -106,10 +125,12 @@ Full text in `documentation/audits/R106-R109-recipe-completeness-2026-07-30.md` |---|---| | **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 | | **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here | +| **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) | | **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` | ## 5. Not done, and why - **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done. - **The backup machinery** — untouched. This corrects the record, not the doing. -- **demo-felhom's agent was not upgraded past what the deploy step covers** — see the audit §6. +- **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6. +- **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy. diff --git a/documentation/audits/R106-R109-recipe-completeness-2026-07-30.md b/documentation/audits/R106-R109-recipe-completeness-2026-07-30.md new file mode 100644 index 0000000..43f3cc5 --- /dev/null +++ b/documentation/audits/R106-R109-recipe-completeness-2026-07-30.md @@ -0,0 +1,250 @@ +# AUDIT — R-106 + R-109 (+ R-122): the DR recipe stops guessing (2026-07-30) + +Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0** +(`felhom.eu` `acfc2b7`). Live on **both** demo boxes. + +--- + +## 1. Baselines, established separately at the start of the session + +| component | value | how | +|---|---|---| +| agent — demo-felhom | **0.115.0** | `ssh felhom-pve "felhom-agent --version"` | +| agent — demo-hp | **0.117.0** | `ssh demo-hp "felhom-agent --version"` — R-117's §5.1 step HAD landed here | +| agent — source HEAD | v0.117.0 | top of `felhom-agent/CHANGELOG.md` | +| controller (live/golden) | 0.186.0 vouched as golden | `hub_settings.artifact_golden_version` | +| hub | **0.82.0** | `style.css?v=` + `deploy/hub` image tag | +| `felhom.eu` HEAD | **`3d504d5`** | as the task expected | +| R-121 | not moved this session | — | + +Fleet was split three ways (0.115.0 / 0.116.0 / 0.117.0) before this session; both demo boxes are now +0.118.1. `sess-f` (0.116.0) and `drill-r50` (0.113.0) were deliberately **not** touched. + +--- + +## 2. Part 0 — the set and the owner + +### 2.1 The real open set, and two items missing from the register + +`OPEN-ITEMS.md:1` declares itself "the single source of truth for open work"; `ROADMAP.md` keeps +"the full history and reasoning". + +| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict | +|---|---|---|---| +| R-105 | `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** | +| R-106 | `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** | +| R-108 | `:111` | `:50` | registered | +| R-109 | `:112` | `:61` | registered | + +R-109's own cell — *"third recipe-completeness defect beside R-105/R-106"* — was the **only** mention of +either in the register. That is precisely the thread-loss the register exists to prevent → filed **R-123**. + +**The set closed here is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`). +**R-105 is NOT in it and was not worked**: it is M-sized and about three *hub-held* DR records being `{}` +(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third already closed by the +2026-07-28 target move). Different fields, different owner, different size. It still needs an OPEN-ITEMS row. + +### 2.2 Where the recipe is generated — THREE producers, not two + +| half | repo | function | +|---|---|---| +| host — guests/pbs/drives/pve_storage | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` | +| app — customer/apps/offsite_restic | `felhom-controller` | `controller/internal/report/dr_recipe.go:35-42` | +| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`; route `internal/web/server.go:439` | + +R-109's "host-half" is the **agent** — and the field must ALSO pass the **hub's** allow-list (§4). + +### 2.3 What the namespace field contained — the eleven-session-old brief was RIGHT + +Live pre-fix, both boxes: `"namespace": "root"`. Against `/etc/pve/storage.cfg` on the same boxes: +`namespace demo-felhom` / `namespace demo-hp`. + +Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo +per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises +empty → `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in. + +**Authority taken:** storage.cfg's `namespace` on the pbs storage — the same field `vzdump --storage ` +makes PVE read, and the source of the agent's own verify client (`cmd/felhom-agent/main.go:1164`). + +--- + +## 3. R-109's ambiguity, in the boxes' own pre-fix recipe + +```json +{ "name": "felhom-backup", "type": "local-dir", "content": "backup" } +{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" } +``` + +No `backup_target` key. **And the ambiguity is not theoretical** — on disk, same day: + +| box | `felhom-backup` newest archive | `local` newest archive | +|---|---|---| +| demo-felhom | `2026-07-30 04:36` (6.1 GB) | **`2026-07-28 17:58`** (5.9 GB) | +| demo-hp | `2026-07-30 04:36` (2.3 GB) | **`2026-07-28 17:44`** (1.7 GB) | + +`local` froze at the target-move date on both boxes and has never been refreshed since. A restorer picking +it gets a guest that restores cleanly and is silently stale — two days at the time of writing, unbounded +thereafter. That is the worst shape a backup defect can take, and it is what the recipe left to a guess. + +--- + +## 4. R-122 — a FOURTH defect, found here, already shipped + +`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists dressed as forward-compat**. The +controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*); +`appHalfShape` never listed the key. Verified both ways: + +- **stored** — `dr_recipe.app_half_json` carries it for all three real customers (`peti-felhom`, + `demo-felhom`, `demo-hp`). +- **delivered** — the pre-fix download's top-level keys were + `recipe_version, customer, guests, pbs, drives, pve_storage, apps`. **No `offsite_restic`.** + +So a restorer had **no offsite location at all**, for the feature's whole life, with a green suite +throughout — because the fixture `drAppHalf` is hand-written and omits the field. R-109's new +`backup_target` would have been swallowed identically, making the fix read as shipped while changing +nothing an operator could see. + +**Deviation from the task's §7.10 ("findings filed, none fixed"), stated rather than absorbed:** I fixed +it. Part 0 authorises working the real set; it is the same symptom the task is named for and the worst +instance of it; and it lives in the same two structs R-109 forced me to edit. Filed as R-122, SHIPPED. + +--- + +## 5. Before / after, quoted, same boxes + +### demo-felhom + +``` +BEFORE agent 0.115.0, hub 0.82.0 + pbs {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"} + backup_target (absent) + offsite_restic (absent) + +AFTER agent 0.118.1, hub 0.83.0 + pbs {"repo_id":"felhom-pbs","namespace":"demo-felhom","namespace_state":"resolved","latest_snapshot_id":"9201"} + backup_target {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/hdd_1"} + offsite_restic {"host":"u629488-sub1.your-storagebox.de","user":"u629488-sub1","port":23,"repo_path":"/home/felhom-repo"} +``` + +### demo-hp + +``` +BEFORE agent 0.117.0, hub 0.82.0 + pbs {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"} + backup_target (absent) + offsite_restic (absent) + +AFTER agent 0.118.1, hub 0.83.0 + pbs {"repo_id":"felhom-pbs","namespace":"demo-hp","namespace_state":"resolved","latest_snapshot_id":"9201"} + backup_target {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/nvme-1tb"} + offsite_restic {"host":"u629488-sub3.your-storagebox.de","user":"u629488-sub3","port":23,"repo_path":"/home/felhom-repo"} +``` + +**The two boxes disagreeing is the point** — different namespaces, different mountpoints, different restic +sub-accounts. Nothing is hardcoded; each value is resolved from that box's own configuration. Method: +`GET /customers//dr-recipe.json` against the hub ClusterIP with operator Basic auth — the exact +endpoint the operator UI's download link invokes (no browser on DooPlex). + +--- + +## 6. v0.118.0 shipped an INERT half of R-106, and live validation is what caught it + +Deployed 0.118.0 to demo-felhom, the recipe read: + +```json +"pbs": { "namespace": "root", "namespace_state": "resolved", ... } +``` + +`resolved` **and wrong** — strictly worse than the original defect, because it asserts confidence in a +wrong answer. The new state field is what made it legible at a glance; without it this would have been +indistinguishable from the pre-fix output and would have shipped as "done". + +**Cause:** `mergeConfig` (`internal/storage/observe.go:457`) overlays the CLUSTER storage config onto the +NODE entry through a hand-listed field set — `Type, Server, Export, Share, Datastore, Fingerprint, VGName, +ThinPool, Path, Content` — and `Namespace` was not on it. `NodeStorage` does not return the namespace at +all, so the merged entry's was always empty. The recipe logic was correct and fed a wrong input. + +**Why the tests missed it:** every v0.118.0 test constructs `StorageTarget` values directly — *including +the two that run `Collector.Collect()`*, because they inject a `fakeObserver`. What I had called "the +production generation path" started one layer below the break. Fixed with +`TestObserve_CarriesPBSNamespaceThroughMerge` (real `Observe`, PVE's actual split returns reproduced) and +`TestMergeConfig_CarriesPBSNamespace` (the merge tabled, fill-if-empty vs never-clobber). + +**The transferable lesson:** "a test through the production path" is only true up to the seam you inject +at. Naming the injected seam — here `fakeObserver` — makes the untested span visible; not naming it is how +a fix ships inert. `Username` is the one remaining unmerged field; the comment now says to add it here the +moment anything consumes it. + +--- + +## 7. Tests + +**Agent (11 new).** Consequence-level: a box with two `content=backup` storages names the LIVE one and not +the frozen one; `assertBackupCandidateAmbiguity` **fails the test if the fixture ever stops posing that +problem**, so it cannot pass hollow. Fixtures are the storage set demo-felhom really had, provenance +recorded in-file (the box's own pre-fix recipe + its `storage.cfg`). Wrong cases: unresolvable → explicit +unknown with no `storage_id`/`mount_path` key at all and no leaked candidate value; missing storage → the +other reason with the configured id recorded; no pbs storage → namespace unknown, never defaulted to root. + +**Hub (3 new).** Built on halves read **verbatim out of the live `dr_recipe` table**. +`TestAssembleDRRecipe_CarriesEveryEmittedSection` enumerates every section both emitters produce and fails +on any dropped — the guard the allow-list needed and never had. + +**Red-proofs: 7**, each mutation asserted to have landed *before* the run: + +| # | mutation | caught by | +|---|---|---| +| 1 | namespace from the snapshot again | the live `"root"` symptom reproduces, unit + production-path | +| 2 | unknown target guesses a `content=backup` storage | 5 assertions | +| 3 | `backup_target` never populated | 3 tests + the cross-repo contract guard | +| 4 | decoy loses `content=backup` (hollow fixture) | the ambiguity guard | +| 5 | `offsite_restic` off the allow-list (R-122 restored) | 2 tests | +| 6 | `backup_target` off the allow-list | 4 tests, naming the section | +| 7 | `Namespace` off `mergeConfig` (the §6 gap) | both new observer tests | + +Suites run **separately from every commit**: agent rc=0 / 29 packages / 0 FAIL; hub `vet` rc=0, suite +rc=0 / 17 packages / 0 FAIL. + +--- + +## 8. Publish observables — treated as a gate + +| # | observable | result | +|---|---|---| +| 1 | published at the new version | `GET .../generic/felhom-agent/0.118.1/felhom-agent` → **HTTP 200**, 14043066 bytes, sha256 `9a74c87d…`; the **downloaded** binary reports `felhom-agent 0.118.1`. Anonymous fetch, independent of the publish script | +| 2 | the manifest vouches it | `hub_settings.artifact_agent_version = 0.118.1`, `artifact_agent_sha256 = 9a74c87d…` — **the hub re-derived the sha from Gitea itself**, matching the build exactly. `golden 0.186.0`, `min_agent 0.113.0`, `wrapper_sha 104db0a4…` all carried through unchanged (the handler rewrites every field; omitting one clears it) | +| 3 | the box reports it running | hub-side `hosts.agent_version`: `demo-felhom-8363b5 = 0.118.1`, `demo-hp-bb76ea = 0.118.1` — asked of the hub, not of the box | + +The vouch normally sits with the operator (the `felhom-build-deploy` skill calls it an operator +follow-up); **it was performed here on the operator's explicit in-session approval**, since the task makes +the three observables a gate. R-120's golden gate was checked before submitting and passes +(golden `0.186.0` == newest fleet-reported controller `0.186.0`). + +Hub deploy: ArgoCD app `felhom` hard-refreshed → `OutOfSync` at `acfc2b7` → synced → **Synced/Healthy**, +rollout complete, live image `felhom-hub:0.83.0`, clean startup log. + +--- + +## 9. Findings + +| id | finding | disposition | +|---|---|---| +| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life | **FIXED** (hub 0.83.0) — deviation stated in §4 | +| **R-123** | R-105 and R-106 were `READY` in ROADMAP with no OPEN-ITEMS row | R-106 registered + shipped; **R-105 still needs a row**, and nothing checks the invariant | +| **R-124** | the recipe spells PBS's root namespace `"root"`, but the API spells it `""` and no namespace is literally named `root` — `pct restore --ns root` would fail | **OPEN (XS)**, deliberately not changed mid-fix; affects only a box with no `namespace` line, i.e. no real customer today | +| **R-125** | v0.118.0 shipped an inert R-106 because "the production path" test injected `fakeObserver` one layer below the break (§6) | **the fix is shipped**; filed for the doctrine point — a production-path claim must name the seam it injects at | + +--- + +## 10. What did NOT happen, and why + +- **R-105, R-108, D5** — out of scope per the task's §6. R-108 blocks D5; starting either would leave both half-done. +- **The backup machinery** — untouched. This corrects the RECORD, not the doing. And the record was never + acted on programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`, *"the + destructive restore is a separate, operator-present step"*), so the blast radius was exactly one + operator reading a wrong value. `plan.PBS` inherits the fix for free. +- **`sess-f` (agent 0.116.0) and `drill-r50` (0.113.0) not upgraded** — neither was named as a venue and + `drill-r50` is fenced by §6. +- **No golden re-bake, no guest restart, no destructive operation** — none was needed. +- **`R-124` not fixed** — changing the wire's spelling of the root namespace mid-R-106 would have shifted + the field's meaning during the very fix meant to make it trustworthy. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index ede7aa7..40e2cfa 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -58,11 +58,12 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — | | **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — | | **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator | -| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — | -| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage ` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — | +| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — | +| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage ` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — | | **R-122** | ~~`AssembleDRRecipe` silently DROPPED `offsite_restic` — the offsite recovery location never reached any recipe~~ | **SHIPPED** (hub v0.83.0, 2026-07-30) | — | **Found 2026-07-30 while scoping R-109's hub half; it had already shipped and nobody knew.** The controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*), the hub stored it for **all three real customers**, and `appHalfShape` never listed the key — so no delivered recipe has ever contained it. No error, no log, green suite, because the fixture `drAppHalf` is hand-written and omits the field. `hostHalfShape`/`appHalfShape` are **ALLOW-LISTS dressed as forward-compat**; `TestAssembleDRRecipe_CarriesEveryEmittedSection` is now the guard, built on halves read verbatim out of the live `dr_recipe` table. `REUSE.md` (both repos) records that a recipe section is a TWO-REPO change | — | | **R-123** | **R-105 and R-106 were `READY` in `ROADMAP.md` with no row on THIS page** — each referenced only inside R-109's prose, which is precisely the thread-loss the register exists to prevent | **PARTLY CLOSED** (2026-07-30) | — | R-106 registered above (and shipped). **R-105 still needs a row** — it is M-sized, is about three hub-held DR records being `{}`, and is NOT part of the recipe-completeness set that shipped today. **The process gap is the real item:** nothing checks that a `READY` ROADMAP row has an OPEN-ITEMS row. A grep-level gate would catch it | CC | | **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC | +| **R-125** | **A "test through the production path" is only true up to the seam it injects at.** agent v0.118.0 shipped an INERT R-106: two tests ran the real `Collector.Collect()`, but both injected a `fakeObserver`, and the break was one layer below in `mergeConfig` — which dropped the pbs namespace, so the recipe still said `"root"`, now with `namespace_state: resolved` beside it (confident AND wrong, worse than the original defect). Caught by LIVE VALIDATION, not by the green suite | **FIXED** (agent v0.118.1) — filed for the DOCTRINE point | — | The fix is shipped and red-proofed. What is open is the practice: a production-path claim should NAME the seam it injects at, so the untested span is visible. Candidate for the `felhom-testing` skill; the state field is also what made the wrong value legible, which argues for the three-state shape generally. `audits/R106-R109-recipe-completeness-2026-07-30.md` §6 | CC | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 1732db4..d14f49c 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -106,10 +106,10 @@ | R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 | | R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 | | R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 | -| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.0, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage ` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | +| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.1, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage ` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | | R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 | | R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed | -| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.0 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | +| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | | E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 | ## P3 — post-alpha