R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125
Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes, the on-disk proof that `local` froze at the 2026-07-28 target move while felhom-backup kept running, all seven red-proofs, and the three publish observables. R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real Collector.Collect() but both injected a fakeObserver, and the break was one layer below in mergeConfig, which dropped the pbs namespace. The recipe still said "root" — now with namespace_state "resolved" beside it, confident and wrong. Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for the doctrine point that a production-path claim must name the seam it injects at.
This commit is contained in:
@@ -0,0 +1,250 @@
|
||||
# AUDIT — R-106 + R-109 (+ R-122): the DR recipe stops guessing (2026-07-30)
|
||||
|
||||
Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
|
||||
(`felhom.eu` `acfc2b7`). Live on **both** demo boxes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines, established separately at the start of the session
|
||||
|
||||
| component | value | how |
|
||||
|---|---|---|
|
||||
| agent — demo-felhom | **0.115.0** | `ssh felhom-pve "felhom-agent --version"` |
|
||||
| agent — demo-hp | **0.117.0** | `ssh demo-hp "felhom-agent --version"` — R-117's §5.1 step HAD landed here |
|
||||
| agent — source HEAD | v0.117.0 | top of `felhom-agent/CHANGELOG.md` |
|
||||
| controller (live/golden) | 0.186.0 vouched as golden | `hub_settings.artifact_golden_version` |
|
||||
| hub | **0.82.0** | `style.css?v=` + `deploy/hub` image tag |
|
||||
| `felhom.eu` HEAD | **`3d504d5`** | as the task expected |
|
||||
| R-121 | not moved this session | — |
|
||||
|
||||
Fleet was split three ways (0.115.0 / 0.116.0 / 0.117.0) before this session; both demo boxes are now
|
||||
0.118.1. `sess-f` (0.116.0) and `drill-r50` (0.113.0) were deliberately **not** touched.
|
||||
|
||||
---
|
||||
|
||||
## 2. Part 0 — the set and the owner
|
||||
|
||||
### 2.1 The real open set, and two items missing from the register
|
||||
|
||||
`OPEN-ITEMS.md:1` declares itself "the single source of truth for open work"; `ROADMAP.md` keeps
|
||||
"the full history and reasoning".
|
||||
|
||||
| item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
|
||||
|---|---|---|---|
|
||||
| R-105 | `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-106 | `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
|
||||
| R-108 | `:111` | `:50` | registered |
|
||||
| R-109 | `:112` | `:61` | registered |
|
||||
|
||||
R-109's own cell — *"third recipe-completeness defect beside R-105/R-106"* — was the **only** mention of
|
||||
either in the register. That is precisely the thread-loss the register exists to prevent → filed **R-123**.
|
||||
|
||||
**The set closed here is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
|
||||
**R-105 is NOT in it and was not worked**: it is M-sized and about three *hub-held* DR records being `{}`
|
||||
(`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third already closed by the
|
||||
2026-07-28 target move). Different fields, different owner, different size. It still needs an OPEN-ITEMS row.
|
||||
|
||||
### 2.2 Where the recipe is generated — THREE producers, not two
|
||||
|
||||
| half | repo | function |
|
||||
|---|---|---|
|
||||
| host — guests/pbs/drives/pve_storage | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
|
||||
| app — customer/apps/offsite_restic | `felhom-controller` | `controller/internal/report/dr_recipe.go:35-42` |
|
||||
| **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`; route `internal/web/server.go:439` |
|
||||
|
||||
R-109's "host-half" is the **agent** — and the field must ALSO pass the **hub's** allow-list (§4).
|
||||
|
||||
### 2.3 What the namespace field contained — the eleven-session-old brief was RIGHT
|
||||
|
||||
Live pre-fix, both boxes: `"namespace": "root"`. Against `/etc/pve/storage.cfg` on the same boxes:
|
||||
`namespace demo-felhom` / `namespace demo-hp`.
|
||||
|
||||
Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
|
||||
per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises
|
||||
empty → `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
|
||||
|
||||
**Authority taken:** storage.cfg's `namespace` on the pbs storage — the same field `vzdump --storage <pbs>`
|
||||
makes PVE read, and the source of the agent's own verify client (`cmd/felhom-agent/main.go:1164`).
|
||||
|
||||
---
|
||||
|
||||
## 3. R-109's ambiguity, in the boxes' own pre-fix recipe
|
||||
|
||||
```json
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
```
|
||||
|
||||
No `backup_target` key. **And the ambiguity is not theoretical** — on disk, same day:
|
||||
|
||||
| box | `felhom-backup` newest archive | `local` newest archive |
|
||||
|---|---|---|
|
||||
| demo-felhom | `2026-07-30 04:36` (6.1 GB) | **`2026-07-28 17:58`** (5.9 GB) |
|
||||
| demo-hp | `2026-07-30 04:36` (2.3 GB) | **`2026-07-28 17:44`** (1.7 GB) |
|
||||
|
||||
`local` froze at the target-move date on both boxes and has never been refreshed since. A restorer picking
|
||||
it gets a guest that restores cleanly and is silently stale — two days at the time of writing, unbounded
|
||||
thereafter. That is the worst shape a backup defect can take, and it is what the recipe left to a guess.
|
||||
|
||||
---
|
||||
|
||||
## 4. R-122 — a FOURTH defect, found here, already shipped
|
||||
|
||||
`AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists dressed as forward-compat**. The
|
||||
controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*);
|
||||
`appHalfShape` never listed the key. Verified both ways:
|
||||
|
||||
- **stored** — `dr_recipe.app_half_json` carries it for all three real customers (`peti-felhom`,
|
||||
`demo-felhom`, `demo-hp`).
|
||||
- **delivered** — the pre-fix download's top-level keys were
|
||||
`recipe_version, customer, guests, pbs, drives, pve_storage, apps`. **No `offsite_restic`.**
|
||||
|
||||
So a restorer had **no offsite location at all**, for the feature's whole life, with a green suite
|
||||
throughout — because the fixture `drAppHalf` is hand-written and omits the field. R-109's new
|
||||
`backup_target` would have been swallowed identically, making the fix read as shipped while changing
|
||||
nothing an operator could see.
|
||||
|
||||
**Deviation from the task's §7.10 ("findings filed, none fixed"), stated rather than absorbed:** I fixed
|
||||
it. Part 0 authorises working the real set; it is the same symptom the task is named for and the worst
|
||||
instance of it; and it lives in the same two structs R-109 forced me to edit. Filed as R-122, SHIPPED.
|
||||
|
||||
---
|
||||
|
||||
## 5. Before / after, quoted, same boxes
|
||||
|
||||
### demo-felhom
|
||||
|
||||
```
|
||||
BEFORE agent 0.115.0, hub 0.82.0
|
||||
pbs {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
|
||||
backup_target (absent)
|
||||
offsite_restic (absent)
|
||||
|
||||
AFTER agent 0.118.1, hub 0.83.0
|
||||
pbs {"repo_id":"felhom-pbs","namespace":"demo-felhom","namespace_state":"resolved","latest_snapshot_id":"9201"}
|
||||
backup_target {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/hdd_1"}
|
||||
offsite_restic {"host":"u629488-sub1.your-storagebox.de","user":"u629488-sub1","port":23,"repo_path":"/home/felhom-repo"}
|
||||
```
|
||||
|
||||
### demo-hp
|
||||
|
||||
```
|
||||
BEFORE agent 0.117.0, hub 0.82.0
|
||||
pbs {"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}
|
||||
backup_target (absent)
|
||||
offsite_restic (absent)
|
||||
|
||||
AFTER agent 0.118.1, hub 0.83.0
|
||||
pbs {"repo_id":"felhom-pbs","namespace":"demo-hp","namespace_state":"resolved","latest_snapshot_id":"9201"}
|
||||
backup_target {"state":"resolved","storage_id":"felhom-backup","mount_path":"/mnt/nvme-1tb"}
|
||||
offsite_restic {"host":"u629488-sub3.your-storagebox.de","user":"u629488-sub3","port":23,"repo_path":"/home/felhom-repo"}
|
||||
```
|
||||
|
||||
**The two boxes disagreeing is the point** — different namespaces, different mountpoints, different restic
|
||||
sub-accounts. Nothing is hardcoded; each value is resolved from that box's own configuration. Method:
|
||||
`GET /customers/<id>/dr-recipe.json` against the hub ClusterIP with operator Basic auth — the exact
|
||||
endpoint the operator UI's download link invokes (no browser on DooPlex).
|
||||
|
||||
---
|
||||
|
||||
## 6. v0.118.0 shipped an INERT half of R-106, and live validation is what caught it
|
||||
|
||||
Deployed 0.118.0 to demo-felhom, the recipe read:
|
||||
|
||||
```json
|
||||
"pbs": { "namespace": "root", "namespace_state": "resolved", ... }
|
||||
```
|
||||
|
||||
`resolved` **and wrong** — strictly worse than the original defect, because it asserts confidence in a
|
||||
wrong answer. The new state field is what made it legible at a glance; without it this would have been
|
||||
indistinguishable from the pre-fix output and would have shipped as "done".
|
||||
|
||||
**Cause:** `mergeConfig` (`internal/storage/observe.go:457`) overlays the CLUSTER storage config onto the
|
||||
NODE entry through a hand-listed field set — `Type, Server, Export, Share, Datastore, Fingerprint, VGName,
|
||||
ThinPool, Path, Content` — and `Namespace` was not on it. `NodeStorage` does not return the namespace at
|
||||
all, so the merged entry's was always empty. The recipe logic was correct and fed a wrong input.
|
||||
|
||||
**Why the tests missed it:** every v0.118.0 test constructs `StorageTarget` values directly — *including
|
||||
the two that run `Collector.Collect()`*, because they inject a `fakeObserver`. What I had called "the
|
||||
production generation path" started one layer below the break. Fixed with
|
||||
`TestObserve_CarriesPBSNamespaceThroughMerge` (real `Observe`, PVE's actual split returns reproduced) and
|
||||
`TestMergeConfig_CarriesPBSNamespace` (the merge tabled, fill-if-empty vs never-clobber).
|
||||
|
||||
**The transferable lesson:** "a test through the production path" is only true up to the seam you inject
|
||||
at. Naming the injected seam — here `fakeObserver` — makes the untested span visible; not naming it is how
|
||||
a fix ships inert. `Username` is the one remaining unmerged field; the comment now says to add it here the
|
||||
moment anything consumes it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Tests
|
||||
|
||||
**Agent (11 new).** Consequence-level: a box with two `content=backup` storages names the LIVE one and not
|
||||
the frozen one; `assertBackupCandidateAmbiguity` **fails the test if the fixture ever stops posing that
|
||||
problem**, so it cannot pass hollow. Fixtures are the storage set demo-felhom really had, provenance
|
||||
recorded in-file (the box's own pre-fix recipe + its `storage.cfg`). Wrong cases: unresolvable → explicit
|
||||
unknown with no `storage_id`/`mount_path` key at all and no leaked candidate value; missing storage → the
|
||||
other reason with the configured id recorded; no pbs storage → namespace unknown, never defaulted to root.
|
||||
|
||||
**Hub (3 new).** Built on halves read **verbatim out of the live `dr_recipe` table**.
|
||||
`TestAssembleDRRecipe_CarriesEveryEmittedSection` enumerates every section both emitters produce and fails
|
||||
on any dropped — the guard the allow-list needed and never had.
|
||||
|
||||
**Red-proofs: 7**, each mutation asserted to have landed *before* the run:
|
||||
|
||||
| # | mutation | caught by |
|
||||
|---|---|---|
|
||||
| 1 | namespace from the snapshot again | the live `"root"` symptom reproduces, unit + production-path |
|
||||
| 2 | unknown target guesses a `content=backup` storage | 5 assertions |
|
||||
| 3 | `backup_target` never populated | 3 tests + the cross-repo contract guard |
|
||||
| 4 | decoy loses `content=backup` (hollow fixture) | the ambiguity guard |
|
||||
| 5 | `offsite_restic` off the allow-list (R-122 restored) | 2 tests |
|
||||
| 6 | `backup_target` off the allow-list | 4 tests, naming the section |
|
||||
| 7 | `Namespace` off `mergeConfig` (the §6 gap) | both new observer tests |
|
||||
|
||||
Suites run **separately from every commit**: agent rc=0 / 29 packages / 0 FAIL; hub `vet` rc=0, suite
|
||||
rc=0 / 17 packages / 0 FAIL.
|
||||
|
||||
---
|
||||
|
||||
## 8. Publish observables — treated as a gate
|
||||
|
||||
| # | observable | result |
|
||||
|---|---|---|
|
||||
| 1 | published at the new version | `GET .../generic/felhom-agent/0.118.1/felhom-agent` → **HTTP 200**, 14043066 bytes, sha256 `9a74c87d…`; the **downloaded** binary reports `felhom-agent 0.118.1`. Anonymous fetch, independent of the publish script |
|
||||
| 2 | the manifest vouches it | `hub_settings.artifact_agent_version = 0.118.1`, `artifact_agent_sha256 = 9a74c87d…` — **the hub re-derived the sha from Gitea itself**, matching the build exactly. `golden 0.186.0`, `min_agent 0.113.0`, `wrapper_sha 104db0a4…` all carried through unchanged (the handler rewrites every field; omitting one clears it) |
|
||||
| 3 | the box reports it running | hub-side `hosts.agent_version`: `demo-felhom-8363b5 = 0.118.1`, `demo-hp-bb76ea = 0.118.1` — asked of the hub, not of the box |
|
||||
|
||||
The vouch normally sits with the operator (the `felhom-build-deploy` skill calls it an operator
|
||||
follow-up); **it was performed here on the operator's explicit in-session approval**, since the task makes
|
||||
the three observables a gate. R-120's golden gate was checked before submitting and passes
|
||||
(golden `0.186.0` == newest fleet-reported controller `0.186.0`).
|
||||
|
||||
Hub deploy: ArgoCD app `felhom` hard-refreshed → `OutOfSync` at `acfc2b7` → synced → **Synced/Healthy**,
|
||||
rollout complete, live image `felhom-hub:0.83.0`, clean startup log.
|
||||
|
||||
---
|
||||
|
||||
## 9. Findings
|
||||
|
||||
| id | finding | disposition |
|
||||
|---|---|---|
|
||||
| **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life | **FIXED** (hub 0.83.0) — deviation stated in §4 |
|
||||
| **R-123** | R-105 and R-106 were `READY` in ROADMAP with no OPEN-ITEMS row | R-106 registered + shipped; **R-105 still needs a row**, and nothing checks the invariant |
|
||||
| **R-124** | the recipe spells PBS's root namespace `"root"`, but the API spells it `""` and no namespace is literally named `root` — `pct restore --ns root` would fail | **OPEN (XS)**, deliberately not changed mid-fix; affects only a box with no `namespace` line, i.e. no real customer today |
|
||||
| **R-125** | v0.118.0 shipped an inert R-106 because "the production path" test injected `fakeObserver` one layer below the break (§6) | **the fix is shipped**; filed for the doctrine point — a production-path claim must name the seam it injects at |
|
||||
|
||||
---
|
||||
|
||||
## 10. What did NOT happen, and why
|
||||
|
||||
- **R-105, R-108, D5** — out of scope per the task's §6. R-108 blocks D5; starting either would leave both half-done.
|
||||
- **The backup machinery** — untouched. This corrects the RECORD, not the doing. And the record was never
|
||||
acted on programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`, *"the
|
||||
destructive restore is a separate, operator-present step"*), so the blast radius was exactly one
|
||||
operator reading a wrong value. `plan.PBS` inherits the fix for free.
|
||||
- **`sess-f` (agent 0.116.0) and `drill-r50` (0.113.0) not upgraded** — neither was named as a venue and
|
||||
`drill-r50` is fenced by §6.
|
||||
- **No golden re-bake, no guest restart, no destructive operation** — none was needed.
|
||||
- **`R-124` not fixed** — changing the wire's spelling of the root namespace mid-R-106 would have shifted
|
||||
the field's meaning during the very fix meant to make it trustworthy.
|
||||
@@ -58,11 +58,12 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — |
|
||||
| **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — |
|
||||
| **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator |
|
||||
| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — |
|
||||
| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — |
|
||||
| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — |
|
||||
| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.1, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — |
|
||||
| **R-122** | ~~`AssembleDRRecipe` silently DROPPED `offsite_restic` — the offsite recovery location never reached any recipe~~ | **SHIPPED** (hub v0.83.0, 2026-07-30) | — | **Found 2026-07-30 while scoping R-109's hub half; it had already shipped and nobody knew.** The controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*), the hub stored it for **all three real customers**, and `appHalfShape` never listed the key — so no delivered recipe has ever contained it. No error, no log, green suite, because the fixture `drAppHalf` is hand-written and omits the field. `hostHalfShape`/`appHalfShape` are **ALLOW-LISTS dressed as forward-compat**; `TestAssembleDRRecipe_CarriesEveryEmittedSection` is now the guard, built on halves read verbatim out of the live `dr_recipe` table. `REUSE.md` (both repos) records that a recipe section is a TWO-REPO change | — |
|
||||
| **R-123** | **R-105 and R-106 were `READY` in `ROADMAP.md` with no row on THIS page** — each referenced only inside R-109's prose, which is precisely the thread-loss the register exists to prevent | **PARTLY CLOSED** (2026-07-30) | — | R-106 registered above (and shipped). **R-105 still needs a row** — it is M-sized, is about three hub-held DR records being `{}`, and is NOT part of the recipe-completeness set that shipped today. **The process gap is the real item:** nothing checks that a `READY` ROADMAP row has an OPEN-ITEMS row. A grep-level gate would catch it | CC |
|
||||
| **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC |
|
||||
| **R-125** | **A "test through the production path" is only true up to the seam it injects at.** agent v0.118.0 shipped an INERT R-106: two tests ran the real `Collector.Collect()`, but both injected a `fakeObserver`, and the break was one layer below in `mergeConfig` — which dropped the pbs namespace, so the recipe still said `"root"`, now with `namespace_state: resolved` beside it (confident AND wrong, worse than the original defect). Caught by LIVE VALIDATION, not by the green suite | **FIXED** (agent v0.118.1) — filed for the DOCTRINE point | — | The fix is shipped and red-proofed. What is open is the practice: a production-path claim should NAME the seam it injects at, so the untested span is visible. Candidate for the `felhom-testing` skill; the state field is also what made the wrong value legible, which argues for the three-state shape generally. `audits/R106-R109-recipe-completeness-2026-07-30.md` §6 | CC |
|
||||
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
||||
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
|
||||
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
||||
|
||||
@@ -106,10 +106,10 @@
|
||||
| R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 |
|
||||
| R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 |
|
||||
| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 |
|
||||
| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.0, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage <pbs>` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.1, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage <pbs>` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
|
||||
| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `<drive>/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed |
|
||||
| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.0 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |
|
||||
|
||||
## P3 — post-alpha
|
||||
|
||||
Reference in New Issue
Block a user