R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)

Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
This commit is contained in:
2026-07-30 13:11:08 +02:00
parent 1913e12031
commit 1c8a67eece
13 changed files with 670 additions and 103 deletions
+53 -80
View File
@@ -1,103 +1,76 @@
# REPORT — v0.117.0: R-117, the liveness signal now tests liveness (2026-07-30)
# REPORT — R-106 + R-109: the DR recipe stops guessing (agent v0.118.0, 2026-07-30)
**Overwritten** per the standing rule. Full evidence:
`felhom.eu/documentation/audits/R117-v0117-2026-07-30.md`. Spike this builds on:
`audits/SPIKE-r117-bind-liveness-2026-07-30.md`.
**Overwritten** per the standing rule. Paired with **hub v0.83.0** — neither half is useful alone, see §5.
| | |
|---|---|
| agent | 0.116.0 → **0.117.0** @ `966d8f4` |
| published + vouched | **0.117.0**, sha `e6768a12eba1c9e0…` (anonymous GET verified, manifest read back) |
| deployed | **demo-hp 0.117.0** (from **0.113.0** — see R-121) · felhom-pve still **0.115.0**, out of scope |
| controller / golden / hub | 0.186.0 / 0.186.0 / 0.82.0 — **untouched** |
| tests | **849 → 863**, 29/29 packages, `build`/`vet`/`test` each `rc=0` read separately |
## 1. What was wrong, confirmed live BEFORE the fix
## The defect
Downloaded from the hub, `GET /customers/<id>/dr-recipe.json`, both demo boxes:
`BoundUnderParent` reported a namespace returning `EIO` on every read and write as healthy, and the gate
restarted the customer's apps onto it. Both existing terms — `GuestSeesMount` and `isHostMountpoint`
parse a mountinfo line and then test only `fields[4]`, the mount POINT. **Field 3, `major:minor`, sat in
the same parsed slice and was discarded.**
```json
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
```
## The fix — one predicate, both halves mandatory
and **no `backup_target` key at all**, while `pve_storage` offered two plausible candidates:
`bindLiveness` (`internal/localapi/intermediary.go`) becomes the third term of the conjunction at both
`/disks` construction sites. `/proc` only, **no block I/O**:
```json
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
```
- **the bind must name the same device as the raw mount** — sound, not heuristic: a stale bind pins the
dead superblock, which keeps the old device index allocated, which forces the returning device onto a
different number;
- **the filesystem must not have aborted** — `shutdown` or `emergency_ro`, both measured.
`/etc/pve/storage.cfg` on those same boxes says `namespace demo-felhom` and `namespace demo-hp`.
**The second is not optional.** A device that fails *without disappearing* gives the identical
all-signals-healthy state with the **devnos equal** and the drive never `Disconnected`, so the gate emits
nothing at all, indefinitely (R-117a). A devno-only fix passes every payload test; red-proof RP3 exists for
exactly that.
## 2. Root causes, at source
**Three states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` the zero value, and
`Usable()` is the single place where unknown counts as **present** — absent stops a working customer's apps.
- **R-106** — `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo per
item once the list is namespace-scoped via `?ns=` (`:118-120`). Always empty → `ToHub` normalises empty
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it into the recipe.
- **R-109** — `BuildDRRecipeHostHalf` emitted `guests`/`drives`/`pve_storage`/`pbs` and nothing naming the
vzdump target. Benign until the 2026-07-28 target move gave every box a second `content=backup` dir
storage holding archives frozen at that date.
## The ruling (task §2.2) — and it is flagged for overrule
## 3. The fix
`StaleDevice`**re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts).
`Aborted`**quiet no-op, and surface via `BoundUnderParent=false`** — a re-bind lands on the *same* dead
superblock and this runs every 20 s, so repairing would be an infinite silent retry that also masks the
state. **No operator decision taken quietly:** it does change customer-visible behaviour (apps stop and an
alarm fires where today nothing happens), but it routes an already-broken state into the **existing** gate,
event types and Hungarian copy — no new concept reaches the customer — and the alternative is applications
writing documents into a filesystem that rejects every write.
The namespace now resolves from the pbs **storage** (`StorageTarget.PBSNamespace`, from storage.cfg's
`namespace`) — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with
the backup that produced the snapshot. `backup_target` resolves from the **primary tier of
`cfg.Backup.BackupTiers()`** — the function the scheduler itself consults — and carries the mountpoint,
which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`.
**No new recovery path.** `AttachDrive`'s normalize leg already did the repair and three call sites already
invoked it (20 s ticker, agent startup, and the controller's `Return` branch **before** `restartStacks`).
All three died on `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about a dead namespace.
**Unresolvable is recorded as unresolvable**, never defaulted: `namespace_state` and `backup_target.state`
are `resolved | unknown`, with distinct reasons (`agent_backup_config_unavailable`, `not_a_known_storage`,
`no_pbs_storage_observed`). An unconfigured namespace still reads `"root"` — that is an answer, not a gap.
**Ordering trap, caught by a test:** reading the abort flag before comparing devices classifies the real
return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still
reporting correctly** — so the abort flag is read off the RAW mount in the stale case.
**The resolver reports the tier IN EFFECT, not the file on disk.** A backup-target move rewrites
`agent.json` and deliberately does not restart the agent (the E-1 lesson), so a disk re-read would name a
storage no archive had reached yet.
## Live validation, demo-hp
## 4. Files
| State | raw | bind | verdict | usable |
|---|---|---|---|---|
| healthy | `8:16` | `8:16` | `live` | true |
| **R-117 return** | `8:32` `/dev/sdc` | `8:16 …,shutdown` | **`stale-device`** | **false** |
| **Q7 in-place abort** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** |
`internal/hub/report.go`, `internal/storage/observe.go`, `internal/hub/dr_recipe.go`,
`internal/hub/collect.go`, `cmd/felhom-agent/main.go`, `internal/hub/testdata/host-report.golden.json`,
`REUSE.md`; tests in `internal/hub/{dr_recipe,contract,report}_test.go`, `internal/pbs/live_reporter_test.go`.
340497 µs per call. **No block I/O proven by `strace`**: only `/proc/self/mountinfo`, **0** `statfs`.
**No regression through the real pipeline:** `GET /disks` with the controller's own credential shows the
live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable
and zero spurious transitions.
## 5. The cross-repo half that is easy to miss
## Tests + red-proofs
The hub's `AssembleDRRecipe` **allow-lists** top-level keys. `backup_target` would have been stored intact
and dropped before any operator saw it. The same mechanism already cost `offsite_restic` its whole
existence (**R-122**, found this session). `REUSE.md` now records this as a two-repo change.
14 new tests; the payload ones drive the real `/disks` handler and the `AttachDrive` ones record the actual
privileged calls. Fixtures are the spike's captured mountinfo, injected as **data** via
`procSelfMountinfo`/`procGuestMountinfo`, so the real parser, predicate, `AttachDrive` and handler all run.
## 6. Tests
**6 red-proofs, each verified to have landed.** RP1/RP2 (term 3 removed per site), RP3 (**the P1-only
fix**), RP4 (`emergency_ro` dropped), RP5 (unknown reported dead), RP6 (aborted arm falls through to a
re-bind).
9 new. Consequence-level: a box with two `content=backup` storages names the **live** one and not the
frozen one; `assertBackupCandidateAmbiguity` fails if the fixture ever stops posing that problem. Fixtures
are the storage set demo-felhom really had, provenance recorded in-file. Two tests run the real
`Collector.Collect()` path, one pinning that an **unwired seam yields unknown**.
**A hollow test was caught, by RP1 failing to fail.** The aborted fixture first used a `/dev/mapper`
device, for which `RoleForStorage` derives `role="system"` — a system row never enters the block that
computes `BoundUnderParent`, so the field stayed false **by default**, the assertion passed, and no mutation
could ever fail it. Fixtures now assert the production row shape before asserting the field.
**4 red-proofs**, each mutation asserted to have landed before running: revert R-106 → the live `"root"`
symptom reproduces; make unknown guess a default → 5 assertions fire; drop the field → the cross-repo
contract guard fires; hollow the fixture → the ambiguity guard fires.
## Not yet done / not covered
Suite: **rc=0, 29 packages ok, 0 FAIL** (run separately from the commit).
- **The stale-bind repair on hardware.** `StablePathForRaw` hardcodes the production
`/mnt/felhom-drives`, so exercising `AttachDrive` for a scratch drive writes a submount into guest
9201's namespace, which the task fences. It happened once inadvertently, was reverted within a minute
and 9201 verified unaffected, and was deliberately not repeated. Filed **R-117h**.
- **Sustained-load behaviour of the predicate** — flagged unmeasured by the spike and **still unmeasured**.
- **felhom-pve not upgraded** (0.115.0). The fleet is non-uniform.
- **R-117g** — an aborted filesystem is never cleared automatically, by design, so it alarms until a human
acts; there is no guided recovery.
- **R-121** — a box's installed agent can sit releases behind the vouched one and nothing notices; the
R-120 gate is golden-vs-fleet-controller and cannot see it.
## 7. Not touched
## Observations (not acted on)
Guest 9201 has 2 stacked entries at `/mnt/felhom-drives/nvme-1tb` while the host has 1 (pre-existing;
`AttachDrive` normalizes the host count only). 14 stale `felhom-agent.bak-*` binaries on demo-hp (~200 MB),
nothing prunes them. 11 files in this repo fail `gofmt -l`, all pre-existing and none in `internal/localapi/`.
The backup machinery. This corrects the record, not the doing — and the record was never acted on
programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`), so the blast radius was
exactly one operator reading a wrong value. `plan.PBS` inherits the fix for free.