R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)
Both defects were live on both demo boxes: the recipe said namespace "root" while storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup dir storages holds the local archives. R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once the list is namespace-scoped, so it was always empty and normalised to "root". It now resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes PVE read, so the recipe cannot disagree with the backup. R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1 from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not agent.json on disk — a target move rewrites the file and deliberately does not restart. Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason, never a default, an empty string, or a placeholder. Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target would otherwise be stored intact and dropped before any operator saw it. 9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
This commit is contained in:
@@ -1,103 +1,76 @@
|
||||
# REPORT — v0.117.0: R-117, the liveness signal now tests liveness (2026-07-30)
|
||||
# REPORT — R-106 + R-109: the DR recipe stops guessing (agent v0.118.0, 2026-07-30)
|
||||
|
||||
**Overwritten** per the standing rule. Full evidence:
|
||||
`felhom.eu/documentation/audits/R117-v0117-2026-07-30.md`. Spike this builds on:
|
||||
`audits/SPIKE-r117-bind-liveness-2026-07-30.md`.
|
||||
**Overwritten** per the standing rule. Paired with **hub v0.83.0** — neither half is useful alone, see §5.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| agent | 0.116.0 → **0.117.0** @ `966d8f4` |
|
||||
| published + vouched | **0.117.0**, sha `e6768a12eba1c9e0…` (anonymous GET verified, manifest read back) |
|
||||
| deployed | **demo-hp 0.117.0** (from **0.113.0** — see R-121) · felhom-pve still **0.115.0**, out of scope |
|
||||
| controller / golden / hub | 0.186.0 / 0.186.0 / 0.82.0 — **untouched** |
|
||||
| tests | **849 → 863**, 29/29 packages, `build`/`vet`/`test` each `rc=0` read separately |
|
||||
## 1. What was wrong, confirmed live BEFORE the fix
|
||||
|
||||
## The defect
|
||||
Downloaded from the hub, `GET /customers/<id>/dr-recipe.json`, both demo boxes:
|
||||
|
||||
`BoundUnderParent` reported a namespace returning `EIO` on every read and write as healthy, and the gate
|
||||
restarted the customer's apps onto it. Both existing terms — `GuestSeesMount` and `isHostMountpoint` —
|
||||
parse a mountinfo line and then test only `fields[4]`, the mount POINT. **Field 3, `major:minor`, sat in
|
||||
the same parsed slice and was discarded.**
|
||||
```json
|
||||
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
||||
```
|
||||
|
||||
## The fix — one predicate, both halves mandatory
|
||||
and **no `backup_target` key at all**, while `pve_storage` offered two plausible candidates:
|
||||
|
||||
`bindLiveness` (`internal/localapi/intermediary.go`) becomes the third term of the conjunction at both
|
||||
`/disks` construction sites. `/proc` only, **no block I/O**:
|
||||
```json
|
||||
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
|
||||
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
||||
```
|
||||
|
||||
- **the bind must name the same device as the raw mount** — sound, not heuristic: a stale bind pins the
|
||||
dead superblock, which keeps the old device index allocated, which forces the returning device onto a
|
||||
different number;
|
||||
- **the filesystem must not have aborted** — `shutdown` or `emergency_ro`, both measured.
|
||||
`/etc/pve/storage.cfg` on those same boxes says `namespace demo-felhom` and `namespace demo-hp`.
|
||||
|
||||
**The second is not optional.** A device that fails *without disappearing* gives the identical
|
||||
all-signals-healthy state with the **devnos equal** and the drive never `Disconnected`, so the gate emits
|
||||
nothing at all, indefinitely (R-117a). A devno-only fix passes every payload test; red-proof RP3 exists for
|
||||
exactly that.
|
||||
## 2. Root causes, at source
|
||||
|
||||
**Three states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` the zero value, and
|
||||
`Usable()` is the single place where unknown counts as **present** — absent stops a working customer's apps.
|
||||
- **R-106** — `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo per
|
||||
item once the list is namespace-scoped via `?ns=` (`:118-120`). Always empty → `ToHub` normalises empty
|
||||
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it into the recipe.
|
||||
- **R-109** — `BuildDRRecipeHostHalf` emitted `guests`/`drives`/`pve_storage`/`pbs` and nothing naming the
|
||||
vzdump target. Benign until the 2026-07-28 target move gave every box a second `content=backup` dir
|
||||
storage holding archives frozen at that date.
|
||||
|
||||
## The ruling (task §2.2) — and it is flagged for overrule
|
||||
## 3. The fix
|
||||
|
||||
`StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts).
|
||||
`Aborted` ⇒ **quiet no-op, and surface via `BoundUnderParent=false`** — a re-bind lands on the *same* dead
|
||||
superblock and this runs every 20 s, so repairing would be an infinite silent retry that also masks the
|
||||
state. **No operator decision taken quietly:** it does change customer-visible behaviour (apps stop and an
|
||||
alarm fires where today nothing happens), but it routes an already-broken state into the **existing** gate,
|
||||
event types and Hungarian copy — no new concept reaches the customer — and the alternative is applications
|
||||
writing documents into a filesystem that rejects every write.
|
||||
The namespace now resolves from the pbs **storage** (`StorageTarget.PBSNamespace`, from storage.cfg's
|
||||
`namespace`) — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with
|
||||
the backup that produced the snapshot. `backup_target` resolves from the **primary tier of
|
||||
`cfg.Backup.BackupTiers()`** — the function the scheduler itself consults — and carries the mountpoint,
|
||||
which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`.
|
||||
|
||||
**No new recovery path.** `AttachDrive`'s normalize leg already did the repair and three call sites already
|
||||
invoked it (20 s ticker, agent startup, and the controller's `Return` branch **before** `restartStacks`).
|
||||
All three died on `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about a dead namespace.
|
||||
**Unresolvable is recorded as unresolvable**, never defaulted: `namespace_state` and `backup_target.state`
|
||||
are `resolved | unknown`, with distinct reasons (`agent_backup_config_unavailable`, `not_a_known_storage`,
|
||||
`no_pbs_storage_observed`). An unconfigured namespace still reads `"root"` — that is an answer, not a gap.
|
||||
|
||||
**Ordering trap, caught by a test:** reading the abort flag before comparing devices classifies the real
|
||||
return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still
|
||||
reporting correctly** — so the abort flag is read off the RAW mount in the stale case.
|
||||
**The resolver reports the tier IN EFFECT, not the file on disk.** A backup-target move rewrites
|
||||
`agent.json` and deliberately does not restart the agent (the E-1 lesson), so a disk re-read would name a
|
||||
storage no archive had reached yet.
|
||||
|
||||
## Live validation, demo-hp
|
||||
## 4. Files
|
||||
|
||||
| State | raw | bind | verdict | usable |
|
||||
|---|---|---|---|---|
|
||||
| healthy | `8:16` | `8:16` | `live` | true |
|
||||
| **R-117 return** | `8:32` `/dev/sdc` | `8:16 …,shutdown` | **`stale-device`** | **false** |
|
||||
| **Q7 in-place abort** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** |
|
||||
`internal/hub/report.go`, `internal/storage/observe.go`, `internal/hub/dr_recipe.go`,
|
||||
`internal/hub/collect.go`, `cmd/felhom-agent/main.go`, `internal/hub/testdata/host-report.golden.json`,
|
||||
`REUSE.md`; tests in `internal/hub/{dr_recipe,contract,report}_test.go`, `internal/pbs/live_reporter_test.go`.
|
||||
|
||||
340–497 µs per call. **No block I/O proven by `strace`**: only `/proc/self/mountinfo`, **0** `statfs`.
|
||||
**No regression through the real pipeline:** `GET /disks` with the controller's own credential shows the
|
||||
live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable
|
||||
and zero spurious transitions.
|
||||
## 5. The cross-repo half that is easy to miss
|
||||
|
||||
## Tests + red-proofs
|
||||
The hub's `AssembleDRRecipe` **allow-lists** top-level keys. `backup_target` would have been stored intact
|
||||
and dropped before any operator saw it. The same mechanism already cost `offsite_restic` its whole
|
||||
existence (**R-122**, found this session). `REUSE.md` now records this as a two-repo change.
|
||||
|
||||
14 new tests; the payload ones drive the real `/disks` handler and the `AttachDrive` ones record the actual
|
||||
privileged calls. Fixtures are the spike's captured mountinfo, injected as **data** via
|
||||
`procSelfMountinfo`/`procGuestMountinfo`, so the real parser, predicate, `AttachDrive` and handler all run.
|
||||
## 6. Tests
|
||||
|
||||
**6 red-proofs, each verified to have landed.** RP1/RP2 (term 3 removed per site), RP3 (**the P1-only
|
||||
fix**), RP4 (`emergency_ro` dropped), RP5 (unknown reported dead), RP6 (aborted arm falls through to a
|
||||
re-bind).
|
||||
9 new. Consequence-level: a box with two `content=backup` storages names the **live** one and not the
|
||||
frozen one; `assertBackupCandidateAmbiguity` fails if the fixture ever stops posing that problem. Fixtures
|
||||
are the storage set demo-felhom really had, provenance recorded in-file. Two tests run the real
|
||||
`Collector.Collect()` path, one pinning that an **unwired seam yields unknown**.
|
||||
|
||||
**A hollow test was caught, by RP1 failing to fail.** The aborted fixture first used a `/dev/mapper`
|
||||
device, for which `RoleForStorage` derives `role="system"` — a system row never enters the block that
|
||||
computes `BoundUnderParent`, so the field stayed false **by default**, the assertion passed, and no mutation
|
||||
could ever fail it. Fixtures now assert the production row shape before asserting the field.
|
||||
**4 red-proofs**, each mutation asserted to have landed before running: revert R-106 → the live `"root"`
|
||||
symptom reproduces; make unknown guess a default → 5 assertions fire; drop the field → the cross-repo
|
||||
contract guard fires; hollow the fixture → the ambiguity guard fires.
|
||||
|
||||
## Not yet done / not covered
|
||||
Suite: **rc=0, 29 packages ok, 0 FAIL** (run separately from the commit).
|
||||
|
||||
- **The stale-bind repair on hardware.** `StablePathForRaw` hardcodes the production
|
||||
`/mnt/felhom-drives`, so exercising `AttachDrive` for a scratch drive writes a submount into guest
|
||||
9201's namespace, which the task fences. It happened once inadvertently, was reverted within a minute
|
||||
and 9201 verified unaffected, and was deliberately not repeated. Filed **R-117h**.
|
||||
- **Sustained-load behaviour of the predicate** — flagged unmeasured by the spike and **still unmeasured**.
|
||||
- **felhom-pve not upgraded** (0.115.0). The fleet is non-uniform.
|
||||
- **R-117g** — an aborted filesystem is never cleared automatically, by design, so it alarms until a human
|
||||
acts; there is no guided recovery.
|
||||
- **R-121** — a box's installed agent can sit releases behind the vouched one and nothing notices; the
|
||||
R-120 gate is golden-vs-fleet-controller and cannot see it.
|
||||
## 7. Not touched
|
||||
|
||||
## Observations (not acted on)
|
||||
|
||||
Guest 9201 has 2 stacked entries at `/mnt/felhom-drives/nvme-1tb` while the host has 1 (pre-existing;
|
||||
`AttachDrive` normalizes the host count only). 14 stale `felhom-agent.bak-*` binaries on demo-hp (~200 MB),
|
||||
nothing prunes them. 11 files in this repo fail `gofmt -l`, all pre-existing and none in `internal/localapi/`.
|
||||
The backup machinery. This corrects the record, not the doing — and the record was never acted on
|
||||
programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`), so the blast radius was
|
||||
exactly one operator reading a wrong value. `plan.PBS` inherits the fix for free.
|
||||
|
||||
Reference in New Issue
Block a user