R-109 + R-122: the recipe assembly stops dropping sections (hub v0.83.0)
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the forward-compat their comment advertised: a section an emitter adds is silently discarded until it is named in both the shape struct and AssembledRecipe. No error, no log, no failing test. R-122 (found this session): that already happened and shipped. The controller has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the hub stored it for all three real customers, and appHalfShape never listed the key, so no delivered recipe has ever contained it. It stayed green because the fixture drAppHalf is hand-written and omits the field. R-109: the agent's new backup_target is a new top-level host-half section and would have been dropped identically, making the fix read as shipped while changing nothing an operator can see. 3 tests built on halves read verbatim out of the live dr_recipe table, plus 2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok. Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling.
This commit is contained in:
@@ -58,7 +58,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — |
|
||||
| **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — |
|
||||
| **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator |
|
||||
| **R-109** | **The DR recipe records no backup target.** It lists every storage's name/type/content but never which one holds the local archives — and each demo box now carries TWO `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives) | READY (XS) | — | Add the resolved `BackupTarget()` to the host-half. Third recipe-completeness defect beside R-105/R-106 | CC |
|
||||
| **R-109** | ~~The DR recipe records no backup target~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0 + hub v0.83.0, 2026-07-30) | — | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`. Three states, and unresolvable is recorded as unresolvable (`agent_backup_config_unavailable` / `not_a_known_storage`), never a default. **The resolver reads the daemon-start config on purpose:** a target move rewrites `agent.json` and deliberately does NOT restart, so a disk re-read would name a storage no archive had reached. **Needed a HUB half nobody had scoped** — `AssembleDRRecipe` allow-lists top-level keys, so the field would have been stored intact and dropped before any operator saw it (→ **R-122**). Evidence: `audits/R106-R109-recipe-completeness-2026-07-30.md` | — |
|
||||
| **R-106** | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | **SHIPPED + PROVEN-LIVE** (agent v0.118.0, 2026-07-30) | — | **Was open-but-UNREGISTERED on this page until 2026-07-30 (→ R-123)** — `ROADMAP.md:109` had it READY and the only mention here was inside R-109's prose. Namespace now resolves from the pbs STORAGE (storage.cfg's `namespace`), the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. An unconfigured namespace still reads `"root"` — that is an ANSWER, and `namespace_state` separates it from not knowing. Live: `demo-felhom` and `demo-hp` now report their own namespaces. Evidence: same audit | — |
|
||||
| **R-122** | ~~`AssembleDRRecipe` silently DROPPED `offsite_restic` — the offsite recovery location never reached any recipe~~ | **SHIPPED** (hub v0.83.0, 2026-07-30) | — | **Found 2026-07-30 while scoping R-109's hub half; it had already shipped and nobody knew.** The controller has emitted `offsite_restic` since fork-4 (*"so DR knows WHERE to recover from"*), the hub stored it for **all three real customers**, and `appHalfShape` never listed the key — so no delivered recipe has ever contained it. No error, no log, green suite, because the fixture `drAppHalf` is hand-written and omits the field. `hostHalfShape`/`appHalfShape` are **ALLOW-LISTS dressed as forward-compat**; `TestAssembleDRRecipe_CarriesEveryEmittedSection` is now the guard, built on halves read verbatim out of the live `dr_recipe` table. `REUSE.md` (both repos) records that a recipe section is a TWO-REPO change | — |
|
||||
| **R-123** | **R-105 and R-106 were `READY` in `ROADMAP.md` with no row on THIS page** — each referenced only inside R-109's prose, which is precisely the thread-loss the register exists to prevent | **PARTLY CLOSED** (2026-07-30) | — | R-106 registered above (and shipped). **R-105 still needs a row** — it is M-sized, is about three hub-held DR records being `{}`, and is NOT part of the recipe-completeness set that shipped today. **The process gap is the real item:** nothing checks that a `READY` ROADMAP row has an OPEN-ITEMS row. A grep-level gate would catch it | CC |
|
||||
| **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC |
|
||||
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
||||
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
|
||||
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
||||
|
||||
@@ -106,10 +106,10 @@
|
||||
| R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 |
|
||||
| R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 |
|
||||
| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 |
|
||||
| R-106 | **The DR recipe records the PBS namespace as `"root"` on every box** while the real namespaces are `demo-felhom` / `demo-hp` | XS | READY — 2026-07-28 | Traced to source: `Snapshot.Namespace` decodes a `ns` field (`felhom-agent/internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so it is empty and `ToHub` normalises empty → `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` writes into the recipe (`internal/hub/dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as a coordinate (`:56`). Verified against ep0's real namespaces and a live `pct restore` command line carrying `--ns demo-felhom`. Flips: `07` §4 |
|
||||
| R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.0, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage <pbs>` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
|
||||
| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `<drive>/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed |
|
||||
| R-109 | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` (`felhom-agent/internal/hub/dr_recipe.go:86`) emits `guests`, `drives`, `pve_storage` and the PBS coordinate, but nothing that says WHICH storage holds the local whole-guest archives. Harmless while the target was the well-known `local`; not harmless now — after the 2026-07-28 vzdump-target move each demo box carries **two** `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives, never refreshed again) | XS | READY — 2026-07-28 | Found by `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` Phase 0 §5 and confirmed live in the recipe both boxes emit. A restorer reading the recipe cannot distinguish the live target from the stale one, and picking wrong yields a silently outdated guest. Add the resolved `BackupTarget()` to the host-half. Sits with **R-105**/**R-106** as the third recipe-completeness defect. Flips: `07` §4 |
|
||||
| R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.0 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 |
|
||||
| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |
|
||||
|
||||
## P3 — post-alpha
|
||||
|
||||
Reference in New Issue
Block a user