From b5a73e050bc9800b4849e4d6900dd9b4cecfcd64 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 21:38:13 +0200 Subject: [PATCH] Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom) Supervised operational run. No code, no version bump, nothing deleted. Primary backup tier on both demo boxes moved from `local` (a dir storage on /var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir storage on each box's secondary drive: demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B Both proven end to end via the real UI path: archive lands on the secondary drive (df delta matches the archive byte-for-byte), restore-test auto-selects it and passes with mount_parity: ok, and freshness survives an agent restart with an empty in-memory store -- so the age can only have come from the new storage. Phase 0: the target is CONFIGURATION, not converged (the sole writer of agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path. Findings: - F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount and the target reports disconnected permanently (observe.go:321) - F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a non-mounted path reports active with the ROOT filesystem's free space and had already created dump/ on pve-root -- a silent retarget onto the very device this change escapes - F-3 FelhomAgentStore is granted per storage path; without it every backup 403s. felhom-host-install.sh must issue it for new installs - R-109 (new) the DR recipe records no backup target, and each box now carries two content=backup dir storages, one live and one frozen - R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the enrolled drives were never PVE storages, so isUserDataDrive never saw them. Both boxes now populate drives; SMART on the backup drives too Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half live-proven; agent half source-traced). That is NOT the intended fall-back-and- alarm design -- filed as E-2 with the honest single-drive label. Reported in full in the record: the agent was restarted with a felhom-pbs backup in flight, producing a spurious tier failure. The backup had in fact succeeded (PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no channel -- R-84 ground truth superseded it. Outstanding: full drive-loss recovery (needs physical access) and the agent half of the absent-drive behaviour. --- REPORT.md | 153 +++-- documentation/backlog/OPEN-ITEMS.md | 2 + documentation/backlog/ROADMAP.md | 4 +- .../RUNBOOK-vzdump-target-move-2026-07-29.md | 563 ++++++++++++++++++ 4 files changed, 641 insertions(+), 81 deletions(-) create mode 100644 documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md diff --git a/REPORT.md b/REPORT.md index 2040b59..1213962 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,103 +1,96 @@ -# REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28) +# REPORT — the local whole-guest backup moved off the guest's own device (2026-07-28) -**Overwritten** per the standing rule. **Shipped: controller v0.183.0**, live on **both** demo boxes. -Fleet: hub v0.80.0, agent v0.110.0, controller **0.183.0**. `peti-felhom` untouched. +**Overwritten** per the standing rule (the prior C9-F1/C9-F2 text stays in git history). +**Supervised operational run — no code, no version bump.** Fleet unchanged: hub v0.80.0, +agent v0.110.0, controller v0.183.0. `peti-felhom` untouched. **Nothing was deleted.** -Both defects are the same shape — the system reporting healthy while the customer is not — and both -live in the same status-derivation code. +Record: `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`. -## Phase 0 — the asymmetry, sized before designing anything +--- -Tier-2 writes **two** things on every run: the capture legs (`hdd/`, `userdata/`) and, always, a full -`recovery-unit/` (DB dumps + named-volume tarballs). `RestoreTier2Files` reads **only the two legs** -(`tier2_restore.go:101-104`) and has never opened `recovery-unit/`. +## What changed -All 53 catalog templates enumerated, cross-checked against both boxes' actual copies: +On both demo boxes the primary backup tier moved from `local` — a `dir` storage on `/var/lib/vz`, +i.e. **the same physical device as the guest itself** — to **`felhom-backup`**, a `dir` storage on +the secondary drive. -| class | count | what the restore can return | +| | demo-felhom | demo-hp | |---|---|---| -| **A** | **9** | the file legs only — **never** their database or named volumes | -| **B** | **43** | **nothing at all** — a guaranteed no-op, forever | -| C | 1 | `bentopdf`, stateless | +| Target drive | `/dev/sdb` USB HDD → `/mnt/hdd_1` | `/dev/nvme0n1` → `/mnt/nvme-1tb` | +| Durable id | `uuid:47a3361a-91e0-4831-a69d-27f540ed3f48` | `uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` | +| Archive proven | 5,957,878,962 B | 2,256,044,492 B | +| Restore-test | `pass`, **`mount_parity: ok`**, 1 m 24 s | `pass`, **`mount_parity: ok`**, 1 m 54 s | -Four apps (`plex`, `jellyfin`, `emby`, `navidrome`) are in B only because their single bind is a -`:ro` media mount, which `ClassifyBinds` correctly excludes. **81% of the catalog.** +Three steps per box: `pvesm add dir … --is_mountpoint 1`, two `pveum acl modify`, and a one-line +`local_backup_target` edit in `agent.json`. -**The asymmetry is Tier-2's alone.** Tier-1 (`RestoreFromRecoveryUnit`) and offsite both restore the -unit and replay volume dumps — so BookStack already had a working restore; only this button lied. +**Drive loss is now locally recoverable in principle on both boxes** — the whole-guest archive lives +on different hardware from the guest, and a restore from it boots and passes mount parity. The proof +that closes matrix row 4 (pull the system drive and restore for real) needs physical access and is +outstanding. -## Shipped — Part 1a (honesty) +## Phase 0 verdict -- **Refuses UP FRONT.** `Tier2RestoreCoverage` is consulted before any op begins; a class-B app is - refused **without being stopped**. Live: BookStack uptime stayed `Up About an hour` (Campaign 9 - left it at `Up 25 seconds`). -- **Names the working action** rather than dead-ending 81% of the catalog: - > „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem - > állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon." -- **Claims only what was examined** (the QUIET half — immich's 1.3 GB Postgres unit is not covered, - so the old blanket sentence was a clean bill of health over data never opened): - > „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem - > tartoznak ebbe a visszaállításba." +**The target is configuration, not converged — the runbook held and the §3 STOP did not fire.** +Exactly one writer of `agent.json` exists (`pbsdr.seedEscrowStorageID`); it touches only +`escrow.pbs_storage_id` and preserves unknown keys verbatim via a `map[string]json.RawMessage` +read-modify-write. Nothing on the hub pushes agent config. No consumer hardcodes `"local"` on the +backup path — runners are one-per-tier, `NewestArchiveTime` reads its own target, and +`restoreTierForArchive` classifies from the archive, not from config. -## Shipped — Part 2 (C9-F2) +## What the run found -`StateRestarting` is deliberately **NOT** added to `IsDownState` — that alarms on every deploy and -update fleet-wide. A **sustained** run becomes down after `crashLoopAfter = 5m`, set above the three -real numbers already in the codebase: the deploy flow's **120 s** health timeout, Mealie's **60 s** -`start_period`, and R-97b's **180 s** quiesce grace (so the windows compose into one bounded delay -instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside -it. Carried by `Stack.RestartingSince` (not persisted) + `Stack.CrashLooping(now)`, used by **both** -the alarm and the dashboard counter — which previously counted `restarting` as running and so -contradicted the alarm on the same screen. +- **F-1 (before any command).** The storage `path` must **be** the drive's mountpoint. A + subdirectory fails `exactMount` (`internal/storage/observe.go:321`), so the target would report + `disconnected` **permanently** and its durable id would degrade off the filesystem UUID. +- **F-2 (before any command).** `--is_mountpoint 1` is load-bearing. **Proven live:** an unguarded + `dir` storage on a non-mounted path reports `active`, advertises the **root filesystem's** free + space, and had already created `dump/` on `/dev/mapper/pve-root` — a silent retarget onto the + exact device this change exists to escape. The guarded one refuses outright. +- **F-3 (found by the first real backup).** `FelhomAgentStore` is granted **per storage path**; a + new target without its own grant 403s every backup. **`felhom-host-install.sh` must issue it for + new installs**, or a new box ships with a tier that fails on its first run. +- **R-109 (new).** The DR recipe records **no backup target** — and each box now carries two + `content=backup` dir storages, one live and one holding frozen 2026-07-28 archives. +- **R-105 narrowed and traced.** `dr_recipe.host_half.drives` was `[]` fleet-wide with the cause + untraced. Cause: the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them. + Both demo boxes now populate `drives`, and the backup drives gained SMART reporting. R-105's other + two fields are untouched. -## Live replay (demo-hp + demo-felhom) +## Absent-drive behaviour (Part 3) -| # | scenario | result | -|---|---|---| -| 1 | **crash loop alarms** | Ten consecutive 30 s samples **silent** through the threshold window, then `17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma` — **5m25s** after the loop began (5 min + one scan). | -| 1b | **heartbeat COUNTS it** | `17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down**` — Campaign 9's evidence was `0 currently down` while an app looped. | -| 2 | **normal deploy is silent** | A real `uptime-kuma` deploy produced only `app_deployed (info)`; no alarm, with deadapp-check running every 30 s throughout. | -| 3 | **restore refuses without an outage** | The honest message rendered; `[WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped`; BookStack uptime unbroken. | -| 4 | **paperless still restores** (regression guard on Campaign 9's headline) | 3 files deleted → restored → **43/43 byte-identical to the pre-deletion sha256 set**, `documents_ok 16 of 16 problems []`. | +Today: **fail loudly, no silent retarget.** The PVE half is live-proven with throwaway storages (no +live drive was unmounted). The agent half is source-traced: `targetStoragePresent` checks name +presence only, never `Reachable`, so the tier stays DUE, the controller quiesces, vzdump is refused, +and the run fails and alarms. -## Filed, not fixed +**This is not §6's intended design** (fall back to the system drive and alarm). There is no fallback +at all, so an absent drive means no local backup until a human intervenes. Filed as **E-2**, together +with the honest single-drive label — a one-drive box protects against corruption only, and two +drives is effectively a hardware requirement for drive-loss protection. -- **C9-F1b** — route class-B apps to the Tier-1 unit restore from the card the customer already - opened. Its own task **deliberately**: it puts a DESTRUCTIVE operation (overwrites live data with - the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry - that difference. -- **C9-F4** — **nothing reads the Tier-2 copy's `recovery-unit/` mirror.** Written by every Tier-2 run - (`tier2.go:369`), read by no path: `RecoveryUnitPath` resolves to `backups/**primary**/` - (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`. - Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary - unit is gone while this mirror survives on the second drive, unreachable by any customer action, - leaving offsite as the only route. **Potentially larger than C9-F1.** +## One operational error, reported in full -## Tests +**The agent was restarted on demo-hp with a `felhom-pbs` backup in flight**, against §4.3. The +in-flight check was done before the first restart and not repeated before the second. It produced a +**spurious tier failure** (`context canceled` while waiting) — the F-A1 class the project already +fixed once. -`go test ./...` **rc=0, 27 packages** — run and `rc` read *separately* from the commit. Six red-proofs -all observed, including the one that matters most: adding `StateRestarting` to `IsDownState` fails the -brief-restart test with *"every deploy and update would page the operator"*. +**The backup had not failed:** the PVE task returned OK and the PBS snapshot `2026-07-28T19:19:45Z` +is 6,264,034,053 B. The spurious failure **reached no channel** — the agent died with its in-memory +record and R-84 ground truth superseded it; the controller's event trail for the window shows only +the real 403 failure and its recovery. The design absorbed it; the mistake was still a mistake, and +on a slower tier the same slip could have aborted a multi-hour WAN upload. -## Observations +Also recorded: `pgrep -f vzdump` self-matches a polling script's own command line and is not a safe +in-flight check — use the PVE task list. -- **A seventh shipped-invariant-comment.** `controller/README.md` stated that faults "still surface as - `exited`/`degraded`/**`restarting`**/`unhealthy`" — but `restarting` was in no down set at all. The - sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented - beside it. -- **Pre-existing gate failure, not mine:** `scripts/docker_run_volume_path_gate.py` fails on - `internal/appexport/estimate.go:179` (an unreviewed `docker run -v`). Verified it fails identically - on clean HEAD; left alone as out of scope rather than silently "fixed". -- The other six gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`, `app_row_dedup`, - `offbox_rename`). +## State at close -## NOT yet live-validated — carried forward +Both boxes healthy. Primary and offsite tiers `due=false` with `age_state=known` on both; breakers +clear; agents `active`; no thrash and no spurious staleness. Target drives at 1–2 % used with SMART +`PASSED`. The old archives (18 GB demo-felhom, 5.2 GB demo-hp) are **left in place** as the rollback +and as the only evidence of what the previous configuration produced. -- **Tier-1 content recovery after real loss** — still the most valuable unproven item (Campaign 9's A2 - ran against an intact app; A3 used Tier-2). Unchanged by this work. -- The C9-F2 threshold under a **quiesce** cycle (Scenario C) is unit-proven but was not replayed live; - it needs a backup window plus an app that fails to come back. -- C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not - app-by-app. -- Host reboot mid-backup, three-way concurrency with GC, Scenario C live, `offsite_stale` firing, - F-HUB — all still open from Campaign 9. +**Outstanding:** full drive-loss recovery (physical access), and the agent half of the absent-drive +behaviour (needs a drive unmount that would break the guest bind on a remote box). diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index ab80940..06751ec 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -39,6 +39,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | | **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — | | **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — | +| **E-2** | Drive-role machinery around the moved vzdump target: backup-target role on `StoragePath`, wizard assignment (never by transport/`removable`), no automatic roles, stickiness, `felhom-host-install.sh` creating the target **+ its `FelhomAgentStore` ACL**, absent-target policy, retention/space on a shared drive, honest single-drive label, remaining fleet migration | **READY (M)** | — | Architecture change PROVEN by hand on both demo boxes 2026-07-28 (`runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`); this is the machinery around a proven thing. Flips matrix row 4 | CC | +| **R-109** | **The DR recipe records no backup target.** It lists every storage's name/type/content but never which one holds the local archives — and each demo box now carries TWO `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives) | READY (XS) | — | Add the resolved `BackupTarget()` to the host-half. Third recipe-completeness defect beside R-105/R-106 | CC | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 12aa472..268a9ac 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -99,10 +99,12 @@ | R-102 | **Tier-2 writes a full `recovery-unit/` mirror on every run and no code path reads it.** Written at `internal/backup/tier2.go:368-369` ("Unit leg (always)"); `RecoveryUnitPath` resolves to `backups/**primary**/` (`internal/appbackup/paths.go:46-48`) and the only reader of the secondary tree is `internal/backup/tier2_restore.go`, which reads `hdd/`+`userdata/` only (`:101-104`) | M | READY — 2026-07-28 | **Was C9-F4** (`OPEN-ITEMS.md`). The sharp edge is *when* it bites: Tier-2 exists for primary-drive loss, and in exactly that failure the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action — leaving offsite as the only route. LIVE: demo-felhom's `backups/secondary/{bookstack,docmost}/` hold `recovery-unit` and nothing else, at 156 MB and 86 MB. Flips: the Tier-2 row in map §C; `07` §6.3, §7.2 | | R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 | | R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 | -| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. Flips: `07` §4 | +| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 | | R-106 | **The DR recipe records the PBS namespace as `"root"` on every box** while the real namespaces are `demo-felhom` / `demo-hp` | XS | READY — 2026-07-28 | Traced to source: `Snapshot.Namespace` decodes a `ns` field (`felhom-agent/internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so it is empty and `ToHub` normalises empty → `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` writes into the recipe (`internal/hub/dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as a coordinate (`:56`). Verified against ep0's real namespaces and a live `pct restore` command line carrying `--ns demo-felhom`. Flips: `07` §4 | | R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 | | R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed | +| R-109 | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` (`felhom-agent/internal/hub/dr_recipe.go:86`) emits `guests`, `drives`, `pve_storage` and the PBS coordinate, but nothing that says WHICH storage holds the local whole-guest archives. Harmless while the target was the well-known `local`; not harmless now — after the 2026-07-28 vzdump-target move each demo box carries **two** `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives, never refreshed again) | XS | READY — 2026-07-28 | Found by `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` Phase 0 §5 and confirmed live in the recipe both boxes emit. A restorer reading the recipe cannot distinguish the live target from the stale one, and picking wrong yields a silently outdated guest. Add the resolved `BackupTarget()` to the host-half. Sits with **R-105**/**R-106** as the third recipe-completeness defect. Flips: `07` §4 | +| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 | ## P3 — post-alpha diff --git a/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md b/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md new file mode 100644 index 0000000..7fe3b64 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md @@ -0,0 +1,563 @@ +# RUNBOOK RECORD — the local whole-guest backup moved off the guest's own device (2026-07-28) + +**Class:** supervised operational change, one box at a time. **No code, no version bump** — the +change is a PVE storage definition, a per-storage ACL grant, and a one-line target switch in +`agent.json`. **Nothing was deleted.** + +**Boxes:** `demo-hp` (`felhom-host`, HP t740) first, then `demo-felhom` (N100). +**Change:** primary backup tier `local` → **`felhom-backup`**, a `dir` storage on each box's +secondary drive. +**Window:** 2026-07-28 21:11 → 21:35 CEST. + +> **Deliverable note.** The spec is `RUNBOOK-vzdump-target-move-2026-07-29.md`; the work ran on the +> evening of 07-28. The filename is kept as specified. + +--- + +## 1. Phase 0 — ground truth, established read-only before anything was touched + +### 1.1 The premise holds: `local` and `local-lvm` are the same physical device + +| | demo-felhom | demo-hp | +|---|---|---| +| `local` (dir, `/var/lib/vz`) | `pve-root` → **`sda3`** | `pve-root` → **`sda3`** | +| `local-lvm` (lvmthin, `pve/data`) | **`sda3`** | **`sda3`** | +| System drive | AirDisk 512 GB SATA SSD (`sda`) | SanDisk X600 M.2 128 GB (`sda`) | + +A drive failure took the guest and its only local backup together. That is the fault being closed. + +### 1.2 Physical inventory and the durable id + +| | demo-felhom | demo-hp | +|---|---|---| +| Secondary drive | `/dev/sdb` TOSHIBA MQ04ABF100, **USB**, rotational | `/dev/nvme0n1` KXG50PNV1T02, **NVMe** | +| Size / free before | 916 GB / 866 GB | 938 GB / 891 GB | +| Mountpoint | `/mnt/hdd_1` | `/mnt/nvme-1tb` | +| **Durable id (fs UUID)** | `47a3361a-91e0-4831-a69d-27f540ed3f48` | `91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` | +| `/sys/block/*/removable` | `0` | `0` | +| SMART at close | — | `PASSED`, 50 °C, 7 % wear, 0 media errors | + +Both drives are already enrolled Felhom drives, mounted **by filesystem UUID** through +agent-generated systemd units (`mnt-hdd_1.mount`, `mnt-nvme\x2d1tb.mount`), each marked +*"Managed by felhom-agent — do not edit by hand"*, `WantedBy=multi-user.target`. **Neither is in +`/etc/fstab`.** The UUID in the unit is the same durable id the DR recipe carries, so the target +drive was already durably identified before this change. + +This is §2's principle meeting the reference hardware head-on: demo-felhom's target **is** an +external USB HDD and both drives report `removable=0`. A rule keyed on transport would disqualify +demo-felhom; a rule keyed on the `removable` flag would see no removable drive at all on either box. +Neither attribute can decide a role. Both are fine as *hints*. + +### 1.3 Space + +The runbook's ~14.5 GB figure is the **PBS-side** measurement. The local `zstd` archive is far +smaller: + +| | newest local archive before | × `keep-last=3` | share of the target drive's free space | +|---|---|---|---| +| demo-felhom (9201) | 5.93 GB (trend 5.82 → 5.84 → 5.93) | **~17.8 GB** | **2.1 %** of 866 GB | +| demo-hp (9201) | 1.68 GB (1.61 → 1.63 → 1.68) | **~5.0 GB** | **0.6 %** of 891 GB | + +Retention is `local_backup_retention: 3` on both boxes and is unchanged by this runbook. + +The move also relieves the system drive — 18 GB on demo-felhom, 5.2 GB on demo-hp once the old +archives are eventually removed. On demo-hp that matters: its root is 40 GB with 24 GB free. + +### 1.4 How the target is set — **configuration, not converged. The runbook holds.** + +The primary tier is built from `backup.local_backup_target` (`felhom-agent/internal/config/config.go:541`, +`BackupTarget()`), which was `"local"` on both boxes. The extra tier is `felhom-pbs`, weekly. + +Every writer of `agent.json` was enumerated. **There is exactly one:** +`pbsdr.seedEscrowStorageID` (`internal/pbsdr/manager.go:454`). It sets only `escrow.pbs_storage_id`, +refuses to clobber a differing existing value (warn-and-keep), and performs a +`map[string]json.RawMessage` read-modify-write that **preserves unknown keys verbatim**. Nothing on +the hub pushes agent configuration. Seven prior hand-edit backups sit beside the live file. + +**A hand edit sticks. No code change was required, so the STOP in §3 did not fire.** + +### 1.5 What else reads the target — no hardcoded `"local"` on the backup path + +| Consumer | Verdict | +|---|---| +| Backup runners | One runner **per tier**, each constructed with its own `TargetID` (`cmd/felhom-agent/main.go:1355`). Parameterised. | +| `NewestArchiveTime` | Reads `r.target` (`internal/backup/runner.go:390`). Parameterised. | +| Restore-test rotation | Iterates `BackupTiers()`; `restoreTierForArchive` classifies from the **archive**, never from config (the R-82/R-85 fix). Parameterised. | +| Per-run prune | `localPruneSpec` suppresses only for **PBS**-type targets; a `dir` target still prunes at `keep-last=3`. Correct. | +| Customer-facing label | `backupIsPBS` (`felhom-controller/.../backup_handlers.go:174`) matches the substring `pbs`. `felhom-backup` correctly renders „Helyi tároló (felhom-backup)". **Naming trap recorded** — a target named with `pbs` in it would be mislabelled as offsite. | + +Two Phase 0 findings that shaped the change are in §4 and §6. + +--- + +## 2. The two findings Phase 0 turned up before any command ran + +### F-1 — the storage path must BE the mountpoint, or the target reports disconnected forever + +`reportType` (`internal/storage/observe.go:365`) calls a `dir` storage `local` only when it is +literally named `local`; anything else becomes `usb` or `local-dir`. For those two types +`reachable()` (`:321`) requires **`exactMount`** — the storage `path` must exactly equal a +mountpoint (`exactMountDevice`, `:422`). + +Point the storage at a **subdirectory** of the drive and `exactMount` is false, so +`reachable=false`, `state=disconnected`, **permanently** — a false alarm designed in on day one, +and the durable id degrades to a path fallback instead of the filesystem UUID. + +**Decision: `path` = the drive's mountpoint.** PVE creates `dump/` at the drive root, beside the +existing `felhom-data/`. This also gives a real safety property: the guest is bind-mounted only the +drive's `felhom-data` subpath (`/mnt/felhom-drives/`), so **the archives sit outside anything +the customer's guest can reach.** + +### F-2 — `is_mountpoint` is what stops a silent retarget onto the system drive + +`man pvesm`: *"Assume the given path is an externally managed mountpoint and consider the storage +offline if it is not mounted."* Without it, an unplugged or late-mounting drive leaves +`/mnt/` a bare directory on the root filesystem and vzdump writes the backup **onto the +system drive** — the exact device this change exists to escape, silently. Proven live in §6. + +**Decision: `--is_mountpoint 1` on both boxes.** + +--- + +## 3. The exact commands used, per box + +Identical on both, differing only in the mountpoint. Run as `root` on the PVE host. + +```bash +# 1. the storage, on the drive's OWN mountpoint, guarded (F-1 + F-2) +pvesm add dir felhom-backup --path /mnt/nvme-1tb --content backup --is_mountpoint 1 # demo-hp +pvesm add dir felhom-backup --path /mnt/hdd_1 --content backup --is_mountpoint 1 # demo-felhom + +# 2. the per-storage grant — REQUIRED, and the reason the first backup failed (§4) +pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --users felhom-agent@pve +pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent' + +# 3. back up the config, then repoint the primary tier +cd /etc/felhom-agent +cp -p agent.json agent.json.pre-e-target-move +# in-place truncate-write (NOT sed -i): the directory is root-owned while the file is +# agent-owned 0600, so a rename would flip ownership and the non-root agent could not read it. +python3 - <<'EOF' +import json +p='/etc/felhom-agent/agent.json' +raw=open(p).read() +old='"local_backup_target": "local"' +new='"local_backup_target": "felhom-backup"' +assert raw.count(old)==1, 'occurrences: %d' % raw.count(old) +out=raw.replace(old,new); json.loads(out) # validate BEFORE writing +with open(p,'r+') as f: f.seek(0); f.write(out); f.truncate() +EOF + +# 4. restart only with nothing in flight — check the PVE task list, NOT pgrep (see §8) +pvesh get /nodes//tasks --limit 20 --output-format json | python3 -c \ + 'import sys,json;print([t for t in json.load(sys.stdin) if "endtime" not in t] or "none running")' +systemctl restart felhom-agent +``` + +Resulting stanza, both boxes: + +``` +dir: felhom-backup + path /mnt/ + content backup + is_mountpoint 1 +``` + +**Verification that Part 1 took**, both boxes — the diff was exactly one line, ownership and mode +preserved (`-rw------- felhom-agent felhom-agent`), and both tiers armed with no rejection: + +``` +backup tier armed target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s primary=true +backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s primary=false +``` + +`GET /backup/due` echoes the new tier, and the **old target no longer exists** — the cleanest proof +the switch took: + +``` +GET /backup/due → {"due":true,"reason":"no successful backup recorded yet","age_state":"absent"} +GET /backup/due?target=felhom-backup → {..., "target":"felhom-backup", "age_state":"absent"} +GET /backup/due?target=felhom-pbs → {"due":false,"reason":"within cadence window","age_seconds":44362} +GET /backup/due?target=local → {"ok":false,"error":"unknown backup target: local"} +``` + +--- + +## 4. Finding F-3 — the agent had no PVE privilege on the new storage (found by the first real backup) + +The first triggered backup on demo-hp failed: + +``` +local-api: backup job failed vmid=9201 target=felhom-backup + err="proxmox: POST /nodes/felhom-host/vzdump -> HTTP 403: permission denied + at /storage/felhom-backup (missing privilege Datastore.Allocate)" +``` + +`FelhomAgentStore` (`Datastore.Allocate,Datastore.AllocateSpace`) is granted **per storage path**, +to both `felhom-agent@pve` and the `felhom-agent@pve!agent` token — `local`, `local-lvm` and +`felhom-pbs` each had their own pair. A new backup target needs the same pair or every backup 403s. + +The grant in §3 step 2 mirrors the existing ones exactly; **no privilege was widened**, and the +storage-scoped shape (rather than a grant at `/`) is preserved. + +**This is an E-2 requirement, not a one-off:** `felhom-host-install.sh` must issue the grant for +whatever target it provisions on a new install, or every new box ships with a backup tier that +403s on its first run. + +--- + +## 5. Part 2 — the proof + +### 5.1 A backup lands on the secondary drive — via the real path + +Triggered through the endpoint the UI's „Mentés most" button invokes, not `--selftest`. + +**Route correction worth recording:** the backups page has two triggers. +`POST /api/backup/run` is the **Tier-1 app-data** backup (DB dumps + volume dumps); the +**whole-guest** vzdump is `POST /api/guest-backup/trigger`, which goes through the controller's +quiesce loop (`ServeBackupAPI`, `backup_handlers.go`). The first attempt used the former and +correctly produced an app-data run and no vzdump. + +| | demo-hp | demo-felhom | +|---|---|---| +| Archive | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst` | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst` | +| Size | **2,256,044,492 B (2.26 GB)** | **5,957,878,962 B (5.96 GB)** | +| Duration | 86.3 s | ~197 s | +| Mode | `snapshot` — `snapshotted` at 21:18:16, app resumed before the copy finished | `snapshot` — `snapshotted` at 21:26:29 | +| `df` on the target drive | used 92,344,320 → **2,348,572,672** (**+2,256,228,352 B**) | used 3,600,314,368 → **9,558,208,512** (**+5,957,894,144 B**) | + +On both boxes the `df` delta matches the archive size, so the bytes demonstrably landed on the +secondary drive and not somewhere else. + +On demo-felhom the backup was taken by the controller's **first-backup safety valve** rather than by +the manual trigger: the agent restart made the new tier report `age_state: absent`, the controller +correctly refused to withhold a first backup, and the manual trigger that followed was answered +`{"error":"mentés már folyamatban van"}`. Same code path, same quiesce loop — the trigger simply +lost the race to the mechanism that exists for exactly this state. + +`uncovered_volumes: ["/etc/felhom-bootstrap", "/mnt/felhom-drives"]` — both are host binds and are +correctly outside the guest archive. Pre-existing behaviour, unchanged by this move, but see §7. + +### 5.2 A restore works from that archive + +Run with **no** `-archive`, so the candidate *selection* is part of the proof. The rotation needed +no nudge — it picked the new storage on its own. + +``` +restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst + into scratch band [990000,990009] on local-lvm … +{ + "source_archive": "felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst", + "source_tier": "local", + "scratch_vmid": 990000, + "pass": true, + "verified": "boot+running", + "mount_parity": "ok", + "duration_seconds": 114.39, + "mount_inventory": [ + "mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)", + "mp8=/mnt/felhom-drives (throwaway for the archived bind)", + "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)" + ] +} +=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m54s) === +``` + +**`mount_parity: ok`, `pass: true`, scratch torn down cleanly.** + +demo-felhom, same method, same result — the bigger guest (200 G + 50 G volumes): + +``` +restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst … + "pass": true, "verified": "boot+running", "mount_parity": "ok", + "duration_seconds": 84.10, "scratch_vmid": 990000, + "mount_inventory": ["mp0=/var/lib/docker (200G)", "mp1=/mnt/sys_drive (50G)", …] +=== selftest=restore-test OK (restored+booted+verified+torn-down in 1m24s) === +``` + +| | demo-hp | demo-felhom | +|---|---|---| +| Candidate picked automatically from the new storage | yes | yes | +| `pass` / `verified` | true / `boot+running` | true / `boot+running` | +| **`mount_parity`** | **ok** | **ok** | +| Duration | 1 m 54 s | 1 m 24 s | +| Scratch torn down | yes (990000) | yes (990000) | + +Run with **no** `-archive` on both boxes, so the candidate *selection* is part of the proof. The +rotation needed no nudge on either box — it picked the new storage on its own. + +`source_tier: "local"` is **correct, not a stale reference to the old storage name.** +`restoreTierForArchive` classifies by the archive's storage *type*; `felhom-backup` is a `dir` +storage, so the restore correctly earns the local (10-minute) wait bound rather than the PBS one. + +**Method caveat:** `--selftest=restore-test` is a separate one-shot process, so it does **not** +write the daemon's `rtState`. `GET /restore-test/status` still reads `null` on both boxes and the +customer-visible „utoljára ellenőrizve" is unchanged. The proof is real; the daemon's own 84 h +rotation is undisturbed and will record its next run normally. + +### 5.3 Freshness follows the target + +Immediately after the backup: `{"due":false,"age_seconds":159,"target":"felhom-backup","age_state":"known"}`. + +That alone does not separate the storage read from this process's in-memory record, so the agent was +**restarted** — which empties the in-memory store (R-84) — and re-asked. Both boxes: + +``` +demo-hp GET /backup/status → {"phase":"idle"} ← no in-memory record at all + GET /backup/due → {"due":false,"age_seconds":188, + "target":"felhom-backup","age_state":"known"} +demo-felhom GET /backup/status → {"phase":"idle"} + GET /backup/due → {"due":false,"age_seconds":267, + "target":"felhom-backup","age_state":"known"} +``` + +A `known` age with an empty in-memory store **can only** have come from reading the new storage. +The tier neither looks absent (which would re-backup every cycle) nor stale. This is the positive +observable, not the absence of a complaint. + +**This restart is also where the one operational error of the run happened — see §8.** + +### 5.4 The DR recipe does **not** record the new target — and one field improved + +**The finding:** `BuildDRRecipeHostHalf` (`internal/hub/dr_recipe.go:86`) records `guests`, +`drives`, `pve_storage` and the PBS coordinate. **Nothing in the recipe says which storage holds the +local archives.** While the target was `local` that was guessable; it is not any more, and it is now +actively misleading — after this change `local` still holds the *stale* archives while the live ones +are on `felhom-backup`. A restorer reading the recipe sees two backup-capable storages and cannot +tell which is which. **Filed as R-109**, alongside R-105/R-106. + +**The improvement — this change traces and partly closes R-105.** R-105 recorded +`dr_recipe.host_half.drives = []` on every customer *including two with enrolled data drives*, with +the cause explicitly untraced. It is now traced: **the enrolled drives were never PVE storages at +all**, so they never entered `report.StorageTargets`, so `isUserDataDrive` (which needs type +`usb`/`local-dir` **and** a durable id **and** a mount path) never saw them. Making the drive a +storage populated it. Live on demo-hp after the change: + +```json +"drives": [{ "durable_id": "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae", + "mount_path": "/mnt/nvme-1tb", "intent": "enrolled", + "total_bytes": 1006980812800 }] +``` + +The storage observation also gained a real device, a real UUID and **SMART** for the backup drive — +`health=PASSED temp=50 poh=28451 media_err=0 pct_used=7` — none of which existed when the drive was +not a PVE storage: + +``` +- felhom-backup type=local-dir state=attached reach=true class=fast + durable=uuid:91d2dc2d-… mount="/mnt/nvme-1tb" dev="/dev/nvme0n1" +``` + +demo-felhom shows the identical shape — `drives` populated, SMART now readable on the USB HDD: + +``` +- felhom-backup type=local-dir state=attached reach=true class=slow + durable=uuid:47a3361a-… mount="/mnt/hdd_1" dev="/dev/sdb" + smart: health=PASSED temp=35 poh=3581 realloc=0 pending=0 offline_unc=0 + +"drives": [{ "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48", + "mount_path": "/mnt/hdd_1", "intent": "enrolled", + "total_bytes": 983351140352 }] +``` + +**`dr_recipe.host_half.drives` is now non-empty on both demo boxes**, where R-105 recorded `[]` +fleet-wide. Note the classification detail: demo-felhom's drive is USB-attached but reports +`removable=0`, so it is typed `local-dir` rather than `usb`. That does not matter here — both types +take the same reachability branch and both satisfy `isUserDataDrive` — but it is one more reason +§2's "do not classify by transport" rule is right. + +R-106 is untouched and still live on both boxes: the recipe still records `"namespace": "root"` +where the real namespaces are `demo-hp` / `demo-felhom`. + +--- + +## 6. Part 3 — what happens when the target drive is absent + +Established with **throwaway storages**, so neither live drive was unmounted (unmounting would break +the guest's `felhom-data` bind and the customer's app data on a remote box). + +### Live-proven: the PVE half + +| | storage on a non-mount path, **no** `is_mountpoint` | same path, **`is_mountpoint 1`** | +|---|---|---| +| Creation | succeeded silently | **refused**: `unable to activate storage 'absent-test-b' - directory is expected to be a mount point but is not mounted: '/mnt/absent-b'` | +| `pvesm status` | **`active`**, 40516856 KiB total | never created | +| Backing device | **`/dev/mapper/pve-root`** — the system drive | — | +| What it created | `/mnt/absent-a/dump/` **on the root filesystem** | nothing | + +The unguarded storage reports `active` with plausible free space that is really the **root +filesystem's**, and it had already created a `dump/` directory there. A vzdump aimed at it would +write the whole-guest backup onto the system drive with no error and no alarm — **silent +retargeting onto the exact device this change exists to escape**, which §6 names as the worst +outcome. `is_mountpoint 1` converts that into a loud refusal. Both test storages were removed and +`/mnt/absent-a` deleted. + +### Source-traced, not live-proven: the agent half + +With the guard in place and the drive absent, the storage stays in `storage.cfg` but goes inactive. +`targetStoragePresent` (`internal/localapi/server.go:1212`) checks the target's **name presence +only**, never `Reachable`, so the tier is **not** deferred; `newestArchiveOn` cannot read the +storage and degrades to `archiveUnknown`, which stays **DUE** by design. The controller therefore +quiesces the apps and fires a vzdump, PVE refuses because the storage is not active, and the run +**fails loudly** — arming the R-88 breaker and the `backup_failed` operator mail. + +So today's behaviour is: **fail and alarm; no silent retarget.** That is the safe half. + +**It does not match §6's intended design** (*fall back to the system drive and alarm*): there is no +fallback at all, so an absent drive means **no local backup whatsoever** until a human intervenes, +and each attempt costs an app quiesce until the breaker backs off. Whether a corruption-only copy on +the system drive beats no copy is a product decision; the honest label matters either way. +**Filed as E-2.** + +This half is traced through source, not reproduced on hardware — the boxes are remote and unmounting +a live enrolled drive would break the guest bind and the customer's data. Recorded as outstanding +proof in §9. + +### Single-drive boxes + +A single-drive box has nowhere to move the target. There the backup stays on the system drive and +protects against **corruption only** — it is not drive-loss protection and must not be presented as +a backup tier that survives hardware failure. **Two drives is effectively a hardware requirement for +drive-loss protection**, and that belongs in the sales/onboarding material as well as in the UI +label. Filed under E-2. + +--- + +## 7. Scope recorded for E-2 — filed, not built + +| # | Item | +|---|---| +| 1 | A **backup-target role** on `StoragePath`, alongside `Schedulable` / `IsDefault` / `Kind`. | +| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). | +| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. | +| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. | +| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. | +| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. | +| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. | +| 8 | The honest **single-drive label**. | +| 9 | **Migrating the remaining fleet** — `peti-felhom` and any box not covered here. | +| 10 | Naming guard: `backupIsPBS` matches the substring `pbs`, so a target so named would be mislabelled to the customer as offsite (§1.5). | + +--- + +## 8. Observations + +### 8.1 An operational error in this run: the agent was restarted with a backup in flight + +§4.3 and §8 of the spec both say not to. It happened anyway, on demo-hp at **21:21:12**. + +The manual „Mentés most" correctly fires **every** tier, not just the primary, so the trigger at +21:18 started the local vzdump *and* a `felhom-pbs` one. The local tier finished at 21:19:41. The +PBS tier was still running. The in-flight check had been done before the **first** restart (21:12, +clean) and **was not repeated** before the second one — the restart taken for the §5.3 freshness +proof — so it landed on top of a live PBS backup. + +What it produced: + +``` +level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs + err="backup: vzdump task vmid 9201: proxmox: waiting for task UPID:…:vzdump:9201:… : context canceled" +``` + +**The backup had not failed.** `context canceled` is the agent being shut down while waiting, not +the task failing. Checked rather than assumed, against two independent authorities: + +- the PVE task `2026-07-28T21:19:45 … vzdump 9201` → **OK**; +- the PBS snapshot `felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z` → **6,264,034,053 B**, a real, + plausibly-complete archive (nine orders of magnitude above the F-CRIT-2 1 MiB floor). + +So the run recorded a **spurious tier failure** — a self-inflicted instance of exactly the F-A1 +class ("a healthy backup reported as FAILED") that the project fixed in controller v0.179.0. + +**It reached no channel, and the reason is worth keeping.** The controller's event trail for the +whole window is: + +``` +19:15:43Z whole_guest_backup_failed (error) felhom-backup tier — the REAL 403, backoff 15m +19:19:45Z whole_guest_backup_recovered (info) felhom-backup tier — cleared after the ACL grant +``` + +— and **nothing for the 21:21 PBS failure.** The agent process died with its own in-memory failure +record, and by the time the controller next polled, `NewestArchiveTime` had read ground truth from +PBS and found the successful 21:19:45 archive. The R-84 ground-truth design absorbed it: no breaker +armed on the offsite tier, no operator mail, no spurious staleness. That is the design working, not +a reason the mistake was harmless — on a slower tier, or with the restart a minute earlier, the same +slip could have aborted a multi-hour WAN upload. + +**The rule that would have caught it:** re-check in flight immediately before *every* restart, not +once at the start of the procedure — and remember that a manual trigger fires all tiers, so the +offsite tier can still be running long after the local one has finished. + +### 8.2 Method and tooling + +- **`pgrep -f vzdump` is not a safe in-flight check.** A polling script whose own command line + contains the string matches itself, and it reported a backup in flight when the PVE task list + showed none. It caused one false "ABORT: vzdump in flight" and one wrong reading in this run. + **Use the PVE task list** (`pvesh get /nodes//tasks`, filter for a missing `endtime`). +- **`sed -i` must not be used on `agent.json`.** The directory is root-owned and the file is + agent-owned `0600`; `sed -i` renames, which would flip the file to `root:root` and leave the + non-root agent unable to read its own config. The in-place truncate-write in §3 preserves both. +- A `felhom-pbs` vzdump failed at **17:58**, hours before this change, with + `Can't connect to 10.77.0.1:8007 (Connection refused)`. **Pre-existing and unrelated** — checked + rather than assumed: ep0 is up (uptime 1 d 2 h, `proxmox-backup-proxy` active), the WG handshake + is fresh, and 8007 is open from demo-felhom now. +- The controller's first-backup safety valve behaved exactly as designed on demo-felhom: the agent + restart made the new tier report `age_state: absent`, and the controller took a backup on its own + before the manual trigger could (`{"error":"mentés már folyamatban van"}`). +- The archives are outside the customer's reach by construction — the guest is bound only the + drive's `felhom-data` subpath, while `dump/` sits at the drive root. + +--- + +## 9. State at close + +### 9.1 Old archives — left in place, as instructed + +Nothing was deleted. They remain the rollback and the only evidence of what the previous +configuration produced. + +| Box | Path | Contents | Size | +|---|---|---|---| +| demo-felhom | `/var/lib/vz/dump/` | 3 × 9201 (5.82 / 5.84 / 5.93 GB) + 2 × 9100 (612 / 649 MB) | **18 GB** | +| demo-hp | `/var/lib/vz/dump/` | 3 × 9201 (1.61 / 1.63 / 1.68 GB) + 1 × 9100 (649 MB) | **5.2 GB** | + +**These are now stale and will never be refreshed** — the tier that wrote them no longer points +here, and `keep-last=3` prunes only the *new* target. They are also the reason R-109 matters: a +restorer reading the DR recipe sees both `local` and `felhom-backup` carrying `content=backup` and +cannot tell that one holds live archives and the other holds a frozen snapshot of 2026-07-28. +Deleting them is a separate, deliberate decision. + +### 9.2 Fleet state — both boxes healthy, no thrash, no spurious staleness + +| | demo-felhom | demo-hp | +|---|---|---| +| Agent | `active`, v0.110.0 | `active`, v0.110.0 | +| Primary tier | `felhom-backup`, `due=false`, `age_state=known` | `felhom-backup`, `due=false`, `age_state=known` | +| Offsite tier | `felhom-pbs`, `due=false`, `age_state=known` | `felhom-pbs`, `due=false`, `age_state=known` | +| New target | 5.96 GB used of 916 GB (**2 %**) | 2.26 GB used of 938 GB (**1 %**) | +| Target drive SMART | `PASSED`, 35 °C | `PASSED`, 50 °C, 7 % wear | +| System drive | 24 G / 94 G (27 %) | 14 G / 39 G (38 %) | +| Agent errors, last 15 min | **0** | 2, both accounted for (§4 403, §8.1 spurious) | +| Breaker | clear (`whole_guest_backup_recovered` 19:19:45Z) | clear | + +No tier is stale, no tier is thrashing, and both offsite tiers report a known, recent age. + +### 9.3 Outstanding proof + +1. **Full drive-loss recovery** — pull the system drive, restore the guest from the vzdump on the + secondary drive, confirm it returns with named volumes, secrets and config. Needs physical + access; the boxes are remote until ~08-02. This is the proof that flips **matrix row 4** from + PARTIAL to a real local route. Everything up to it is now proven: the archive exists on separate + hardware, and a restore from it boots and passes `mount_parity`. +2. **The agent half of §6** — with the drive absent, that the tier fails loudly rather than silently + retargeting. The **PVE half is live-proven** (§6); the agent half is traced through source only, + because unmounting a live enrolled drive would break the guest's `felhom-data` bind and the + customer's app data on a remote box. + +### 9.4 Backlog rows filed + +| ID | What | +|---|---| +| **E-2** | The machinery around this change — backup-target role on `StoragePath`, wizard assignment, no automatic roles, stickiness, `felhom-host-install.sh` creating the target **and** its ACL, absent-target policy, retention/space accounting on a shared drive, the honest single-drive label, remaining fleet migration. Full scope in §7. | +| **R-109** | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` lists every storage's name/type/content but never says which one holds the local archives — and after this change the box carries two `content=backup` dir storages, one live and one stale. §5.4. | +| **R-105** | Partially closed and **fully traced** for its third field: `dr_recipe.host_half.drives` is now populated on both demo boxes, and the cause of the fleet-wide `[]` is identified — the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them. The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched by this run. |