admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)

Browse Source 
Supervised operational run. No code, no version bump, nothing deleted.

Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:

  demo-hp      /mnt/nvme-1tb  uuid:91d2dc2d-...  archive 2,256,044,492 B
  demo-felhom  /mnt/hdd_1     uuid:47a3361a-...  archive 5,957,878,962 B

Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.

Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.

Findings:
- F-1  the storage path must BE the mountpoint; a subdirectory fails exactMount
       and the target reports disconnected permanently (observe.go:321)
- F-2  --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
       non-mounted path reports active with the ROOT filesystem's free space and
       had already created dump/ on pve-root -- a silent retarget onto the very
       device this change escapes
- F-3  FelhomAgentStore is granted per storage path; without it every backup
       403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
       two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
       enrolled drives were never PVE storages, so isUserDataDrive never saw
       them. Both boxes now populate drives; SMART on the backup drives too

Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.

Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.

Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
This commit is contained in:
authentik Default Admin
2026-07-28 21:38:13 +02:00
parent 070b0ce2dc
commit b5a73e050b
4 changed files with 641 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+73 -80
View File
@@ -1,103 +1,96 @@
# REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)
# REPORT — the local whole-guest backup moved off the guest's own device (2026-07-28)
**Overwritten** per the standing rule. **Shipped: controller v0.183.0**, live on **both** demo boxes.
Fleet: hub v0.80.0, agent v0.110.0, controller **0.183.0**. `peti-felhom` untouched.
**Overwritten** per the standing rule (the prior C9-F1/C9-F2 text stays in git history).
**Supervised operational run — no code, no version bump.** Fleet unchanged: hub v0.80.0,
agent v0.110.0, controller v0.183.0. `peti-felhom` untouched. **Nothing was deleted.**
Both defects are the same shape — the system reporting healthy while the customer is not — and both
live in the same status-derivation code.
Record: `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.
## Phase 0 — the asymmetry, sized before designing anything
---
Tier-2 writes **two** things on every run: the capture legs (`hdd/`, `userdata/`) and, always, a full
`recovery-unit/` (DB dumps + named-volume tarballs). `RestoreTier2Files` reads **only the two legs**
(`tier2_restore.go:101-104`) and has never opened `recovery-unit/`.
## What changed
All 53 catalog templates enumerated, cross-checked against both boxes' actual copies:
On both demo boxes the primary backup tier moved from `local` — a `dir` storage on `/var/lib/vz`,
i.e. **the same physical device as the guest itself** — to **`felhom-backup`**, a `dir` storage on
the secondary drive.
| class | count | what the restore can return |
| | demo-felhom | demo-hp |
|---|---|---|
| **A** | **9** | the file legs only — **never** their database or named volumes |
| **B** | **43** | **nothing at all** — a guaranteed no-op, forever |
| C | 1 | `bentopdf`, stateless |
| Target drive | `/dev/sdb` USB HDD → `/mnt/hdd_1` | `/dev/nvme0n1` → `/mnt/nvme-1tb` |
| Durable id | `uuid:47a3361a-91e0-4831-a69d-27f540ed3f48` | `uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
| Archive proven | 5,957,878,962 B | 2,256,044,492 B |
| Restore-test | `pass`, **`mount_parity: ok`**, 1 m 24 s | `pass`, **`mount_parity: ok`**, 1 m 54 s |
Four apps (`plex`, `jellyfin`, `emby`, `navidrome`) are in B only because their single bind is a
`:ro` media mount, which `ClassifyBinds` correctly excludes. **81% of the catalog.**
Three steps per box: `pvesm add dir … --is_mountpoint 1`, two `pveum acl modify`, and a one-line
`local_backup_target` edit in `agent.json`.
**The asymmetry is Tier-2's alone.** Tier-1 (`RestoreFromRecoveryUnit`) and offsite both restore the
unit and replay volume dumps — so BookStack already had a working restore; only this button lied.
**Drive loss is now locally recoverable in principle on both boxes** — the whole-guest archive lives
on different hardware from the guest, and a restore from it boots and passes mount parity. The proof
that closes matrix row 4 (pull the system drive and restore for real) needs physical access and is
outstanding.
## Shipped — Part 1a (honesty)
## Phase 0 verdict
- **Refuses UP FRONT.** `Tier2RestoreCoverage` is consulted before any op begins; a class-B app is
refused **without being stopped**. Live: BookStack uptime stayed `Up About an hour` (Campaign 9
left it at `Up 25 seconds`).
- **Names the working action** rather than dead-ending 81% of the catalog:
> „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem
> állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
- **Claims only what was examined** (the QUIET half — immich's 1.3 GB Postgres unit is not covered,
so the old blanket sentence was a clean bill of health over data never opened):
> „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem
> tartoznak ebbe a visszaállításba."
**The target is configuration, not converged — the runbook held and the §3 STOP did not fire.**
Exactly one writer of `agent.json` exists (`pbsdr.seedEscrowStorageID`); it touches only
`escrow.pbs_storage_id` and preserves unknown keys verbatim via a `map[string]json.RawMessage`
read-modify-write. Nothing on the hub pushes agent config. No consumer hardcodes `"local"` on the
backup path — runners are one-per-tier, `NewestArchiveTime` reads its own target, and
`restoreTierForArchive` classifies from the archive, not from config.
## Shipped — Part 2 (C9-F2)
## What the run found
`StateRestarting` is deliberately **NOT** added to `IsDownState` — that alarms on every deploy and
update fleet-wide. A **sustained** run becomes down after `crashLoopAfter = 5m`, set above the three
real numbers already in the codebase: the deploy flow's **120 s** health timeout, Mealie's **60 s**
`start_period`, and R-97b's **180 s** quiesce grace (so the windows compose into one bounded delay
instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside
it. Carried by `Stack.RestartingSince` (not persisted) + `Stack.CrashLooping(now)`, used by **both**
the alarm and the dashboard counter — which previously counted `restarting` as running and so
contradicted the alarm on the same screen.
- **F-1 (before any command).** The storage `path` must **be** the drive's mountpoint. A
subdirectory fails `exactMount` (`internal/storage/observe.go:321`), so the target would report
`disconnected` **permanently** and its durable id would degrade off the filesystem UUID.
- **F-2 (before any command).** `--is_mountpoint 1` is load-bearing. **Proven live:** an unguarded
`dir` storage on a non-mounted path reports `active`, advertises the **root filesystem's** free
space, and had already created `dump/` on `/dev/mapper/pve-root` — a silent retarget onto the
exact device this change exists to escape. The guarded one refuses outright.
- **F-3 (found by the first real backup).** `FelhomAgentStore` is granted **per storage path**; a
new target without its own grant 403s every backup. **`felhom-host-install.sh` must issue it for
new installs**, or a new box ships with a tier that fails on its first run.
- **R-109 (new).** The DR recipe records **no backup target** — and each box now carries two
`content=backup` dir storages, one live and one holding frozen 2026-07-28 archives.
- **R-105 narrowed and traced.** `dr_recipe.host_half.drives` was `[]` fleet-wide with the cause
untraced. Cause: the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them.
Both demo boxes now populate `drives`, and the backup drives gained SMART reporting. R-105's other
two fields are untouched.
## Live replay (demo-hp + demo-felhom)
## Absent-drive behaviour (Part 3)
| # | scenario | result |
|---|---|---|
| 1 | **crash loop alarms** | Ten consecutive 30 s samples **silent** through the threshold window, then `17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma` — **5m25s** after the loop began (5 min + one scan). |
| 1b | **heartbeat COUNTS it** | `17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down**` — Campaign 9's evidence was `0 currently down` while an app looped. |
| 2 | **normal deploy is silent** | A real `uptime-kuma` deploy produced only `app_deployed (info)`; no alarm, with deadapp-check running every 30 s throughout. |
| 3 | **restore refuses without an outage** | The honest message rendered; `[WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped`; BookStack uptime unbroken. |
| 4 | **paperless still restores** (regression guard on Campaign 9's headline) | 3 files deleted → restored → **43/43 byte-identical to the pre-deletion sha256 set**, `documents_ok 16 of 16 problems []`. |
Today: **fail loudly, no silent retarget.** The PVE half is live-proven with throwaway storages (no
live drive was unmounted). The agent half is source-traced: `targetStoragePresent` checks name
presence only, never `Reachable`, so the tier stays DUE, the controller quiesces, vzdump is refused,
and the run fails and alarms.
## Filed, not fixed
**This is not §6's intended design** (fall back to the system drive and alarm). There is no fallback
at all, so an absent drive means no local backup until a human intervenes. Filed as **E-2**, together
with the honest single-drive label — a one-drive box protects against corruption only, and two
drives is effectively a hardware requirement for drive-loss protection.
- **C9-F1b** — route class-B apps to the Tier-1 unit restore from the card the customer already
opened. Its own task **deliberately**: it puts a DESTRUCTIVE operation (overwrites live data with
the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry
that difference.
- **C9-F4** — **nothing reads the Tier-2 copy's `recovery-unit/` mirror.** Written by every Tier-2 run
(`tier2.go:369`), read by no path: `RecoveryUnitPath` resolves to `backups/**primary**/`
(`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`.
Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary
unit is gone while this mirror survives on the second drive, unreachable by any customer action,
leaving offsite as the only route. **Potentially larger than C9-F1.**
## One operational error, reported in full
## Tests
**The agent was restarted on demo-hp with a `felhom-pbs` backup in flight**, against §4.3. The
in-flight check was done before the first restart and not repeated before the second. It produced a
**spurious tier failure** (`context canceled` while waiting) — the F-A1 class the project already
fixed once.
`go test ./...` **rc=0, 27 packages** — run and `rc` read *separately* from the commit. Six red-proofs
all observed, including the one that matters most: adding `StateRestarting` to `IsDownState` fails the
brief-restart test with *"every deploy and update would page the operator"*.
**The backup had not failed:** the PVE task returned OK and the PBS snapshot `2026-07-28T19:19:45Z`
is 6,264,034,053 B. The spurious failure **reached no channel** — the agent died with its in-memory
record and R-84 ground truth superseded it; the controller's event trail for the window shows only
the real 403 failure and its recovery. The design absorbed it; the mistake was still a mistake, and
on a slower tier the same slip could have aborted a multi-hour WAN upload.
## Observations
Also recorded: `pgrep -f vzdump` self-matches a polling script's own command line and is not a safe
in-flight check — use the PVE task list.
- **A seventh shipped-invariant-comment.** `controller/README.md` stated that faults "still surface as
`exited`/`degraded`/**`restarting`**/`unhealthy`" — but `restarting` was in no down set at all. The
sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented
beside it.
- **Pre-existing gate failure, not mine:** `scripts/docker_run_volume_path_gate.py` fails on
`internal/appexport/estimate.go:179` (an unreviewed `docker run -v`). Verified it fails identically
on clean HEAD; left alone as out of scope rather than silently "fixed".
- The other six gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`, `app_row_dedup`,
`offbox_rename`).
## State at close
## NOT yet live-validated — carried forward
Both boxes healthy. Primary and offsite tiers `due=false` with `age_state=known` on both; breakers
clear; agents `active`; no thrash and no spurious staleness. Target drives at 1–2 % used with SMART
`PASSED`. The old archives (18 GB demo-felhom, 5.2 GB demo-hp) are **left in place** as the rollback
and as the only evidence of what the previous configuration produced.
- **Tier-1 content recovery after real loss** — still the most valuable unproven item (Campaign 9's A2
ran against an intact app; A3 used Tier-2). Unchanged by this work.
- The C9-F2 threshold under a **quiesce** cycle (Scenario C) is unit-proven but was not replayed live;
it needs a backup window plus an app that fails to come back.
- C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not
app-by-app.
- Host reboot mid-backup, three-way concurrency with GC, Scenario C live, `offsite_stale` firing,
F-HUB — all still open from Campaign 9.
**Outstanding:** full drive-loss recovery (physical access), and the agent half of the absent-drive
behaviour (needs a drive unmount that would break the guest bind on a remote box).
documentation/backlog/OPEN-ITEMS.md
+2
View File
@@ -39,6 +39,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
| **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/<vmid>` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — |
| **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — |
| **E-2** | Drive-role machinery around the moved vzdump target: backup-target role on `StoragePath`, wizard assignment (never by transport/`removable`), no automatic roles, stickiness, `felhom-host-install.sh` creating the target **+ its `FelhomAgentStore` ACL**, absent-target policy, retention/space on a shared drive, honest single-drive label, remaining fleet migration | **READY (M)** | — | Architecture change PROVEN by hand on both demo boxes 2026-07-28 (`runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`); this is the machinery around a proven thing. Flips matrix row 4 | CC |
| **R-109** | **The DR recipe records no backup target.** It lists every storage's name/type/content but never which one holds the local archives — and each demo box now carries TWO `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives) | READY (XS) | — | Add the resolved `BackupTarget()` to the host-half. Third recipe-completeness defect beside R-105/R-106 | CC |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
documentation/backlog/ROADMAP.md
+3 -1
View File
@@ -99,10 +99,12 @@
| R-102 | **Tier-2 writes a full `recovery-unit/` mirror on every run and no code path reads it.** Written at `internal/backup/tier2.go:368-369` ("Unit leg (always)"); `RecoveryUnitPath` resolves to `backups/**primary**/` (`internal/appbackup/paths.go:46-48`) and the only reader of the secondary tree is `internal/backup/tier2_restore.go`, which reads `hdd/`+`userdata/` only (`:101-104`) | M | READY — 2026-07-28 | **Was C9-F4** (`OPEN-ITEMS.md`). The sharp edge is *when* it bites: Tier-2 exists for primary-drive loss, and in exactly that failure the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action — leaving offsite as the only route. LIVE: demo-felhom's `backups/secondary/{bookstack,docmost}/` hold `recovery-unit` and nothing else, at 156 MB and 86 MB. Flips: the Tier-2 row in map §C; `07` §6.3, §7.2 |
| R-103 | **The Tier-2 no-coverage refusal names the working action but does not route to it.** v0.183.0 refuses up front without stopping the app and tells the customer to use „Visszaállítás indítása" on the other page; it does not take them there | S | READY — 2026-07-28 | **Was C9-F1b.** Deliberately its own item: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference. Flips: nothing until shipped; `07` §10.2 |
| R-104 | **An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach.** `resticStep` has `unlock --remove-all` (`internal/backup/offbox.go:634-648`) but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`:77-93`) has no lock case → `"other"` → fail-fast; `ClassifyOffsiteFailure` likewise, so the operator is told *„A távoli mentés ismeretlen okból nem sikerült"* for a precisely-known, self-healable condition | S | READY — 2026-07-28 | **Was C9-F3.** Reachable by any interruption — container restart, OOM, network drop, host reboot mid-backup. The tier stays dead until a human runs `restic unlock --remove-all`. Flips: the offsite row in map §C; `07` §8 row 15 |
| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. Flips: `07` §4 |
| R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 |
| R-106 | **The DR recipe records the PBS namespace as `"root"` on every box** while the real namespaces are `demo-felhom` / `demo-hp` | XS | READY — 2026-07-28 | Traced to source: `Snapshot.Namespace` decodes a `ns` field (`felhom-agent/internal/pbs/client.go:97`) that PBS does not echo per item when the request is already namespace-scoped via `?ns=` (`:117-120`), so it is empty and `ToHub` normalises empty → `"root"` (`internal/pbs/report.go:23-27`), which `latestPBSCoord` writes into the recipe (`internal/hub/dr_recipe.go:149`). `latest_snapshot_id: "9201"` is **not** a defect — it is `BackupID`, documented as a coordinate (`:56`). Verified against ep0's real namespaces and a live `pct restore` command line carrying `--ns demo-felhom`. Flips: `07` §4 |
| R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 |
| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `<drive>/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed |
| R-109 | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` (`felhom-agent/internal/hub/dr_recipe.go:86`) emits `guests`, `drives`, `pve_storage` and the PBS coordinate, but nothing that says WHICH storage holds the local whole-guest archives. Harmless while the target was the well-known `local`; not harmless now — after the 2026-07-28 vzdump-target move each demo box carries **two** `content=backup` dir storages, `felhom-backup` (live) and `local` (frozen 2026-07-28 archives, never refreshed again) | XS | READY — 2026-07-28 | Found by `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` Phase 0 §5 and confirmed live in the recipe both boxes emit. A restorer reading the recipe cannot distinguish the live target from the stale one, and picking wrong yields a silently outdated guest. Add the resolved `BackupTarget()` to the host-half. Sits with **R-105**/**R-106** as the third recipe-completeness defect. Flips: `07` §4 |
| E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |
## P3 — post-alpha
documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md
+563
View File
@@ -0,0 +1,563 @@
# RUNBOOK RECORD — the local whole-guest backup moved off the guest's own device (2026-07-28)
**Class:** supervised operational change, one box at a time. **No code, no version bump** — the
change is a PVE storage definition, a per-storage ACL grant, and a one-line target switch in
`agent.json`. **Nothing was deleted.**
**Boxes:** `demo-hp` (`felhom-host`, HP t740) first, then `demo-felhom` (N100).
**Change:** primary backup tier `local` → **`felhom-backup`**, a `dir` storage on each box's
secondary drive.
**Window:** 2026-07-28 21:11 → 21:35 CEST.
> **Deliverable note.** The spec is `RUNBOOK-vzdump-target-move-2026-07-29.md`; the work ran on the
> evening of 07-28. The filename is kept as specified.
---
## 1. Phase 0 — ground truth, established read-only before anything was touched
### 1.1 The premise holds: `local` and `local-lvm` are the same physical device
| | demo-felhom | demo-hp |
|---|---|---|
| `local` (dir, `/var/lib/vz`) | `pve-root` → **`sda3`** | `pve-root` → **`sda3`** |
| `local-lvm` (lvmthin, `pve/data`) | **`sda3`** | **`sda3`** |
| System drive | AirDisk 512 GB SATA SSD (`sda`) | SanDisk X600 M.2 128 GB (`sda`) |
A drive failure took the guest and its only local backup together. That is the fault being closed.
### 1.2 Physical inventory and the durable id
| | demo-felhom | demo-hp |
|---|---|---|
| Secondary drive | `/dev/sdb` TOSHIBA MQ04ABF100, **USB**, rotational | `/dev/nvme0n1` KXG50PNV1T02, **NVMe** |
| Size / free before | 916 GB / 866 GB | 938 GB / 891 GB |
| Mountpoint | `/mnt/hdd_1` | `/mnt/nvme-1tb` |
| **Durable id (fs UUID)** | `47a3361a-91e0-4831-a69d-27f540ed3f48` | `91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
| `/sys/block/*/removable` | `0` | `0` |
| SMART at close | — | `PASSED`, 50 °C, 7 % wear, 0 media errors |
Both drives are already enrolled Felhom drives, mounted **by filesystem UUID** through
agent-generated systemd units (`mnt-hdd_1.mount`, `mnt-nvme\x2d1tb.mount`), each marked
*"Managed by felhom-agent — do not edit by hand"*, `WantedBy=multi-user.target`. **Neither is in
`/etc/fstab`.** The UUID in the unit is the same durable id the DR recipe carries, so the target
drive was already durably identified before this change.
This is §2's principle meeting the reference hardware head-on: demo-felhom's target **is** an
external USB HDD and both drives report `removable=0`. A rule keyed on transport would disqualify
demo-felhom; a rule keyed on the `removable` flag would see no removable drive at all on either box.
Neither attribute can decide a role. Both are fine as *hints*.
### 1.3 Space
The runbook's ~14.5 GB figure is the **PBS-side** measurement. The local `zstd` archive is far
smaller:
| | newest local archive before | × `keep-last=3` | share of the target drive's free space |
|---|---|---|---|
| demo-felhom (9201) | 5.93 GB (trend 5.82 → 5.84 → 5.93) | **~17.8 GB** | **2.1 %** of 866 GB |
| demo-hp (9201) | 1.68 GB (1.61 → 1.63 → 1.68) | **~5.0 GB** | **0.6 %** of 891 GB |
Retention is `local_backup_retention: 3` on both boxes and is unchanged by this runbook.
The move also relieves the system drive — 18 GB on demo-felhom, 5.2 GB on demo-hp once the old
archives are eventually removed. On demo-hp that matters: its root is 40 GB with 24 GB free.
### 1.4 How the target is set — **configuration, not converged. The runbook holds.**
The primary tier is built from `backup.local_backup_target` (`felhom-agent/internal/config/config.go:541`,
`BackupTarget()`), which was `"local"` on both boxes. The extra tier is `felhom-pbs`, weekly.
Every writer of `agent.json` was enumerated. **There is exactly one:**
`pbsdr.seedEscrowStorageID` (`internal/pbsdr/manager.go:454`). It sets only `escrow.pbs_storage_id`,
refuses to clobber a differing existing value (warn-and-keep), and performs a
`map[string]json.RawMessage` read-modify-write that **preserves unknown keys verbatim**. Nothing on
the hub pushes agent configuration. Seven prior hand-edit backups sit beside the live file.
**A hand edit sticks. No code change was required, so the STOP in §3 did not fire.**
### 1.5 What else reads the target — no hardcoded `"local"` on the backup path
| Consumer | Verdict |
|---|---|
| Backup runners | One runner **per tier**, each constructed with its own `TargetID` (`cmd/felhom-agent/main.go:1355`). Parameterised. |
| `NewestArchiveTime` | Reads `r.target` (`internal/backup/runner.go:390`). Parameterised. |
| Restore-test rotation | Iterates `BackupTiers()`; `restoreTierForArchive` classifies from the **archive**, never from config (the R-82/R-85 fix). Parameterised. |
| Per-run prune | `localPruneSpec` suppresses only for **PBS**-type targets; a `dir` target still prunes at `keep-last=3`. Correct. |
| Customer-facing label | `backupIsPBS` (`felhom-controller/.../backup_handlers.go:174`) matches the substring `pbs`. `felhom-backup` correctly renders „Helyi tároló (felhom-backup)". **Naming trap recorded** — a target named with `pbs` in it would be mislabelled as offsite. |
Two Phase 0 findings that shaped the change are in §4 and §6.
---
## 2. The two findings Phase 0 turned up before any command ran
### F-1 — the storage path must BE the mountpoint, or the target reports disconnected forever
`reportType` (`internal/storage/observe.go:365`) calls a `dir` storage `local` only when it is
literally named `local`; anything else becomes `usb` or `local-dir`. For those two types
`reachable()` (`:321`) requires **`exactMount`** — the storage `path` must exactly equal a
mountpoint (`exactMountDevice`, `:422`).
Point the storage at a **subdirectory** of the drive and `exactMount` is false, so
`reachable=false`, `state=disconnected`, **permanently** — a false alarm designed in on day one,
and the durable id degrades to a path fallback instead of the filesystem UUID.
**Decision: `path` = the drive's mountpoint.** PVE creates `dump/` at the drive root, beside the
existing `felhom-data/`. This also gives a real safety property: the guest is bind-mounted only the
drive's `felhom-data` subpath (`/mnt/felhom-drives/<drive>`), so **the archives sit outside anything
the customer's guest can reach.**
### F-2 — `is_mountpoint` is what stops a silent retarget onto the system drive
`man pvesm`: *"Assume the given path is an externally managed mountpoint and consider the storage
offline if it is not mounted."* Without it, an unplugged or late-mounting drive leaves
`/mnt/<drive>` a bare directory on the root filesystem and vzdump writes the backup **onto the
system drive** — the exact device this change exists to escape, silently. Proven live in §6.
**Decision: `--is_mountpoint 1` on both boxes.**
---
## 3. The exact commands used, per box
Identical on both, differing only in the mountpoint. Run as `root` on the PVE host.
```bash
# 1. the storage, on the drive's OWN mountpoint, guarded (F-1 + F-2)
pvesm add dir felhom-backup --path /mnt/nvme-1tb --content backup --is_mountpoint 1 # demo-hp
pvesm add dir felhom-backup --path /mnt/hdd_1 --content backup --is_mountpoint 1 # demo-felhom
# 2. the per-storage grant — REQUIRED, and the reason the first backup failed (§4)
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --users felhom-agent@pve
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
# 3. back up the config, then repoint the primary tier
cd /etc/felhom-agent
cp -p agent.json agent.json.pre-e-target-move
# in-place truncate-write (NOT sed -i): the directory is root-owned while the file is
# agent-owned 0600, so a rename would flip ownership and the non-root agent could not read it.
python3 - <<'EOF'
import json
p='/etc/felhom-agent/agent.json'
raw=open(p).read()
old='"local_backup_target": "local"'
new='"local_backup_target": "felhom-backup"'
assert raw.count(old)==1, 'occurrences: %d' % raw.count(old)
out=raw.replace(old,new); json.loads(out) # validate BEFORE writing
with open(p,'r+') as f: f.seek(0); f.write(out); f.truncate()
EOF
# 4. restart only with nothing in flight — check the PVE task list, NOT pgrep (see §8)
pvesh get /nodes/<node>/tasks --limit 20 --output-format json | python3 -c \
'import sys,json;print([t for t in json.load(sys.stdin) if "endtime" not in t] or "none running")'
systemctl restart felhom-agent
```
Resulting stanza, both boxes:
```
dir: felhom-backup
path /mnt/<drive>
content backup
is_mountpoint 1
```
**Verification that Part 1 took**, both boxes — the diff was exactly one line, ownership and mode
preserved (`-rw------- felhom-agent felhom-agent`), and both tiers armed with no rejection:
```
backup tier armed target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s primary=true
backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s primary=false
```
`GET /backup/due` echoes the new tier, and the **old target no longer exists** — the cleanest proof
the switch took:
```
GET /backup/due → {"due":true,"reason":"no successful backup recorded yet","age_state":"absent"}
GET /backup/due?target=felhom-backup → {..., "target":"felhom-backup", "age_state":"absent"}
GET /backup/due?target=felhom-pbs → {"due":false,"reason":"within cadence window","age_seconds":44362}
GET /backup/due?target=local → {"ok":false,"error":"unknown backup target: local"}
```
---
## 4. Finding F-3 — the agent had no PVE privilege on the new storage (found by the first real backup)
The first triggered backup on demo-hp failed:
```
local-api: backup job failed vmid=9201 target=felhom-backup
err="proxmox: POST /nodes/felhom-host/vzdump -> HTTP 403: permission denied
at /storage/felhom-backup (missing privilege Datastore.Allocate)"
```
`FelhomAgentStore` (`Datastore.Allocate,Datastore.AllocateSpace`) is granted **per storage path**,
to both `felhom-agent@pve` and the `felhom-agent@pve!agent` token — `local`, `local-lvm` and
`felhom-pbs` each had their own pair. A new backup target needs the same pair or every backup 403s.
The grant in §3 step 2 mirrors the existing ones exactly; **no privilege was widened**, and the
storage-scoped shape (rather than a grant at `/`) is preserved.
**This is an E-2 requirement, not a one-off:** `felhom-host-install.sh` must issue the grant for
whatever target it provisions on a new install, or every new box ships with a backup tier that
403s on its first run.
---
## 5. Part 2 — the proof
### 5.1 A backup lands on the secondary drive — via the real path
Triggered through the endpoint the UI's „Mentés most" button invokes, not `--selftest`.
**Route correction worth recording:** the backups page has two triggers.
`POST /api/backup/run` is the **Tier-1 app-data** backup (DB dumps + volume dumps); the
**whole-guest** vzdump is `POST /api/guest-backup/trigger`, which goes through the controller's
quiesce loop (`ServeBackupAPI`, `backup_handlers.go`). The first attempt used the former and
correctly produced an app-data run and no vzdump.
| | demo-hp | demo-felhom |
|---|---|---|
| Archive | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst` | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst` |
| Size | **2,256,044,492 B (2.26 GB)** | **5,957,878,962 B (5.96 GB)** |
| Duration | 86.3 s | ~197 s |
| Mode | `snapshot` — `snapshotted` at 21:18:16, app resumed before the copy finished | `snapshot` — `snapshotted` at 21:26:29 |
| `df` on the target drive | used 92,344,320 → **2,348,572,672** (**+2,256,228,352 B**) | used 3,600,314,368 → **9,558,208,512** (**+5,957,894,144 B**) |
On both boxes the `df` delta matches the archive size, so the bytes demonstrably landed on the
secondary drive and not somewhere else.
On demo-felhom the backup was taken by the controller's **first-backup safety valve** rather than by
the manual trigger: the agent restart made the new tier report `age_state: absent`, the controller
correctly refused to withhold a first backup, and the manual trigger that followed was answered
`{"error":"mentés már folyamatban van"}`. Same code path, same quiesce loop — the trigger simply
lost the race to the mechanism that exists for exactly this state.
`uncovered_volumes: ["/etc/felhom-bootstrap", "/mnt/felhom-drives"]` — both are host binds and are
correctly outside the guest archive. Pre-existing behaviour, unchanged by this move, but see §7.
### 5.2 A restore works from that archive
Run with **no** `-archive`, so the candidate *selection* is part of the proof. The rotation needed
no nudge — it picked the new storage on its own.
```
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst
into scratch band [990000,990009] on local-lvm …
{
"source_archive": "felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst",
"source_tier": "local",
"scratch_vmid": 990000,
"pass": true,
"verified": "boot+running",
"mount_parity": "ok",
"duration_seconds": 114.39,
"mount_inventory": [
"mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)",
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
]
}
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m54s) ===
```
**`mount_parity: ok`, `pass: true`, scratch torn down cleanly.**
demo-felhom, same method, same result — the bigger guest (200 G + 50 G volumes):
```
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst …
"pass": true, "verified": "boot+running", "mount_parity": "ok",
"duration_seconds": 84.10, "scratch_vmid": 990000,
"mount_inventory": ["mp0=/var/lib/docker (200G)", "mp1=/mnt/sys_drive (50G)", …]
=== selftest=restore-test OK (restored+booted+verified+torn-down in 1m24s) ===
```
| | demo-hp | demo-felhom |
|---|---|---|
| Candidate picked automatically from the new storage | yes | yes |
| `pass` / `verified` | true / `boot+running` | true / `boot+running` |
| **`mount_parity`** | **ok** | **ok** |
| Duration | 1 m 54 s | 1 m 24 s |
| Scratch torn down | yes (990000) | yes (990000) |
Run with **no** `-archive` on both boxes, so the candidate *selection* is part of the proof. The
rotation needed no nudge on either box — it picked the new storage on its own.
`source_tier: "local"` is **correct, not a stale reference to the old storage name.**
`restoreTierForArchive` classifies by the archive's storage *type*; `felhom-backup` is a `dir`
storage, so the restore correctly earns the local (10-minute) wait bound rather than the PBS one.
**Method caveat:** `--selftest=restore-test` is a separate one-shot process, so it does **not**
write the daemon's `rtState`. `GET /restore-test/status` still reads `null` on both boxes and the
customer-visible „utoljára ellenőrizve" is unchanged. The proof is real; the daemon's own 84 h
rotation is undisturbed and will record its next run normally.
### 5.3 Freshness follows the target
Immediately after the backup: `{"due":false,"age_seconds":159,"target":"felhom-backup","age_state":"known"}`.
That alone does not separate the storage read from this process's in-memory record, so the agent was
**restarted** — which empties the in-memory store (R-84) — and re-asked. Both boxes:
```
demo-hp GET /backup/status → {"phase":"idle"} ← no in-memory record at all
GET /backup/due → {"due":false,"age_seconds":188,
"target":"felhom-backup","age_state":"known"}
demo-felhom GET /backup/status → {"phase":"idle"}
GET /backup/due → {"due":false,"age_seconds":267,
"target":"felhom-backup","age_state":"known"}
```
A `known` age with an empty in-memory store **can only** have come from reading the new storage.
The tier neither looks absent (which would re-backup every cycle) nor stale. This is the positive
observable, not the absence of a complaint.
**This restart is also where the one operational error of the run happened — see §8.**
### 5.4 The DR recipe does **not** record the new target — and one field improved
**The finding:** `BuildDRRecipeHostHalf` (`internal/hub/dr_recipe.go:86`) records `guests`,
`drives`, `pve_storage` and the PBS coordinate. **Nothing in the recipe says which storage holds the
local archives.** While the target was `local` that was guessable; it is not any more, and it is now
actively misleading — after this change `local` still holds the *stale* archives while the live ones
are on `felhom-backup`. A restorer reading the recipe sees two backup-capable storages and cannot
tell which is which. **Filed as R-109**, alongside R-105/R-106.
**The improvement — this change traces and partly closes R-105.** R-105 recorded
`dr_recipe.host_half.drives = []` on every customer *including two with enrolled data drives*, with
the cause explicitly untraced. It is now traced: **the enrolled drives were never PVE storages at
all**, so they never entered `report.StorageTargets`, so `isUserDataDrive` (which needs type
`usb`/`local-dir` **and** a durable id **and** a mount path) never saw them. Making the drive a
storage populated it. Live on demo-hp after the change:
```json
"drives": [{ "durable_id": "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae",
"mount_path": "/mnt/nvme-1tb", "intent": "enrolled",
"total_bytes": 1006980812800 }]
```
The storage observation also gained a real device, a real UUID and **SMART** for the backup drive —
`health=PASSED temp=50 poh=28451 media_err=0 pct_used=7` — none of which existed when the drive was
not a PVE storage:
```
- felhom-backup type=local-dir state=attached reach=true class=fast
durable=uuid:91d2dc2d-… mount="/mnt/nvme-1tb" dev="/dev/nvme0n1"
```
demo-felhom shows the identical shape — `drives` populated, SMART now readable on the USB HDD:
```
- felhom-backup type=local-dir state=attached reach=true class=slow
durable=uuid:47a3361a-… mount="/mnt/hdd_1" dev="/dev/sdb"
smart: health=PASSED temp=35 poh=3581 realloc=0 pending=0 offline_unc=0
"drives": [{ "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
"mount_path": "/mnt/hdd_1", "intent": "enrolled",
"total_bytes": 983351140352 }]
```
**`dr_recipe.host_half.drives` is now non-empty on both demo boxes**, where R-105 recorded `[]`
fleet-wide. Note the classification detail: demo-felhom's drive is USB-attached but reports
`removable=0`, so it is typed `local-dir` rather than `usb`. That does not matter here — both types
take the same reachability branch and both satisfy `isUserDataDrive` — but it is one more reason
§2's "do not classify by transport" rule is right.
R-106 is untouched and still live on both boxes: the recipe still records `"namespace": "root"`
where the real namespaces are `demo-hp` / `demo-felhom`.
---
## 6. Part 3 — what happens when the target drive is absent
Established with **throwaway storages**, so neither live drive was unmounted (unmounting would break
the guest's `felhom-data` bind and the customer's app data on a remote box).
### Live-proven: the PVE half
| | storage on a non-mount path, **no** `is_mountpoint` | same path, **`is_mountpoint 1`** |
|---|---|---|
| Creation | succeeded silently | **refused**: `unable to activate storage 'absent-test-b' - directory is expected to be a mount point but is not mounted: '/mnt/absent-b'` |
| `pvesm status` | **`active`**, 40516856 KiB total | never created |
| Backing device | **`/dev/mapper/pve-root`** — the system drive | — |
| What it created | `/mnt/absent-a/dump/` **on the root filesystem** | nothing |
The unguarded storage reports `active` with plausible free space that is really the **root
filesystem's**, and it had already created a `dump/` directory there. A vzdump aimed at it would
write the whole-guest backup onto the system drive with no error and no alarm — **silent
retargeting onto the exact device this change exists to escape**, which §6 names as the worst
outcome. `is_mountpoint 1` converts that into a loud refusal. Both test storages were removed and
`/mnt/absent-a` deleted.
### Source-traced, not live-proven: the agent half
With the guard in place and the drive absent, the storage stays in `storage.cfg` but goes inactive.
`targetStoragePresent` (`internal/localapi/server.go:1212`) checks the target's **name presence
only**, never `Reachable`, so the tier is **not** deferred; `newestArchiveOn` cannot read the
storage and degrades to `archiveUnknown`, which stays **DUE** by design. The controller therefore
quiesces the apps and fires a vzdump, PVE refuses because the storage is not active, and the run
**fails loudly** — arming the R-88 breaker and the `backup_failed` operator mail.
So today's behaviour is: **fail and alarm; no silent retarget.** That is the safe half.
**It does not match §6's intended design** (*fall back to the system drive and alarm*): there is no
fallback at all, so an absent drive means **no local backup whatsoever** until a human intervenes,
and each attempt costs an app quiesce until the breaker backs off. Whether a corruption-only copy on
the system drive beats no copy is a product decision; the honest label matters either way.
**Filed as E-2.**
This half is traced through source, not reproduced on hardware — the boxes are remote and unmounting
a live enrolled drive would break the guest bind and the customer's data. Recorded as outstanding
proof in §9.
### Single-drive boxes
A single-drive box has nowhere to move the target. There the backup stays on the system drive and
protects against **corruption only** — it is not drive-loss protection and must not be presented as
a backup tier that survives hardware failure. **Two drives is effectively a hardware requirement for
drive-loss protection**, and that belongs in the sales/onboarding material as well as in the UI
label. Filed under E-2.
---
## 7. Scope recorded for E-2 — filed, not built
| # | Item |
|---|---|
| 1 | A **backup-target role** on `StoragePath`, alongside `Schedulable` / `IsDefault` / `Kind`. |
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. |
| 9 | **Migrating the remaining fleet** — `peti-felhom` and any box not covered here. |
| 10 | Naming guard: `backupIsPBS` matches the substring `pbs`, so a target so named would be mislabelled to the customer as offsite (§1.5). |
---
## 8. Observations
### 8.1 An operational error in this run: the agent was restarted with a backup in flight
§4.3 and §8 of the spec both say not to. It happened anyway, on demo-hp at **21:21:12**.
The manual „Mentés most" correctly fires **every** tier, not just the primary, so the trigger at
21:18 started the local vzdump *and* a `felhom-pbs` one. The local tier finished at 21:19:41. The
PBS tier was still running. The in-flight check had been done before the **first** restart (21:12,
clean) and **was not repeated** before the second one — the restart taken for the §5.3 freshness
proof — so it landed on top of a live PBS backup.
What it produced:
```
level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs
err="backup: vzdump task vmid 9201: proxmox: waiting for task UPID:…:vzdump:9201:… : context canceled"
```
**The backup had not failed.** `context canceled` is the agent being shut down while waiting, not
the task failing. Checked rather than assumed, against two independent authorities:
- the PVE task `2026-07-28T21:19:45 … vzdump 9201` → **OK**;
- the PBS snapshot `felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z` → **6,264,034,053 B**, a real,
plausibly-complete archive (nine orders of magnitude above the F-CRIT-2 1 MiB floor).
So the run recorded a **spurious tier failure** — a self-inflicted instance of exactly the F-A1
class ("a healthy backup reported as FAILED") that the project fixed in controller v0.179.0.
**It reached no channel, and the reason is worth keeping.** The controller's event trail for the
whole window is:
```
19:15:43Z whole_guest_backup_failed (error) felhom-backup tier — the REAL 403, backoff 15m
19:19:45Z whole_guest_backup_recovered (info) felhom-backup tier — cleared after the ACL grant
```
— and **nothing for the 21:21 PBS failure.** The agent process died with its own in-memory failure
record, and by the time the controller next polled, `NewestArchiveTime` had read ground truth from
PBS and found the successful 21:19:45 archive. The R-84 ground-truth design absorbed it: no breaker
armed on the offsite tier, no operator mail, no spurious staleness. That is the design working, not
a reason the mistake was harmless — on a slower tier, or with the restart a minute earlier, the same
slip could have aborted a multi-hour WAN upload.
**The rule that would have caught it:** re-check in flight immediately before *every* restart, not
once at the start of the procedure — and remember that a manual trigger fires all tiers, so the
offsite tier can still be running long after the local one has finished.
### 8.2 Method and tooling
- **`pgrep -f vzdump` is not a safe in-flight check.** A polling script whose own command line
contains the string matches itself, and it reported a backup in flight when the PVE task list
showed none. It caused one false "ABORT: vzdump in flight" and one wrong reading in this run.
**Use the PVE task list** (`pvesh get /nodes/<node>/tasks`, filter for a missing `endtime`).
- **`sed -i` must not be used on `agent.json`.** The directory is root-owned and the file is
agent-owned `0600`; `sed -i` renames, which would flip the file to `root:root` and leave the
non-root agent unable to read its own config. The in-place truncate-write in §3 preserves both.
- A `felhom-pbs` vzdump failed at **17:58**, hours before this change, with
`Can't connect to 10.77.0.1:8007 (Connection refused)`. **Pre-existing and unrelated** — checked
rather than assumed: ep0 is up (uptime 1 d 2 h, `proxmox-backup-proxy` active), the WG handshake
is fresh, and 8007 is open from demo-felhom now.
- The controller's first-backup safety valve behaved exactly as designed on demo-felhom: the agent
restart made the new tier report `age_state: absent`, and the controller took a backup on its own
before the manual trigger could (`{"error":"mentés már folyamatban van"}`).
- The archives are outside the customer's reach by construction — the guest is bound only the
drive's `felhom-data` subpath, while `dump/` sits at the drive root.
---
## 9. State at close
### 9.1 Old archives — left in place, as instructed
Nothing was deleted. They remain the rollback and the only evidence of what the previous
configuration produced.
| Box | Path | Contents | Size |
|---|---|---|---|
| demo-felhom | `/var/lib/vz/dump/` | 3 × 9201 (5.82 / 5.84 / 5.93 GB) + 2 × 9100 (612 / 649 MB) | **18 GB** |
| demo-hp | `/var/lib/vz/dump/` | 3 × 9201 (1.61 / 1.63 / 1.68 GB) + 1 × 9100 (649 MB) | **5.2 GB** |
**These are now stale and will never be refreshed** — the tier that wrote them no longer points
here, and `keep-last=3` prunes only the *new* target. They are also the reason R-109 matters: a
restorer reading the DR recipe sees both `local` and `felhom-backup` carrying `content=backup` and
cannot tell that one holds live archives and the other holds a frozen snapshot of 2026-07-28.
Deleting them is a separate, deliberate decision.
### 9.2 Fleet state — both boxes healthy, no thrash, no spurious staleness
| | demo-felhom | demo-hp |
|---|---|---|
| Agent | `active`, v0.110.0 | `active`, v0.110.0 |
| Primary tier | `felhom-backup`, `due=false`, `age_state=known` | `felhom-backup`, `due=false`, `age_state=known` |
| Offsite tier | `felhom-pbs`, `due=false`, `age_state=known` | `felhom-pbs`, `due=false`, `age_state=known` |
| New target | 5.96 GB used of 916 GB (**2 %**) | 2.26 GB used of 938 GB (**1 %**) |
| Target drive SMART | `PASSED`, 35 °C | `PASSED`, 50 °C, 7 % wear |
| System drive | 24 G / 94 G (27 %) | 14 G / 39 G (38 %) |
| Agent errors, last 15 min | **0** | 2, both accounted for (§4 403, §8.1 spurious) |
| Breaker | clear (`whole_guest_backup_recovered` 19:19:45Z) | clear |
No tier is stale, no tier is thrashing, and both offsite tiers report a known, recent age.
### 9.3 Outstanding proof
1. **Full drive-loss recovery** — pull the system drive, restore the guest from the vzdump on the
secondary drive, confirm it returns with named volumes, secrets and config. Needs physical
access; the boxes are remote until ~08-02. This is the proof that flips **matrix row 4** from
PARTIAL to a real local route. Everything up to it is now proven: the archive exists on separate
hardware, and a restore from it boots and passes `mount_parity`.
2. **The agent half of §6** — with the drive absent, that the tier fails loudly rather than silently
retargeting. The **PVE half is live-proven** (§6); the agent half is traced through source only,
because unmounting a live enrolled drive would break the guest's `felhom-data` bind and the
customer's app data on a remote box.
### 9.4 Backlog rows filed
| ID | What |
|---|---|
| **E-2** | The machinery around this change — backup-target role on `StoragePath`, wizard assignment, no automatic roles, stickiness, `felhom-host-install.sh` creating the target **and** its ACL, absent-target policy, retention/space accounting on a shared drive, the honest single-drive label, remaining fleet migration. Full scope in §7. |
| **R-109** | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` lists every storage's name/type/content but never says which one holds the local archives — and after this change the box carries two `content=backup` dir storages, one live and one stale. §5.4. |
| **R-105** | Partially closed and **fully traced** for its third field: `dr_recipe.host_half.drives` is now populated on both demo boxes, and the cause of the fleet-wide `[]` is identified — the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them. The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched by this run. |