Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
This commit is contained in:
@@ -0,0 +1,563 @@
|
||||
# RUNBOOK RECORD — the local whole-guest backup moved off the guest's own device (2026-07-28)
|
||||
|
||||
**Class:** supervised operational change, one box at a time. **No code, no version bump** — the
|
||||
change is a PVE storage definition, a per-storage ACL grant, and a one-line target switch in
|
||||
`agent.json`. **Nothing was deleted.**
|
||||
|
||||
**Boxes:** `demo-hp` (`felhom-host`, HP t740) first, then `demo-felhom` (N100).
|
||||
**Change:** primary backup tier `local` → **`felhom-backup`**, a `dir` storage on each box's
|
||||
secondary drive.
|
||||
**Window:** 2026-07-28 21:11 → 21:35 CEST.
|
||||
|
||||
> **Deliverable note.** The spec is `RUNBOOK-vzdump-target-move-2026-07-29.md`; the work ran on the
|
||||
> evening of 07-28. The filename is kept as specified.
|
||||
|
||||
---
|
||||
|
||||
## 1. Phase 0 — ground truth, established read-only before anything was touched
|
||||
|
||||
### 1.1 The premise holds: `local` and `local-lvm` are the same physical device
|
||||
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| `local` (dir, `/var/lib/vz`) | `pve-root` → **`sda3`** | `pve-root` → **`sda3`** |
|
||||
| `local-lvm` (lvmthin, `pve/data`) | **`sda3`** | **`sda3`** |
|
||||
| System drive | AirDisk 512 GB SATA SSD (`sda`) | SanDisk X600 M.2 128 GB (`sda`) |
|
||||
|
||||
A drive failure took the guest and its only local backup together. That is the fault being closed.
|
||||
|
||||
### 1.2 Physical inventory and the durable id
|
||||
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| Secondary drive | `/dev/sdb` TOSHIBA MQ04ABF100, **USB**, rotational | `/dev/nvme0n1` KXG50PNV1T02, **NVMe** |
|
||||
| Size / free before | 916 GB / 866 GB | 938 GB / 891 GB |
|
||||
| Mountpoint | `/mnt/hdd_1` | `/mnt/nvme-1tb` |
|
||||
| **Durable id (fs UUID)** | `47a3361a-91e0-4831-a69d-27f540ed3f48` | `91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
|
||||
| `/sys/block/*/removable` | `0` | `0` |
|
||||
| SMART at close | — | `PASSED`, 50 °C, 7 % wear, 0 media errors |
|
||||
|
||||
Both drives are already enrolled Felhom drives, mounted **by filesystem UUID** through
|
||||
agent-generated systemd units (`mnt-hdd_1.mount`, `mnt-nvme\x2d1tb.mount`), each marked
|
||||
*"Managed by felhom-agent — do not edit by hand"*, `WantedBy=multi-user.target`. **Neither is in
|
||||
`/etc/fstab`.** The UUID in the unit is the same durable id the DR recipe carries, so the target
|
||||
drive was already durably identified before this change.
|
||||
|
||||
This is §2's principle meeting the reference hardware head-on: demo-felhom's target **is** an
|
||||
external USB HDD and both drives report `removable=0`. A rule keyed on transport would disqualify
|
||||
demo-felhom; a rule keyed on the `removable` flag would see no removable drive at all on either box.
|
||||
Neither attribute can decide a role. Both are fine as *hints*.
|
||||
|
||||
### 1.3 Space
|
||||
|
||||
The runbook's ~14.5 GB figure is the **PBS-side** measurement. The local `zstd` archive is far
|
||||
smaller:
|
||||
|
||||
| | newest local archive before | × `keep-last=3` | share of the target drive's free space |
|
||||
|---|---|---|---|
|
||||
| demo-felhom (9201) | 5.93 GB (trend 5.82 → 5.84 → 5.93) | **~17.8 GB** | **2.1 %** of 866 GB |
|
||||
| demo-hp (9201) | 1.68 GB (1.61 → 1.63 → 1.68) | **~5.0 GB** | **0.6 %** of 891 GB |
|
||||
|
||||
Retention is `local_backup_retention: 3` on both boxes and is unchanged by this runbook.
|
||||
|
||||
The move also relieves the system drive — 18 GB on demo-felhom, 5.2 GB on demo-hp once the old
|
||||
archives are eventually removed. On demo-hp that matters: its root is 40 GB with 24 GB free.
|
||||
|
||||
### 1.4 How the target is set — **configuration, not converged. The runbook holds.**
|
||||
|
||||
The primary tier is built from `backup.local_backup_target` (`felhom-agent/internal/config/config.go:541`,
|
||||
`BackupTarget()`), which was `"local"` on both boxes. The extra tier is `felhom-pbs`, weekly.
|
||||
|
||||
Every writer of `agent.json` was enumerated. **There is exactly one:**
|
||||
`pbsdr.seedEscrowStorageID` (`internal/pbsdr/manager.go:454`). It sets only `escrow.pbs_storage_id`,
|
||||
refuses to clobber a differing existing value (warn-and-keep), and performs a
|
||||
`map[string]json.RawMessage` read-modify-write that **preserves unknown keys verbatim**. Nothing on
|
||||
the hub pushes agent configuration. Seven prior hand-edit backups sit beside the live file.
|
||||
|
||||
**A hand edit sticks. No code change was required, so the STOP in §3 did not fire.**
|
||||
|
||||
### 1.5 What else reads the target — no hardcoded `"local"` on the backup path
|
||||
|
||||
| Consumer | Verdict |
|
||||
|---|---|
|
||||
| Backup runners | One runner **per tier**, each constructed with its own `TargetID` (`cmd/felhom-agent/main.go:1355`). Parameterised. |
|
||||
| `NewestArchiveTime` | Reads `r.target` (`internal/backup/runner.go:390`). Parameterised. |
|
||||
| Restore-test rotation | Iterates `BackupTiers()`; `restoreTierForArchive` classifies from the **archive**, never from config (the R-82/R-85 fix). Parameterised. |
|
||||
| Per-run prune | `localPruneSpec` suppresses only for **PBS**-type targets; a `dir` target still prunes at `keep-last=3`. Correct. |
|
||||
| Customer-facing label | `backupIsPBS` (`felhom-controller/.../backup_handlers.go:174`) matches the substring `pbs`. `felhom-backup` correctly renders „Helyi tároló (felhom-backup)". **Naming trap recorded** — a target named with `pbs` in it would be mislabelled as offsite. |
|
||||
|
||||
Two Phase 0 findings that shaped the change are in §4 and §6.
|
||||
|
||||
---
|
||||
|
||||
## 2. The two findings Phase 0 turned up before any command ran
|
||||
|
||||
### F-1 — the storage path must BE the mountpoint, or the target reports disconnected forever
|
||||
|
||||
`reportType` (`internal/storage/observe.go:365`) calls a `dir` storage `local` only when it is
|
||||
literally named `local`; anything else becomes `usb` or `local-dir`. For those two types
|
||||
`reachable()` (`:321`) requires **`exactMount`** — the storage `path` must exactly equal a
|
||||
mountpoint (`exactMountDevice`, `:422`).
|
||||
|
||||
Point the storage at a **subdirectory** of the drive and `exactMount` is false, so
|
||||
`reachable=false`, `state=disconnected`, **permanently** — a false alarm designed in on day one,
|
||||
and the durable id degrades to a path fallback instead of the filesystem UUID.
|
||||
|
||||
**Decision: `path` = the drive's mountpoint.** PVE creates `dump/` at the drive root, beside the
|
||||
existing `felhom-data/`. This also gives a real safety property: the guest is bind-mounted only the
|
||||
drive's `felhom-data` subpath (`/mnt/felhom-drives/<drive>`), so **the archives sit outside anything
|
||||
the customer's guest can reach.**
|
||||
|
||||
### F-2 — `is_mountpoint` is what stops a silent retarget onto the system drive
|
||||
|
||||
`man pvesm`: *"Assume the given path is an externally managed mountpoint and consider the storage
|
||||
offline if it is not mounted."* Without it, an unplugged or late-mounting drive leaves
|
||||
`/mnt/<drive>` a bare directory on the root filesystem and vzdump writes the backup **onto the
|
||||
system drive** — the exact device this change exists to escape, silently. Proven live in §6.
|
||||
|
||||
**Decision: `--is_mountpoint 1` on both boxes.**
|
||||
|
||||
---
|
||||
|
||||
## 3. The exact commands used, per box
|
||||
|
||||
Identical on both, differing only in the mountpoint. Run as `root` on the PVE host.
|
||||
|
||||
```bash
|
||||
# 1. the storage, on the drive's OWN mountpoint, guarded (F-1 + F-2)
|
||||
pvesm add dir felhom-backup --path /mnt/nvme-1tb --content backup --is_mountpoint 1 # demo-hp
|
||||
pvesm add dir felhom-backup --path /mnt/hdd_1 --content backup --is_mountpoint 1 # demo-felhom
|
||||
|
||||
# 2. the per-storage grant — REQUIRED, and the reason the first backup failed (§4)
|
||||
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --users felhom-agent@pve
|
||||
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
|
||||
|
||||
# 3. back up the config, then repoint the primary tier
|
||||
cd /etc/felhom-agent
|
||||
cp -p agent.json agent.json.pre-e-target-move
|
||||
# in-place truncate-write (NOT sed -i): the directory is root-owned while the file is
|
||||
# agent-owned 0600, so a rename would flip ownership and the non-root agent could not read it.
|
||||
python3 - <<'EOF'
|
||||
import json
|
||||
p='/etc/felhom-agent/agent.json'
|
||||
raw=open(p).read()
|
||||
old='"local_backup_target": "local"'
|
||||
new='"local_backup_target": "felhom-backup"'
|
||||
assert raw.count(old)==1, 'occurrences: %d' % raw.count(old)
|
||||
out=raw.replace(old,new); json.loads(out) # validate BEFORE writing
|
||||
with open(p,'r+') as f: f.seek(0); f.write(out); f.truncate()
|
||||
EOF
|
||||
|
||||
# 4. restart only with nothing in flight — check the PVE task list, NOT pgrep (see §8)
|
||||
pvesh get /nodes/<node>/tasks --limit 20 --output-format json | python3 -c \
|
||||
'import sys,json;print([t for t in json.load(sys.stdin) if "endtime" not in t] or "none running")'
|
||||
systemctl restart felhom-agent
|
||||
```
|
||||
|
||||
Resulting stanza, both boxes:
|
||||
|
||||
```
|
||||
dir: felhom-backup
|
||||
path /mnt/<drive>
|
||||
content backup
|
||||
is_mountpoint 1
|
||||
```
|
||||
|
||||
**Verification that Part 1 took**, both boxes — the diff was exactly one line, ownership and mode
|
||||
preserved (`-rw------- felhom-agent felhom-agent`), and both tiers armed with no rejection:
|
||||
|
||||
```
|
||||
backup tier armed target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s primary=true
|
||||
backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s primary=false
|
||||
```
|
||||
|
||||
`GET /backup/due` echoes the new tier, and the **old target no longer exists** — the cleanest proof
|
||||
the switch took:
|
||||
|
||||
```
|
||||
GET /backup/due → {"due":true,"reason":"no successful backup recorded yet","age_state":"absent"}
|
||||
GET /backup/due?target=felhom-backup → {..., "target":"felhom-backup", "age_state":"absent"}
|
||||
GET /backup/due?target=felhom-pbs → {"due":false,"reason":"within cadence window","age_seconds":44362}
|
||||
GET /backup/due?target=local → {"ok":false,"error":"unknown backup target: local"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Finding F-3 — the agent had no PVE privilege on the new storage (found by the first real backup)
|
||||
|
||||
The first triggered backup on demo-hp failed:
|
||||
|
||||
```
|
||||
local-api: backup job failed vmid=9201 target=felhom-backup
|
||||
err="proxmox: POST /nodes/felhom-host/vzdump -> HTTP 403: permission denied
|
||||
at /storage/felhom-backup (missing privilege Datastore.Allocate)"
|
||||
```
|
||||
|
||||
`FelhomAgentStore` (`Datastore.Allocate,Datastore.AllocateSpace`) is granted **per storage path**,
|
||||
to both `felhom-agent@pve` and the `felhom-agent@pve!agent` token — `local`, `local-lvm` and
|
||||
`felhom-pbs` each had their own pair. A new backup target needs the same pair or every backup 403s.
|
||||
|
||||
The grant in §3 step 2 mirrors the existing ones exactly; **no privilege was widened**, and the
|
||||
storage-scoped shape (rather than a grant at `/`) is preserved.
|
||||
|
||||
**This is an E-2 requirement, not a one-off:** `felhom-host-install.sh` must issue the grant for
|
||||
whatever target it provisions on a new install, or every new box ships with a backup tier that
|
||||
403s on its first run.
|
||||
|
||||
---
|
||||
|
||||
## 5. Part 2 — the proof
|
||||
|
||||
### 5.1 A backup lands on the secondary drive — via the real path
|
||||
|
||||
Triggered through the endpoint the UI's „Mentés most" button invokes, not `--selftest`.
|
||||
|
||||
**Route correction worth recording:** the backups page has two triggers.
|
||||
`POST /api/backup/run` is the **Tier-1 app-data** backup (DB dumps + volume dumps); the
|
||||
**whole-guest** vzdump is `POST /api/guest-backup/trigger`, which goes through the controller's
|
||||
quiesce loop (`ServeBackupAPI`, `backup_handlers.go`). The first attempt used the former and
|
||||
correctly produced an app-data run and no vzdump.
|
||||
|
||||
| | demo-hp | demo-felhom |
|
||||
|---|---|---|
|
||||
| Archive | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst` | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst` |
|
||||
| Size | **2,256,044,492 B (2.26 GB)** | **5,957,878,962 B (5.96 GB)** |
|
||||
| Duration | 86.3 s | ~197 s |
|
||||
| Mode | `snapshot` — `snapshotted` at 21:18:16, app resumed before the copy finished | `snapshot` — `snapshotted` at 21:26:29 |
|
||||
| `df` on the target drive | used 92,344,320 → **2,348,572,672** (**+2,256,228,352 B**) | used 3,600,314,368 → **9,558,208,512** (**+5,957,894,144 B**) |
|
||||
|
||||
On both boxes the `df` delta matches the archive size, so the bytes demonstrably landed on the
|
||||
secondary drive and not somewhere else.
|
||||
|
||||
On demo-felhom the backup was taken by the controller's **first-backup safety valve** rather than by
|
||||
the manual trigger: the agent restart made the new tier report `age_state: absent`, the controller
|
||||
correctly refused to withhold a first backup, and the manual trigger that followed was answered
|
||||
`{"error":"mentés már folyamatban van"}`. Same code path, same quiesce loop — the trigger simply
|
||||
lost the race to the mechanism that exists for exactly this state.
|
||||
|
||||
`uncovered_volumes: ["/etc/felhom-bootstrap", "/mnt/felhom-drives"]` — both are host binds and are
|
||||
correctly outside the guest archive. Pre-existing behaviour, unchanged by this move, but see §7.
|
||||
|
||||
### 5.2 A restore works from that archive
|
||||
|
||||
Run with **no** `-archive`, so the candidate *selection* is part of the proof. The rotation needed
|
||||
no nudge — it picked the new storage on its own.
|
||||
|
||||
```
|
||||
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst
|
||||
into scratch band [990000,990009] on local-lvm …
|
||||
{
|
||||
"source_archive": "felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst",
|
||||
"source_tier": "local",
|
||||
"scratch_vmid": 990000,
|
||||
"pass": true,
|
||||
"verified": "boot+running",
|
||||
"mount_parity": "ok",
|
||||
"duration_seconds": 114.39,
|
||||
"mount_inventory": [
|
||||
"mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)",
|
||||
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
|
||||
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
|
||||
]
|
||||
}
|
||||
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m54s) ===
|
||||
```
|
||||
|
||||
**`mount_parity: ok`, `pass: true`, scratch torn down cleanly.**
|
||||
|
||||
demo-felhom, same method, same result — the bigger guest (200 G + 50 G volumes):
|
||||
|
||||
```
|
||||
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst …
|
||||
"pass": true, "verified": "boot+running", "mount_parity": "ok",
|
||||
"duration_seconds": 84.10, "scratch_vmid": 990000,
|
||||
"mount_inventory": ["mp0=/var/lib/docker (200G)", "mp1=/mnt/sys_drive (50G)", …]
|
||||
=== selftest=restore-test OK (restored+booted+verified+torn-down in 1m24s) ===
|
||||
```
|
||||
|
||||
| | demo-hp | demo-felhom |
|
||||
|---|---|---|
|
||||
| Candidate picked automatically from the new storage | yes | yes |
|
||||
| `pass` / `verified` | true / `boot+running` | true / `boot+running` |
|
||||
| **`mount_parity`** | **ok** | **ok** |
|
||||
| Duration | 1 m 54 s | 1 m 24 s |
|
||||
| Scratch torn down | yes (990000) | yes (990000) |
|
||||
|
||||
Run with **no** `-archive` on both boxes, so the candidate *selection* is part of the proof. The
|
||||
rotation needed no nudge on either box — it picked the new storage on its own.
|
||||
|
||||
`source_tier: "local"` is **correct, not a stale reference to the old storage name.**
|
||||
`restoreTierForArchive` classifies by the archive's storage *type*; `felhom-backup` is a `dir`
|
||||
storage, so the restore correctly earns the local (10-minute) wait bound rather than the PBS one.
|
||||
|
||||
**Method caveat:** `--selftest=restore-test` is a separate one-shot process, so it does **not**
|
||||
write the daemon's `rtState`. `GET /restore-test/status` still reads `null` on both boxes and the
|
||||
customer-visible „utoljára ellenőrizve" is unchanged. The proof is real; the daemon's own 84 h
|
||||
rotation is undisturbed and will record its next run normally.
|
||||
|
||||
### 5.3 Freshness follows the target
|
||||
|
||||
Immediately after the backup: `{"due":false,"age_seconds":159,"target":"felhom-backup","age_state":"known"}`.
|
||||
|
||||
That alone does not separate the storage read from this process's in-memory record, so the agent was
|
||||
**restarted** — which empties the in-memory store (R-84) — and re-asked. Both boxes:
|
||||
|
||||
```
|
||||
demo-hp GET /backup/status → {"phase":"idle"} ← no in-memory record at all
|
||||
GET /backup/due → {"due":false,"age_seconds":188,
|
||||
"target":"felhom-backup","age_state":"known"}
|
||||
demo-felhom GET /backup/status → {"phase":"idle"}
|
||||
GET /backup/due → {"due":false,"age_seconds":267,
|
||||
"target":"felhom-backup","age_state":"known"}
|
||||
```
|
||||
|
||||
A `known` age with an empty in-memory store **can only** have come from reading the new storage.
|
||||
The tier neither looks absent (which would re-backup every cycle) nor stale. This is the positive
|
||||
observable, not the absence of a complaint.
|
||||
|
||||
**This restart is also where the one operational error of the run happened — see §8.**
|
||||
|
||||
### 5.4 The DR recipe does **not** record the new target — and one field improved
|
||||
|
||||
**The finding:** `BuildDRRecipeHostHalf` (`internal/hub/dr_recipe.go:86`) records `guests`,
|
||||
`drives`, `pve_storage` and the PBS coordinate. **Nothing in the recipe says which storage holds the
|
||||
local archives.** While the target was `local` that was guessable; it is not any more, and it is now
|
||||
actively misleading — after this change `local` still holds the *stale* archives while the live ones
|
||||
are on `felhom-backup`. A restorer reading the recipe sees two backup-capable storages and cannot
|
||||
tell which is which. **Filed as R-109**, alongside R-105/R-106.
|
||||
|
||||
**The improvement — this change traces and partly closes R-105.** R-105 recorded
|
||||
`dr_recipe.host_half.drives = []` on every customer *including two with enrolled data drives*, with
|
||||
the cause explicitly untraced. It is now traced: **the enrolled drives were never PVE storages at
|
||||
all**, so they never entered `report.StorageTargets`, so `isUserDataDrive` (which needs type
|
||||
`usb`/`local-dir` **and** a durable id **and** a mount path) never saw them. Making the drive a
|
||||
storage populated it. Live on demo-hp after the change:
|
||||
|
||||
```json
|
||||
"drives": [{ "durable_id": "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae",
|
||||
"mount_path": "/mnt/nvme-1tb", "intent": "enrolled",
|
||||
"total_bytes": 1006980812800 }]
|
||||
```
|
||||
|
||||
The storage observation also gained a real device, a real UUID and **SMART** for the backup drive —
|
||||
`health=PASSED temp=50 poh=28451 media_err=0 pct_used=7` — none of which existed when the drive was
|
||||
not a PVE storage:
|
||||
|
||||
```
|
||||
- felhom-backup type=local-dir state=attached reach=true class=fast
|
||||
durable=uuid:91d2dc2d-… mount="/mnt/nvme-1tb" dev="/dev/nvme0n1"
|
||||
```
|
||||
|
||||
demo-felhom shows the identical shape — `drives` populated, SMART now readable on the USB HDD:
|
||||
|
||||
```
|
||||
- felhom-backup type=local-dir state=attached reach=true class=slow
|
||||
durable=uuid:47a3361a-… mount="/mnt/hdd_1" dev="/dev/sdb"
|
||||
smart: health=PASSED temp=35 poh=3581 realloc=0 pending=0 offline_unc=0
|
||||
|
||||
"drives": [{ "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
|
||||
"mount_path": "/mnt/hdd_1", "intent": "enrolled",
|
||||
"total_bytes": 983351140352 }]
|
||||
```
|
||||
|
||||
**`dr_recipe.host_half.drives` is now non-empty on both demo boxes**, where R-105 recorded `[]`
|
||||
fleet-wide. Note the classification detail: demo-felhom's drive is USB-attached but reports
|
||||
`removable=0`, so it is typed `local-dir` rather than `usb`. That does not matter here — both types
|
||||
take the same reachability branch and both satisfy `isUserDataDrive` — but it is one more reason
|
||||
§2's "do not classify by transport" rule is right.
|
||||
|
||||
R-106 is untouched and still live on both boxes: the recipe still records `"namespace": "root"`
|
||||
where the real namespaces are `demo-hp` / `demo-felhom`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Part 3 — what happens when the target drive is absent
|
||||
|
||||
Established with **throwaway storages**, so neither live drive was unmounted (unmounting would break
|
||||
the guest's `felhom-data` bind and the customer's app data on a remote box).
|
||||
|
||||
### Live-proven: the PVE half
|
||||
|
||||
| | storage on a non-mount path, **no** `is_mountpoint` | same path, **`is_mountpoint 1`** |
|
||||
|---|---|---|
|
||||
| Creation | succeeded silently | **refused**: `unable to activate storage 'absent-test-b' - directory is expected to be a mount point but is not mounted: '/mnt/absent-b'` |
|
||||
| `pvesm status` | **`active`**, 40516856 KiB total | never created |
|
||||
| Backing device | **`/dev/mapper/pve-root`** — the system drive | — |
|
||||
| What it created | `/mnt/absent-a/dump/` **on the root filesystem** | nothing |
|
||||
|
||||
The unguarded storage reports `active` with plausible free space that is really the **root
|
||||
filesystem's**, and it had already created a `dump/` directory there. A vzdump aimed at it would
|
||||
write the whole-guest backup onto the system drive with no error and no alarm — **silent
|
||||
retargeting onto the exact device this change exists to escape**, which §6 names as the worst
|
||||
outcome. `is_mountpoint 1` converts that into a loud refusal. Both test storages were removed and
|
||||
`/mnt/absent-a` deleted.
|
||||
|
||||
### Source-traced, not live-proven: the agent half
|
||||
|
||||
With the guard in place and the drive absent, the storage stays in `storage.cfg` but goes inactive.
|
||||
`targetStoragePresent` (`internal/localapi/server.go:1212`) checks the target's **name presence
|
||||
only**, never `Reachable`, so the tier is **not** deferred; `newestArchiveOn` cannot read the
|
||||
storage and degrades to `archiveUnknown`, which stays **DUE** by design. The controller therefore
|
||||
quiesces the apps and fires a vzdump, PVE refuses because the storage is not active, and the run
|
||||
**fails loudly** — arming the R-88 breaker and the `backup_failed` operator mail.
|
||||
|
||||
So today's behaviour is: **fail and alarm; no silent retarget.** That is the safe half.
|
||||
|
||||
**It does not match §6's intended design** (*fall back to the system drive and alarm*): there is no
|
||||
fallback at all, so an absent drive means **no local backup whatsoever** until a human intervenes,
|
||||
and each attempt costs an app quiesce until the breaker backs off. Whether a corruption-only copy on
|
||||
the system drive beats no copy is a product decision; the honest label matters either way.
|
||||
**Filed as E-2.**
|
||||
|
||||
This half is traced through source, not reproduced on hardware — the boxes are remote and unmounting
|
||||
a live enrolled drive would break the guest bind and the customer's data. Recorded as outstanding
|
||||
proof in §9.
|
||||
|
||||
### Single-drive boxes
|
||||
|
||||
A single-drive box has nowhere to move the target. There the backup stays on the system drive and
|
||||
protects against **corruption only** — it is not drive-loss protection and must not be presented as
|
||||
a backup tier that survives hardware failure. **Two drives is effectively a hardware requirement for
|
||||
drive-loss protection**, and that belongs in the sales/onboarding material as well as in the UI
|
||||
label. Filed under E-2.
|
||||
|
||||
---
|
||||
|
||||
## 7. Scope recorded for E-2 — filed, not built
|
||||
|
||||
| # | Item |
|
||||
|---|---|
|
||||
| 1 | A **backup-target role** on `StoragePath`, alongside `Schedulable` / `IsDefault` / `Kind`. |
|
||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||
| 8 | The honest **single-drive label**. |
|
||||
| 9 | **Migrating the remaining fleet** — `peti-felhom` and any box not covered here. |
|
||||
| 10 | Naming guard: `backupIsPBS` matches the substring `pbs`, so a target so named would be mislabelled to the customer as offsite (§1.5). |
|
||||
|
||||
---
|
||||
|
||||
## 8. Observations
|
||||
|
||||
### 8.1 An operational error in this run: the agent was restarted with a backup in flight
|
||||
|
||||
§4.3 and §8 of the spec both say not to. It happened anyway, on demo-hp at **21:21:12**.
|
||||
|
||||
The manual „Mentés most" correctly fires **every** tier, not just the primary, so the trigger at
|
||||
21:18 started the local vzdump *and* a `felhom-pbs` one. The local tier finished at 21:19:41. The
|
||||
PBS tier was still running. The in-flight check had been done before the **first** restart (21:12,
|
||||
clean) and **was not repeated** before the second one — the restart taken for the §5.3 freshness
|
||||
proof — so it landed on top of a live PBS backup.
|
||||
|
||||
What it produced:
|
||||
|
||||
```
|
||||
level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs
|
||||
err="backup: vzdump task vmid 9201: proxmox: waiting for task UPID:…:vzdump:9201:… : context canceled"
|
||||
```
|
||||
|
||||
**The backup had not failed.** `context canceled` is the agent being shut down while waiting, not
|
||||
the task failing. Checked rather than assumed, against two independent authorities:
|
||||
|
||||
- the PVE task `2026-07-28T21:19:45 … vzdump 9201` → **OK**;
|
||||
- the PBS snapshot `felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z` → **6,264,034,053 B**, a real,
|
||||
plausibly-complete archive (nine orders of magnitude above the F-CRIT-2 1 MiB floor).
|
||||
|
||||
So the run recorded a **spurious tier failure** — a self-inflicted instance of exactly the F-A1
|
||||
class ("a healthy backup reported as FAILED") that the project fixed in controller v0.179.0.
|
||||
|
||||
**It reached no channel, and the reason is worth keeping.** The controller's event trail for the
|
||||
whole window is:
|
||||
|
||||
```
|
||||
19:15:43Z whole_guest_backup_failed (error) felhom-backup tier — the REAL 403, backoff 15m
|
||||
19:19:45Z whole_guest_backup_recovered (info) felhom-backup tier — cleared after the ACL grant
|
||||
```
|
||||
|
||||
— and **nothing for the 21:21 PBS failure.** The agent process died with its own in-memory failure
|
||||
record, and by the time the controller next polled, `NewestArchiveTime` had read ground truth from
|
||||
PBS and found the successful 21:19:45 archive. The R-84 ground-truth design absorbed it: no breaker
|
||||
armed on the offsite tier, no operator mail, no spurious staleness. That is the design working, not
|
||||
a reason the mistake was harmless — on a slower tier, or with the restart a minute earlier, the same
|
||||
slip could have aborted a multi-hour WAN upload.
|
||||
|
||||
**The rule that would have caught it:** re-check in flight immediately before *every* restart, not
|
||||
once at the start of the procedure — and remember that a manual trigger fires all tiers, so the
|
||||
offsite tier can still be running long after the local one has finished.
|
||||
|
||||
### 8.2 Method and tooling
|
||||
|
||||
- **`pgrep -f vzdump` is not a safe in-flight check.** A polling script whose own command line
|
||||
contains the string matches itself, and it reported a backup in flight when the PVE task list
|
||||
showed none. It caused one false "ABORT: vzdump in flight" and one wrong reading in this run.
|
||||
**Use the PVE task list** (`pvesh get /nodes/<node>/tasks`, filter for a missing `endtime`).
|
||||
- **`sed -i` must not be used on `agent.json`.** The directory is root-owned and the file is
|
||||
agent-owned `0600`; `sed -i` renames, which would flip the file to `root:root` and leave the
|
||||
non-root agent unable to read its own config. The in-place truncate-write in §3 preserves both.
|
||||
- A `felhom-pbs` vzdump failed at **17:58**, hours before this change, with
|
||||
`Can't connect to 10.77.0.1:8007 (Connection refused)`. **Pre-existing and unrelated** — checked
|
||||
rather than assumed: ep0 is up (uptime 1 d 2 h, `proxmox-backup-proxy` active), the WG handshake
|
||||
is fresh, and 8007 is open from demo-felhom now.
|
||||
- The controller's first-backup safety valve behaved exactly as designed on demo-felhom: the agent
|
||||
restart made the new tier report `age_state: absent`, and the controller took a backup on its own
|
||||
before the manual trigger could (`{"error":"mentés már folyamatban van"}`).
|
||||
- The archives are outside the customer's reach by construction — the guest is bound only the
|
||||
drive's `felhom-data` subpath, while `dump/` sits at the drive root.
|
||||
|
||||
---
|
||||
|
||||
## 9. State at close
|
||||
|
||||
### 9.1 Old archives — left in place, as instructed
|
||||
|
||||
Nothing was deleted. They remain the rollback and the only evidence of what the previous
|
||||
configuration produced.
|
||||
|
||||
| Box | Path | Contents | Size |
|
||||
|---|---|---|---|
|
||||
| demo-felhom | `/var/lib/vz/dump/` | 3 × 9201 (5.82 / 5.84 / 5.93 GB) + 2 × 9100 (612 / 649 MB) | **18 GB** |
|
||||
| demo-hp | `/var/lib/vz/dump/` | 3 × 9201 (1.61 / 1.63 / 1.68 GB) + 1 × 9100 (649 MB) | **5.2 GB** |
|
||||
|
||||
**These are now stale and will never be refreshed** — the tier that wrote them no longer points
|
||||
here, and `keep-last=3` prunes only the *new* target. They are also the reason R-109 matters: a
|
||||
restorer reading the DR recipe sees both `local` and `felhom-backup` carrying `content=backup` and
|
||||
cannot tell that one holds live archives and the other holds a frozen snapshot of 2026-07-28.
|
||||
Deleting them is a separate, deliberate decision.
|
||||
|
||||
### 9.2 Fleet state — both boxes healthy, no thrash, no spurious staleness
|
||||
|
||||
| | demo-felhom | demo-hp |
|
||||
|---|---|---|
|
||||
| Agent | `active`, v0.110.0 | `active`, v0.110.0 |
|
||||
| Primary tier | `felhom-backup`, `due=false`, `age_state=known` | `felhom-backup`, `due=false`, `age_state=known` |
|
||||
| Offsite tier | `felhom-pbs`, `due=false`, `age_state=known` | `felhom-pbs`, `due=false`, `age_state=known` |
|
||||
| New target | 5.96 GB used of 916 GB (**2 %**) | 2.26 GB used of 938 GB (**1 %**) |
|
||||
| Target drive SMART | `PASSED`, 35 °C | `PASSED`, 50 °C, 7 % wear |
|
||||
| System drive | 24 G / 94 G (27 %) | 14 G / 39 G (38 %) |
|
||||
| Agent errors, last 15 min | **0** | 2, both accounted for (§4 403, §8.1 spurious) |
|
||||
| Breaker | clear (`whole_guest_backup_recovered` 19:19:45Z) | clear |
|
||||
|
||||
No tier is stale, no tier is thrashing, and both offsite tiers report a known, recent age.
|
||||
|
||||
### 9.3 Outstanding proof
|
||||
|
||||
1. **Full drive-loss recovery** — pull the system drive, restore the guest from the vzdump on the
|
||||
secondary drive, confirm it returns with named volumes, secrets and config. Needs physical
|
||||
access; the boxes are remote until ~08-02. This is the proof that flips **matrix row 4** from
|
||||
PARTIAL to a real local route. Everything up to it is now proven: the archive exists on separate
|
||||
hardware, and a restore from it boots and passes `mount_parity`.
|
||||
2. **The agent half of §6** — with the drive absent, that the tier fails loudly rather than silently
|
||||
retargeting. The **PVE half is live-proven** (§6); the agent half is traced through source only,
|
||||
because unmounting a live enrolled drive would break the guest's `felhom-data` bind and the
|
||||
customer's app data on a remote box.
|
||||
|
||||
### 9.4 Backlog rows filed
|
||||
|
||||
| ID | What |
|
||||
|---|---|
|
||||
| **E-2** | The machinery around this change — backup-target role on `StoragePath`, wizard assignment, no automatic roles, stickiness, `felhom-host-install.sh` creating the target **and** its ACL, absent-target policy, retention/space accounting on a shared drive, the honest single-drive label, remaining fleet migration. Full scope in §7. |
|
||||
| **R-109** | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` lists every storage's name/type/content but never says which one holds the local archives — and after this change the box carries two `content=backup` dir storages, one live and one stale. §5.4. |
|
||||
| **R-105** | Partially closed and **fully traced** for its third field: `dr_recipe.host_half.drives` is now populated on both demo boxes, and the cause of the fleet-wide `[]` is identified — the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them. The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched by this run. |
|
||||
Reference in New Issue
Block a user