# RUNBOOK RECORD — the local whole-guest backup moved off the guest's own device (2026-07-28)

**Class:** supervised operational change, one box at a time. **No code, no version bump** — the
change is a PVE storage definition, a per-storage ACL grant, and a one-line target switch in
`agent.json`. **Nothing was deleted.**

**Boxes:** `demo-hp` (`felhom-host`, HP t740) first, then `demo-felhom` (N100).
**Change:** primary backup tier `local` → **`felhom-backup`**, a `dir` storage on each box's
secondary drive.
**Window:** 2026-07-28 21:11 → 21:35 CEST.

> **Deliverable note.** The spec is `RUNBOOK-vzdump-target-move-2026-07-29.md`; the work ran on the
> evening of 07-28. The filename is kept as specified.

---

## 1. Phase 0 — ground truth, established read-only before anything was touched

### 1.1 The premise holds: `local` and `local-lvm` are the same physical device

| | demo-felhom | demo-hp |
|---|---|---|
| `local` (dir, `/var/lib/vz`) | `pve-root` → **`sda3`** | `pve-root` → **`sda3`** |
| `local-lvm` (lvmthin, `pve/data`) | **`sda3`** | **`sda3`** |
| System drive | AirDisk 512 GB SATA SSD (`sda`) | SanDisk X600 M.2 128 GB (`sda`) |

A drive failure took the guest and its only local backup together. That is the fault being closed.

### 1.2 Physical inventory and the durable id

| | demo-felhom | demo-hp |
|---|---|---|
| Secondary drive | `/dev/sdb` TOSHIBA MQ04ABF100, **USB**, rotational | `/dev/nvme0n1` KXG50PNV1T02, **NVMe** |
| Size / free before | 916 GB / 866 GB | 938 GB / 891 GB |
| Mountpoint | `/mnt/hdd_1` | `/mnt/nvme-1tb` |
| **Durable id (fs UUID)** | `47a3361a-91e0-4831-a69d-27f540ed3f48` | `91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae` |
| `/sys/block/*/removable` | `0` | `0` |
| SMART at close | — | `PASSED`, 50 °C, 7 % wear, 0 media errors |

Both drives are already enrolled Felhom drives, mounted **by filesystem UUID** through
agent-generated systemd units (`mnt-hdd_1.mount`, `mnt-nvme\x2d1tb.mount`), each marked
*"Managed by felhom-agent — do not edit by hand"*, `WantedBy=multi-user.target`. **Neither is in
`/etc/fstab`.** The UUID in the unit is the same durable id the DR recipe carries, so the target
drive was already durably identified before this change.

This is §2's principle meeting the reference hardware head-on: demo-felhom's target **is** an
external USB HDD and both drives report `removable=0`. A rule keyed on transport would disqualify
demo-felhom; a rule keyed on the `removable` flag would see no removable drive at all on either box.
Neither attribute can decide a role. Both are fine as *hints*.

### 1.3 Space

The runbook's ~14.5 GB figure is the **PBS-side** measurement. The local `zstd` archive is far
smaller:

| | newest local archive before | × `keep-last=3` | share of the target drive's free space |
|---|---|---|---|
| demo-felhom (9201) | 5.93 GB (trend 5.82 → 5.84 → 5.93) | **~17.8 GB** | **2.1 %** of 866 GB |
| demo-hp (9201) | 1.68 GB (1.61 → 1.63 → 1.68) | **~5.0 GB** | **0.6 %** of 891 GB |

Retention is `local_backup_retention: 3` on both boxes and is unchanged by this runbook.

The move also relieves the system drive — 18 GB on demo-felhom, 5.2 GB on demo-hp once the old
archives are eventually removed. On demo-hp that matters: its root is 40 GB with 24 GB free.

### 1.4 How the target is set — **configuration, not converged. The runbook holds.**

The primary tier is built from `backup.local_backup_target` (`felhom-agent/internal/config/config.go:541`,
`BackupTarget()`), which was `"local"` on both boxes. The extra tier is `felhom-pbs`, weekly.

Every writer of `agent.json` was enumerated. **There is exactly one:**
`pbsdr.seedEscrowStorageID` (`internal/pbsdr/manager.go:454`). It sets only `escrow.pbs_storage_id`,
refuses to clobber a differing existing value (warn-and-keep), and performs a
`map[string]json.RawMessage` read-modify-write that **preserves unknown keys verbatim**. Nothing on
the hub pushes agent configuration. Seven prior hand-edit backups sit beside the live file.

**A hand edit sticks. No code change was required, so the STOP in §3 did not fire.**

### 1.5 What else reads the target — no hardcoded `"local"` on the backup path

| Consumer | Verdict |
|---|---|
| Backup runners | One runner **per tier**, each constructed with its own `TargetID` (`cmd/felhom-agent/main.go:1355`). Parameterised. |
| `NewestArchiveTime` | Reads `r.target` (`internal/backup/runner.go:390`). Parameterised. |
| Restore-test rotation | Iterates `BackupTiers()`; `restoreTierForArchive` classifies from the **archive**, never from config (the R-82/R-85 fix). Parameterised. |
| Per-run prune | `localPruneSpec` suppresses only for **PBS**-type targets; a `dir` target still prunes at `keep-last=3`. Correct. |
| Customer-facing label | `backupIsPBS` (`felhom-controller/.../backup_handlers.go:174`) matches the substring `pbs`. `felhom-backup` correctly renders „Helyi tároló (felhom-backup)". **Naming trap recorded** — a target named with `pbs` in it would be mislabelled as offsite. |

Two Phase 0 findings that shaped the change are in §4 and §6.

---

## 2. The two findings Phase 0 turned up before any command ran

### F-1 — the storage path must BE the mountpoint, or the target reports disconnected forever

`reportType` (`internal/storage/observe.go:365`) calls a `dir` storage `local` only when it is
literally named `local`; anything else becomes `usb` or `local-dir`. For those two types
`reachable()` (`:321`) requires **`exactMount`** — the storage `path` must exactly equal a
mountpoint (`exactMountDevice`, `:422`).

Point the storage at a **subdirectory** of the drive and `exactMount` is false, so
`reachable=false`, `state=disconnected`, **permanently** — a false alarm designed in on day one,
and the durable id degrades to a path fallback instead of the filesystem UUID.

**Decision: `path` = the drive's mountpoint.** PVE creates `dump/` at the drive root, beside the
existing `felhom-data/`. This also gives a real safety property: the guest is bind-mounted only the
drive's `felhom-data` subpath (`/mnt/felhom-drives/<drive>`), so **the archives sit outside anything
the customer's guest can reach.**

### F-2 — `is_mountpoint` is what stops a silent retarget onto the system drive

`man pvesm`: *"Assume the given path is an externally managed mountpoint and consider the storage
offline if it is not mounted."* Without it, an unplugged or late-mounting drive leaves
`/mnt/<drive>` a bare directory on the root filesystem and vzdump writes the backup **onto the
system drive** — the exact device this change exists to escape, silently. Proven live in §6.

**Decision: `--is_mountpoint 1` on both boxes.**

---

## 3. The exact commands used, per box

Identical on both, differing only in the mountpoint. Run as `root` on the PVE host.

```bash
# 1. the storage, on the drive's OWN mountpoint, guarded (F-1 + F-2)
pvesm add dir felhom-backup --path /mnt/nvme-1tb --content backup --is_mountpoint 1   # demo-hp
pvesm add dir felhom-backup --path /mnt/hdd_1    --content backup --is_mountpoint 1   # demo-felhom

# 2. the per-storage grant — REQUIRED, and the reason the first backup failed (§4)
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --users  felhom-agent@pve
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'

# 3. back up the config, then repoint the primary tier
cd /etc/felhom-agent
cp -p agent.json agent.json.pre-e-target-move
#   in-place truncate-write (NOT sed -i): the directory is root-owned while the file is
#   agent-owned 0600, so a rename would flip ownership and the non-root agent could not read it.
python3 - <<'EOF'
import json
p='/etc/felhom-agent/agent.json'
raw=open(p).read()
old='"local_backup_target": "local"'
new='"local_backup_target": "felhom-backup"'
assert raw.count(old)==1, 'occurrences: %d' % raw.count(old)
out=raw.replace(old,new); json.loads(out)          # validate BEFORE writing
with open(p,'r+') as f: f.seek(0); f.write(out); f.truncate()
EOF

# 4. restart only with nothing in flight — check the PVE task list, NOT pgrep (see §8)
pvesh get /nodes/<node>/tasks --limit 20 --output-format json | python3 -c \
  'import sys,json;print([t for t in json.load(sys.stdin) if "endtime" not in t] or "none running")'
systemctl restart felhom-agent
```

Resulting stanza, both boxes:

```
dir: felhom-backup
	path /mnt/<drive>
	content backup
	is_mountpoint 1
```

**Verification that Part 1 took**, both boxes — the diff was exactly one line, ownership and mode
preserved (`-rw------- felhom-agent felhom-agent`), and both tiers armed with no rejection:

```
backup tier armed target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s primary=true
backup tier armed target=felhom-pbs    cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s primary=false
```

`GET /backup/due` echoes the new tier, and the **old target no longer exists** — the cleanest proof
the switch took:

```
GET /backup/due                        → {"due":true,"reason":"no successful backup recorded yet","age_state":"absent"}
GET /backup/due?target=felhom-backup   → {..., "target":"felhom-backup", "age_state":"absent"}
GET /backup/due?target=felhom-pbs      → {"due":false,"reason":"within cadence window","age_seconds":44362}
GET /backup/due?target=local           → {"ok":false,"error":"unknown backup target: local"}
```

---

## 4. Finding F-3 — the agent had no PVE privilege on the new storage (found by the first real backup)

The first triggered backup on demo-hp failed:

```
local-api: backup job failed vmid=9201 target=felhom-backup
  err="proxmox: POST /nodes/felhom-host/vzdump -> HTTP 403: permission denied
       at /storage/felhom-backup (missing privilege Datastore.Allocate)"
```

`FelhomAgentStore` (`Datastore.Allocate,Datastore.AllocateSpace`) is granted **per storage path**,
to both `felhom-agent@pve` and the `felhom-agent@pve!agent` token — `local`, `local-lvm` and
`felhom-pbs` each had their own pair. A new backup target needs the same pair or every backup 403s.

The grant in §3 step 2 mirrors the existing ones exactly; **no privilege was widened**, and the
storage-scoped shape (rather than a grant at `/`) is preserved.

**This is an E-2 requirement, not a one-off:** `felhom-host-install.sh` must issue the grant for
whatever target it provisions on a new install, or every new box ships with a backup tier that
403s on its first run.

---

## 5. Part 2 — the proof

### 5.1 A backup lands on the secondary drive — via the real path

Triggered through the endpoint the UI's „Mentés most" button invokes, not `--selftest`.

**Route correction worth recording:** the backups page has two triggers.
`POST /api/backup/run` is the **Tier-1 app-data** backup (DB dumps + volume dumps); the
**whole-guest** vzdump is `POST /api/guest-backup/trigger`, which goes through the controller's
quiesce loop (`ServeBackupAPI`, `backup_handlers.go`). The first attempt used the former and
correctly produced an app-data run and no vzdump.

| | demo-hp | demo-felhom |
|---|---|---|
| Archive | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst` | `felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst` |
| Size | **2,256,044,492 B (2.26 GB)** | **5,957,878,962 B (5.96 GB)** |
| Duration | 86.3 s | ~197 s |
| Mode | `snapshot` — `snapshotted` at 21:18:16, app resumed before the copy finished | `snapshot` — `snapshotted` at 21:26:29 |
| `df` on the target drive | used 92,344,320 → **2,348,572,672** (**+2,256,228,352 B**) | used 3,600,314,368 → **9,558,208,512** (**+5,957,894,144 B**) |

On both boxes the `df` delta matches the archive size, so the bytes demonstrably landed on the
secondary drive and not somewhere else.

On demo-felhom the backup was taken by the controller's **first-backup safety valve** rather than by
the manual trigger: the agent restart made the new tier report `age_state: absent`, the controller
correctly refused to withhold a first backup, and the manual trigger that followed was answered
`{"error":"mentés már folyamatban van"}`. Same code path, same quiesce loop — the trigger simply
lost the race to the mechanism that exists for exactly this state.

`uncovered_volumes: ["/etc/felhom-bootstrap", "/mnt/felhom-drives"]` — both are host binds and are
correctly outside the guest archive. Pre-existing behaviour, unchanged by this move, but see §7.

### 5.2 A restore works from that archive

Run with **no** `-archive`, so the candidate *selection* is part of the proof. The rotation needed
no nudge — it picked the new storage on its own.

```
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst
          into scratch band [990000,990009] on local-lvm …
{
  "source_archive": "felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst",
  "source_tier": "local",
  "scratch_vmid": 990000,
  "pass": true,
  "verified": "boot+running",
  "mount_parity": "ok",
  "duration_seconds": 114.39,
  "mount_inventory": [
    "mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)",
    "mp8=/mnt/felhom-drives (throwaway for the archived bind)",
    "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
  ]
}
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m54s) ===
```

**`mount_parity: ok`, `pass: true`, scratch torn down cleanly.**

demo-felhom, same method, same result — the bigger guest (200 G + 50 G volumes):

```
restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst …
  "pass": true, "verified": "boot+running", "mount_parity": "ok",
  "duration_seconds": 84.10, "scratch_vmid": 990000,
  "mount_inventory": ["mp0=/var/lib/docker (200G)", "mp1=/mnt/sys_drive (50G)", …]
=== selftest=restore-test OK (restored+booted+verified+torn-down in 1m24s) ===
```

| | demo-hp | demo-felhom |
|---|---|---|
| Candidate picked automatically from the new storage | yes | yes |
| `pass` / `verified` | true / `boot+running` | true / `boot+running` |
| **`mount_parity`** | **ok** | **ok** |
| Duration | 1 m 54 s | 1 m 24 s |
| Scratch torn down | yes (990000) | yes (990000) |

Run with **no** `-archive` on both boxes, so the candidate *selection* is part of the proof. The
rotation needed no nudge on either box — it picked the new storage on its own.

`source_tier: "local"` is **correct, not a stale reference to the old storage name.**
`restoreTierForArchive` classifies by the archive's storage *type*; `felhom-backup` is a `dir`
storage, so the restore correctly earns the local (10-minute) wait bound rather than the PBS one.

**Method caveat:** `--selftest=restore-test` is a separate one-shot process, so it does **not**
write the daemon's `rtState`. `GET /restore-test/status` still reads `null` on both boxes and the
customer-visible „utoljára ellenőrizve" is unchanged. The proof is real; the daemon's own 84 h
rotation is undisturbed and will record its next run normally.

### 5.3 Freshness follows the target

Immediately after the backup: `{"due":false,"age_seconds":159,"target":"felhom-backup","age_state":"known"}`.

That alone does not separate the storage read from this process's in-memory record, so the agent was
**restarted** — which empties the in-memory store (R-84) — and re-asked. Both boxes:

```
demo-hp       GET /backup/status → {"phase":"idle"}            ← no in-memory record at all
              GET /backup/due    → {"due":false,"age_seconds":188,
                                    "target":"felhom-backup","age_state":"known"}
demo-felhom   GET /backup/status → {"phase":"idle"}
              GET /backup/due    → {"due":false,"age_seconds":267,
                                    "target":"felhom-backup","age_state":"known"}
```

A `known` age with an empty in-memory store **can only** have come from reading the new storage.
The tier neither looks absent (which would re-backup every cycle) nor stale. This is the positive
observable, not the absence of a complaint.

**This restart is also where the one operational error of the run happened — see §8.**

### 5.4 The DR recipe does **not** record the new target — and one field improved

**The finding:** `BuildDRRecipeHostHalf` (`internal/hub/dr_recipe.go:86`) records `guests`,
`drives`, `pve_storage` and the PBS coordinate. **Nothing in the recipe says which storage holds the
local archives.** While the target was `local` that was guessable; it is not any more, and it is now
actively misleading — after this change `local` still holds the *stale* archives while the live ones
are on `felhom-backup`. A restorer reading the recipe sees two backup-capable storages and cannot
tell which is which. **Filed as R-109**, alongside R-105/R-106.

**The improvement — this change traces and partly closes R-105.** R-105 recorded
`dr_recipe.host_half.drives = []` on every customer *including two with enrolled data drives*, with
the cause explicitly untraced. It is now traced: **the enrolled drives were never PVE storages at
all**, so they never entered `report.StorageTargets`, so `isUserDataDrive` (which needs type
`usb`/`local-dir` **and** a durable id **and** a mount path) never saw them. Making the drive a
storage populated it. Live on demo-hp after the change:

```json
"drives": [{ "durable_id": "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae",
             "mount_path": "/mnt/nvme-1tb", "intent": "enrolled",
             "total_bytes": 1006980812800 }]
```

The storage observation also gained a real device, a real UUID and **SMART** for the backup drive —
`health=PASSED temp=50 poh=28451 media_err=0 pct_used=7` — none of which existed when the drive was
not a PVE storage:

```
- felhom-backup type=local-dir state=attached reach=true class=fast
    durable=uuid:91d2dc2d-… mount="/mnt/nvme-1tb" dev="/dev/nvme0n1"
```

demo-felhom shows the identical shape — `drives` populated, SMART now readable on the USB HDD:

```
- felhom-backup type=local-dir state=attached reach=true class=slow
    durable=uuid:47a3361a-… mount="/mnt/hdd_1" dev="/dev/sdb"
    smart: health=PASSED temp=35 poh=3581 realloc=0 pending=0 offline_unc=0

"drives": [{ "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
             "mount_path": "/mnt/hdd_1", "intent": "enrolled",
             "total_bytes": 983351140352 }]
```

**`dr_recipe.host_half.drives` is now non-empty on both demo boxes**, where R-105 recorded `[]`
fleet-wide. Note the classification detail: demo-felhom's drive is USB-attached but reports
`removable=0`, so it is typed `local-dir` rather than `usb`. That does not matter here — both types
take the same reachability branch and both satisfy `isUserDataDrive` — but it is one more reason
§2's "do not classify by transport" rule is right.

R-106 is untouched and still live on both boxes: the recipe still records `"namespace": "root"`
where the real namespaces are `demo-hp` / `demo-felhom`.

---

## 6. Part 3 — what happens when the target drive is absent

Established with **throwaway storages**, so neither live drive was unmounted (unmounting would break
the guest's `felhom-data` bind and the customer's app data on a remote box).

### Live-proven: the PVE half

| | storage on a non-mount path, **no** `is_mountpoint` | same path, **`is_mountpoint 1`** |
|---|---|---|
| Creation | succeeded silently | **refused**: `unable to activate storage 'absent-test-b' - directory is expected to be a mount point but is not mounted: '/mnt/absent-b'` |
| `pvesm status` | **`active`**, 40516856 KiB total | never created |
| Backing device | **`/dev/mapper/pve-root`** — the system drive | — |
| What it created | `/mnt/absent-a/dump/` **on the root filesystem** | nothing |

The unguarded storage reports `active` with plausible free space that is really the **root
filesystem's**, and it had already created a `dump/` directory there. A vzdump aimed at it would
write the whole-guest backup onto the system drive with no error and no alarm — **silent
retargeting onto the exact device this change exists to escape**, which §6 names as the worst
outcome. `is_mountpoint 1` converts that into a loud refusal. Both test storages were removed and
`/mnt/absent-a` deleted.

### Source-traced, not live-proven: the agent half

With the guard in place and the drive absent, the storage stays in `storage.cfg` but goes inactive.
`targetStoragePresent` (`internal/localapi/server.go:1212`) checks the target's **name presence
only**, never `Reachable`, so the tier is **not** deferred; `newestArchiveOn` cannot read the
storage and degrades to `archiveUnknown`, which stays **DUE** by design. The controller therefore
quiesces the apps and fires a vzdump, PVE refuses because the storage is not active, and the run
**fails loudly** — arming the R-88 breaker and the `backup_failed` operator mail.

So today's behaviour is: **fail and alarm; no silent retarget.** That is the safe half.

**It does not match §6's intended design** (*fall back to the system drive and alarm*): there is no
fallback at all, so an absent drive means **no local backup whatsoever** until a human intervenes,
and each attempt costs an app quiesce until the breaker backs off. Whether a corruption-only copy on
the system drive beats no copy is a product decision; the honest label matters either way.
**Filed as E-2.**

This half is traced through source, not reproduced on hardware — the boxes are remote and unmounting
a live enrolled drive would break the guest bind and the customer's data. Recorded as outstanding
proof in §9.

### Single-drive boxes

A single-drive box has nowhere to move the target. There the backup stays on the system drive and
protects against **corruption only** — it is not drive-loss protection and must not be presented as
a backup tier that survives hardware failure. **Two drives is effectively a hardware requirement for
drive-loss protection**, and that belongs in the sales/onboarding material as well as in the UI
label. Filed under E-2.

---

## 7. Scope recorded for E-2 — filed, not built

| # | Item |
|---|---|
| 1 | A **backup-target role** on `StoragePath`, alongside `Schedulable` / `IsDefault` / `Kind`. |
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. |
| 9 | **Migrating the remaining fleet** — `peti-felhom` and any box not covered here. |
| 10 | Naming guard: `backupIsPBS` matches the substring `pbs`, so a target so named would be mislabelled to the customer as offsite (§1.5). |

---

## 8. Observations

### 8.1 An operational error in this run: the agent was restarted with a backup in flight

§4.3 and §8 of the spec both say not to. It happened anyway, on demo-hp at **21:21:12**.

The manual „Mentés most" correctly fires **every** tier, not just the primary, so the trigger at
21:18 started the local vzdump *and* a `felhom-pbs` one. The local tier finished at 21:19:41. The
PBS tier was still running. The in-flight check had been done before the **first** restart (21:12,
clean) and **was not repeated** before the second one — the restart taken for the §5.3 freshness
proof — so it landed on top of a live PBS backup.

What it produced:

```
level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs
  err="backup: vzdump task vmid 9201: proxmox: waiting for task UPID:…:vzdump:9201:… : context canceled"
```

**The backup had not failed.** `context canceled` is the agent being shut down while waiting, not
the task failing. Checked rather than assumed, against two independent authorities:

- the PVE task `2026-07-28T21:19:45 … vzdump 9201` → **OK**;
- the PBS snapshot `felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z` → **6,264,034,053 B**, a real,
  plausibly-complete archive (nine orders of magnitude above the F-CRIT-2 1 MiB floor).

So the run recorded a **spurious tier failure** — a self-inflicted instance of exactly the F-A1
class ("a healthy backup reported as FAILED") that the project fixed in controller v0.179.0.

**It reached no channel, and the reason is worth keeping.** The controller's event trail for the
whole window is:

```
19:15:43Z  whole_guest_backup_failed     (error)  felhom-backup tier — the REAL 403, backoff 15m
19:19:45Z  whole_guest_backup_recovered  (info)   felhom-backup tier — cleared after the ACL grant
```

— and **nothing for the 21:21 PBS failure.** The agent process died with its own in-memory failure
record, and by the time the controller next polled, `NewestArchiveTime` had read ground truth from
PBS and found the successful 21:19:45 archive. The R-84 ground-truth design absorbed it: no breaker
armed on the offsite tier, no operator mail, no spurious staleness. That is the design working, not
a reason the mistake was harmless — on a slower tier, or with the restart a minute earlier, the same
slip could have aborted a multi-hour WAN upload.

**The rule that would have caught it:** re-check in flight immediately before *every* restart, not
once at the start of the procedure — and remember that a manual trigger fires all tiers, so the
offsite tier can still be running long after the local one has finished.

### 8.2 Method and tooling

- **`pgrep -f vzdump` is not a safe in-flight check.** A polling script whose own command line
  contains the string matches itself, and it reported a backup in flight when the PVE task list
  showed none. It caused one false "ABORT: vzdump in flight" and one wrong reading in this run.
  **Use the PVE task list** (`pvesh get /nodes/<node>/tasks`, filter for a missing `endtime`).
- **`sed -i` must not be used on `agent.json`.** The directory is root-owned and the file is
  agent-owned `0600`; `sed -i` renames, which would flip the file to `root:root` and leave the
  non-root agent unable to read its own config. The in-place truncate-write in §3 preserves both.
- A `felhom-pbs` vzdump failed at **17:58**, hours before this change, with
  `Can't connect to 10.77.0.1:8007 (Connection refused)`. **Pre-existing and unrelated** — checked
  rather than assumed: ep0 is up (uptime 1 d 2 h, `proxmox-backup-proxy` active), the WG handshake
  is fresh, and 8007 is open from demo-felhom now.
- The controller's first-backup safety valve behaved exactly as designed on demo-felhom: the agent
  restart made the new tier report `age_state: absent`, and the controller took a backup on its own
  before the manual trigger could (`{"error":"mentés már folyamatban van"}`).
- The archives are outside the customer's reach by construction — the guest is bound only the
  drive's `felhom-data` subpath, while `dump/` sits at the drive root.

---

## 9. State at close

### 9.1 Old archives — left in place, as instructed

Nothing was deleted. They remain the rollback and the only evidence of what the previous
configuration produced.

| Box | Path | Contents | Size |
|---|---|---|---|
| demo-felhom | `/var/lib/vz/dump/` | 3 × 9201 (5.82 / 5.84 / 5.93 GB) + 2 × 9100 (612 / 649 MB) | **18 GB** |
| demo-hp | `/var/lib/vz/dump/` | 3 × 9201 (1.61 / 1.63 / 1.68 GB) + 1 × 9100 (649 MB) | **5.2 GB** |

**These are now stale and will never be refreshed** — the tier that wrote them no longer points
here, and `keep-last=3` prunes only the *new* target. They are also the reason R-109 matters: a
restorer reading the DR recipe sees both `local` and `felhom-backup` carrying `content=backup` and
cannot tell that one holds live archives and the other holds a frozen snapshot of 2026-07-28.
Deleting them is a separate, deliberate decision.

### 9.2 Fleet state — both boxes healthy, no thrash, no spurious staleness

| | demo-felhom | demo-hp |
|---|---|---|
| Agent | `active`, v0.110.0 | `active`, v0.110.0 |
| Primary tier | `felhom-backup`, `due=false`, `age_state=known` | `felhom-backup`, `due=false`, `age_state=known` |
| Offsite tier | `felhom-pbs`, `due=false`, `age_state=known` | `felhom-pbs`, `due=false`, `age_state=known` |
| New target | 5.96 GB used of 916 GB (**2 %**) | 2.26 GB used of 938 GB (**1 %**) |
| Target drive SMART | `PASSED`, 35 °C | `PASSED`, 50 °C, 7 % wear |
| System drive | 24 G / 94 G (27 %) | 14 G / 39 G (38 %) |
| Agent errors, last 15 min | **0** | 2, both accounted for (§4 403, §8.1 spurious) |
| Breaker | clear (`whole_guest_backup_recovered` 19:19:45Z) | clear |

No tier is stale, no tier is thrashing, and both offsite tiers report a known, recent age.

### 9.3 Outstanding proof

1. **Full drive-loss recovery** — pull the system drive, restore the guest from the vzdump on the
   secondary drive, confirm it returns with named volumes, secrets and config. Needs physical
   access; the boxes are remote until ~08-02. This is the proof that flips **matrix row 4** from
   PARTIAL to a real local route. Everything up to it is now proven: the archive exists on separate
   hardware, and a restore from it boots and passes `mount_parity`.
2. **The agent half of §6** — with the drive absent, that the tier fails loudly rather than silently
   retargeting. The **PVE half is live-proven** (§6); the agent half is traced through source only,
   because unmounting a live enrolled drive would break the guest's `felhom-data` bind and the
   customer's app data on a remote box.

### 9.4 Backlog rows filed

| ID | What |
|---|---|
| **E-2** | The machinery around this change — backup-target role on `StoragePath`, wizard assignment, no automatic roles, stickiness, `felhom-host-install.sh` creating the target **and** its ACL, absent-target policy, retention/space accounting on a shared drive, the honest single-drive label, remaining fleet migration. Full scope in §7. |
| **R-109** | **The DR recipe records no backup target.** `BuildDRRecipeHostHalf` lists every storage's name/type/content but never says which one holds the local archives — and after this change the box carries two `content=backup` dir storages, one live and one stale. §5.4. |
| **R-105** | Partially closed and **fully traced** for its third field: `dr_recipe.host_half.drives` is now populated on both demo boxes, and the cause of the fleet-wide `[]` is identified — the enrolled drives were never PVE storages, so `isUserDataDrive` never saw them. The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched by this run. |
