# Phase 3 — vzdump exclusion (B2) & agent operator role + root boundary (B3): Findings

**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via
`pvesh get /nodes` → `demo-felhom`. **Date:** 2026-06-08. Throwaway resources (VMIDs
9010-9023, role/user `FelhomAgent`/`felhom-agent@pve`); all torn down (only the pre-existing
9000/9001 remain, stopped). Every Proxmox op polled to `task exitstatus` (not the POST
return).

> Validates the two items the design review (`_design-review.md`) flagged as unvalidated:
> **B2** (what vzdump includes/excludes per LXC mount type + how to keep bulk out) and **B3**
> (the least-privilege operator role + the root-vs-API boundary). Data only.

---

## B2 — vzdump inclusion/exclusion matrix

**Setup:** one unprivileged LXC `9010` (`nesting=1,keyctl=1`, overlayfs), Docker 29.5.3
installed, with five sentinel locations:

| # | location | config |
|---|---|---|
| 1 | rootfs file `/SENTINEL_ROOTFS` | rootfs (`local-lvm:8`) |
| 2 | Docker **named** volume `b2vol` → `SENTINEL_DOCKERVOL` | default driver |
| 3 | `mp1` volume mount `/mnt/mp1` `SENTINEL_MP1` | `local-lvm:1,backup=1` |
| 4 | `mp2` volume mount `/mnt/mp2` `SENTINEL_MP2` | `local-lvm:1,backup=0` |
| 5 | `mp3` **bind** mount `/mnt/mp3` `SENTINEL_MP3` | host `/root/b2-bindsrc` |
| 6 | bulk Docker vol `bulkvol` bound onto mp2 → `SENTINEL_BULK` | `--driver local -o type=none -o o=bind -o device=/mnt/mp2` |

**The "trap" confirmed at setup:** the Docker named volume's on-disk path is
`/var/lib/docker/volumes/b2vol/_data` — **inside the LXC rootfs**.

### Result matrix (stop-mode vzdump → `local`, verified 3 ways: vzdump log, archive grep, restore to 9011)

| Sentinel | location | flag | **in archive?** | restored 9011 |
|---|---|---|---|---|
| `SENTINEL_ROOTFS` | rootfs | — | **INCLUDED** | present |
| `SENTINEL_DOCKERVOL` | Docker named vol (in rootfs) | — | **INCLUDED** ⚠️ the trap | present |
| `SENTINEL_MP1` | volume mp | `backup=1` | **INCLUDED** | present |
| `SENTINEL_MP2` | volume mp | `backup=0` | **EXCLUDED** | absent (vol recreated empty) |
| `SENTINEL_MP3` | bind mount | n/a | **EXCLUDED** | reappears via re-bind only¹ |
| `SENTINEL_BULK` | Docker vol on mp2 | `backup=0` | **EXCLUDED** | absent |

¹ The bind-mount **data is not in the archive** (archive grep shows no mp3 path). It
reappears in the restored 9011 only because `pct restore` preserves the bind config
`mp3: /root/b2-bindsrc` and re-attaches the **same host dir**. On a *different* host (true DR)
the bind data would be gone unless backed up separately — important for DR planning.

**vzdump log (verbatim) — the authoritative per-mount decision:**
```
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp1 ('/mnt/mp1') in backup
INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled)
INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume)
```
**Archive contents (verbatim) — `tar --zstd -tf … | grep SENTINEL`:**
```
./var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL
./SENTINEL_ROOTFS
./mnt/mp1/SENTINEL_MP1
```
**Restore verification (verbatim) — sentinels in restored 9011:**
```
PRESENT : /SENTINEL_ROOTFS
PRESENT : /var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL
PRESENT : /mnt/mp1/SENTINEL_MP1
ABSENT  : /mnt/mp2/SENTINEL_MP2
ABSENT  : /mnt/mp2/SENTINEL_BULK
PRESENT : /mnt/mp3/SENTINEL_MP3   # via re-bind to same host dir, NOT from archive
```

### Proven bulk-exclusion recipe
A "bulk" Docker volume is kept out of the guest vzdump by binding it onto a **volume
mountpoint with `backup=0`**:
1. Attach a Proxmox volume mountpoint with the flag:
   `pct set <id> -mpN <storage>:<size>,mp=/mnt/bulk,backup=0`
2. Realize the Docker volume on that path:
   `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol`
   (or a compose bind to `/mnt/bulk`).
3. Data written through `bulkvol` lands on the `backup=0` mountpoint → **excluded** from
   vzdump, while rootfs/hot sentinels are **included**. Verified: `SENTINEL_BULK` absent from
   archive and restore; `SENTINEL_ROOTFS` present.

### The trap, stated for the placement component
`backup=<boolean>` is **only honoured for volume mount points** (confirmed: pct manpage +
vzdump log "excluding volume mount point … (disabled)"). A Docker **named volume uses the
default driver and lands in the rootfs**, which is **always backed up** — so a "bulk" volume
left as an ordinary named volume is **silently swept into the whole-guest image**. The
per-volume placement component **must** realize every `bulk` volume as a dedicated `backup=0`
mountpoint (or external bind mount), never a default named volume.

---

## B3 — agent operator role + root-vs-API boundary

**Caveat applied (Phase 1):** privsep token needs the role on **both** user and token. Setup:
user `felhom-agent@pve` + privsep token `agent`, role `FelhomAgent`, dual-granted at `/`.
All ops driven **as the token** via the REST API; task `exitstatus` polled.

> ⚠️ **Terminology:** the Phase-1 `FelhomSelfBackup` role is the discarded **guest-side
> self-backup** role (scoped to one guest, *denied* create/allocate). `FelhomAgent` here is
> its **operator-tier replacement** — a different, broader role. Do not conflate.

### Op matrix (as the scoped token)

| # | Operation | API call | Result |
|---|---|---|---|
| read | host status | `GET /nodes/$N/status` | **200** (needs `Sys.Audit`) |
| read | storage list | `GET /storage` | **200** (`Datastore.Audit`) |
| 1 | **create LXC, `nesting=1,keyctl=1`** | `POST /nodes/$N/lxc` | **403** — `changing feature flags (except nesting) is only allowed for root@pam` |
| 1′ | create LXC, **nesting-only** | `POST /nodes/$N/lxc` | **200 / OK** |
| 2 | set config (mem/cpu/options + mountpoint w/ `backup` flag) | `PUT /nodes/$N/lxc/<id>/config` | **200** |
| 3 | allocate volume | `POST /nodes/$N/storage/local-lvm/content` | **200** (`Datastore.AllocateSpace`) |
| 4 | start | `POST …/status/start` | **OK** (`VM.PowerMgmt`) |
| 5 | stop | `POST …/status/stop` | **OK** |
| 6a | snapshot | `POST …/snapshot` | **OK** (`VM.Snapshot`) |
| 6b | rollback | `POST …/snapshot/s1/rollback` | **OK** (`VM.Snapshot.Rollback`) |
| 7 | stop-mode backup | `POST /nodes/$N/vzdump mode=stop` | **OK** (`VM.Backup`) |
| 8 | restore → fresh vmid | `POST /nodes/$N/lxc restore=1` | **OK** — and **restored CT kept `features: nesting=1,keyctl=1`** |
| 9 | destroy CT | `DELETE /nodes/$N/lxc/<id>?purge=1` | **OK** (`VM.Allocate`) |
| 9b | add storage definition (dir) | `POST /storage` | **200** (`Datastore.Allocate`, **no root**) |

**The two headline results:**
1. **`keyctl=1` on create is `root@pam`-only.** Verbatim:
   `Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)`.
   Confirmed this is **not** token-fixable: a **non-privsep `root@pam` token** got the **same
   403**. Only an actual `root@pam` session (OS root / `pct create` as root) can set it.
   `nesting` alone is allowed for a scoped token.
2. **Restore preserves `keyctl`.** A token-authorized `vzrestore` of a keyctl archive produced
   `9021` with `features: nesting=1,keyctl=1, unprivileged: 1`. So the **DR/restore path is
   fully token-covered**; only *fresh provisioning* needs root for the keyctl flag.

### Paring (each drop shown to still pass, or proven needed)

| Privilege | Verdict | Evidence |
|---|---|---|
| `Datastore.AllocateTemplate` | **DROP** (unnecessary) | create-from-template succeeded without it (200/OK) |
| `Sys.Audit` | **KEEP** | `GET /nodes/$N/status` → **403** without it (host metrics, `03` §5) |
| `VM.Config.Network` | **KEEP** | create with `net0` → **403 (/vms/…, VM.Config.Network)** without it |
| `VM.Config.Options` | **KEEP** | config `onboot=1` → **403 (/vms/…, VM.Config.Options)** without it |
| `SDN.Use` | **KEEP (added vs review sketch)** | create → **403 (/sdn/zones/localnetwork/vmbr0, SDN.Use)** without it |

> Corrections to the review's candidate sketch: `VM.Config.CPUMemory` is **not a real
> privilege** — split into `VM.Config.CPU` + `VM.Config.Memory`. `SDN.Use` was **missing** and
> is **required** (PVE 9 gates bridge use behind it). `Datastore.AllocateTemplate` is **not
> needed**.

### Final minimal `FelhomAgent` role (proven sufficient for ops 1′–9b)
```
VM.Allocate  VM.Audit  VM.Config.Disk  VM.Config.CPU  VM.Config.Memory
VM.Config.Network  VM.Config.Options  VM.PowerMgmt  VM.Snapshot  VM.Snapshot.Rollback
VM.Backup  Datastore.Allocate  Datastore.AllocateSpace  Datastore.Audit  Sys.Audit  SDN.Use
```
(16 privileges. `Datastore.Allocate` is for the storage-definition add; drop it if the agent
never creates Proxmox storage entries via the API. `VM.PowerMgmt` is for start/stop lifecycle
— not for the backup itself, consistent with `proxmox-platform.md` §3.4.)

### Root-vs-API boundary table (answers `03` §3)

| Agent host operation | Coverage | Notes |
|---|---|---|
| Create unprivileged LXC, **nesting-only** | **API token** | `VM.Allocate`+`VM.Config.*`+`Datastore.AllocateSpace`+`SDN.Use` |
| **Create with `keyctl=1` (Docker needs it — Phase 0)** | **OS root `root@pam`** (`pct create` as root / sudoers) | no API token works, incl. a root@pam token |
| Set config (mem/cpu/net/options/mountpoint + `backup` flag) | API token | |
| Allocate guest volume | API token | `Datastore.AllocateSpace` |
| Start / stop / snapshot / rollback | API token | `VM.PowerMgmt` / `VM.Snapshot(.Rollback)` |
| vzdump backup (stop/snapshot mode) | API token | `VM.Backup` |
| **Restore from vzdump (preserves keyctl)** | **API token** | DR path needs no root |
| Destroy guest (scratch + compensating rollback, B1) | API token | `VM.Allocate` |
| Add Proxmox **storage definition** (dir/nfs/cifs/pbs) | API token | `Datastore.Allocate`; the *definition* only |
| Host status / metrics report | API token | `Sys.Audit` |
| **USB physical mount-by-UUID / systemd mount unit / fstab** | **OS root / narrow sudoers** | not a Proxmox API op (host-level mount; not tested here) |
| **SMART / hardware sensors** | OS root | not API-exposed |

**Boundary summary:** nearly the entire guest lifecycle — including **restore** — is covered
by the scoped token. The genuine OS-root residual is narrow: **(1) fresh creation of a
Docker-capable LXC (the `keyctl` flag), (2) physical USB mount-by-UUID / systemd mount units /
fstab, (3) hardware/SMART.** This supports `03` §3's "non-root service + scoped token + narrow
sudoers" model — with the **specific** sudoers/root entries being: `pct create` (or just the
keyctl-setting step) and the host mount operations.

---

## Raw command log (appendix)

### B2
```
pct create 9010 ... --features nesting=1,keyctl=1 --unprivileged 1   # rootfs local-lvm:8
pct set 9010 -mp1 local-lvm:1,mp=/mnt/mp1,backup=1
pct set 9010 -mp2 local-lvm:1,mp=/mnt/mp2,backup=0
pct set 9010 -mp3 /root/b2-bindsrc,mp=/mnt/mp3
# docker named vol: docker volume inspect b2vol -> /var/lib/docker/volumes/b2vol/_data
# bulk: docker volume create --driver local -o type=none -o o=bind -o device=/mnt/mp2 bulkvol
vzdump 9010 --mode stop --storage local --compress zstd
#   INFO: including mount point rootfs ('/') in backup
#   INFO: including mount point mp1 ('/mnt/mp1') in backup
#   INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled)
#   INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume)
tar --zstd -tf <archive> | grep SENTINEL   # -> rootfs, dockervol, mp1 only
pct restore 9011 <archive> --storage local-lvm   # -> mp2/bulk absent, mp3 via re-bind
```

### B3
```
pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Sys.Audit"   # candidate (pre-SDN)
pveum user add felhom-agent@pve ; pveum user token add felhom-agent@pve agent --privsep 1
pveum acl modify / -user  'felhom-agent@pve'        -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent'  -role FelhomAgent

# token create with keyctl:
POST /nodes/demo-felhom/lxc ... features=nesting=1,keyctl=1
  -> 403 "changing feature flags (except nesting) is only allowed for root@pam"
# + SDN.Use missing initially:
  -> 403 "Permission check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)"
# root@pam non-privsep token, keyctl create:
  -> 403 (same "only allowed for root@pam")   # tokens never qualify

# token nesting-only create / config(PUT) / start / stop / snapshot / rollback /
# vzdump(stop) / restore->9021 (kept keyctl) / destroy / POST /storage  -> all 200/OK

# paring:
GET /nodes/$N/status  without Sys.Audit            -> 403   (KEEP)
create net0           without VM.Config.Network     -> 403   (KEEP)
config onboot=1       without VM.Config.Options      -> 403   (KEEP)
create from template  without Datastore.AllocateTemplate -> OK (DROP)
```

### Teardown
```
pct destroy 9010 9011 9021 --purge   # 9020/9022/9023 already destroyed during tests
pveum user token remove felhom-agent@pve agent ; pveum user delete felhom-agent@pve
pveum role delete FelhomAgent        # ACLs at / auto-invalidated
rm -f /var/lib/vz/dump/vzdump-lxc-9010-* /var/lib/vz/dump/vzdump-lxc-9020-*
# verified: only 9000/9001 remain (stopped-but-present); no felhom-agent user/role; dump dir empty
```