307 lines
17 KiB
Markdown
307 lines
17 KiB
Markdown
# Phase 1 + 2 — Privilege Model & Backup/Restore Round-Trip: Findings
|
||
|
||
**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via
|
||
`pvesh get /nodes` → `demo-felhom`. Storage: `local` (dir, content
|
||
`iso,vztmpl,backup,import`), `local-lvm` (LVM-thin, `rootdir,images`).
|
||
**Subject:** LXC `9001` (`spike-lxc`, unprivileged, `nesting=1,keyctl=1`, Docker +
|
||
postgres/redis/nginx stack). **Date:** 2026-06-07.
|
||
|
||
> Data and observations only — **no recommendation or verdict**.
|
||
|
||
## Hypotheses — verdicts at a glance
|
||
| | Hypothesis | Result |
|
||
|---|---|---|
|
||
| **H1** | Backup scopes to one VMID; restore/create needs node/pool allocate → denied to narrow token | **CONFIRMED** (create CT = 403) |
|
||
| **H2** | An LXC vzdump captures the Docker volumes (they live in the container rootfs) | **CONFIRMED** (sentinel survived both restores) |
|
||
| **H3** | Crash-consistent (running) *and* quiesced (stopped) backups both restore cleanly | **CONFIRMED** (A via WAL recovery, B clean start) |
|
||
| **H4** | Running unprivileged LXC snapshots on LVM-thin; restored CT keeps unprivileged+nesting/keyctl | **CONFIRMED** (live snapshot OK; config survived) |
|
||
|
||
---
|
||
|
||
## 1. Phase 1 — Privilege model
|
||
|
||
### 1.1 Setup (operator side, root)
|
||
```
|
||
pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit"
|
||
pveum user add felhom-ctl@pve --comment "spike in-guest controller"
|
||
pveum user token add felhom-ctl@pve ctl --privsep 1 # secret: b6547d9d-... (ephemeral, spike-only)
|
||
pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
|
||
pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
|
||
```
|
||
Privilege names were verified against `PVEVMAdmin` / `PVEDatastoreUser` via
|
||
`pveum role list` first. **Note:** the reference doc's introspection command
|
||
`pveum role info <role>` **does not exist in PVE 9** — only `pveum role list` works.
|
||
|
||
### 1.2 ⚠️ Privsep gotcha — the doc's runbook is incomplete
|
||
With `--privsep 1`, a token's effective rights are the **intersection of the backing
|
||
user's permissions AND the token's own ACLs**. The reference doc (§3) grants ACLs to the
|
||
**token only**. With the user `felhom-ctl@pve` holding **no** permissions, the
|
||
intersection was **empty** — the first self-audit call returned:
|
||
```
|
||
HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"}
|
||
```
|
||
**Fix applied:** also grant the user the role on the same paths
|
||
(`pveum acl modify /vms/9001 -user felhom-ctl@pve -role FelhomSelfBackup`, same for
|
||
`/storage/local`). After that the self-calls succeeded. **A privsep token needs the
|
||
permission present on *both* the user and the token** (the token ACL is what keeps the
|
||
token ≤ user / narrowly scoped). This must be reflected in the controller provisioning.
|
||
|
||
### 1.3 Test matrix (every call run from **inside** the unprivileged LXC, `pct exec 9001`)
|
||
`H=192.168.0.162 N=demo-felhom AUTH="PVEAPIToken=felhom-ctl@pve!ctl=<secret>"`
|
||
|
||
| # | Call | Expected | **Actual** | Notes |
|
||
|---|---|---|---|---|
|
||
| 1 | `GET /version` | 200 | **200** | reachable + auth from inside LXC (no privilege needed) |
|
||
| 2 | `GET /nodes/$N/lxc/9001/status/current` | 200 | **200**¹ | self audit (after privsep fix) |
|
||
| 3 | `POST /nodes/$N/lxc/9001/snapshot snapname=spk1` | 200/UPID→OK | **200, task exitstatus OK** | **running-LXC self-snapshot (H4)** |
|
||
| 4 | `POST /nodes/$N/vzdump vmid=9001 storage=local mode=snapshot` | 200/UPID→OK | **200, task exitstatus OK** | self backup, archive produced |
|
||
| 5 | `GET /nodes/$N/qemu/9000/status/current` | 403 | **403** | `Permission check failed (/vms/9000, VM.Audit)` |
|
||
| 6 | `POST /nodes/$N/vzdump vmid=9000 storage=local` | 403 | **200 POST → task exitstatus 403**² | see note |
|
||
| 7 | `POST /nodes/$N/lxc` (create CT) | 403 | **403** | `Permission check failed` — **proves create/allocate is operator-tier (H1)** |
|
||
|
||
¹ before the privsep fix this was 403; see §1.2.
|
||
² **Important nuance:** the `vzdump` endpoint accepts the POST and returns a UPID even for
|
||
an unauthorized vmid; the authorization failure surfaces at **task execution**, not at the
|
||
HTTP layer. Polled from root:
|
||
`exitstatus: "403 Permission check failed (/vms/9000, VM.Backup)"`, and **no 9000 archive
|
||
was created**. The boundary holds — but a controller must **poll the task exitstatus**, not
|
||
trust the POST's 200, to know a cross-guest backup was actually refused.
|
||
|
||
**Pass criteria met:** self-ops (1–4) succeed; cross-guest read (5), cross-guest backup
|
||
(6, at task level), and create/allocate (7) are denied. The controller-as-guest boundary
|
||
and the two-tier split are validated.
|
||
|
||
### 1.4 Final minimal role — `VM.PowerMgmt` **not** required
|
||
The doc's open question ("does Tier A need `VM.PowerMgmt` for stop-mode backups? Likely
|
||
yes"). **Tested and refuted:** a **stop-mode** self-vzdump submitted by the token
|
||
(`vmid=9001 mode=stop`) completed with **`exitstatus: OK`** using the role *without*
|
||
`VM.PowerMgmt`. `vzdump` performs the guest shutdown/restart internally under
|
||
`VM.Backup`; no separate power privilege is needed.
|
||
|
||
> **Final minimal role (`FelhomSelfBackup`) — satisfies self-audit, self-snapshot, and
|
||
> both `snapshot`- and `stop`-mode self-backup:**
|
||
> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit`
|
||
> (`VM.PowerMgmt` deliberately omitted — confirmed unnecessary.)
|
||
|
||
### 1.5 TLS observation
|
||
From inside the LXC, `curl` **without** `-k`:
|
||
```
|
||
curl: (60) SSL certificate problem: unable to get local issuer certificate
|
||
```
|
||
The host serves the default self-signed PVE cert; all tests used `-k`. Production trust
|
||
(pin the PVE CA / issue a proper cert) is a separate design decision, flagged here.
|
||
|
||
### 1.6 Running-LXC snapshot (H4)
|
||
Call #3 snapshotted the **running** unprivileged LXC on LVM-thin (`exitstatus OK`).
|
||
`pct listsnapshot 9001` shows `spk1` with `pct status 9001 = running`. **No stop
|
||
required** — the snapshot-before-update rollback flow is viable on a live container.
|
||
|
||
---
|
||
|
||
## 2. Phase 2 — Backup → real restore round-trip
|
||
|
||
Sentinel written pre-flight into the `pgdata` volume:
|
||
`restore_check(42,'phase2-sentinel')` → clean read `42|phase2-sentinel`.
|
||
|
||
### 2.1 Backups (operator/root side)
|
||
| Variant | Mode | Stack state | Task time | Wall | Archive | Size (zstd) |
|
||
|---|---|---|---|---|---|---|
|
||
| **A — crash-consistent** | `snapshot` | **running** | 00:00:24 | 25 s | `vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst` | **934 MB** (979,718,569 B) |
|
||
| **B — quiesced** | `snapshot` | **stopped** (`docker compose stop`) | 00:00:21 | 22 s | `vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst` | **934 MB** (979,671,582 B) |
|
||
|
||
Both from a 2.5 GiB source; zstd → ~934 MB (~2.7:1). The stack was restarted after
|
||
Variant B. **LXC snapshot-mode vzdump does *not* fsfreeze** (no guest agent in an LXC —
|
||
consistent with the Phase 0 finding) → Variant A is genuinely crash-consistent.
|
||
|
||
### 2.2 Restore → fresh VMID → boot → verify
|
||
| Check | 9002 (Variant A) | 9003 (Variant B) |
|
||
|---|---|---|
|
||
| Restore time (`pct restore … --storage local-lvm`) | **12 s** | **11 s** |
|
||
| `unprivileged: 1` survived | **yes** | **yes** |
|
||
| `features: nesting=1,keyctl=1` survived | **yes** | **yes** |
|
||
| Containers after boot | `exited` (no restart policy) → `docker compose up -d` | same |
|
||
| 3 containers healthy | **yes** | **yes** |
|
||
| `curl localhost:8080` | **HTTP 200** | **HTTP 200** |
|
||
| **Sentinel `(42,'phase2-sentinel')`** | **PRESENT** | **PRESENT** |
|
||
| Postgres first-start | **WAL crash recovery** (see below) | **clean start, no recovery** |
|
||
|
||
> Restored CTs inherit 9001's fixed `hwaddr`. To avoid a MAC clash with the still-running
|
||
> 9001 on `vmbr0`, `net0` was reset to auto-generate a fresh MAC before boot. All
|
||
> verification (stack health, `curl localhost`, sentinel) is guest-internal and needs no
|
||
> external network — and the Docker images are inside the restored rootfs, so no pulls.
|
||
|
||
**Variant A — Postgres automatic WAL recovery on 9002 (verbatim, post-restore boot):**
|
||
```
|
||
LOG: database system was interrupted; last known up at 2026-06-07 18:13:21 UTC
|
||
LOG: database system was not properly shut down; automatic recovery in progress
|
||
LOG: redo starts at 0/CB12838
|
||
LOG: invalid record length at 0/CB12870: expected at least 24, got 0 # normal end-of-WAL
|
||
LOG: redo done at 0/CB12838 ...
|
||
LOG: checkpoint starting: end-of-recovery immediate wait
|
||
LOG: database system is ready to accept connections
|
||
```
|
||
**Variant B — clean start on 9003 (verbatim, post-restore boot):**
|
||
```
|
||
LOG: database system was shut down at 2026-06-07 18:14:39 UTC
|
||
LOG: database system is ready to accept connections
|
||
```
|
||
|
||
**H2 confirmed:** one LXC vzdump captured the whole customer including the Docker named
|
||
volume — the sentinel data restored in both guests. **H3 confirmed:** both variants
|
||
restored to a bootable guest with intact data; the crash-consistent one recovered via WAL
|
||
with no manual intervention, the quiesced one started clean. **H4 confirmed:** restored
|
||
config preserved `unprivileged` + `nesting/keyctl`, so Docker ran in the restored CT.
|
||
|
||
---
|
||
|
||
## 3. Observations & confounds
|
||
1. **Privsep token needs perms on user *and* token** (§1.2) — the single most important
|
||
correction to the reference runbook; without it every scoped call 403s.
|
||
2. **vzdump authorization is task-level, not POST-level** (§1.3 note ²) — a 200 + UPID
|
||
does **not** mean authorized. The controller must poll `exitstatus`. This is also the
|
||
general async-task lesson: every backup/snapshot/restore returns a UPID and the real
|
||
result is in the task status.
|
||
3. **`pveum role info` is gone in PVE 9** — use `pveum role list`. Minor doc drift.
|
||
4. **`VM.PowerMgmt` not needed for stop-mode backup** (§1.4) — narrower role than the doc
|
||
assumed.
|
||
5. **No fsfreeze for LXC** — Variant A relied on Postgres's own WAL crash recovery, which
|
||
worked here for an idle-at-backup DB. Under heavy write load, app-consistency for LXC
|
||
still rests on the controller quiescing first (or stop-mode), exactly as the reference
|
||
warned. This single test is not a durability guarantee under load.
|
||
6. **Restore MAC collision** (§2.2) — `pct restore` preserves the source `hwaddr`;
|
||
restoring while the original runs needs a MAC reset (or the original stopped). The
|
||
controller's restore flow must handle identity (MAC/hostname/IP) to avoid clashes.
|
||
7. **No restart policy on the compose services** — restored containers came up `exited`;
|
||
`docker compose up -d` (or a restart policy / systemd unit) is required for the stack
|
||
to return automatically after a restore or guest reboot.
|
||
8. **Restore is fast, backup dominated by I/O** — restores were 11–12 s (extract at
|
||
~524 MiB/s); backups ~22–25 s (read 2.5 GiB at ~108–119 MiB/s + zstd). Single runs,
|
||
idle host, ~150 MB DB; not a throughput benchmark.
|
||
9. **Sequencing artifact:** a Phase-1 stop-mode self-backup ran before Phase 2 and
|
||
stopped/started 9001; the stack was brought back up and the sentinel re-verified
|
||
before the Variant A/B backups, so it does not affect the round-trip results.
|
||
|
||
---
|
||
|
||
## 4. Raw command log (appendix)
|
||
|
||
### 4.1 Pre-flight
|
||
```
|
||
$ pvesh get /nodes -> node: demo-felhom
|
||
$ cat /etc/pve/storage.cfg
|
||
dir: local ... content iso,vztmpl,backup,import # 'backup' present
|
||
lvmthin: local-lvm ... content rootdir,images # no backup (expected)
|
||
$ pct start 9001 ; docker compose up -d -> 3 containers Started
|
||
$ curl localhost:8080 -> HTTP 200
|
||
# sentinel:
|
||
CREATE TABLE ; INSERT 0 1 ; SELECT count -> 1 ; SELECT * -> 42 | phase2-sentinel
|
||
```
|
||
|
||
### 4.2 Phase 1 — role/user/token/ACL
|
||
```
|
||
$ pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" -> role-ok
|
||
$ pveum user add felhom-ctl@pve --comment "spike in-guest controller" -> user-ok
|
||
$ pveum user token add felhom-ctl@pve ctl --privsep 1
|
||
{"full-tokenid":"felhom-ctl@pve!ctl","info":{"privsep":"1"},"value":"b6547d9d-08ec-4f22-beb8-a551dc2cd69d"}
|
||
$ pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok
|
||
$ pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok
|
||
$ pveum role list | grep FelhomSelfBackup
|
||
FelhomSelfBackup | Datastore.AllocateSpace,Datastore.Audit,VM.Audit,VM.Backup,VM.Snapshot
|
||
$ pveum role info FelhomSelfBackup -> ERROR: unknown command 'pveum role info' # PVE9 has no 'role info'
|
||
```
|
||
|
||
### 4.3 Phase 1 — matrix (from inside LXC)
|
||
```
|
||
# TLS without -k:
|
||
curl: (60) SSL certificate problem: unable to get local issuer certificate
|
||
|
||
# BEFORE privsep fix:
|
||
#2 GET self status -> HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"}
|
||
|
||
# privsep fix:
|
||
$ pveum acl modify /vms/9001 -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok
|
||
$ pveum acl modify /storage/local -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok
|
||
|
||
# AFTER fix:
|
||
#1 GET /version -> HTTP 200
|
||
#2 GET /nodes/.../lxc/9001/status/current -> HTTP 200 {"data":{...,"status":"running",...}}
|
||
#5 GET /nodes/.../qemu/9000/status/current -> HTTP 403 (/vms/9000, VM.Audit)
|
||
#6 POST vzdump vmid=9000 -> HTTP 200 {"data":"UPID:...vzdump:9000:felhom-ctl@pve!ctl:"}
|
||
root poll: exitstatus="403 Permission check failed (/vms/9000, VM.Backup)"
|
||
task log: TASK ERROR: 403 Permission check failed (/vms/9000, VM.Backup)
|
||
/var/lib/vz/dump: no 9000 archive created
|
||
#7 POST /nodes/.../lxc (create CT vmid=9009) -> HTTP 403 {"message":"Permission check failed\n"}
|
||
|
||
#3 POST lxc/9001/snapshot snapname=spk1 -> HTTP 200 UPID:...vzsnapshot:9001...
|
||
root: exitstatus "OK" ; pct listsnapshot 9001 -> spk1 ; pct status 9001 -> running
|
||
#4 POST vzdump vmid=9001 storage=local mode=snapshot -> HTTP 200 UPID:...vzdump:9001...
|
||
root: exitstatus "OK"
|
||
token can read own task status: HTTP 200 {"...exitstatus":"OK"} # earlier poll TIMEOUTs were a shell-quoting bug in the helper, not a perms issue
|
||
|
||
# stop-mode self-backup (VM.PowerMgmt test):
|
||
$ token POST vzdump vmid=9001 storage=local mode=stop -> HTTP 200 UPID:...vzdump:9001...
|
||
root poll: exitstatus "OK" # SUCCEEDED without VM.PowerMgmt in the role
|
||
```
|
||
|
||
### 4.4 Phase 2 — backups
|
||
```
|
||
# Variant A (running):
|
||
$ vzdump 9001 --mode snapshot --storage local --compress zstd
|
||
INFO: Total bytes written: 2585589760 (2.5GiB, 108MiB/s)
|
||
INFO: archive file size: 934MB
|
||
INFO: Finished Backup of VM 9001 (00:00:24) ; WALL_SECONDS=25
|
||
-> vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst (979718569 B)
|
||
|
||
# Variant B (stopped):
|
||
$ docker compose stop (cache,db,web Stopped)
|
||
$ vzdump 9001 --mode snapshot --storage local --compress zstd
|
||
INFO: Total bytes written: 2585825280 (2.5GiB, 119MiB/s)
|
||
INFO: Finished Backup of VM 9001 (00:00:21) ; WALL_SECONDS=22
|
||
-> vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst (979671582 B)
|
||
$ docker compose start (db,cache,web Started)
|
||
```
|
||
|
||
### 4.5 Phase 2 — restores + verification
|
||
```
|
||
# A -> 9002:
|
||
$ pct restore 9002 .../20_13_43.tar.zst --storage local-lvm
|
||
Total bytes read: 2585589760 (2.5GiB, 524MiB/s) ; RESTORE_A_SECONDS=12
|
||
$ pct config 9002 -> features: nesting=1,keyctl=1 ; unprivileged: 1
|
||
$ pct set 9002 -net0 name=eth0,bridge=vmbr0,ip=dhcp # fresh MAC BC:24:11:E3:F4:64
|
||
$ pct start 9002 ; docker compose up -d -> 3 running ; curl -> HTTP 200
|
||
$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel
|
||
db log: "was interrupted ... not properly shut down; automatic recovery in progress
|
||
redo starts/redo done ... database system is ready to accept connections"
|
||
|
||
# B -> 9003:
|
||
$ pct restore 9003 .../20_14_40.tar.zst --storage local-lvm
|
||
Total bytes read: 2585825280 (2.5GiB, 524MiB/s) ; RESTORE_B_SECONDS=11
|
||
$ pct config 9003 -> features: nesting=1,keyctl=1 ; unprivileged: 1
|
||
$ pct set 9003 -net0 ... (fresh MAC) ; pct start 9003 ; docker compose up -d -> 3 running ; curl 200
|
||
$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel
|
||
db log: "database system was shut down at ... ; database system is ready to accept connections" # clean
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Teardown (executed — see §6 for what was left)
|
||
Restore targets destroyed; Phase 1 objects and spike artifacts removed; `9000`/`9001`
|
||
left **stopped-but-present**.
|
||
|
||
```bash
|
||
pct destroy 9002 --purge ; pct destroy 9003 --purge
|
||
pveum acl delete /vms/9001 -user 'felhom-ctl@pve' ; pveum acl delete /vms/9001 -token 'felhom-ctl@pve!ctl'
|
||
pveum acl delete /storage/local -user 'felhom-ctl@pve' ; pveum acl delete /storage/local -token 'felhom-ctl@pve!ctl'
|
||
pveum user token remove felhom-ctl@pve ctl ; pveum user delete felhom-ctl@pve ; pveum role delete FelhomSelfBackup
|
||
pct delsnapshot 9001 spk1
|
||
rm -f /var/lib/vz/dump/vzdump-lxc-9001-*.tar.zst /var/lib/vz/dump/vzdump-lxc-9001-*.log
|
||
pct stop 9001 # back to stopped-but-present
|
||
```
|
||
|
||
## 6. To destroy 9000/9001 later (NOT run — left stopped-but-present)
|
||
```bash
|
||
qm destroy 9000 --purge # VM (Phase 0 subject)
|
||
pct destroy 9001 --purge # LXC (Phase 0/1/2 subject)
|
||
# Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
|
||
```
|