admin
94a236328b
Empirical PBS validation before the slice-6 Phase B spec. Records: PBS install on Debian-13 DooPlex (trixie key ships in proxmox-archive-keyring, no standalone .gpg), datastore + cert fingerprint, the PBS privsep gotcha (grant role on user AND token), the encrypted pbs storage + key location (/etc/pve/priv/storage/<id>.enc), the snapshot volid format + native fields (→ PBSSnapshot shape), restore-from-PBS works unchanged, the verify mechanism (server-side; agent drives it remotely via the PBS API, result read from snapshot verification.state), no operator-token privilege gap, and zero-knowledge confirmed (server can't decrypt without the client key). PBS+datastore+storage left up for Phase B; no secrets committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
164 lines
11 KiB
Markdown
164 lines
11 KiB
Markdown
# Phase 5 spike — PBS mechanism validation (DooPlex server ← N100 client)
|
||
|
||
> **Status: empirical findings from a live spike (2026-06-09).** PBS was never validated in
|
||
> any prior spike (proxmox-platform.md §4.6). This establishes the *real* mechanisms before
|
||
> the slice-6 **Phase B** spec is written. No production data; probe → record → teardown. The
|
||
> PBS server + datastore + the N100's `felhom-pbs` storage are **left in place** for Phase B's
|
||
> live runbook to reuse.
|
||
|
||
## Topology under test
|
||
|
||
- **PBS server: DooPlex** `192.168.0.180` (Debian 13 trixie, separate box, backup-only — runs
|
||
no guests). PBS `proxmox-backup-server 4.2.1-1`.
|
||
- **Client: the N100** `demo-felhom` `192.168.0.162` (PVE 9.2.2, `proxmox-backup-client 4.2.0`).
|
||
It backs up to DooPlex and restores from it.
|
||
|
||
## Pre-flight
|
||
|
||
| | Result |
|
||
|---|---|
|
||
| P1 DooPlex can host PBS | ✅ Debian 13 trixie; `proxmox-backup-server` **not** in stock apt — needed the Proxmox **PBS no-subscription** repo (`deb http://download.proxmox.com/debian/pbs trixie pbs-no-subscription`). **Surprise:** there is **no standalone `proxmox-release-trixie.gpg`** (404; only bookworm/bullseye are published) — the trixie key ships in the **`proxmox-archive-keyring`** package (key `24B30F06…0BFE778E`). I copied that keyring from the N100 (PVE9/trixie already has it). 126 GB free on `/`. |
|
||
| P2 N100 → DooPlex:8007 | ✅ reachable (8007 closed pre-install, open after). |
|
||
| P3 N100 PBS client | ✅ `proxmox-backup-client 4.2.0`, PVE `PBSPlugin.pm` present. |
|
||
|
||
## Stand-up (DooPlex)
|
||
|
||
- **S1** — `proxmox-backup-manager datastore create felhom-spike /var/lib/pbs-spike` → `TASK OK`.
|
||
Services `proxmox-backup-proxy` + `proxmox-backup` active, listening on `:8007`.
|
||
**Server cert fingerprint:** `3b:95:5a:fa:9e:0e:4a:54:f3:64:08:e5:a2:a2:6c:66:e9:86:44:64:40:8e:c2:f7:6e:41:d2:c2:1e:86:48:c4`.
|
||
- **S2** — created PBS user `felhom@pbs` + **API token** `felhom@pbs!n100`, ACL `DatastoreAdmin`
|
||
on `/datastore/felhom-spike`.
|
||
**⚠️ PBS privsep gotcha (mirrors PVE):** an API token's effective rights = token-ACL ∩
|
||
**user**-ACL. Granting only the token wasn't enough — `pvesm add` failed with *"Cannot find
|
||
datastore"* until `DatastoreAdmin` was **also** granted to the `felhom@pbs` user. Phase B's
|
||
enrollment must grant the role on **both** the user and the token.
|
||
|
||
## Adding as an encrypted PVE storage (N100)
|
||
|
||
- **A1** — `pvesm add pbs felhom-pbs --server 192.168.0.180 --datastore felhom-spike
|
||
--fingerprint <fp> --username 'felhom@pbs!n100' --password <token-secret> --encryption-key
|
||
autogen --content backup`. Resulting `/etc/pve/storage.cfg`:
|
||
```
|
||
pbs: felhom-pbs
|
||
datastore felhom-spike
|
||
server 192.168.0.180
|
||
content backup
|
||
encryption-key 01:36:e9:fe:e1:ee:3d:7a:9d:bf:3d:63:d0:68:fd:24:45:b7:5f:bc:b6:82:bc:6d:d2:b4:7a:b0:1a:86:6d:a1
|
||
fingerprint 3b:95:5a:fa:…:48:c4
|
||
username felhom@pbs!n100
|
||
```
|
||
**Where the keys live on the box (the "live key on box"):**
|
||
- **client encryption key** → `/etc/pve/priv/storage/felhom-pbs.enc` (root:www-data **0600**, 255 B).
|
||
The `encryption-key` line in storage.cfg is only the key's **fingerprint** (`01:36:e9:fe…`),
|
||
not the key.
|
||
- **PBS token secret** → `/etc/pve/priv/storage/felhom-pbs.pw` (0600, 37 B).
|
||
- **A2** — the slice-5 agent observe (`--selftest=storage`) sees the target with the
|
||
**fingerprint-pinned durable_id** exactly as designed:
|
||
`durable=192.168.0.180:felhom-spike#3b:95:5a:fa:…:48:c4`, `type=pbs`, `state=attached`.
|
||
No agent change needed for observation.
|
||
|
||
## Probes (B1–B6)
|
||
|
||
### B1 — backup to PBS
|
||
`vzdump 9001 --storage felhom-pbs --mode snapshot` →
|
||
- PBS snapshot id **`ct/9001/2026-06-09T14:18:33Z`** (`<type>/<id>/<RFC3339Z>`); the underlying
|
||
`proxmox-backup-client backup … --repository felhom@pbs!n100@192.168.0.180:felhom-spike`.
|
||
- **Encrypted client-side**: `--crypt-mode=encrypt`, *"Using encryption key from file
|
||
descriptor"*, *"Encryption key fingerprint: 01:36:e9:fe:e1:ee:3d:7a"* (matches the storage
|
||
key). Incremental/deduped (*"reused 41 MiB"*). ~19 s for ~1 GiB.
|
||
- **Surprise vs Phase A:** vzdump **chose `stop` mode** for the (stopped) guest even though
|
||
`snapshot` was requested (`INFO: backup mode: stop`). PVE picks the actual mode; the
|
||
reported `Backup.mode` is what was *requested*. For a running guest on lvm-thin it would
|
||
snapshot. (Still crash-consistent only — no fsfreeze, per slice 6.)
|
||
|
||
### B2 — snapshot inventory → the `PBSSnapshot` wire shape
|
||
- **PVE volid** (`pvesm list felhom-pbs`): **`felhom-pbs:backup/ct/9001/2026-06-09T14:18:33Z`**,
|
||
format `pbs-ct`, type `backup`. This is the exact volid `pct restore` consumes (B3).
|
||
- **PBS native** (`proxmox-backup-client snapshot list --output-format json`) per snapshot:
|
||
`backup-type` (ct|vm), `backup-id`, `backup-time` (epoch int), `size`, `owner`
|
||
(`felhom@pbs!n100`), `protected` (bool), `fingerprint` (the encryption-key fp), and
|
||
`files[]` each with `filename` + `size` + **`crypt-mode`** (`encrypt` for data, `sign-only`
|
||
for `index.json`). **`verification` is ABSENT until a verify runs** (see B4). **Namespace**:
|
||
not shown → the default (root) namespace; a `ns` field appears only for non-root namespaces.
|
||
- → **Proposed `PBSSnapshot`**: `namespace`, `backup_type`, `backup_id`, `backup_time`,
|
||
`size_bytes`, `owner`, `protected`, `encrypted` (derive from `files[].crypt-mode`),
|
||
`verify_state` (ok|failed|none), `verified_at`/`verify_upid`.
|
||
|
||
### B3 — restore from PBS
|
||
`pct restore 990001 'felhom-pbs:backup/ct/9001/2026-06-09T14:18:33Z' --storage local-lvm` →
|
||
restored + booted to `running`. **The existing restore path works UNCHANGED against a
|
||
pbs-sourced volid** — same `volid` + `--storage` shape the agent's `RestoreLXC` already uses
|
||
(`ostemplate=<volid>`, `restore=1`). **No agent restore code change needed for PBS.** PVE pulls
|
||
+ decrypts using the storage's `.enc` key automatically.
|
||
|
||
### B4 — verify mechanism (the big unknown — resolved)
|
||
- **`proxmox-backup-client` has NO `verify` subcommand** — verify is **server-side**.
|
||
- Triggers: server CLI `proxmox-backup-manager verify <store> [--ignore-verified]
|
||
[--outdated-after N]` **on DooPlex**, OR the **PBS API** `POST /api2/json/admin/datastore/
|
||
<ds>/verify` (whole datastore; per-snapshot params available).
|
||
- **The agent on the N100 CAN drive it remotely** via the PBS API + token (no DooPlex shell
|
||
needed). Proven: `curl -X POST …/admin/datastore/felhom-spike/verify` with header
|
||
`Authorization: PBSAPIToken=felhom@pbs!n100:<secret>` returned a task UPID
|
||
`UPID:dooplex:…:verify:felhom\x2dspike:felhom@pbs!n100:`. Needs `Datastore.Verify` (in
|
||
`DatastoreAdmin`).
|
||
- **Result read-back:** after verify, the snapshot's **`verification` field** appears:
|
||
`{"state":"ok","upid":"UPID:dooplex:…"}` (read via `snapshot list`). So the agent triggers
|
||
via API → polls/re-lists → reads `verification.state` (`ok`/`failed`). (Task-status polling
|
||
needs the PBS **node name** — it's `dooplex`, embedded in the UPID; `localhost` returns
|
||
`exitstatus: unknown`.)
|
||
|
||
### B5 — agent-token (`felhom-agent@pve`) privileges — **no gap**
|
||
Driven by the agent (operator token, not root@pam):
|
||
- **Backup to PBS** (`--selftest=backup`): ✅ `felhom-pbs:backup/ct/9001/2026-06-09T14:22:30Z`,
|
||
crash_consistent, success.
|
||
- **Restore from PBS** (`--selftest=restore-test`): ✅ restored into scratch 990000, booted,
|
||
verified `running`, torn down — pass.
|
||
- The **FelhomAgent role's existing `Datastore.{Audit,Allocate,AllocateSpace}` + `VM.Backup`
|
||
suffice** for both backup-to-PBS and restore-from-PBS. **No role widening needed.** (Two auth
|
||
layers: the *PVE* operator token authorizes the vzdump/restore API call; the *PBS* token in
|
||
storage.cfg authenticates PVE→PBS. The spike exercised both.)
|
||
|
||
### B6 — zero-knowledge confirmed
|
||
- All data files are `crypt-mode=encrypt` (B2); `index.json` is `sign-only`.
|
||
- **Without the key**, an authenticated restore **fails to decrypt**:
|
||
`proxmox-backup-client restore … pct.conf.blob -` (no `--keyfile`) →
|
||
`Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a`.
|
||
- **With** `--keyfile /etc/pve/priv/storage/felhom-pbs.enc` → decrypts (returns the guest
|
||
config). The key is the *only* gate.
|
||
- **The PBS server holds no client key** — `find /etc/proxmox-backup /var/lib/pbs-spike` for
|
||
key material returns only the server's own `csrf.key`, never the client encryption key. So
|
||
DooPlex can store + serve chunks but cannot read guest data. Zero-knowledge holds: the live
|
||
key on the N100 is the irreducible residual (the operator/hub can't read the data).
|
||
|
||
## Implications for the Phase B spec (flagged surprises vs the dir-storage assumptions)
|
||
|
||
1. **Enrollment must grant the PBS role on BOTH the user AND the token** (PBS privsep), and add
|
||
the `pbs` storage with `--encryption-key autogen` → the live key lands at
|
||
`/etc/pve/priv/storage/<id>.enc` (the "live PBS key on the box", doc 03 §8). The hub holds
|
||
only the recovery-code-wrapped escrow (out of scope here).
|
||
2. **Backup + restore need NO new code** beyond targeting a `pbs` storage — `Vzdump` and
|
||
`RestoreLXC`/`pct restore` work against pbs volids unchanged. The agent's `LatestBackupVolID`
|
||
(StorageContent filter) already resolved the pbs volid.
|
||
3. **Verify is a NEW capability to build**: a server-side op the agent triggers **remotely via
|
||
the PBS API** (`POST …/datastore/<ds>/verify`) using the storage's token, then reads back
|
||
`verification.state` from the snapshot list. This is the "lighter frequent integrity check"
|
||
(§8) — it does NOT need the encryption key (ciphertext-level), unlike the full self-restore-
|
||
test. Phase B needs a small PBS-API client (token auth, fingerprint pin) for verify +
|
||
snapshot-list-with-verify-state; the existing `proxmox.Client` (PVE API) does not cover it.
|
||
4. **`PBSSnapshot` wire shape** = the B2 fields; `verify_state` is the load-bearing one and is
|
||
`none` until a verify runs.
|
||
5. **vzdump mode** is PVE's choice (stop for stopped guests) — report requested-vs-actual if it
|
||
matters, or read the actual mode from the task log.
|
||
|
||
## Teardown / left-in-place
|
||
|
||
- Throwaway restore guest **990001 destroyed**; agent restore-test scratch self-torn-down;
|
||
`pct list` → **no leftover guests**. Agent config reverted (`backup.local_backup_target` →
|
||
`local`). Token-secret temp files removed from both boxes.
|
||
- **Left in place for Phase B:** the PBS server on DooPlex, the `felhom-spike` datastore (with
|
||
two test snapshots of 9001), the `felhom@pbs!n100` token + ACLs, and the N100's `felhom-pbs`
|
||
encrypted storage (+ its `.enc`/`.pw` under `/etc/pve/priv/storage/`).
|
||
- **No secrets committed** — the encryption key, token secret, and PBS password live only in
|
||
`/etc/pve/priv/storage/` (0600) on the N100; this doc references them by location/fingerprint
|
||
only.
|