# Phase 5 spike — PBS mechanism validation (DooPlex server ← N100 client)

> **Status: empirical findings from a live spike (2026-06-09).** PBS was never validated in
> any prior spike (proxmox-platform.md §4.6). This establishes the *real* mechanisms before
> the slice-6 **Phase B** spec is written. No production data; probe → record → teardown. The
> PBS server + datastore + the N100's `felhom-pbs` storage are **left in place** for Phase B's
> live runbook to reuse.

## Topology under test

- **PBS server: DooPlex** `192.168.0.180` (Debian 13 trixie, separate box, backup-only — runs
  no guests). PBS `proxmox-backup-server 4.2.1-1`.
- **Client: the N100** `demo-felhom` `192.168.0.162` (PVE 9.2.2, `proxmox-backup-client 4.2.0`).
  It backs up to DooPlex and restores from it.

## Pre-flight

| | Result |
|---|---|
| P1 DooPlex can host PBS | ✅ Debian 13 trixie; `proxmox-backup-server` **not** in stock apt — needed the Proxmox **PBS no-subscription** repo (`deb http://download.proxmox.com/debian/pbs trixie pbs-no-subscription`). **Surprise:** there is **no standalone `proxmox-release-trixie.gpg`** (404; only bookworm/bullseye are published) — the trixie key ships in the **`proxmox-archive-keyring`** package (key `24B30F06…0BFE778E`). I copied that keyring from the N100 (PVE9/trixie already has it). 126 GB free on `/`. |
| P2 N100 → DooPlex:8007 | ✅ reachable (8007 closed pre-install, open after). |
| P3 N100 PBS client | ✅ `proxmox-backup-client 4.2.0`, PVE `PBSPlugin.pm` present. |

## Stand-up (DooPlex)

- **S1** — `proxmox-backup-manager datastore create felhom-spike /var/lib/pbs-spike` → `TASK OK`.
  Services `proxmox-backup-proxy` + `proxmox-backup` active, listening on `:8007`.
  **Server cert fingerprint:** `3b:95:5a:fa:9e:0e:4a:54:f3:64:08:e5:a2:a2:6c:66:e9:86:44:64:40:8e:c2:f7:6e:41:d2:c2:1e:86:48:c4`.
- **S2** — created PBS user `felhom@pbs` + **API token** `felhom@pbs!n100`, ACL `DatastoreAdmin`
  on `/datastore/felhom-spike`.
  **⚠️ PBS privsep gotcha (mirrors PVE):** an API token's effective rights = token-ACL ∩
  **user**-ACL. Granting only the token wasn't enough — `pvesm add` failed with *"Cannot find
  datastore"* until `DatastoreAdmin` was **also** granted to the `felhom@pbs` user. Phase B's
  enrollment must grant the role on **both** the user and the token.

## Adding as an encrypted PVE storage (N100)

- **A1** — `pvesm add pbs felhom-pbs --server 192.168.0.180 --datastore felhom-spike
  --fingerprint <fp> --username 'felhom@pbs!n100' --password <token-secret> --encryption-key
  autogen --content backup`. Resulting `/etc/pve/storage.cfg`:
  ```
  pbs: felhom-pbs
      datastore felhom-spike
      server 192.168.0.180
      content backup
      encryption-key 01:36:e9:fe:e1:ee:3d:7a:9d:bf:3d:63:d0:68:fd:24:45:b7:5f:bc:b6:82:bc:6d:d2:b4:7a:b0:1a:86:6d:a1
      fingerprint 3b:95:5a:fa:…:48:c4
      username felhom@pbs!n100
  ```
  **Where the keys live on the box (the "live key on box"):**
  - **client encryption key** → `/etc/pve/priv/storage/felhom-pbs.enc` (root:www-data **0600**, 255 B).
    The `encryption-key` line in storage.cfg is only the key's **fingerprint** (`01:36:e9:fe…`),
    not the key.
  - **PBS token secret** → `/etc/pve/priv/storage/felhom-pbs.pw` (0600, 37 B).
- **A2** — the slice-5 agent observe (`--selftest=storage`) sees the target with the
  **fingerprint-pinned durable_id** exactly as designed:
  `durable=192.168.0.180:felhom-spike#3b:95:5a:fa:…:48:c4`, `type=pbs`, `state=attached`.
  No agent change needed for observation.

## Probes (B1–B6)

### B1 — backup to PBS
`vzdump 9001 --storage felhom-pbs --mode snapshot` →
- PBS snapshot id **`ct/9001/2026-06-09T14:18:33Z`** (`<type>/<id>/<RFC3339Z>`); the underlying
  `proxmox-backup-client backup … --repository felhom@pbs!n100@192.168.0.180:felhom-spike`.
- **Encrypted client-side**: `--crypt-mode=encrypt`, *"Using encryption key from file
  descriptor"*, *"Encryption key fingerprint: 01:36:e9:fe:e1:ee:3d:7a"* (matches the storage
  key). Incremental/deduped (*"reused 41 MiB"*). ~19 s for ~1 GiB.
- **Surprise vs Phase A:** vzdump **chose `stop` mode** for the (stopped) guest even though
  `snapshot` was requested (`INFO: backup mode: stop`). PVE picks the actual mode; the
  reported `Backup.mode` is what was *requested*. For a running guest on lvm-thin it would
  snapshot. (Still crash-consistent only — no fsfreeze, per slice 6.)

### B2 — snapshot inventory → the `PBSSnapshot` wire shape
- **PVE volid** (`pvesm list felhom-pbs`): **`felhom-pbs:backup/ct/9001/2026-06-09T14:18:33Z`**,
  format `pbs-ct`, type `backup`. This is the exact volid `pct restore` consumes (B3).
- **PBS native** (`proxmox-backup-client snapshot list --output-format json`) per snapshot:
  `backup-type` (ct|vm), `backup-id`, `backup-time` (epoch int), `size`, `owner`
  (`felhom@pbs!n100`), `protected` (bool), `fingerprint` (the encryption-key fp), and
  `files[]` each with `filename` + `size` + **`crypt-mode`** (`encrypt` for data, `sign-only`
  for `index.json`). **`verification` is ABSENT until a verify runs** (see B4). **Namespace**:
  not shown → the default (root) namespace; a `ns` field appears only for non-root namespaces.
  - → **Proposed `PBSSnapshot`**: `namespace`, `backup_type`, `backup_id`, `backup_time`,
    `size_bytes`, `owner`, `protected`, `encrypted` (derive from `files[].crypt-mode`),
    `verify_state` (ok|failed|none), `verified_at`/`verify_upid`.

### B3 — restore from PBS
`pct restore 990001 'felhom-pbs:backup/ct/9001/2026-06-09T14:18:33Z' --storage local-lvm` →
restored + booted to `running`. **The existing restore path works UNCHANGED against a
pbs-sourced volid** — same `volid` + `--storage` shape the agent's `RestoreLXC` already uses
(`ostemplate=<volid>`, `restore=1`). **No agent restore code change needed for PBS.** PVE pulls
+ decrypts using the storage's `.enc` key automatically.

### B4 — verify mechanism (the big unknown — resolved)
- **`proxmox-backup-client` has NO `verify` subcommand** — verify is **server-side**.
- Triggers: server CLI `proxmox-backup-manager verify <store> [--ignore-verified]
  [--outdated-after N]` **on DooPlex**, OR the **PBS API** `POST /api2/json/admin/datastore/
  <ds>/verify` (whole datastore; per-snapshot params available).
- **The agent on the N100 CAN drive it remotely** via the PBS API + token (no DooPlex shell
  needed). Proven: `curl -X POST …/admin/datastore/felhom-spike/verify` with header
  `Authorization: PBSAPIToken=felhom@pbs!n100:<secret>` returned a task UPID
  `UPID:dooplex:…:verify:felhom\x2dspike:felhom@pbs!n100:`. Needs `Datastore.Verify` (in
  `DatastoreAdmin`).
- **Result read-back:** after verify, the snapshot's **`verification` field** appears:
  `{"state":"ok","upid":"UPID:dooplex:…"}` (read via `snapshot list`). So the agent triggers
  via API → polls/re-lists → reads `verification.state` (`ok`/`failed`). (Task-status polling
  needs the PBS **node name** — it's `dooplex`, embedded in the UPID; `localhost` returns
  `exitstatus: unknown`.)

### B5 — agent-token (`felhom-agent@pve`) privileges — **no gap**
Driven by the agent (operator token, not root@pam):
- **Backup to PBS** (`--selftest=backup`): ✅ `felhom-pbs:backup/ct/9001/2026-06-09T14:22:30Z`,
  crash_consistent, success.
- **Restore from PBS** (`--selftest=restore-test`): ✅ restored into scratch 990000, booted,
  verified `running`, torn down — pass.
- The **FelhomAgent role's existing `Datastore.{Audit,Allocate,AllocateSpace}` + `VM.Backup`
  suffice** for both backup-to-PBS and restore-from-PBS. **No role widening needed.** (Two auth
  layers: the *PVE* operator token authorizes the vzdump/restore API call; the *PBS* token in
  storage.cfg authenticates PVE→PBS. The spike exercised both.)

### B6 — zero-knowledge confirmed
- All data files are `crypt-mode=encrypt` (B2); `index.json` is `sign-only`.
- **Without the key**, an authenticated restore **fails to decrypt**:
  `proxmox-backup-client restore … pct.conf.blob -` (no `--keyfile`) →
  `Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a`.
- **With** `--keyfile /etc/pve/priv/storage/felhom-pbs.enc` → decrypts (returns the guest
  config). The key is the *only* gate.
- **The PBS server holds no client key** — `find /etc/proxmox-backup /var/lib/pbs-spike` for
  key material returns only the server's own `csrf.key`, never the client encryption key. So
  DooPlex can store + serve chunks but cannot read guest data. Zero-knowledge holds: the live
  key on the N100 is the irreducible residual (the operator/hub can't read the data).

## Implications for the Phase B spec (flagged surprises vs the dir-storage assumptions)

1. **Enrollment must grant the PBS role on BOTH the user AND the token** (PBS privsep), and add
   the `pbs` storage with `--encryption-key autogen` → the live key lands at
   `/etc/pve/priv/storage/<id>.enc` (the "live PBS key on the box", doc 03 §8). The hub holds
   only the recovery-code-wrapped escrow (out of scope here).
2. **Backup + restore need NO new code** beyond targeting a `pbs` storage — `Vzdump` and
   `RestoreLXC`/`pct restore` work against pbs volids unchanged. The agent's `LatestBackupVolID`
   (StorageContent filter) already resolved the pbs volid.
3. **Verify is a NEW capability to build**: a server-side op the agent triggers **remotely via
   the PBS API** (`POST …/datastore/<ds>/verify`) using the storage's token, then reads back
   `verification.state` from the snapshot list. This is the "lighter frequent integrity check"
   (§8) — it does NOT need the encryption key (ciphertext-level), unlike the full self-restore-
   test. Phase B needs a small PBS-API client (token auth, fingerprint pin) for verify +
   snapshot-list-with-verify-state; the existing `proxmox.Client` (PVE API) does not cover it.
4. **`PBSSnapshot` wire shape** = the B2 fields; `verify_state` is the load-bearing one and is
   `none` until a verify runs.
5. **vzdump mode** is PVE's choice (stop for stopped guests) — report requested-vs-actual if it
   matters, or read the actual mode from the task log.

## Teardown / left-in-place

- Throwaway restore guest **990001 destroyed**; agent restore-test scratch self-torn-down;
  `pct list` → **no leftover guests**. Agent config reverted (`backup.local_backup_target` →
  `local`). Token-secret temp files removed from both boxes.
- **Left in place for Phase B:** the PBS server on DooPlex, the `felhom-spike` datastore (with
  two test snapshots of 9001), the `felhom@pbs!n100` token + ACLs, and the N100's `felhom-pbs`
  encrypted storage (+ its `.enc`/`.pw` under `/etc/pve/priv/storage/`).
- **No secrets committed** — the encryption key, token secret, and PBS password live only in
  `/etc/pve/priv/storage/` (0600) on the N100; this doc references them by location/fingerprint
  only.