diff --git a/documentation/architecture/01-topology-and-trust.md b/documentation/architecture/01-topology-and-trust.md new file mode 100644 index 0000000..6dc8099 --- /dev/null +++ b/documentation/architecture/01-topology-and-trust.md @@ -0,0 +1,224 @@ +# Felhom Controller Architecture — Part 1: Topology & Trust + +**Status:** draft (decisions from the topology/trust design sessions). +**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document +records *Felhom's decisions*, not Proxmox behaviour. + +--- + +## 1. Model at a glance + +Three components. **Control is always box-initiated** — the hub never connects *into* a +customer box. + +``` + operator side customer box (per Proxmox host) + ┌───────────────────┐ ┌───────────────────────────────────────────┐ + │ HUB │ │ Proxmox host │ + │ (dooplex.hu, k3s) │ │ ┌──────────────┐ │ + │ - report sink │◀──poll──┤ │ HOST AGENT │ operator-tier │ + │ - signed jobs │ signed │ │ (Proxmox │ • all Proxmox ops │ + │ - dashboard │ jobs │ │ token) │ • provision / restore │ + │ - customer record│ │ └──────┬───────┘ • storage mgmt │ + │ - PBS namespace │ │ │ local constrained API │ + └─────────▲─────────┘ │ ┌──────▼───────────────────────────────┐ │ + │ │ │ customer LXC (one per customer) │ │ + │ direct, app- │ │ ┌──────────────┐ Docker: │ │ + └───────────────────┼───┤ │ IN-GUEST │ [app] [app] ... │ │ + domain reports │ │ │ CONTROLLER │ (Docker containers)│ + │ │ │ (Docker-only)│ │ │ + │ │ └──────────────┘ │ │ + │ └───────────────────────────────────────┘ │ + └───────────────────────────────────────────┘ + PBS (offsite) ◀── outbound, client-side-encrypted backups ── customer box + end-users / customer ◀── Cloudflare Tunnel ── apps + controller UI +``` + +--- + +## 2. The customer node + +- One **Proxmox host** per box (PVE 9.2, Debian 13, LVM-thin). +- **Default workload topology:** one **customer LXC**, Docker inside it, each app a Docker + container/stack. Apps are isolated at the Docker layer (separate containers, networks, + volumes, cgroup limits); they share one LXC/kernel/Docker daemon. +- **Escape hatch:** promote an individual app to its own guest (LXC or VM) only for a + specific reason — a non-Linux/Windows app, a genuinely untrusted or exposed app needing + hard isolation, or a resource hog needing guarantees. +- **Multi-tenant:** one customer per host is the home default; multiple customer LXCs on + one host (a company environment) is **not precluded** — the agent manages a *set* of + guests. The only multi-tenant-specific work deferred to "if it becomes real" is resource + fairness (per-guest disk/RAM/CPU quotas). + +--- + +## 3. Components & responsibilities + +| | **Hub** | **Host agent** | **In-guest controller** | +|---|---|---|---| +| Runs on | dooplex.hu (k3s) | the Proxmox host | the customer LXC | +| Tier | operator backend | operator (high-privilege) | customer-facing (app) | +| Holds | customer records, signed-job source, PBS namespaces, escrowed keys | the **only** Proxmox API token; per-host operator identity | **no Proxmox creds**; its own hub API key + a local-API token to the agent | +| Does | reporting sink, dashboard, job queue, source of durable truth | all Proxmox ops (provision, restore, snapshot, backup, storage mgmt, LXC lifecycle); polls hub for signed jobs; exposes a constrained local API to the controller; **per-guest authorization gate** | Docker/app lifecycle, catalog deploy, customer UI, app-level (data-layer) backup; reports app-domain to the hub directly | +| Never does | initiate a connection *into* a box | — | touch the Proxmox API directly | + +**Key separation:** the controller manages Docker; the agent manages Proxmox. The controller's +only path to guest-level operations (snapshot-before-deploy, "grow my RAM") is a constrained +**local API call to the agent**, which the agent authorizes (scoped to that controller's own +guest) and executes with its operator-tier token. This consolidates all Proxmox access and +all per-guest authorization in one auditable place and leaves the guest with zero Proxmox +credentials. + +--- + +## 4. Control plane — box-initiated + +- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through + Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the + box exposes no control endpoint at all. +- The agent and the controller **poll** the hub; the hub never initiates inbound. +- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature + before executing, so a compromised hub database alone cannot forge commands. +- All operator-initiated actions are recorded in a **customer-visible audit log**. + +--- + +## 5. Trust boundaries + +| Boundary | What crosses | Mechanism | Blast radius if breached | +|---|---|---|---| +| end-user ↔ apps | app traffic | Cloudflare Tunnel → Traefik (Host routing) | that app | +| customer ↔ controller UI | management UI | Cloudflare Tunnel; UI auth (bcrypt) | the customer's own box | +| controller ↔ agent | snapshot/resize/backup requests | local constrained RPC; agent authorizes per-guest | the controller's own guest only | +| agent ↔ hub | reports + signed jobs | outbound poll; signed jobs | one box; signed jobs limit forgery | +| controller ↔ hub | app-domain reports/jobs (incl. geo desired-state) | outbound, own API key | app-domain of one customer | +| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) | +| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — | +| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF | + +--- + +## 6. Enrollment & identity + +- **Physical presence at provisioning** (on-site install, or pre-imaged-and-delivered). + This removes any zero-touch remote-enrollment problem. +- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful + config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure + case (fetch fails before anything is persisted). After the fetch, the code is irrelevant — + everything downstream runs on durable credentials, so retries don't need it. +- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped + operator-tier Proxmox token), then provisions the customer LXC from the golden template and + deploys the controller into it — injecting the controller's hub API key and its local-API + token. The controller is the agent's product, never the other way around. +- The **hub customer record is the durable source of truth**, and it survives box loss: + identity, domain, **Cloudflare tunnel token**, **PBS namespace**, **storage manifest**, a + **mirrored app inventory** (bottom-up reality, not operator-declared intent — apps themselves + restore from the PBS guest snapshot, never re-deployed from this record; see `05` §1/§9), and the + **escrowed (zero-knowledge) backup key**. This is what makes hardware replacement possible. + +--- + +## 7. Networking + +- **Cloudflare Tunnel** provides inbound access to apps and the controller UI (the CGNAT + solution). Tunnel token lives in the hub record → **reused on new hardware during DR**, so + DNS/routing stay intact through an outage. +- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control + endpoint exists in the chosen model. +- **Tunnel placement: host** (resolved, Part 3 §3/§5). `cloudflared` runs on the Proxmox host + as its own **agent-managed systemd service** — not inside the guest — so the data path + survives control-plane death by construction. Geo-restriction WAF is **hub-enforced** (the + hub holds the CF API token; the controller only reports geo desired-state). + +--- + +## 8. Storage & backup + +**Tiers** (escalating failure scope): + +| Layer | Mechanism | Survives | Note | +|---|---|---|---| +| Snapshot | LVM-thin snapshot (transient) | *logical* loss only | whole-LXC rollback; **not a backup** | +| Local — second storage | vzdump to `dir`/`nfs`/`cifs` | primary-disk failure (USB) / box death (NAS) | first *real* backup tier | +| Offsite — PBS | dedup'd, incremental, encrypted | site loss | the DR substrate; paid tier | + +- **Storage manifest** (hub-held, agent-reconciled): per target → type, durable identity + (UUID / `server:/export` / repo+fingerprint), **class** (fast/slow + rough IOPS, set once + at attach), role, encrypted credentials, schedule/retention. The agent creates the Proxmox + storages, continuously checks presence/reachability, and reports per-target status (a + disconnected target → actionable notification). +- **App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume + **hot** (DB/config/cache → fast storage, enforced) vs **bulk** (media/files → may be slow). + A photo app's DB stays on SSD while its blobs go to the USB. +- **Backup scoping:** hot data (LXC rootfs) rides the guest `vzdump` → tiers + PBS. Bulk data + on external mount points is **excluded** from the guest vzdump (per-mount `backup` flag) and + gets its own per-volume policy (file-level to a tier, slower cadence — or explicitly *not* + backed up for re-downloadable content, with the customer informed). +- **Tiers double as the DR restore-source priority:** restore from the fastest *surviving* + source (local if still attachable, PBS on true site loss). +- **Key custody (zero-knowledge default):** three tiers the customer chooses — + *customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows + the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery + code** the operator can't open; DR needs the customer's code. Access-notification is an + audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native + keyfile passphrase.) + +--- + +## 9. Disaster recovery + +- **Guest-loss (host + agent alive):** the agent restores the guest from the fastest + surviving tier, **resets identity** (MAC/hostname — see `proxmox-platform.md`), boots it, + controller returns. Validated mechanics: Phase 2. +- **Host / hardware-loss (agent gone):** re-provision (§6) in **restore mode** — the hub, + knowing the customer has PBS backups, hands the freshly-enrolled agent the existing identity + + PBS namespace + a restore directive instead of a clean-provision directive. The agent + restores from PBS; the controller returns on the same domain (tunnel reused from the hub + record). DR = provisioning + a restore mode, not a separate mechanism. +- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its + post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) + +--- + +## 10. How this embodies the product values + +- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it. +- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise + alone can't forge commands. +- **Customer-visible audit log** — every operator action is visible to the customer. +- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite, + support, new deployments); the customer's data and deployed apps remain recoverable by the + customer (recovery code), with nothing locked behind the operator. + +--- + +## 11. Open sub-decisions (carried into later parts) + +- **RTO/RPO targets** → drive the backup + offsite-replication schedule (§8). +- Offboarding / decommission (scenario 6) — not yet designed; must honour "never hold data + hostage" in credential revocation + data hand-off. +- Multi-tenant resource fairness — deferred until multi-tenant is real (§2). + +--- + +## Appendix — relationship to the spike + +- **Phase 0** → §2: LXC-default for the workload; overhead numbers. +- **Phase 1** → §3/§5: validated the privilege boundary (create/allocate is operator-tier). + The guest-side scoped-backup-token it proved possible is **not** used — we chose the + agent-mediated path — but it confirmed restore = operator-tier, which shapes the agent. +- **Phase 2** → §8/§9: backup→restore round-trip; identity reset on restore. + +--- + +## Changelog — design-review + Phase-3 fold-in (2026-06-08) + +- §5 trust boundaries: **added `hub ↔ Cloudflare API`** row (hub holds the CF token, enforces + geo→WAF); controller↔hub row notes it carries geo desired-state (S4). +- §7 networking: **tunnel placement resolved → host** (agent-managed systemd service); geo is + hub-enforced (S4/S5). +- §11 open items: removed the now-resolved **tunnel placement** and **self-update flow** entries + (S5; self-update designed in 03 §11). +- §6 durable record: **"declarative app inventory" → "mirrored app inventory"** — aligns the wording + with the locked two-driver model (`05` §1: apps are bottom-up mirror, never operator-declared; + `05` §9: apps restore from the PBS guest snapshot, not re-deployed from this record). \ No newline at end of file diff --git a/documentation/architecture/02-controller-module-map.md b/documentation/architecture/02-controller-module-map.md new file mode 100644 index 0000000..b1e538b --- /dev/null +++ b/documentation/architecture/02-controller-module-map.md @@ -0,0 +1,374 @@ +# Felhom Controller Architecture — Part 2: Controller Module Map + +**Status:** audit (keep / port / delete / modify / add), grounded in the v0.33 source. +**Subject:** the v0.33 controller in `deploy-felhom-compose/controller/` (110 `.go` files, +~40 K LOC) audited against [01-topology-and-trust.md](01-topology-and-trust.md) and +[../proxmox-platform.md](../proxmox-platform.md). + +> This is a **planning map, not the port.** No controller code was changed. Source +> citations use `controller/internal/...:line` (a different repo, so links are not +> clickable). Classifications reflect the **target model**: the in-guest controller is +> **Docker-only and holds no Proxmox credentials**; everything host/disk/Proxmox moves to +> a new **host agent** (out of scope here); the controller reaches the agent through a +> constrained **local API**. + +## Classification scheme +**KEEP** (host-agnostic, ~unchanged) · **PORT** (survives, needs rework) · +**DELETE (→agent)** (responsibility moves to the host agent) · +**DELETE (obsolete)** (no longer needed) · **MODIFY** (stays, materially changes) · +**NEW** (no v0.33 equivalent). +Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-target with a keep/port target). + +--- + +## 0. Executive summary + +- The **app domain is largely intact and portable**: stack lifecycle (`stacks/`), catalog + git-sync (`sync/`), app-to-app integrations (`integrations/`), `.fab` export/import + (`appexport/`), the scheduler, crypto, asset sync, the hub report/notify *channels*, and + most of the web UI **KEEP/PORT cleanly**. +- The **disk/storage/host half deletes wholesale to the agent**: all of `storage/`, + `monitor/watchdog.go`, the restic/cross-drive/disk-layout/drive-mount parts of `backup/`, + `report/infra_backup*`+`infra_pull`, and the host-physical parts of `system/`. +- The **setup wizard (`setup/`) is obsolete** — the agent provisions the controller. +- **The single biggest hazard is `backup/`**: the keep side (DB dumps, Docker-volume + archive, per-app restore — needed by `appexport/` and the backup UI) and the delete side + (restic, cross-drive, drive-mount) are **interleaved inside the same files** + (`backup.go`, `restore.go`, `paths.go`), not cleanly file-separated. Extracting the + app-data-backup subset into a clean retained package is the critical refactor. +- **Intent-vs-reality corrections** (vs the task's provisional split): `monitor/pinger.go` + is already **dead** (legacy Healthchecks.io, "deprecated… now handled by Hub" per + `main.go`) → DELETE(obsolete), not keep. `backup.go`/`restore.go`/`paths.go` do **not** + split on file boundaries — they split *within* the file. `settings/` is **not** pure app + domain — it stores disk/disconnect/decommission state. `system/` is genuinely + mixed-per-function, not per-file. + +--- + +## 1. v0.33 module inventory (package → purpose, key deps) + +| Package | Purpose | Key internal deps | +|---|---|---| +| `cmd/controller/main.go` | Entry point; wires all subsystems; 6 adapters break import cycles; branches into setup mode | imports **every** package | +| `api/` | REST API (`router.go`) + geo endpoints (`geo.go`) | stacks, backup, metrics, notify, selfupdate, sync, system, assets, integrations, cloudflare, config, settings | +| `appexport/` | `.fab` app export/import (config+DB+volumes, AES-256-CTR+scrypt) | **backup** (DB dump), (provider iface → stacks) | +| `assets/` | Download/cache app assets from Hub API | — (HTTP only) | +| `backup/` | DB dumps, Docker-volume archive, **restic**, **cross-drive rsync**, per-app restore, **drive mount**, disk-layout, infra-backup metadata | config, monitor, settings, system, util | +| `cloudflare/` | Geo-restriction via Cloudflare WAF (zone/waf/geosync/countries) — **enforcement → hub** (S4) | settings | +| `config/` | `controller.yaml` schema + load | — | +| `crypto/` | AES-256-GCM for app.yaml secrets | — | +| `integrations/` | App-to-app (OnlyOffice→FileBrowser/Nextcloud) via docker exec / config patch | stacks, crypto, settings | +| `metrics/` | SQLite time-series: system + container metrics, log scan | system | +| `monitor/` | App health (`healthcheck`,`pinger`) + **storage/USB watchdog** | config, notify, settings, system | +| `notify/` | Hub event push (direct, own API key) | settings | +| `recovery/` | Generate `recovery-info.txt` (DR guide) | — | +| `report/` | Build+push hub report; **infra-backup payload**; **recovery pull** | backup, config, metrics, monitor, scheduler, settings, stacks, system | +| `scheduler/` | Cron/interval jobs, Budapest TZ | — | +| `selftest/` | Startup checks (docker/dirs/catalog/hub/**restic repos**/mountpoint) | backup, config, settings, system | +| `selfupdate/` | Self-update: pull image, edit compose, `up -d` | config | +| `settings/` | `settings.json` persistent state: **storage paths/disconnect/decommission**, cross-drive cfg, notif prefs, geo, integration state, DB-validation cache | — | +| `setup/` | **First-run wizard** (scan drives, hub-restore, manual config) | backup, config, report, settings, web | +| `stacks/` | Docker Compose lifecycle, deploy + memory validation, metadata (`.felhom.yml`), HDD-data delete | config, crypto, system | +| `storage/` | **Physical disk** scan/format/attach/mount/migrate/fstab/safety | backup, settings, util | +| `sync/` | Catalog git-sync (pull templates) | config | +| `system/` | Resource info: mem/cpu/load (guest) + **temp/disk-model/USB/mount topology (host)** | — | +| `util/` | String helper | — | +| `web/` | Hungarian dashboard: pages, auth, deploy, backup UI, **storage/disk UI**, DR restore UI, export UI, debug | appexport, backup, config, crypto, integrations, monitor, notify, scheduler, selfupdate, settings, stacks, storage, system | + +--- + +## 2. Classification table (per package/file) + +### `cmd/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `cmd/controller/main.go` | **MODIFY** | Wiring stays, but drop the setup-mode branch, the storage/watchdog/drive-migrator/restic/cross-drive/infra-backup wiring, and add the **agent local-API client**. 6 adapters shrink. | hazard | + +### `api/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `api/router.go` | **PORT/MODIFY** | Keep stacks/deploy/integrations/metrics/sync/assets/selfupdate routes; **remove `/api/storage/*` (disk)**; backup routes become **agent-coordinated guest-backup** requests; `config/apply` (hub-pushes-yaml) changes since the **agent** now injects config at provision. | needs-rework | +| `api/geo.go` | **PORT/MODIFY** | Keep the customer-facing geo **preference** endpoints (set/get global + per-app); **drop the Cloudflare-sync trigger** — enforcement → hub (S4). The controller reports geo desired-state up instead of calling the CF API. | needs-rework | + +### `appexport/` — KEEP/PORT (Docker-volume + DB level, no disk ops) +| File | Class | Reason | Risk | +|---|---|---|---| +| `crypto.go` | **KEEP** | Self-contained AES-256-CTR+HMAC+scrypt for `.fab`. | clean | +| `manifest.go`, `provider.go` | **KEEP** | Bundle metadata; provider interface (impl in main). | clean | +| `export.go` | **PORT** | Docker-volume `tar`, DB dump via `backup.DumpOne`, config copy. Depends on the **retained** app-data-backup subset of `backup/`; HDD-mount enumeration reworked to **per-volume placement**. | needs-rework | +| `restore.go` | **PORT** | `docker volume create`/`tar xf`, DB import, compose up. Same per-volume rework. | needs-rework | +| `estimate.go` | **PORT** | `du`/`df` on mounts → per-volume sizing. | clean | + +### `assets/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `syncer.go` | **KEEP** | Hub API download + checksum cache; already a direct hub channel. | clean | + +### `backup/` — THE SPLIT (delete side interleaved with keep side; see §3) +| File | Class | Reason | Risk | +|---|---|---|---| +| `dbdump.go` | **KEEP** | Pure `docker exec pg_dump`/`mariadb-dump` — app/DB data layer; the retained per-app backup. | clean | +| `appdata.go` | **PORT** | App-data discovery (stacks/volumes/DB containers, `du`). "HDD mount" concept → per-volume. | needs-rework | +| `backup.go` (1478 L) | **MODIFY (split)** | Mixes **keep** (`RunDBDumps`, `DumpAppVolumes(Safe)`, app restore) with **delete→agent** (`RunBackup`/`backupDrive`/restic snapshot/prune/check on per-drive repos). Must be torn in two. | hazard | +| `restore.go` (442 L) | **MODIFY (split)** | `RestoreApp` restic path → agent; Docker-volume + Tier-2 rsync restore (app layer) → keep. | hazard | +| `restore_app_linux.go`/`_other.go` | **PORT** | Per-app restore: compose pull/up, rsync app data, DB-dump restore. App layer; depends on backup location that changes. | needs-rework | +| `paths.go` | **MODIFY (split)** | `AppDBDumpPath`/`AppVolumeDumpPath` keep; `Primary/SecondaryResticRepoPath`, `InfraBackupDir` → agent. | needs-rework | +| `restic.go` | **DELETE (→agent)** | restic repos on drives = infra backup tier; agent does vzdump/PBS. | hazard | +| `crossdrive.go` | **DELETE (→agent)** | Tier-2 cross-drive rsync to secondary storage = storage-tier (agent + storage manifest). | hazard | +| `restore_drives_linux.go`/`_other.go` | **DELETE (→agent)** | `lsblk`/`blkid`/`mount`/fstab — pure host disk. | hazard | +| `disk_layout.go` | **DELETE (→agent)** | Disk topology for DR → agent. | clean | +| `local_infra.go` | **DELETE (→agent)** | Per-drive infra-backup metadata → agent. | clean | +| `restore_scan.go` | **DELETE (→agent)** | Scans drives to build a DR restore plan = agent-tier DR. | needs-rework | + +### `cloudflare/` — DELETE (→hub): CF-API enforcement moves to the hub (S4) +| File | Class | Reason | Risk | +|---|---|---|---| +| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework | + +### `config/`, `crypto/`, `util/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `config/config.go` | **MODIFY** | Drop `BackupConfig` (restic/retention), storage-drive keys, and `InfrastructureConfig.cf_api_token` (→hub, S4); keep customer/paths/web/git/stacks/monitoring/hub/assets/system; **add agent local-API endpoint+token**. | needs-rework | +| `crypto/crypto.go` | **KEEP** | App.yaml secret encryption. | clean | +| `util/strings.go` | **KEEP** | Trivial helper. | clean | + +### `integrations/` — all KEEP (pure app-domain) +| File | Class | Reason | Risk | +|---|---|---|---| +| `integrations.go`,`lifecycle.go`,`manager.go`,`onlyoffice_filebrowser.go`,`onlyoffice_nextcloud.go` | **KEEP** | App-to-app via `docker exec` / compose-config patch; no host ops. | clean | + +### `metrics/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `store.go`,`logscanner.go`,`telemetry.go`,`types.go` | **KEEP** | SQLite store, `docker logs` scan, container telemetry — app-domain. | clean | +| `collector.go` | **PORT** | Container metrics (`docker stats`) keep; host metrics via `system.GetInfo` (temp, physical disk) become **agent-provided or dropped**. | needs-rework | +| `sysinfo.go`/`sysinfo_other.go` | **MODIFY** | Reads `/host/etc`, `/proc/cpuinfo`, uptime — host static info; in-guest some is meaningful, hardware identity via agent. | needs-rework | + +### `monitor/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `healthcheck.go` | **PORT (split)** | Keep guest health (mem/cpu/docker/protected-containers); host health (temp, **physical disk**, storage-path mount status) becomes **agent-fed**. | needs-rework | +| `pinger.go` | **DELETE (obsolete)** | Legacy Healthchecks.io; `main.go` itself marks it "deprecated… now handled by Hub". *(Corrects the task's KEEP/PORT guess.)* | clean | +| `watchdog.go` (902 L) | **DELETE (→agent)** | Storage/USB disconnect monitoring: `umount -l`, `mount -T /host-fstab`, UUID probing, restic-lock cleanup — pure host storage. | hazard | + +### `notify/`, `recovery/`, `scheduler/`, `selftest/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `notify/notifier.go` | **KEEP/MODIFY** | Direct hub event channel (own API key) — keep; prune infra event types that move to the agent (`storage_disconnected`, `crossdrive_*`, `disaster_recovery_*`). | clean | +| `recovery/info.go` | **DELETE (obsolete)** | Generates a DR text guide (OS install, docker-setup.sh, hub restore UI); DR is now agent+hub provisioning. | clean | +| `scheduler/scheduler.go` | **KEEP** | Generic cron/interval, Budapest TZ. | clean | +| `selftest/selftest.go` | **PORT** | Keep docker/dirs/catalog/hub checks; drop restic-repo + system-data **mountpoint** checks (→agent). | needs-rework | + +### `report/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `pusher.go` | **KEEP** | Direct hub push (`/api/v1/report`, Bearer). | clean | +| `telemetry.go` | **KEEP** | Per-app telemetry section. | clean | +| `builder.go` (326 L) | **MODIFY** | Keep containers/telemetry/stacks/geo/app-health; drop/relocate host system info, physical storage, **restic backup status incl. restic password**. | hazard | +| `types.go` | **MODIFY** | Schema: drop infra fields (`restic password`, physical storage), keep app-domain. | needs-rework | +| `infra_backup.go`/`_linux.go`/`_other.go` | **DELETE (→agent)** | Builds infra-backup payload (disk layout, restic/enc passwords) for hub. | hazard | +| `infra_pull.go` | **DELETE (→agent)** | Pulls recovery config + infra backup from hub (setup-wizard DR). | needs-rework | + +### `selfupdate/` — controller is agent-managed (doc 03 §11) +| File | Class | Reason | Risk | +|---|---|---|---| +| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean | +| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean | +| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean | + +### `settings/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `settings/settings.go` (1101 L) | **MODIFY (split)** | Keep notif prefs, integration state, geo, DB-validation cache, cross-drive *intent*. The **storage-path registry** (`StoragePath` with `Disconnected`/`DisconnectedAt`/`StoppedStacks`/decommission) is disk-management state → reshape to **per-volume placement** fed by the agent's storage manifest; disconnect/decommission/migrate state leaves. (UUID is *not* a persisted field — runtime-derived from fstab.) | hazard | + +### `setup/` — all DELETE (obsolete); the agent provisions the controller +| File | Class | Reason | Risk | +|---|---|---|---| +| `handlers.go`,`setup.go`,`csrf.go`,`network.go` | **DELETE (obsolete)** | First-run wizard (hub-restore, manual config, LAN-IP detection). | needs-rework | +| `scanner.go` | **DELETE (→agent)** | Drive scan (`lsblk`+temp mounts) for backup discovery — host op; its capability informs the agent. | clean | + +### `stacks/` — core app domain (KEEP/PORT) +| File | Class | Reason | Risk | +|---|---|---|---| +| `manager.go` (1074 L) | **KEEP/PORT** | Docker Compose orchestration, scan/state/start/stop/logs — the heart. Minor port. | clean | +| `deploy.go` | **PORT** | Memory validation (`system.GetMemoryMB` — **guest** mem, fine in LXC), secret gen, encrypted app.yaml. **Add snapshot-before-deploy → agent** hook. | needs-rework | +| `healthprobe.go` | **KEEP** | TCP/HTTP app probes. | clean | +| `metadata.go` | **PORT** | `.felhom.yml` parse. **Add per-volume hot/bulk classification** (doc 01 §8). | needs-rework | +| `delete.go` | **PORT** | Stack delete + HDD-data `os.RemoveAll` on bind mounts → per-volume cleanup. | needs-rework | + +### `storage/` — entire package DELETE (→agent) +| File | Class | Reason | Risk | +|---|---|---|---| +| `scan*`,`format*`,`attach*`,`migrate*`,`migrate_drive*`,`safety*` | **DELETE (→agent)** | Physical disk: `lsblk`/`sfdisk`/`wipefs`/`mkfs.ext4`/`partprobe`/`mount`/`umount`/fstab/`blkid`/drive-rsync. The agent owns all of this (doc 01 §3, §8). | hazard | + +### `sync/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset, copy compose+`.felhom.yml`, never overwrite app.yaml). | clean | + +### `system/` — split per-function (not per-file) +| File | Class | Reason | Risk | +|---|---|---|---| +| `cpu_linux.go`/`cpu_other.go` | **KEEP** | `/proc/stat` works inside an LXC. | clean | +| `info.go`/`info_other.go` | **KEEP** | Structs/stubs. | clean | +| `info_linux.go` | **MODIFY (split)** | Keep mem (`/proc/meminfo`)/load/statfs (guest); **temp via `/host/sys`, hwmon → agent**. | needs-rework | +| `mounts_linux.go`/`mounts_other.go` | **DELETE (→agent)** mostly | Mount-point detection, USB, disk model, fstab, probe — host/disk. Guest-meaningful `statfs` disk-usage is the only keep-candidate → fold into the kept `info`. | hazard | + +### `web/` — split by UI surface +| File | Class | Reason | Risk | +|---|---|---|---| +| `auth.go`,`csrf.go`,`logbuffer.go`,`embed.go`,`templates.go` | **KEEP** | Session/CSRF, log ring buffer, embeds/logo. | clean | +| `funcmap.go` | **KEEP/PORT** | Template helpers; a few backup/state labels track the backup rework. | clean | +| `server.go` (559 L) | **MODIFY** | Routing/wiring; remove storage/DR-restore/watchdog wiring; keep app/deploy/backup/settings/export/debug. | needs-rework | +| `handlers.go` (1883 L) | **PORT/MODIFY** | Core pages keep; the embedded **storage-path management** (add/remove/label/schedulable, storage bars, FileBrowser mount sync) → per-volume / agent-fed. | hazard | +| `handler_export.go` | **KEEP/PORT** | `.fab` UI. | clean | +| `handler_debug.go` (823 L) | **PORT** | Drop storage-simulate/infra-push/DR debug; keep the rest. | needs-rework | +| `alerts.go` | **PORT/MODIFY** | Storage-disconnect alert now sourced from **agent** status; backup/update alerts keep. | needs-rework | +| `handler_restore.go` | **DELETE (→agent) / MODIFY** | DR restore-mode UI; DR is agent-tier — replace with an agent-status view or remove. | needs-rework | +| `storage_handlers.go` (1600 L) | **DELETE (→agent)** | Format/attach/mount/disconnect/migrate-drive/decommission disk UI. Any survivor is a **thin client calling the agent API** (e.g. per-volume placement requests). | hazard | +| `templates/` (HTML, non-Go) | **PORT** | Remove disk-wizard + DR pages; keep app/deploy/backup/settings pages. | needs-rework | + +### `scripts/` +| File | Class | Reason | Risk | +|---|---|---|---| +| `scripts/hashpass.go` | **KEEP** | Standalone bcrypt helper. | clean | + +--- + +## 3. Coupling hazards (delete-targets depended on by keep/port) + +1. **`backup/` is half-deleted but split *inside files*, not across them.** `backup.go` + contains both `RunDBDumps`/`DumpAppVolumesSafe`/app-restore (keep) and + `RunBackup`/`backupDrive` + restic (delete→agent); `restore.go` and `paths.go` are + likewise mixed. **Keep/port consumers reach into this same package:** + - `appexport/export.go:295` → `backup.DiscoverDatabases`/`DumpOne` (DB dump is app-layer — must survive) + - `report/builder.go:buildBackupReport` → backup status (MODIFY) + - `web/handlers.go` (backups page, `buildAppBackupRows`), `web/funcmap.go`, `web/alerts.go`, `web/handler_restore.go`, `web/handler_debug.go` + - `selftest/selftest.go:217` → `checkResticRepos` (restic path — delete) + - `main.go` scheduler chain `RunFullBackup` (DB→volume→restic→infra-push) interleaves both sides. + **Action:** extract the app-data-backup subset (DB dump, volume archive, per-app + restore) into a clean retained package *before* deleting the restic/cross-drive code, + or every keep consumer breaks. + +2. **`backup/crossdrive.go` (delete→agent) is wired as `crossDriveRunner` into** + `main.go`, `api/router.go`, `web/server.go`, and surfaced by `report/builder.go` and the + backups page. Removing it requires reworking the backup UI/report to the agent's + guest-backup status. + +3. **`storage/` (delete→agent) depended on by keep/port UI:** `web/storage_handlers.go` + (delete) and `web/server.go`/`web/handlers.go` (port) — the latter renders storage + labels/bars and runs **FileBrowser mount sync** off the storage-path registry. + `storage/migrate*.go` also imports `backup` (also being split). Untangle the per-volume + placement UI from the disk-management UI. + +4. **`monitor/watchdog.go` (delete→agent) depended on by** `web/alerts.go` (port), + `web/server.go`, `web/handler_debug.go`, `main.go`. The disconnect **alert** must instead + consume agent-reported storage status. + +5. **`system/` mixed-per-function, consumed by both sides.** Keep consumers — + `stacks/deploy.go` (`GetMemoryMB`, guest), `metrics/collector.go` (container) — must not + drag in the host-disk/temp/USB code that goes to the agent (`mounts_linux.go`, + `info_linux.go` temp). Also consumed by `report/builder.go` (MODIFY), `monitor/healthcheck.go` + (PORT), `selftest`, `crossdrive` (delete). **Split `system/` cleanly into guest-info vs + host-info first.** + +6. **`settings/StoragePath` carries disk state into an app-domain store.** Disk fields + (`Disconnected`,`DisconnectedAt`,`StoppedStacks`, decommission — UUID is *not* persisted, it's runtime-derived from fstab via `system.ParseFstabUUID`/`watchdog.go`) are written by + `watchdog.go`/`storage_handlers.go`/`crossdrive.go` (all delete) but the same struct is + read by `stacks`/`web` for labels and **placement** (keep). Reshape `StoragePath` to a + placement record fed by the agent manifest. + +7. **`report/builder.go` imports almost everything** (backup, monitor, scheduler, stacks, + system, metrics, settings, config). Its MODIFY must land *after* the backup and system + splits, or it pulls deleted code along. + +8. **`backup/paths.go` shared both ways** — `appexport` + `selftest` + the kept DB-dump + flow use the app-dump path helpers; the same file holds the restic/secondary helpers + that leave. + +9. **DR/provisioning chain is cross-cut:** `setup/` (obsolete) → `report/infra_pull` + + `recovery/info` + `backup.MountDrivesFromLayout` + `backup.ReadLocalInfraBackup`. All + obsolete/→agent, but `main.go`'s setup branch and `web/handler_restore.go` reference + them; remove together. + +--- + +## 4. Moves to the host agent (consolidated — feeds the future agent design) + +> Reporting only; **not** designing the agent here. + +- **All physical-disk management** — `storage/` in full: scan/classify, format + (`wipefs`/`sfdisk`/`mkfs.ext4`/`partprobe`), attach (raw mount + bind + fstab), per-app + and full-drive migration (rsync), safety checks (system-disk detection). +- **Storage/USB watchdog** — `monitor/watchdog.go`: disconnect/reconnect detection, + `umount -l`, `mount -T /host-fstab`, UUID-by-id probing, safe-disconnect, restic-lock + cleanup. +- **Infra/disk backup tier** — `backup/restic.go`, `crossdrive.go`, + `restore_drives_*`, `disk_layout.go`, `local_infra.go`, `restore_scan.go`, plus the + restic-snapshot half of `backup.go`, the restic-restore half of `restore.go`, and the + restic/secondary path helpers in `paths.go`. (Maps to the agent's `vzdump`→tiers→PBS in + doc 01 §8.) +- **Infra-backup payload + recovery pull** — `report/infra_backup*`, `report/infra_pull`. +- **Host-physical telemetry** — `system/mounts_linux.go` (mount topology, USB, disk + model), the temp/hwmon parts of `system/info_linux.go`, and the host-hardware parts of + `metrics/sysinfo.go`. +- **Drive scanning for provisioning/DR** — `setup/scanner.go`. +- **Self-restore-test execution** — the agent performs the restore-to-scratch-guest; the + controller only orchestrates/validates (see §5). + +--- + +## 5. New components to build (no v0.33 equivalent) + +1. **Agent local-API client** — the controller's only path to guest-level Proxmox + operations (doc 01 §3, §5): `snapshot-before-deploy` + rollback, "grow my RAM", request + guest backup/restore, read the storage manifest / mount placement, query per-target + storage status. Replaces the deleted direct host/disk code with constrained RPC. The + controller holds **no Proxmox creds** — only a local-API token. +2. **Per-volume storage placement** (doc 01 §8) — `.felhom.yml` `hot`/`bulk` volume + classification (extend `stacks/metadata.go`), enforcement at deploy (extend + `stacks/deploy.go`), and a placement record in `settings`. Replaces the per-app + HDD-path + cross-drive model. A `bulk` volume must be realized as a `backup=0` mount point, + **never** a rootfs Docker named volume (validated recipe: `phase3-findings.md` B2 / doc 03 §7). +3. **Self-restore-test status display** (read-only) — the **agent owns orchestration** (it + holds the PBS key and creates the scratch guest — operator-tier, doc 03 §8); the controller + only surfaces `GET /restore-test/status` in its UI. (Round-trip validated: Phase 2, + [../proxmox-platform.md](../proxmox-platform.md) §4.) +4. **Snapshot-before-deploy/rollback flow** in the deploy path — wraps the existing + compose deploy with agent snapshot → health check → agent rollback-on-failure + (doc 01 §9). New behaviour on top of `stacks/deploy.go` + `stacks/healthprobe.go`. +5. **Agent-provisioning bootstrap receiver** — the controller accepts its injected hub API + key + local-API token from the agent at provision time (doc 01 §6), replacing the + deleted `setup/` wizard. + +--- + +## 6. Open / blocked items + +- **Geo — resolved (S4):** CF-API **enforcement moves to the hub** (it holds the CF token and + reconciles geo → WAF); the controller keeps the geo **preference UI/data** and reports + desired-state up. Tunnel placement is settled (host, agent-managed, doc 03 §3/§5). The + `cloudflare/` package + `api/geo.go`'s CF-sync are DELETE-from-controller → hub. +- **Self-update — resolved (doc 03 §11):** the controller is agent-managed; its self-update + path is removed. +- **`settings`/`stacks` per-volume reshape** — depends on the storage-manifest contract + between hub ↔ agent ↔ controller (doc 01 §8), not yet specified. +- **Backup UI/report surface** — depends on the agent's guest-backup status API shape + (what the controller can see about vzdump/PBS state) — undefined. +- **Notification event taxonomy** — which infra events (`storage_disconnected`, + `crossdrive_*`, `disaster_recovery_*`) the **agent** emits vs the controller, once those + responsibilities move. + +--- + +## Changelog — design-review + Phase-3 fold-in (2026-06-08) + +- **M1:** removed `UUID` from the `settings.StoragePath` field lists (§ settings, hazard #6) — + it is runtime-derived from fstab, not persisted. +- **S4 (geo):** `cloudflare/` reclassified **PORT(blocked) → DELETE(→hub)** (CF-API enforcement + moves to the hub); `api/geo.go` → **PORT/MODIFY** (keep geo *preference* endpoints, drop the + CF-sync trigger); `config/config.go` also drops `cf_api_token`. §6 + §1 updated. +- **S5:** cloudflare/geo no longer "blocked on tunnel placement" (resolved). +- **S6:** §5(3) self-restore-test → **status-display only**; the agent owns orchestration. +- **Self-update resolved (03 §11):** `updater.go` → **DELETE(→agent)**, `state.go` → + DELETE(obsolete), `version.go` KEEP; §6 + §5(2) updated (bulk = `backup=0` mountpoint recipe). diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md new file mode 100644 index 0000000..644864d --- /dev/null +++ b/documentation/architecture/03-host-agent.md @@ -0,0 +1,299 @@ +# Architecture Part 3 — The Host Agent + +> Status: design draft (decision content). To be grounded by Claude Code against +> `docs/proxmox-platform.md` and `docs/architecture/02-controller-module-map.md`, +> then placed at `docs/architecture/03-host-agent.md`. +> +> Builds on Part 1 (`01-topology-and-trust.md`) and Part 2 (`02-controller-module-map.md`). +> Where this doc and the locked decisions disagree, the locked decisions win and this +> draft is wrong — flag it. + +## 1. Purpose & scope + +The **host agent** is the operator-tier component that runs on each Proxmox host and +owns *all* Proxmox interaction. It is the trusted host actor: it provisions and restores +guests, manages host storage, orchestrates backups and restore-tests, watches the host +and the tunnel, talks to the hub, and exposes a narrow local API to the in-guest +controllers it deploys. + +It is the privileged tier. The controller deliberately holds **no** Proxmox credentials +(Part 1) — the privilege the controller shed by losing `storage/` did not disappear, it +**moved here**. That makes the agent's hardening and blast-radius discipline the most +security-sensitive part of the platform. + +The agent manages a **set** of guests on its host (usually one customer = one guest, but +the multi-tenant/company case is not precluded — the agent's data model is per-host, +N-guests, never "the guest"). + +## 2. Responsibilities (and explicit non-responsibilities) + +Owns: + +1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable. +2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests. +3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**. +4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. +5. **Provisioning** — provision a guest **by restoring the golden base image** (§9), deploy the controller into it, hand it its bootstrap config; also **build and refresh the golden base image** itself. +6. **Hub control loop** — poll for desired state + signed jobs, reconcile, execute, report, heartbeat. +7. **Local API** — the per-guest authorization gate the controller calls. +8. **Self-update** — update itself (carefully — it is a host service) and update the controllers it owns. + +Explicitly does **not**: + +- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat. +- Hold or proxy customer application data. +- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them. +- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules. + +## 3. Process model & host integration + +- **Native Go binary, systemd service** on the host: boot-start, `Restart=always`, systemd watchdog (kill+restart on hang), journald logging, resource limits. +- **Root-minimized (boundary settled — Phase 3 B3).** The agent runs as a **non-root** service user with the scoped `FelhomAgent` token for all API-covered work + a **narrow `sudoers` allowlist** for true host ops. Per Phase 3 (B3) the boundary is settled: the entire per-customer guest lifecycle — provision (by restore, §9), config, start/stop, snapshot, backup, **restore**, destroy — is token-covered. Genuine OS-root is confined to: (1) building/refreshing the **golden base image** (`keyctl` create is `root@pam`-only — one-time at enrollment + a maintenance cadence, §9); (2) **host mounts** (USB mount-by-UUID, systemd mount units / fstab); (3) **SMART / hardware sensors**. Root therefore never sits on the per-customer path. See `proxmox-platform.md` §3.6 for the role + boundary table. +- **`cloudflared` is a separate systemd service**, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process. + +## 4. Control model — reconcile + signed destructive ops + +Two channels, split by **reversibility**, not by transport. + +**(a) Desired-state reconciliation — steady state.** +The hub holds desired state for the host: which guests should exist (and at what spec), +the storage manifest, backup/retention policies, controller image versions. The agent +runs a reconcile loop converging actual Proxmox state → desired: idempotent, self-healing, +and tolerant of missed polls (drift is corrected on the next loop). Provisioning retries, +re-attach of a flapping USB target, redeploy of a crashed controller — all fall out of +reconciliation for free. + +**(b) Signed one-shot jobs — operator actions.** +Restore-now, decommission, force-backup, break-glass-enable. Discrete, run-once +(idempotency key), written to the customer-visible audit log, and **outside** the reconcile +loop — they are point-in-time and often destructive, and a reconciler must never re-run a +restore because it "sees drift." A one-shot job names a **target** ("restore guest X from +snapshot S"), not a procedure; the agent owns the *how*. + +**The reversibility gate (security-critical).** +"Signed jobs resist hub compromise" only holds if the agent also distrusts hub-supplied +*desired state* for destructive changes. The gate is by **provenance + data-bearing-ness, not +by verb**: + +- **The reconciler MAY act without an operator signature** when: (a) creating/starting/restarting; (b) destroying resources it created earlier **within the same journaled transaction** (compensating rollback, §10); (c) destroying resources it **tagged ephemeral/scratch** (e.g. restore-test scratch guests, §8). The ephemeral/scratch tag is **agent-internal provenance and is never accepted from the hub** — else a compromised hub could relabel a data-bearing guest as scratch to walk the gate. +- **An operator signature is always required** to destroy/overwrite any resource holding the only/primary copy of customer data — live-guest destroy, storage detach/wipe, restore-overwrite, decommission — *regardless of whether it arrives as a job or as a desired-state delta*. A compromised hub cannot forge them because the signing key is **not held by the hub** (it lives with the operator / a separate signing path; the hub only queues opaque signed blobs). +- **Healing a crashed controller is non-destructive by construction:** it is reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. (v0.33 precedent: `watchdog.go` restarts stopped stacks, it never destroys the guest.) + +Signed payloads carry a **nonce + expiry** (anti-replay: a captured "restore" job cannot be +re-injected later) and a target binding (host + guest id) so a signature can't be retargeted. +Notification-on-destructive-op is an **audit signal, never the guard** — a compromised hub +could both issue and suppress the notice, which is exactly why the *signature* (not the +notification) is the control. + +## 5. Hub ↔ agent protocol (host domain) + +**Box-initiated poll.** The hub never connects inbound. Each poll cycle exchanges: + +- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health, agent + controller versions, audit-log tail. +- **Down:** the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes). + +**Dead-man's-switch (essential, not optional).** In a box-initiated model the heartbeat +*is* the liveness signal — a box that stops checking in is otherwise invisible. The hub +alerts the operator when an agent misses its expected check-in window. This is the worst +failure mode for a managed service, so it gets first-class treatment hub-side. + +**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged +(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit, +**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via +the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed, +logged operation; it auto-expires. + +## 6. Agent ↔ controller local API + +The controller (in its LXC) reaches the agent (on the host) over the local bridge. + +- **Transport:** HTTPS to the host's bridge IP on a fixed port. +- **Auth:** a per-guest local token, minted by the agent when it deploys the controller and written into the guest's bootstrap config. The agent maps token → guest and **authorizes per guest**: a controller can only act on *its own* guest. This is the agent acting as the per-guest authorization gate from Part 1. +- **Surface (minimal, all scoped to the caller's own guest):** + - `GET /storage` — mounts available to this guest and their **class** (fast/slow), so the controller can place hot vs bulk volumes per `.felhom.yml`. (The agent owns the actual mounts; the controller just binds to the paths it's given.) + - `POST /snapshot` — snapshot *this* guest (the snapshot-before-deploy primitive). + - `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery). + - `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive). + - `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8). + - `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. + +Note what is *absent*: nothing here lets a controller touch another guest, the host, storage +attachment, or restore-overwrite. Destructive/cross-guest power stays operator-signed (§4). + +A controller can only `POST /rollback` (or snapshot/backup) **its own** guest — the agent maps +token → guest and authorizes per guest, so a compromised controller's blast radius is +**self-scoped and bounded** to its own guest. + +## 7. Storage manifest & reconciliation + +The manifest is the load-bearing contract. It absorbs the **persisted** disk-state fields that +`settings.StoragePath` carries today **and adds** `durable_id`/UUID — today the controller +re-derives the UUID from fstab each boot (Part 2 / Phase-3), so persisting it is an +improvement. Held in the hub, reconciled by the agent. + +Per target: + +| field | meaning | +|---|---| +| `type` | `local-dir` / `usb` / `nfs` / `cifs` / `pbs` | +| `durable_id` | UUID (USB), `server:export` (NFS/CIFS), `repo+fingerprint` (PBS) — survives box loss | +| `class` | `fast` or `slow`, set **once at attach**, with an IOPS marker; no runtime speed-test | +| `role` | `primary` / `vzdump-target` / `pbs-offsite` / `bulk-data` | +| `creds` | encrypted (NFS/CIFS/PBS); USB has none | +| `policy` | schedule + retention for this target | +| `state` | `attached` / `disconnected` / `decommissioned` | + +Reconciliation: ensure each `attached` target is mounted (USB-by-UUID via the sudoers +allowlist), each Proxmox storage entry matches, and `disconnected` targets are surfaced to +the hub (the storage watchdog — detect a USB drop in seconds, not at the next health cycle). + +**Placement is per-volume, not per-app.** Hot volumes (DB/config) → a `fast` target, +**enforced**; bulk volumes (media) → may live on `slow`, declared in `.felhom.yml`. + +A `bulk` volume **MUST** be realized as a `backup=0` **volume mount point** (or an external +bind mount) — **never** a Docker named volume in rootfs, which `vzdump` always captures +(verified, `phase3-findings.md` B2). Proven recipe: attach +`-mpN :,mp=/mnt/bulk,backup=0`, then +`docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk ` (or a +compose bind). The per-volume placement component (Part 2 §5(2)) enforces this at deploy. The +**DR consequence** of excluding bulk is covered in §8. + +**Field re-homing (from `settings.StoragePath`, Part 2):** `Label` → manifest (canonical); +`IsDefault`/`Schedulable` → manifest `policy`; `MigratedTo` + decommission → manifest `state`; +`StoppedStacks` → the **controller's `settings`** (app-domain: which apps to restart on +reconnect, not a host concern). + +## 8. Backup/restore orchestration + +Tiers double as backup *and* restore-source priority (fastest surviving source first), +per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a backup) → +**local second storage** (vzdump to dir/NFS/CIFS) → **PBS offsite** (the DR substrate). + +- **Quiescing (controller-driven for app-consistency):** an LXC has no fsfreeze + (`proxmox-platform.md` §4.2), so app-consistency is the controller's job: it learns a backup + is due (`GET /backup/due`, §6, or via its hub channel) → **quiesces** the app stack → + `POST /backup` → polls `GET /backup/status` → unquiesces. **An agent-initiated vzdump is + crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5). + Every Proxmox op is async → the agent polls `task exitstatus`, never trusts the POST return. +- **Bulk volumes have no DR coverage from the guest vzdump** — they are excluded (§7). Every + `bulk` volume needs an explicit own-backup decision: its own backup target per the manifest + `policy`, **or deliberately none** when the data is re-downloadable (customer informed). On + host-loss, un-backed-up bulk is gone; a **bind-mounted** bulk volume re-attaches only on the + *same* host, so cross-host DR needs the separate backup. A deliberate per-volume choice, + never a silent loss. +- **Key custody (PBS):** the **live** PBS key sits on the box so the agent can both back up + *and* run restore-tests. The hub holds only the **recovery-code-wrapped escrow** copy it + cannot open (zero-knowledge default). So: the box can restore-test; the operator cannot + read the data; the customer's offsite recovery code is the irreducible residual. +- **Self-restore-test:** the closing of the "tested restore is the critical gap" theme. The + agent periodically restores a backup into a **throwaway scratch guest**, boots it, runs + health checks, reports pass/fail, and tears it down. Zero-knowledge backups can *only* be + restore-tested by the box (the operator lacks the key) — so this lives in the agent by + necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often + as the lighter check. + +## 9. Provisioning & DR flows + +**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs +the `keyctl=1` feature flag, which Proxmox permits only for `root@pam` (Phase 3, B3) — not the +scoped token. But a token-authorized **restore preserves `keyctl`** (Phase 3, B3), so the agent +provisions **by restoring a golden base image**, never by `pct create` on the per-customer path: + +- A **golden base archive** — minimal Debian + Docker, `nesting=1,keyctl=1`, overlayfs — is + built once as `root@pam` **at enrollment** (when the agent legitimately holds root to mint its + Proxmox token) and refreshed on a maintenance cadence. This is the one place `keyctl`/root + provisioning lives — off the per-customer path. +- To provision guest G: restore the golden archive → new VMID (token-covered: `VM.Allocate` + + `Datastore.AllocateSpace`; `keyctl` preserved) → reset identity (MAC/hostname) → size the guest + (CPU/mem config + `pct resize` rootfs, token-covered) → attach storage mounts per the manifest + → deploy the controller → hand it bootstrap config. A mid-flight failure is journaled and + compensating-rolled-back (destroy the just-restored guest — allowed without a signature per §4, + same-transaction provenance). + +**Unified bring-up primitive.** Provisioning and DR-restore share the same token-covered front +half — *restore an archive → reset identity* — and differ only in the archive and the back half: +provisioning restores the **golden base** then deploys a fresh controller; DR-restore restores +the **customer's backup** (already containing controller + data), brings it up, and reattaches +external storage. One code path, exercised by every restore-test (§8). + +**Guest loss.** Agent restores G from the fastest surviving tier and resets identity +(MAC/hostname) so the restored guest rejoins cleanly — this *is* the unified restore primitive +above (customer-backup archive, DR back half). + +**Host/hardware loss.** Re-enroll the new host in **restore mode**; the hub — the durable +source of truth that survives box death — hands the new agent the existing identity, PBS +namespace, tunnel token, storage manifest, and a restore directive. Tunnel is reused from +the hub record, so DNS stays intact. + +## 10. Concurrency, crash-safety, idempotency + +- **Per-guest serialization.** Reconcile, one-shot jobs, and local-API calls all feed a + work queue that serializes mutations **per guest** (Proxmox dislikes concurrent conflicting + ops on the same guest). Independent guests proceed in parallel. +- **Operation journaling.** Multi-step async ops (provision, restore, controller-update, agent + self-update) are journaled with their in-flight Proxmox task ids. On agent restart, the + journal is replayed: resume-or-rollback, so a crash mid-restore never leaves a corrupt or + half-built guest. +- **Idempotency keys** on one-shot jobs (run-once across retries and restarts). + +## 11. Self-update + +- **Agent (the hard case — a host service, no snapshot-rollback).** **A/B layout:** download → + verify signature → stage as the inactive slot → flip a `current → good|new` symlink → restart. + **Revert authority lives outside the swapped binary** — `Restart=always` alone just + crash-loops a bad binary — so a **separate health-gate** (a systemd oneshot `ExecStartPost` + probe, or a tiny supervisor unit) flips `current` back to last-good and restarts on a failed + health window. The new version is **committed as "good" only after a clean health window**. + Triggered by a hub signed job within the update window; manual always allowed. Journaled (§10). +- **Controller (the easy case — it's a guest).** The agent owns the controller's lifecycle, + so the **agent updates the controller**: snapshot-before-update (free rollback, because the + controller *is* a snapshottable guest) → pull new image → redeploy → health-check → rollback + on failure. This resolves the Part-2 `selfupdate/` open: the controller is **agent-managed**, + not self-updating; the controller's old self-update path is removed. + +## 12. Secrets at rest on the host + +The agent holds, root-only on the host fs: the scoped Proxmox token, the hub API key, the +operator's **public** verify key (for §4 signatures — public, low-risk), the Cloudflare +tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**. The privilege +and the secret footprint that left the controller now concentrate here — which is the whole +argument for §3's root-minimization and a small, auditable agent. + +## 13. Open items / what this unblocks + +Resolved here: tunnel placement (host, agent-managed, own systemd service), the +reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update +ownership, the local-API surface, the storage-manifest schema, **provision-by-restore**, and +the **root-vs-API boundary** (Phase 3, B3). + +Still open: + +- Multi-tenant **resource fairness** on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass. +- Operator-side **signing tooling** — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub"). +- Hub-side **desired-state editing UX** and the host-domain report schema details — belong to the hub architecture doc. +- **Golden base image** refresh cadence + fleet versioning — who triggers a rebuild, how the per-host image version is tracked (operational detail, not blocking; §9). + +This doc hands the implementation three contracts it was waiting on: + +1. the **local-API surface** (§6) → the controller's NEW local-API client, snapshot-before-deploy, and self-restore-test wiring (Part 2); +2. the **storage-manifest schema** (§7) → the `settings.StoragePath` reshape and per-volume hot/bulk placement (Part 2); +3. the **backup contract** (§7–8) → the destination for the app-data-backup package extracted in the Part-2 refactor. + +--- + +## Changelog — design-review + Phase-3 fold-in (2026-06-08) + +- **NEW provision-by-restore** (§9): the agent provisions by **restoring a golden base image** + (token-covered, preserves `keyctl`), never `pct create` on the per-customer path; one unified + restore primitive shared with DR. §2 responsibility + §3 boundary updated. +- **B3** (§2/§3): replaced "Phase-1 minimal role" with the validated **`FelhomAgent`** operator + role; root-vs-API boundary **settled** (root only for golden-image build, host mounts, SMART). +- **B1** (§4): reversibility gate rewritten as **provenance + data-bearing** (scratch tag is + agent-internal, never hub-supplied; crashed-controller heal is non-destructive in-place). +- **B2** (§7/§8): validated bulk-as-`backup=0`-mountpoint recipe + the **bulk-DR consequence** + (excluded bulk needs its own backup decision). +- **S1** (§6/§8): `GET /backup/due` added; controller-driven quiescing; agent vzdump is + crash-consistent only. **S2** (§10/§11): A/B self-update with external revert authority; + controller-update + agent self-update journaled. **S3** (§7): `StoragePath` field re-homing. + **S4:** geo non-responsibility added (§2). **M2** (§7): manifest "absorbs + adds durable_id". + **§6:** rollback is self-scoped/bounded. **§13:** golden-image refresh cadence added as open. \ No newline at end of file diff --git a/documentation/architecture/04-control-plane-authorization.md b/documentation/architecture/04-control-plane-authorization.md new file mode 100644 index 0000000..c9abbbb --- /dev/null +++ b/documentation/architecture/04-control-plane-authorization.md @@ -0,0 +1,154 @@ +# Architecture Part 4 — Control-plane authorization (operator signing) + +> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`. +> To be reviewed by Claude Code against that spike + `03` §4, then placed at +> `docs/architecture/04-control-plane-authorization.md`. +> +> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate). +> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge." + +## 1. Purpose & scope + +`03` §4 gates **destructive/irreversible** operations behind an operator signature the hub cannot +forge. That gate is only real if signing is real. This doc defines the signing mechanism: the +primitive, the keys, rotation, the three components' roles, and the operator workflow. The +*policy* (what needs a signature) lives in `03` §4; this is the *how*. + +**Recap of what needs a signature** (from `03` §4, by reversibility, not by verb): destroying or +overwriting any resource holding the only/primary copy of customer data — live-guest destroy, +storage detach/wipe, restore-overwrite, decommission — **regardless of whether it arrives as a job +or a desired-state delta**. Benign convergence (deploy a guest, attach storage, restore to a *new* +guest, bump a version) runs on normal hub auth, unsigned. Most recovery is therefore unsigned; +signed ops are rare and deliberate. + +## 2. Primitive — SSH signatures (SSHSIG) + +Confirmed by Phase 4: destructive ops carry an **SSH signature** (`ssh-keygen -Y sign`, the armored +`SSHSIG` format), verified by the agent in Go (`golang.org/x/crypto/ssh`) — `pem.Decode` → +`ssh.Unmarshal` → `ssh.ParsePublicKey` → `pub.Verify`. ~40 lines of framing, no hand-rolled crypto. + +**Why SSHSIG and not raw Ed25519 / minisign:** SSHSIG verification dispatches on the key type +embedded in the signature, so the **same verifier accepts a software key (`ssh-ed25519`) today and +a FIDO2 hardware key (`sk-ssh-ed25519@openssh.com`) later** — which is exactly the hardware-ready +foundation we want (§7). A raw-Ed25519 verifier cannot consume an sk signature (flags+counter, +different signed-data), so it would force a verifier change on every box at hardware-adoption time. +SSHSIG buys key-type-agnosticism for a one-file framing cost (Phase 4 §5–6). + +### 2.1 The signed object — canonical op blob +The signature covers an op blob (Phase 4 §2): + +``` +{ op, target:{host_id, guest_id}, params, nonce, issued_at, expires_at, key_id } +``` + +- **Canonical form is a *signer-side* requirement** — JSON, keys sorted at every level, no + insignificant whitespace, UTF-8 — so the blob is deterministic and human-auditable. The + **verifier trusts the exact bytes it receives** (it verifies the signature over the raw bytes and + parses those same bytes for fields), so there is no canonicalization-mismatch risk on the verify + side. The canonical form is the shared contract between the operator CLI and the agent (both Go). +- `nonce` ≥128-bit random; `issued_at`/`expires_at` a short window (minutes); `key_id` identifies + the signing key (rotation/audit). + +### 2.2 Domain separation — the namespace +The SSHSIG **namespace** `felhom-op-v1` is a **fixed constant in the verifier**, never +caller-supplied. A signature minted for any other namespace must not verify (proven). This stops a +signature made for one purpose being reused for another. + +### 2.3 Verify pipeline (order is load-bearing) +`namespace → allow-list → crypto verify → target binding → time window → nonce`. The **nonce is +recorded last**, only after everything else passes, so an invalid signature can never consume a +nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4): +- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box + A cannot be replayed at box B); +- **time window** — `now ∈ [issued_at, expires_at]`; +- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts** + and expiry-pruned; a non-persistent store reopens the replay window after a restart). + +The Phase-4 reference verifier (`VerifySignedOp`) is the seed of the agent's implementation. + +## 3. The keys — two-key model, software now + +Both software (SSH-format) keys today; both are also valid FIDO2-resident keys later with no box +change (§7). + +- **Operational signing key** — the "master stamp" for destructive ops. A **dedicated** key (NOT + the operator's daily SSH login key), passphrase-protected, on the operator workstation. Used only + for destructive ops — rare, so its exposure is low. +- **Cold recovery key** — generated once, kept **offline** (password manager / a USB held back / + printed). Never used for ordinary ops; its sole power is to authorize rotating the operational key + if that key is lost or compromised. + +Both **public** keys are pinned onto the agent at enrollment (the allowed-signers set). The +operational key is authorized for ops; the recovery key is authorized **only** for key-rotation +instructions. + +**Allowed-signers is a set** → single signer today; **quorum (N-of-M) for the highest-blast ops is +just set sizing + a threshold policy**, addable later without a redesign (Phase 4 §8). Out of scope +now. + +## 4. Rotation & compromise recovery + +The agents pin the operator public keys. The danger: rotation must **not** flow as plain hub config, +or a compromised hub re-pins its own key and forges everything. So **every re-pin is itself a signed +op the agent verifies** (same pipeline, §2.3) — never unauthenticated config. + +- **Planned rotation:** the *current* operational key signs a "new operational public key = X" op; + the agent accepts it because it's signed by the trusted current key (key-signs-key). +- **Operational key lost/compromised:** the **cold recovery key** signs the re-pin; the agent accepts + it because the recovery key is pinned and authorized for rotation. The compromised key is removed + from the allowed set in the same signed op. +- **Both keys gone:** on-site physical re-enrollment (last resort — re-establishes the trust root the + way initial enrollment did). + +## 5. Component roles + +- **Operator tooling (the workstation).** A signing CLI behind a thin **`Signer` interface** + (`Sign(blob) → signature`). The backend today is a **file key**; a **FIDO2/PIV** backend drops in + later (§7) with no change to the blob format, the hub, or the agent. Holds the operational private + key (passphrase-protected); can reach the cold recovery key when rotation is needed. +- **Hub.** Queues the **opaque** signed blobs and surfaces pending destructive ops + their signature + status in the operator UI. Holds **no** private key and cannot sign — a compromised hub can only + queue blobs the agent rejects. (Matches `03` §4 / box-initiated poll.) +- **Agent (each box).** Pins the allowed-signers set (operational + recovery) at enrollment; runs the + verify pipeline (§2.3) on any destructive op before executing; writes every signed op to the + customer-visible **audit log**. Notification-on-destructive-op is an audit signal, never the guard + (a compromised hub could issue *and* suppress notice — the signature is the control). +- **Enrollment.** Pins the initial operational + recovery public keys onto the agent during the + physical-presence provisioning step (the trust root is established on-site, not via the hub). + +## 6. Operator workflow + +- **Routine work** (deploy, monitor, attach storage, restore to a *new* guest): no signing, zero + overhead. +- **A destructive op** (rare): the operator runs the signing CLI on their workstation — which builds + the canonical blob, signs it (passphrase, or later a hardware touch), and posts it to the hub + queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the + desk. **Never** a site visit. + +## 7. Hardware readiness (Viktor's "build the foundation now") + +Software `ssh-ed25519` now; a FIDO2 `sk-ssh-ed25519@openssh.com` key later is a **no-op on the +boxes** — proven end-to-end against the OpenSSH spec in Phase 4 §5 (the unchanged verifier accepts a +spec-faithful sk signature). At hardware adoption the operator generates an sk-key, points the +`Signer` backend at it, and updates the allowed-signers entry; nothing on the boxes changes. + +Two honest notes: +- **Confirm with a real device at adoption.** §5 was validated to spec, not against live hardware — + a 5-minute real-key round-trip should confirm it (no surprise expected; signer/library/device all + follow the same spec). +- **Optional future hardening:** require the FIDO2 **user-presence (touch) flag**. The verifier is + crypto-only today (correct for software keys); enforcing the flag is a small later option once + hardware is in use. + +## 8. Open items +- **Quorum policy** (N-of-M per op-class, e.g. two signatures for decommission) — deferred; the + allowed-signers-set foundation supports it. +- **Signing-key passphrase UX** on the workstation (ssh-agent / askpass) — minor operator-tooling + detail. +- **Hub-side pending-op UI** (showing ops awaiting signature + audit) — belongs to the hub doc. + +## 9. What this unblocks +Closes the `03` §4 "undesigned signing path." Hands the implementation: the **canonical blob spec** +(§2.1) + the **`VerifySignedOp` reference** (Phase 4 §7) for the agent's verify path, the +**`Signer` interface** for the operator CLI, and the **allowed-signers pinning** step for enrollment. +The hub's signed-job queue + pending-op UI carry into the hub architecture doc. \ No newline at end of file diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md new file mode 100644 index 0000000..61bb238 --- /dev/null +++ b/documentation/architecture/05-hub-architecture.md @@ -0,0 +1,223 @@ +# Architecture Part 5 — The Hub + +> Status: design draft (decision content). To be validated by Claude Code against the **actual +> felhom-hub source** (`felhom.eu` repo, `hub/`) + Parts 01–04, then placed at +> `docs/architecture/05-hub-architecture.md`. +> +> The hub is **not** greenfield — it's a mature service (felhom-hub v0.6.3, Go + SQLite on k3s, +> `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus the new +> data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing). + +## 1. Source-of-truth model — two drivers, two directions + +The single most important framing, and the one that governs everything below: the hub is **not** a +monolithic source of truth. State flows in two directions with opposite drivers. + +- **Operator-driven *intent* — hub authors, agent reconciles (top-down).** Which guests should + exist and their spec, storage *policy* (a target's role/class/backup schedule), controller + + golden-image versions, identity, tunnel. The operator sets these in the hub; the agent converges + toward them. Here the hub *is* the source of truth. +- **Box/customer-driven *reality* — box authors, pushes up, hub mirrors (bottom-up).** Which USB + drive is *physically* attached (and its `durable_id`), what apps are deployed and where, the + customer's controller configs/settings, host/guest health, latest PBS snapshot pointers. The + customer or the physical world drives these; the box reports them; the hub stays an up-to-date + **mirror** but is **never** the driver. + +They meet at a **handshake**, not a tug-of-war. Storage is the clearest case: the customer plugs in +a drive → the agent *detects* it and reports `durable_id X attached` (reality) → the operator +assigns `role=bulk, class=slow, backup=weekly` (policy, intent) → the agent reconciles that policy +*onto the detected drive*. **Apps never enter the reconcile loop** — app deployment is the +controller's domain (customer- or operator-driven, inside the guest); the hub only mirrors the +resulting inventory. **Reconciliation applies to infrastructure; the app/customer layer is mirrored.** + +## 2. Data model (Part 1 decision (b): customer-anchored) + +A customer's deployment is one **Host** (its agent) plus one-or-more **Guests** (its controllers). +1 customer = 1 host + N guests; the shared-host multi-tenant case is deferred (not precluded — the +`hosts` table is the seam it would use). + +- **`customer_configs`** (existing) — the Customer anchor: identity, domain, email, + `retrieval_password`, status, config_json. Unchanged role. +- **`hosts`** (new) — `host_id PK, customer_id, api_key` (the agent's hub key), `agent_version`, + desired-state intent (storage manifest + policies + golden-image version, as JSON), a per-host + **`desired_generation`** counter, the slim DR record (§9), timestamps. +- **`guests`** (new) — `guest_id PK, customer_id, host_id, api_key` (the controller's hub key), + `display_name, controller_version`, per-guest **`desired_spec_json`** (CPU/mem/disk, versions), + timestamps. + +**Per-reporter keys:** today's per-customer `customer_configs.api_key` becomes per-reporter — +`hosts.api_key` (agent) and `guests.api_key` (controller). The hub resolves a presented Bearer key → +host or guest → customer; `customer_configs.api_key` goes unused once auth resolves via the new keys. +**Clean cutover:** no dual-model support; the demo re-enrolls fresh into `host + guests`. + +## 3. Report ingest — two domains + +The single controller report splits. The de-privileged controller no longer sees host disks/storage/ +backup, so its report **slims** (it loses System/Storage/Backup, keeps app-domain). + +- **`POST /api/v1/host-report`** (new, agent) → **`host_reports`**: host CPU/RAM/disk, per-guest + up/down + spec, storage-target status (attached drives + `durable_id` + reachability), last backup + + restore-test per target, latest PBS snapshot pointers, `cloudflared` health, agent + controller + versions. Denormalized columns for the dashboard; full `report_json`. Index `(host_id, received_at + DESC)` + `(customer_id, received_at DESC)`. +- **`POST /api/v1/report`** (existing, slimmed controller) → the renamed **`guest_reports`**: it + gains `guest_id` + `host_id`; its `cpu/memory` denorm now means *guest-level*; `backup_last_snapshot` + goes quiet (backup status lives in `host_reports`). App telemetry / log issues stay. + +These two streams are the bottom-up mirror of §1 — they keep the hub current without a separate push. + +## 4. Liveness / dead-man's-switch + +Evolves the existing staleness checker (60s **cadence**, 30m/1h **thresholds** — OK <30m, down at +2× = >1h; today: controller-report recency → `node_stale`/`down`/`recovered`): + +- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's + liveness signal; a silent agent = the box is gone (the critical alert). +- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster + than waiting for a guest report to go stale. +- **Guest-report recency = secondary** app-level signal. + +**Backup-deadline checker:** today it is *event-based* — it scans for `backup_completed`/`backup_failed` +events since local midnight and alerts if none. Two changes: (1) **mechanism** — move it to a field +check on `host_reports`' last-backup-per-target (cleaner now that backup state arrives in the host +report); (2) **emitter** — the de-privileged controller no longer runs backups, so the **agent** is the +source of the last-backup status (Part 3 §8). Without re-homing the source, the deadline check would go +silent after the controller stops backing up. + +## 5. Desired-state serving + +The operator's **intent** (§1 top-down) lives as JSON on `hosts`/`guests` (storage manifest + +policies + golden version on the host; per-guest spec + versions on the guest) with a per-host +`desired_generation`. The agent pulls its host's desired state on poll (with the generation, so it +reconciles only on change and reports which generation it has converged to). + +- **Benign convergence** (create a guest, attach storage per policy, bump a version, adjust a + non-destructive policy) → the agent reconciles freely. +- **Destructive convergence** (guest removal = destroy, storage detach/wipe, data-losing resize) → + the agent requires a **matching signed op** (§6) before executing that delta; absent/invalid → it + refuses and reports `pending_signature`. + +**Geo is *not* in the agent's desired state** — it's customer→hub→Cloudflare (§7); the agent never +touches WAF. + +## 6. Authorization — signed-op queue + editing flow + +Implements Part 4's gate on the hub side. The hub holds **no signing key**. + +- **`signed_ops`** (new): `op_id, customer_id, host_id, target_guest, op_type, op_blob (canonical + JSON), signature (armored SSHSIG), status (pending_signature → signed → delivered → executed / + failed / expired / rejected), nonce, issued_at, expires_at, executed_at, result`. +- **Editing flow:** the operator edits a customer's desired state, reusing the existing config-form + + diff UX. Note the **transport inverts**: today's "Push" is a hub→box *inbound* POST (forbidden by the + box-initiated model); here "publish" means **write to desired state, delivered on the next agent/ + controller poll**. The form and diff carry over; the push transport does not. The hub diffs vs current + and **classifies each delta** (B1 rule): + - **benign** → published straight to desired state; + - **destructive** → the hub generates the canonical op blob and routes it through signing. +- **Signing hand-off (Part 4 option (b)):** a local operator CLI (`felhom-sign --pending`) fetches + the pending blob from the hub, signs it on the workstation with the dedicated key, and posts the + signature back into `signed_ops`. The hub never sees the key. +- The agent polls `signed_ops` for its host alongside desired state, verifies (Part 4 pipeline), + executes, and reports status → the hub logs to the existing **`events`** audit trail. +- **Classification lives in both places, with different jobs:** the hub classifies at *edit time* + for UX (prompt to sign); the **agent's classification is the authoritative guard** (a compromised + hub could skip the prompt, but the agent still enforces the signature). +- A **pending-ops view** per customer shows the lifecycle (awaiting signature → awaiting agent → + executed). + +## 7. Geo enforcement (Part-2 S4) + +The hub already holds the CF API token and already has a remove-all path +(`internal/web/configs.go` `handleGeoDisable` → `cloudflare.RemoveGeoRules`). **But the token is +dual-purpose today** — DNS-01/ACME *and* WAF/geo — and `configgen.Generate` deep-merges it (via +`config_json`) into the generated `controller.yaml`, so it currently ships **down to the box**. Two +things follow: + +- **ACME assumption (must be stated, not skipped):** in the Cloudflare-Tunnel-default model the edge + terminates TLS, so the box needs no public certificate and the **DNS-01/ACME use of the token goes + away**. Granting that, the token comes fully off the box and lives hub-only. (If any box still does + DNS-01, the token cannot fully come off — so this assumption is load-bearing.) +- **`configgen` must stop emitting `cf_api_token`** into `controller.yaml` (drop it from the merge / + relocate it to a hub-only field). + +The delta: the **customer sets geo in the controller UI → the controller reports the geo desired-state +up → the hub reconciles it into the Cloudflare WAF** (rather than the box calling the CF API). The hub +keeps the remove-all override for self-lockout. The controller no longer calls the CF API. + +## 8. Enrollment (evolution of the existing retrieval-password/config-gen flow) + +Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase) returns a deep-merged +`controller.yaml`. New: + +- Enrollment mints the **agent identity first** (the agent then provisions controllers), pins the + **operator signing public keys** (Part 4 — operational + cold recovery) onto the agent, and the + agent mints each controller's bootstrap (its hub guest key + local-API token). +- A **restore-mode** re-enrollment (§9) hands an existing identity to a fresh agent. + +The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the +agent-first + key-pinning + restore-mode steps. + +## 9. DR model + +The headline: the **old heavy infra-backup push retires** — not because the hub authors everything +(§1 says it doesn't), but because (a) the box-driven mirror already arrives via the §3 report streams, +and (b) the actual app **data + configs live inside the PBS guest snapshot**. So a separate +config+secrets+restic-password infra-backup blob is redundant. + +What remains: +- the **report streams** keep the hub's mirror current (storage layout + `durable_id`s, app inventory, + snapshot pointers) — but this mirror is **convenience, not the DR source of record** (reports are + pruned by age); +- the agent **escrows the recovery-code-wrapped PBS key** to the hub (the one artifact only the box + can produce — zero-knowledge: the hub stores it, cannot open it); +- a **slim DR record** on the `hosts` row (PBS namespace + repo fingerprint + the wrapped escrow key). + These last two are *box-reported* columns on an otherwise operator-intent row — labelled as such so + the §1 two-driver split stays legible per column. + +Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers +hit it) **and** `infra_backups` (the deprecated legacy mirror). The slim DR record folds onto `hosts` +instead. The **controller's infra-backup push is removed** (it's de-privileged). + +**Recovery (host loss):** the new agent re-enrolls in **restore mode**; the hub hands it the durable +record — and DR reads from the **durable sources, not the prunable report mirror**: operator intent +(desired-state on `hosts`/`guests` — identity, tunnel token, storage manifest), the slim DR record +(PBS namespace + repo fingerprint), the **wrapped escrow key**, and **PBS's own snapshot enumeration** +(the agent lists snapshots once it has the namespace + unwrapped key). Guest inventory + app data come +from **inside the PBS guest snapshots**, not from a retained `host_report`, so recovery doesn't degrade +when the last report has aged out. The **customer provides their recovery code at the agent**, which +unwraps the PBS key locally (never sent to the hub); the agent restores guests from PBS, resets +identity, reuses the tunnel. The customer recovery code is the irreducible residual (the premium +operator-managed custody tier avoids it, at the cost of the operator holding the key). The old +controller-targeted `GET /recovery/{id}` is replaced by this agent restore-mode flow. + +## 10. What persists from today (unchanged or lightly adapted) + +The Customer record (`customer_configs`); config generation/retrieval (`configgen`); the two-tier +notification system (operator English / customer Hungarian, Resend, cooldowns); `events` + audit; +`app_telemetry` / `app_log_issues`; customer lifecycle actions (block/unblock, trigger-update, +delete); the asset manager; and the dashboard — adapted to render the **host + guests** view per +customer instead of a single controller. + +## 11. Schema deltas (grounded in store.go's idempotent style; clean cutover) + +- **NEW:** `hosts`, `guests`, `host_reports`, `signed_ops`. +- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data + migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean + guest-level; `backup_last_snapshot` goes quiet. +- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the + slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`. +- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR + record replaces them on `hosts`. +- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`, + `app_telemetry`, `app_log_issues`. +- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth + rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup + push; any that carry over should scope to the resolved host/guest → customer under §2. + +## 12. Open items +- Operator signing-key operational mechanics (Part 4 §8) — the hub-side pending-op UI is here; the + key custody/rotation tooling is Part 4's. +- Multi-tenant resource fairness (deferred shared-host case). +- Hub-side desired-state **editing UX** specifics (form/diff wiring) — to be grounded against + `hub/internal/web/configs.go` at implementation. +- Golden-image refresh cadence / fleet versioning (carried from Part 3 §13). \ No newline at end of file diff --git a/documentation/architecture/_design-review.md b/documentation/architecture/_design-review.md new file mode 100644 index 0000000..91cd684 --- /dev/null +++ b/documentation/architecture/_design-review.md @@ -0,0 +1,260 @@ +# Critical design review — Proxmox re-platform doc set + +> ✅ **RESOLVED (2026-06-08).** All findings folded into 01/02/03 + `proxmox-platform.md` +> (Phase-3 spike run for B2/B3 → `tests/phase3-findings.md`). **Folded:** B1 (03 §4), B2 +> (03 §7/§8 + platform §4.7), B3 (03 §2/§3 + platform §3.6), S1 (03 §6/§8), S2 (03 §10/§11), +> S3 (03 §7), S4 (01 §5/§7 + 02 + 03 §2), S5 (01 §7/§11 + 02 §6), S6 (02 §5), M1 (02 §3), +> M2 (03 §7), M3 (03 §10), §6-residual (03 §6). Plus the two Phase-3 design updates: +> provision-by-restore (03 §9) and the settled root-vs-API boundary (03 §3). **Deferred/none:** +> no finding was deferred; the pre-existing open items (operator signing-key mechanics, +> multi-tenant fairness, hub-side desired-state UX, golden-image refresh cadence) remain +> flagged in 03 §13. This artifact can be deleted once confirmed. + +Working artifact. Review pass over `01-topology-and-trust.md`, `02-controller-module-map.md`, +`03-host-agent.md`, `proxmox-platform.md`, and the Phase 0 / Phase 1-2 findings, grounded +against the v0.33 source (`deploy-felhom-compose/controller/`). Every finding cites a +file+line or a doc section. Severity: **blocking** / **should-fix** / **minor**. + +Two findings are self-corrections of my own earlier work (`02` and `proxmox-platform.md`) — +flagged as such. + +--- + +## Ranked summary + +| # | Severity | Finding | Where | +|---|---|---|---| +| B1 | **blocking** | Reversibility gate contradicts the self-heal reconcile loop — crashed-guest healing can require a signature-gated destroy → reconcile stalls | `03` §4 vs §4(a) | +| B2 | **blocking** | vzdump bulk-exclusion only works for **volume** mount points; Docker **named volumes live in the LXC rootfs and ARE captured** → naive placement silently backs up the 1 TB media drive. Unvalidated by spike. | `03` §7 vs `proxmox-platform.md` §4.3 + pct manpage | +| B3 | **blocking** | Agent's Proxmox role is called "the minimal role from Phase 1" — but that role is the *narrow self-backup* role that Phase 1 proved is **denied** create/allocate/restore. The agent's operator-tier role is undefined. | `03` §2/§3 vs `phase1-2` §1.3-1.4, `01` appendix | +| S1 | should-fix | Quiescing for agent/hub-scheduled backups has **no agent→controller channel** — the local API is controller→agent only | `03` §6, §8 | +| S2 | should-fix | Agent self-update revert authority unspecified — if the new binary won't boot, nothing outside it can flip back | `03` §11 | +| S3 | should-fix | Storage manifest drops fields `settings.StoragePath` carries today (Label, Schedulable/default, StoppedStacks, MigratedTo) with no re-homing stated | `03` §7 vs `settings.go:90-103` | +| S4 | should-fix | Geo-restriction WAF ownership + Cloudflare **API token** placement unspecified after tunnel placement was locked; zone-wide token in a guest is a blast-radius concern | `03` (absent), `01` §3, `config.go` InfrastructureConfig | +| S5 | should-fix | Cross-doc staleness: `01` §11 still lists tunnel placement OPEN; `02` §6 lists geo "blocked on tunnel placement" — both resolved by `03` §13 | `01` §11, `02` §6 vs `03` §13 | +| S6 | should-fix (self-correct) | `02` put self-restore-test **orchestration** in the controller; `03` correctly makes it agent-owned (controller only reads status) | `02` §5(3) vs `03` §6/§8 | +| M1 | minor (self-correct) | `02` §3 lists `UUID` as a `settings.StoragePath` field — it isn't; UUID is derived from fstab at runtime | `02` §3 vs `settings.go:91-103` | +| M2 | minor | `03` §7 says the manifest "absorbs the disk-state fields StoragePath carries today" incl. UUID — UUID isn't persisted today, so the manifest *adds* it (an improvement, not absorption) | `03` §7 | +| M3 | minor | controller-update is not in `03` §10's journaled-ops list, though it's a multi-step async op | `03` §10 vs §11 | + +**Values check: clean.** No DR/key-custody/offboarding path leaves a customer locked out. +Zero-knowledge DR (`03` §8, `01` §8) correctly makes the customer recovery code the +irreducible residual; the operator cannot read data and the box can still restore-test. +No hostage path found. + +**Locked premises:** reviewed for soundness/consistency only; not relitigated. + +--- + +## Blocking findings + +### B1 — The reversibility gate stalls the self-healing reconcile loop +**Where:** `03` §4(a) vs the gate in §4. +**What:** §4(a) lists "redeploy of a crashed controller" as benign convergence that "falls +out of reconciliation for free." The gate then lists **guest destroy** among the +irreversible ops that require an operator signature "*regardless of whether they arrive as a +job or as a desired-state delta*." These collide: if healing a wedged guest requires +destroy+recreate (corrupt rootfs, failed in-place restart, half-built guest from an +interrupted provision), the reconciler hits a signature-gated op and **cannot proceed +without an operator** — the loop either stalls or silently gives up, defeating "self-healing +… tolerant of missed polls." +**Why it matters:** This is the security-critical control model. A fuzzy benign/destructive +line is unimplementable: either the reconciler can destroy (and a compromised hub's desired +state can wipe guests — the exact threat §4 exists to stop), or it can't (and self-heal is a +fiction for the crashed-guest case). +**Grounding:** `03` §4 self-describes the gate as "security-critical"; §9/§10 already rely on +the reconciler rolling back "a half-built guest" — which *is* a destroy of a customer-id-bound +resource, contradicting the blanket "guest destroy needs a signature." +**Suggested fix (crisp, implementable rule):** Scope the reconciler's destructive verbs by +*provenance and data-bearing-ness*, not by verb: +- The reconciler MAY, without a signature: (a) create/start/restart; (b) destroy resources it + **created earlier in the same journaled transaction** (compensating rollback, §10); (c) + destroy resources **tagged ephemeral/scratch** (restore-test scratch guests, §8). +- Destroying or overwriting any resource that **holds the only/primary copy of customer data** + always needs an operator signature. +- **Healing a crashed controller is non-destructive by construction:** the controller is + reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the + LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. State + this explicitly so the two clauses stop colliding. (The v0.33 self-heal precedent is already + in-place restart: `watchdog.go` restarts stopped stacks, it never destroys the guest.) + +### B2 — vzdump bulk-exclusion: the rootfs-Docker-volume trap +**Where:** `03` §7 ("Bulk external mounts are excluded from the guest's vzdump (a per-mount +backup flag)"). +**What:** Two grounded problems: +1. The flag is real but narrow. The pct manpage (verified): `backup=` — + *"Whether to include the mount point in backups (**only used for volume mount points**)."* + It does **not** apply to bind mounts / device mounts (those are handled separately). +2. The trap: `proxmox-platform.md` §4.3 (validated in `phase1-2` §2.2) proved that **Docker + named volumes live inside the LXC rootfs and ARE captured by vzdump** — a sentinel in + `pgdata` survived. The default Felhom app uses Docker named volumes. So unless bulk data is + deliberately placed on a **dedicated Proxmox volume mount point** (backup=0) or a bind + mount, a "bulk" volume will be an ordinary named volume in rootfs and will be **silently + swept into the whole-guest image** — exactly the 1 TB-media-in-every-backup outcome §7 says + it prevents. +**Why it matters:** Backup size/cost and RPO blow up silently; the failure is invisible until +a media drive fills the vzdump target. This is load-bearing for the §8 tier model. +**Grounding:** pct manpage (fetched 2026); `proxmox-platform.md` §4.3; `phase1-2` §2.2. +Not covered by any spike — `proxmox-platform.md` §6 "not yet validated" should gain this row. +**Suggested fix:** Make the placement contract explicit: a `bulk` volume **must** be realized +as a dedicated LXC mount point (volume mountpoint with `backup=0`, or an external bind mount), +**never** a Docker named volume in rootfs. The per-volume placement component (`02` §5(2)) +must enforce this at deploy. Add a Phase-3 spike: create an LXC with a `backup=0` volume +mountpoint + a bind mount, vzdump it, confirm both are excluded and the rootfs+`backup=1` +volume are included. + +### B3 — The agent's Proxmox role is mis-grounded as "the Phase-1 minimal role" +**Where:** `03` §2 ("scoped Proxmox API token (minimal role from Phase 1)"), §3 ("the +Phase-1 minimal role is the API floor"). +**What:** Phase 1's minimal role (`FelhomSelfBackup` = `VM.Audit, VM.Snapshot, VM.Backup, +Datastore.AllocateSpace, Datastore.Audit`) is the **narrow self-backup** role scoped to one +guest, and Phase 1 explicitly proved it is **denied (403)** on create/allocate +(`phase1-2` §1.3 call #7) — i.e. exactly the operator-tier ops the agent's whole job consists +of (provision, restore, storage allocation). Worse, `01` appendix states that guest-side role +"**is not used** — we chose the agent-mediated path." So `03` cites, as the agent's role +floor, a role that (a) the architecture discarded and (b) is provably insufficient for the +agent. +**Why it matters:** The agent's actual operator-tier role is **undefined**. Provisioning, +restore, and storage management cannot be built or hardened against an undefined privilege +set, and §3's root-minimization argument ("the Phase-1 minimal role is the API floor") +collapses because that floor can't create a guest. +**Grounding:** `phase1-2` §1.3 (create CT = 403), §1.4 (role = self-backup only); `01` +appendix ("not used … confirmed restore = operator-tier"); `proxmox-platform.md` §3.4. +**Suggested fix:** Replace the Phase-1 reference with a **new agent operator role** to be +defined and least-privilege-tested in a Phase-3 spike — minimally `VM.Allocate`, `VM.Config.*`, +`VM.PowerMgmt`, `VM.Snapshot(.Rollback)`, `VM.Backup`, `VM.Audit`, `Datastore.Allocate(Space)`, +`Datastore.Audit`, plus whatever storage-attach needs (see S4/root-boundary below). Keep §3's +"API token, not root, where the API suffices" principle — that part is sound — but stop +calling it the Phase-1 role. + +--- + +## Should-fix findings + +### S1 — No agent→controller channel for backup quiescing +**Where:** `03` §6 (local API is controller→agent only) vs §8 ("the controller stops the app +stack … before a guest vzdump where app-consistency matters"). +**What:** App-consistent LXC backup requires the controller to quiesce (no fsfreeze for LXC — +`proxmox-platform.md` §4.2, `phase1-2` §2.1). But the §6 surface is entirely controller→agent; +the box-initiated model forbids the hub calling in, and there is no agent→controller call +defined. For a **hub/agent-scheduled** backup (schedule lives in the manifest `policy`, §7), +the agent has no way to tell the controller "quiesce now." +**Why it matters:** Either scheduled backups silently fall back to crash-consistent (relying +on WAL recovery, which `phase1-2` §3 warns is unvalidated under write load), or the feature +can't be built as drawn. +**Suggested fix:** Make backups **controller-driven for app-consistency**: the controller +learns due/policy via its own hub channel (or a `GET /backup/due` on the local API), quiesces, +calls the existing `POST /backup`, then unquiesces on completion. Document that agent-initiated +vzdump is crash-consistent only. (No inbound-to-guest channel needed — preserves §3/§5.) + +### S2 — Agent self-update revert authority unspecified +**Where:** `03` §11 ("a watchdog reverts to last-good if the new binary fails to come up +healthy"). +**What:** The agent is a single host systemd service with `Restart=always` (§3). If the new +binary crashes on startup, systemd just restarts the **same bad binary** in a loop. "Revert +to last-good" cannot be done *by* the thing that won't boot. §11 doesn't name the actor. +**Why it matters:** A bad self-update can brick the crown-jewel host agent — the one component +that recovers everything else — with no automatic recovery, requiring break-glass. +**Suggested fix:** Put revert authority **outside** the swapped binary: e.g. an A/B symlink +(`current → good|new`) where a separate systemd oneshot health-gate (`ExecStartPost` probe; on +failure flip the symlink back and restart), or a tiny supervisor unit. Boot-into-last-good + +explicit "commit" after a clean health window is the robust pattern. Add agent-update to the +§10 journal so an interrupted swap is resumable. + +### S3 — Manifest schema omits live `StoragePath` fields without re-homing them +**Where:** `03` §7 table vs `settings.go:90-103`. +**What:** Today's `StoragePath` carries `Label`, `IsDefault`, `Schedulable`, `StoppedStacks`, +`Decommissioned`/`DecommissionedAt`/`MigratedTo`. The manifest covers state (attached/ +disconnected/decommissioned) and durable_id, but drops: **Label** (human name, e.g. "Külső +HDD 1TB" — UI), **Schedulable/IsDefault** (default placement target for new apps), +**StoppedStacks** (which apps to restart on reconnect — app-domain), **MigratedTo** (decommission +target pointer). +**Why it matters:** `02` named this manifest as the contract that the `settings.StoragePath` +reshape depends on. Silently dropped fields become lost behavior (no default-drive choice, no +restart-after-reconnect list, no friendly labels). +**Suggested fix:** Either add Label + a placement-default marker to the manifest, or explicitly +state which fields re-home to the controller's `settings` (StoppedStacks and Label are +plausibly controller-side; default/schedulable placement must live wherever placement decisions +are made). Make the split explicit so neither side assumes the other owns it. + +### S4 — Geo-WAF ownership + Cloudflare API token placement unspecified +**Where:** `03` covers `cloudflared` (tunnel) health but is silent on geo-restriction WAF; `02` +§6 had `cloudflare/`+`geo` "blocked on tunnel placement"; `01` §3 lists the controller's creds +as "hub API key + local-API token" only. +**What:** Now that tunnel placement is locked (host), the **geo-restriction WAF** management +(`cloudflare/` package: zone/waf/geosync) still has no home. It requires a Cloudflare **API +token** (`config.go` InfrastructureConfig.cf_api_token) with zone-wide WAF edit rights. If geo +stays in the controller (app-domain, per `02`), a **zone-wide Cloudflare token sits inside the +customer guest** — a real blast-radius concern (compromise → edit/disable WAF for the whole +zone, potentially other customers on the same zone). +**Why it matters:** Trust-boundary gap. `01` §5's boundary table has no row for controller↔ +Cloudflare-API. Unspecified ownership blocks the `02` geo classification from being unblocked. +**Suggested fix:** Decide geo-WAF ownership explicitly and add it to `01` §5. Options: (a) move +WAF management to the **agent/hub** (operator-tier, token off the customer box); (b) keep it in +the controller but scope the CF token per-zone/per-customer if the account model allows. Note +this is now *unblocked* by the tunnel decision and should leave `02` §6's "blocked" state. + +### S5 — Cross-doc staleness on the now-locked tunnel placement +**Where:** `01` §11 ("Cloudflare Tunnel placement: host vs guest (§7)") and `02` §6 +("`cloudflare/` + `api/geo.go` — blocked on tunnel placement") vs `03` §13 ("Resolved here: +tunnel placement (host, agent-managed)") and the LOCKED list. +**What:** `01` and `02` still present as OPEN/blocked a decision `03` and the locked set have +resolved. +**Why it matters:** A dev reading `01`/`02` would treat a settled decision as open (or a +classification as blocked when only geo-ownership, S4, actually remains). +**Suggested fix:** When folding this review in: update `01` §7/§11 to record tunnel=host +(agent-managed systemd service); update `02` §6 to reduce the cloudflare item from "blocked on +tunnel placement" to the narrower "blocked on geo-WAF ownership (S4)." + +### S6 — (self-correction) self-restore-test orchestration belongs to the agent, not the controller +**Where:** `02` §5(3) said "Self-restore-test orchestration — *controller* asks the agent to +restore to scratch guest, validates, reports." `03` §8 makes the **agent** drive it +autonomously; §6 gives the controller only `GET /restore-test/status` (read-only). +**What:** `03` is right and `02` overreached. Zero-knowledge means only the box/agent holds the +PBS key (`03` §8); creating a scratch guest is operator-tier (create/allocate — `phase1-2` +§1.3 #7); the controller cannot do either. The controller's only piece is surfacing status. +**Why it matters:** Keeps the NEW-component list honest — this is not a controller component to +build beyond a status read. +**Suggested fix:** Amend `02` §5(3) to "self-restore-test **status display** (read-only); the +agent owns orchestration." + +--- + +## Minor findings + +- **M1 (self-correction):** `02` §3 lists `UUID` among `settings.StoragePath` fields. It is + **not** there (`settings.go:91-103`: Path, Label, IsDefault, Schedulable, AddedAt, + Disconnected/At, StoppedStacks, Decommissioned/At, MigratedTo). UUID is derived at runtime + from fstab / `/host-dev/disk/by-uuid` by `system.ParseFstabUUID` and `watchdog.go`. The + classification (settings = MODIFY/split) is unaffected; the field list was wrong. +- **M2:** Consequently `03` §7's "absorbs the disk-state fields `settings.StoragePath` carries + today" overstates: `durable_id`/UUID is *not* carried today, so the manifest **adds** durable + identity (a genuine improvement — today the controller re-derives UUID from fstab each boot, + which is fragile). Reword "absorbs" → "absorbs + adds durable_id." +- **M3:** `03` §10 journals "provision, restore" but not **controller-update** (§11), which is + also a multi-step async op (snapshot→pull→redeploy→health→rollback). Add it so an agent crash + mid-controller-update is resume-or-rollback like the others. + +--- + +## Verified-correct (no action) — grounding that held up + +- LXC flags `nesting=1,keyctl=1` + overlayfs (`03` §9) match `proxmox-platform.md` §2.3 / + `phase0` §3. ✓ +- async `task exitstatus`, not POST return (`03` §8) matches `proxmox-platform.md` §3.5. ✓ +- stop-mode backup not requiring `VM.PowerMgmt` (`03` §8 "per Phase 1") matches + `proxmox-platform.md` §3.4. ✓ (applies to the agent role too.) +- running-LXC snapshot on LVM-thin (`03` §6/§8/§11) matches `proxmox-platform.md` §4.5 / + `phase1-2` §1.6. ✓ +- `monitor/pinger.go` deprecation (`02` DELETE-obsolete) confirmed in `main.go:168,175` + ("legacy, will be removed" / "no longer used — monitoring is now handled by the Hub"). ✓ +- backup keep/delete **intra-file tear** (`02` hazard) confirmed: `backup.go` holds both + `RunDBDumps`/`DumpAppVolumes(Safe)` (keep) and `RunBackup`/`RunFullBackup` (restic, delete); + `restore.go` holds `RestoreApp` (restic) + `RestoreAppFromTier2` (app). The §7-8 backup + contract gives the extracted app-data-backup package a coherent destination. ✓ +- Control-plane-not-data-plane (`03` §2/§43): apps keep serving if the agent dies — consistent + with Docker-in-LXC running independently (`phase0` §3). ✓ +- §6 per-guest local-API authorization (token→guest map): sound; a leaked token acts only on + its own guest. Residual: a compromised controller can `POST /rollback` its **own** guest + (blast radius = self) — acceptable per design; worth a one-line note that rollback is + self-scoped and bounded. diff --git a/documentation/architecture/_hub-review.md b/documentation/architecture/_hub-review.md new file mode 100644 index 0000000..b8e91cf --- /dev/null +++ b/documentation/architecture/_hub-review.md @@ -0,0 +1,221 @@ +# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04) + +Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every +cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong / +breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**. + +The two highest-value catches (doc assumes something the code contradicts) are **S1** and **S2**. + +--- + +## Ranked summary + +| # | What | Where (doc → code) | Severity | +|---|---|---|---| +| S1 | §9/§11 name the **wrong infra-backup table as current** — `infra_backup_versions` is the live/primary one; `infra_backups` is the deprecated write-only mirror | 05 §9/§11 → `store.go:198-217,541-578` | should-fix (code-contradiction) | +| S2 | §7 treats the CF token as **geo-only**; it is **dual-purpose (DNS-01/ACME + WAF)** and is injected into the generated `controller.yaml` | 05 §7 → `config_form.html:76-80`, `controller.yaml.default:26`, `configgen.go:28-37`, `configs.go:1041` | should-fix (code-contradiction / unverified assumption) | +| S3 | §6 leans on the existing **"Push"**, but that is a hub→box **inbound** POST — forbidden by the box-initiated model; transport must invert to poll | 05 §6 → `configs.go:569-570,1148-1150`; Part 1 §4/§5/§11; Part 3 §5 | should-fix | +| S4 | Part 1 §6 calls app inventory **"declarative"**; 05 §1 (LOCKED) says apps are mirrored, never declared/reconciled, restored from PBS | Part 1 §6 ↔ 05 §1/§9 | should-fix (cross-doc) | +| S5 | §9 hands "guest inventory + snapshots" **from the prunable report mirror**; DR soundness actually rests on durable sources | 05 §9/§3 → `store.go:809-816` | should-fix (DR robustness) | +| S6 | §4 says backup-deadline checker "maps onto host_reports' last-backup field"; today it is **event-based** and controller-emitted | 05 §4 → `deadline.go:31-86` | should-fix (mechanism) | +| M1 | "60s staleness checker" conflates the 60s **cadence** with the 30m/1h **threshold** | 05 §4 → `main.go:207-217,99-102`, `staleness.go:33-37` | minor | +| M2 | §2 `customer_configs` field list omits `api_key` — the very field the per-reporter plan retires | 05 §2 → `store.go:102-112` | minor | +| M3 | §11 `reports`→`guest_reports` "rename" is really drop+create under the locked clean cutover | 05 §11 → `store.go:55-119` | minor | +| M4 | Pre-existing weak authz on infra-backup GET / `/notify` (any valid key, not customer-scoped) | handler.go:407,536,568,596 | minor | + +No **blocking** findings — the data model and the two-driver framing are sound, and the LOCKED clean +cutover absorbs most schema risk. The items below are gaps/contradictions worth fixing before the doc +drives work. + +--- + +## Highest-value: doc assumes something the code contradicts + +### S1 — `infra_backups` vs `infra_backup_versions` is inverted (should-fix, code-contradiction) +05 §9: *"`infra_backup_versions` retires; `infra_backups` is repurposed into the slim DR record."* +§11 repeats: *"RETIRE `infra_backup_versions`; repurpose `infra_backups`."* + +The code is the other way round: +- `infra_backup_versions` (added v0.7.0, `store.go:198-211`) is the **live/primary** table. **Every read + path hits it**: `GetInfraBackup` (`store.go:565-578`), `GetInfraBackupByID` (`store.go:581-593`), + `GetInfraBackupMeta` (`store.go:604`), `ListInfraBackupVersions` (`store.go:640`), and the recovery + endpoint (`handler.go:670-686`). +- `infra_backups` (original single-row, `store.go:96-100`) is **deprecated**. It is now **written only + as a legacy mirror** ("for backward compatibility during rollback window", `store.go:552-558`) and is + **never read** except as the one-time migration *source* (`store.go:214-217`). + +So the doc proposes retiring the current table and repurposing the dead one. Under the LOCKED clean +cutover both are discarded anyway, so blast radius is low — but an implementer following §9/§11 +literally would point the DR record at the wrong table. +**Fix:** take §11's own alternative — *fold the slim DR record onto `hosts`* and **drop both** +infra-backup tables. If a standalone table is kept, base it on `infra_backup_versions` (the one with the +data/readers), and correct the "which is current" framing. + +### S2 — the CF API token is **not** geo-only; it is the ACME token too, and ships into `controller.yaml` (should-fix, code-contradiction) +05 §7: *"The hub already holds the CF API token (the config form notes Zone WAF:Edit)… rather than +pushing the token down to the controller… The controller no longer calls the CF API."* + +Grounding confirms the hub **does** hold the token and **does** have a remove-all path: +`config_json → infrastructure.cf_api_token` (`configs.go:714-715,1041-1042,1089-1096`) → +`cfClient.RemoveGeoRules(cfToken, cfg.Domain, …)` in `handleGeoDisable` (`configs.go:1112`), route +`/customers/{id}/geo/disable` (`server.go:201-205`). ✓ The §7 framing of geo-enforcement-moves-to-hub +is also consistent with Part 1 §5/§7 and Part 3 §2/§46. + +**But the doc's assumption that the token is *for geo* is contradicted by the code:** the same +`cf_api_token` is **dual-purpose** — +- the config-form hint says **"Zone DNS:Edit (ACME), Zone WAF:Edit (geo)"** (`config_form.html:80`), +- `controller.yaml.default:26` documents it as the **"Cloudflare API token (DNS-01 challenge)"**, +- and it is **deep-merged into the generated `controller.yaml`** via `configgen.Generate` (config_json + overrides, `configgen.go:28-37`), i.e. **today it is shipped down to the box** and served at + `/config/{id}` and `/recovery/{id}`. + +Consequences §7 must address: +1. **"Token off the controller" is incomplete** if the box still does DNS-01/ACME. In the CF-Tunnel + model the box may no longer need a public cert at all (edge-terminated), making the ACME use moot — + but that is an assumption the doc must state, not skip. Either confirm ACME is gone, or the CF token + cannot fully come off the box. +2. **`configgen` must stop emitting `cf_api_token` into `controller.yaml`** (or relocate it to a + hub-only field). As written, the generated config still carries it. + +--- + +## Should-fix + +### S3 — §6 "Push" is an inbound-to-box mechanism the new model forbids +05 §6: *"the operator edits a customer's desired state (building on the existing config-form + +Push/Pull/Diff)."* The form + diff/pull/push handlers exist — `handlePushConfig` (`configs.go:569`), +`handlePullConfig` (`configs.go:952`), `handleConfigDiff` (`configs.go:861`), routes at +`server.go:209-229`. ✓ So the UI base is real. + +The wrinkle: **"Push" today is a hub→controller outbound POST** (`handlePushConfig` "sends the generated +YAML config to the controller", `configs.go:569-570`), as is the geo-disable notify +(`notifyControllerGeoDisable` → `POST controllerURL/api/geo/settings`, `configs.go:1148-1153`). Both are +the hub **connecting into the box** — explicitly disallowed by the box-initiated model (Part 1 §4 +"the hub never initiates inbound"; §5 row `agent↔hub`/`controller↔hub` = outbound poll; Part 3 §5 "The +hub never connects inbound"). 05's own §5 already resolves this (desired state is **pulled** on poll +with a `desired_generation`). So the doc is internally consistent in *mechanism* but loose in *wording*: +**make §6 explicit that "Push" becomes "publish to desired state, delivered on the next agent/controller +poll," not a reuse of the inbound push transport.** The form/diff UX carries over; the transport inverts. +(Same applies to the geo-disable controller-notify path.) + +### S4 — "declarative app inventory" (Part 1 §6) vs "apps are mirrored, never reconciled" (05 §1) +Part 1 §6 lists the durable record as including a **"declarative app inventory"** that survives box loss +— wording that implies an operator-authored, re-deployable spec. 05 §1 (LOCKED two-driver model) is +explicit the opposite way: *"Apps never enter the reconcile loop… the hub only mirrors the resulting +inventory… the app/customer layer is mirrored,"* and 05 §9 restores apps **from the PBS guest snapshot**, +not by re-deploying a declared inventory. These are reconcilable (the mirror *is* durable last-known +truth) but the word "declarative" contradicts the locked framing and the §9 restore-from-snapshot path. +**Fix (align the older doc to the locked model):** in Part 1 §6 change "declarative app inventory" → +"mirrored / last-reported app inventory," and note apps are recovered from the guest snapshot, not +re-declared. (Flagging an internal inconsistency, not relitigating the locked premise.) + +### S5 — §9 reads DR inputs from a prunable mirror; soundness rests on durable sources +05 §9 hands the recovering agent *"identity, tunnel token, storage manifest, PBS namespace, guest +inventory + snapshots."* §3 places "guest inventory" and "latest PBS snapshot pointers" in +`host_reports` — the bottom-up mirror. But reports are **pruned** (`Prune` deletes rows older than +`maxDays`, `store.go:809-816`; the doc keeps this), so after a long pre-DR outage the last `host_report` +can be gone or stale. The actually-durable DR inputs are: desired-state on `hosts`/`guests` (§5), the +slim DR record (PBS namespace + repo fingerprint + wrapped escrow key, §9/§11), and **PBS's own snapshot +enumeration** (the agent lists snapshots once it has the namespace + unwrapped key). The mirrored +inventory/pointers are convenience, not the source of record. +**Fix:** state in §9 that DR reads from the durable sources (desired-state + DR record + PBS), **not** +from prunable `host_reports`, so recovery doesn't degrade when the last report has aged out. This also +keeps §1's two-driver discipline clean: DR must not depend on bottom-up mirror rows being retained. +(Note: the `hosts` row legitimately mixes top-down intent columns with a few box-reported columns — +repo fingerprint, wrapped escrow key. That is fine; just label them as box-reported so the §1 split +stays legible at the column level.) + +### S6 — backup-deadline checker: doc says field-based, code is event-based (and re-emitter changes) +05 §4: *"The existing backup-deadline checker maps onto `host_reports`' last-backup-per-target."* The +existing checker is **event-based**, not field-based: `CheckBackupDeadlines` looks for +`backup_completed` / `backup_failed` (and `db_dump_*`) **events** since Budapest midnight and emits +`expected_backup_missed` if neither is present (`deadline.go:31-86`). Two changes the doc should make +explicit: +1. **Mechanism:** either keep it event-based (someone emits `backup_completed`) or genuinely move it to + a `host_reports.last_backup_per_target` field check — the doc says the latter but the impl is the + former. +2. **Emitter:** today the **controller** emits backup events; in the de-privileged model the **agent** + owns backup/PBS (Part 3 §8), so the agent must now emit `backup_completed`/`backup_failed` (or the + host report carries last-backup-per-target). Without re-homing the emitter, the deadline check goes + silent after the controller stops doing backups. + +--- + +## Minor + +- **M1 — "60s staleness checker" (§4).** 60s is the **check cadence** (`main.go:207-217`, + `ticker := time.NewTicker(60 * time.Second)`); the **staleness threshold** is 30m (default, + `main.go:99-102`) with down at 2× = 60m (`staleness.go:33-37`; CLAUDE.md "OK <30m, DOWN >1h"). The + event-transition mechanism (`node_stale`/`node_down`/`node_recovered`) is described correctly + (`staleness.go:155-185`). Reword to "the staleness checker (60s cadence, 30m/1h thresholds)." +- **M2 — `customer_configs` fields (§2).** The list ("identity, domain, email, retrieval_password, + status, config_json") omits **`api_key`** (`store.go:108`) — the field §2's per-reporter plan + actually retires. Worth noting `customer_configs.api_key` becomes unused once auth resolves via + `hosts.api_key` / `guests.api_key`. +- **M3 — rename under clean cutover (§11).** `migrate()` is all `CREATE TABLE IF NOT EXISTS` + + idempotent `ALTER` (`store.go:55-119,146-149`). §11's claim "grounded in store.go's idempotent style" + is accurate. But a `reports`→`guest_reports` **rename** isn't part of that style; under the LOCKED + clean cutover (demo re-enrolls fresh, §2) it is really **drop `reports` + create `guest_reports`** + with no data migration. Name it as such to avoid implying an in-place rename + backfill. +- **M4 — pre-existing weak authz.** `handleInfraBackupGet`/`Versions` and `handleNotify`/ + `handleSavePreferences`/`handleInfraBackupPush` use `checkAuth` (global **or any** customer key, + `handler.go:63-66`), not customer-scoped `checkAuthCustomer`. Most retire with the infra-backup push + (§9); for any that carry over, the per-reporter model (§2) should scope them to the resolved + host/guest→customer. Not a regression the doc introduces — a cleanup the cutover enables. + +--- + +## Confirmed accurate (grounding that holds — so the rest of the doc can be trusted) + +- **§10 KEEP list** matches the schema exactly: `customer_configs`, `events`, `customer_notifications`, + `notification_log`, `app_telemetry`, `app_log_issues` all present (`store.go:74-189,102-135`). The + asset manager exists (`handler.go:57,834-867`). ✓ +- **§10 two-tier notifications** (operator English / customer Hungarian, Resend, cooldowns) match + `notify/dispatcher.go`: `processOperator` (1h cooldown, `FormatOperatorEmail`, gated by `operatorOn`, + `dispatcher.go:91-114`) + `processCustomer` (prefs-driven, default 6h, `FormatCustomerEmail`, + `dispatcher.go:116-158`); wired in `main.go:134`. ✓ +- **§8 enrollment / §11 configgen** — deep-merge + Hungarian passphrase base is real: + `configgen.deepMerge` (`configgen.go:76-91`), programmatic overrides + `hub.api_key = cfg.APIKey` + (`configgen.go:40-47`), retrieval-password gate (`handler.go:709-753`). The evolution to agent-first + + per-guest keys + key-pinning is a clean extension. ✓ +- **§2 auth extension** (Bearer → reporter → customer) is clean against today's + `checkAuthCustomer` (global key, else `GetCustomerConfigByAPIKey`, `handler.go:72-90`, + `store.go:913-935`); adding host/guest key lookups slots straight in. ✓ +- **§11 "idempotent style"** is accurate (`store.go:55-119`). New tables/columns (`hosts`, `guests`, + `host_reports`, `signed_ops`, `desired_generation`, `desired_spec_json`) follow the existing + `CREATE IF NOT EXISTS` / `ALTER … ` pattern cleanly. +- **§9 escrow/custody** is consistent with Part 1 §8 (three-tier custody, zero-knowledge default, + recovery-code-wrapped PBS keyfile, operator can't open) and Part 3 §8 (live PBS key on the box for + backup + restore-test; hub holds only the wrapped escrow). The "customer recovery code is the + irreducible residual; operator-managed tier avoids it" matches Part 1 §8 verbatim in spirit. ✓ +- **§4 dead-man's-switch** (host-report recency = primary liveness) is consistent with Part 3 §5 + ("the heartbeat *is* the liveness signal… first-class treatment hub-side"). ✓ +- **§5/§6 signed-op + desired-state** are consistent with Part 4 and Part 3 §4: + hub holds **no** signing key and queues opaque blobs (Part 4 §5; 05 §6 "The hub holds no signing + key"); agent runs the verify pipeline and is the authoritative guard (Part 4 §2.3, Part 3 §4; 05 §6 + "the agent's classification is the authoritative guard"); hub classifies at edit-time for UX only. + 05 §6's `signed_ops` columns are a consistent superset of Part 4 §2.1's blob + `{op, target:{host_id,guest_id}, params, nonce, issued_at, expires_at, key_id}` (05 adds hub-side + lifecycle states `delivered`/`rejected` — fine). The local-CLI hand-off (`felhom-sign --pending`) + matches Part 4 §5–6's `Signer`-on-the-workstation model. ✓ + +## Two-driver soundness (axis 3) — holds +No place in 05 has the hub **drive** box/customer-owned state. Desired-state (§5) is all infrastructure +intent (guests, storage *policy*, versions, identity, tunnel) — top-down and legitimate. Apps are +explicitly excluded from reconcile (§1, §5) and mirrored only. Storage is the handshake (detect → +assign policy → reconcile policy onto the detected drive), matching Part 3 §7. The one nuance (S5): the +`hosts` row holds both top-down intent and a few box-reported columns (repo fingerprint, wrapped escrow +key) — acceptable, just label provenance per column. Reconcile (§5) never collides with app/storage +reality because the reality columns (`durable_id` attached, snapshot pointers, app inventory) are +mirror-only and never serve as desired state. + +## DR completeness (axis 4) — safe to retire the heavy push, with S5's clarification +Retiring the controller's infra-backup push is safe **given** that DR reads from durable sources, not +the prunable mirror (S5). What the old push carried — `deployed_stacks` + `disk_layout.mounts` +(`store.go:768-795`, surfaced by `handleRecovery`, `handler.go:620-705`) — is reconstructible: +storage layout/`durable_id`s from the storage manifest (desired-state, durable) + host-report mirror; +app inventory from the guest **inside the PBS snapshot** (so it need not be separately stored); snapshot +list from PBS itself. The one artifact only the box can produce — the recovery-code-wrapped PBS key — is +explicitly escrowed (§9), zero-knowledge, consistent with Part 1 §8 / Part 3 §8. So nothing +DR-essential is lost by removing the push **provided** §9 is amended per S5 to name durable sources and +not lean on `host_reports` retention. diff --git a/documentation/proxmox-platform.md b/documentation/proxmox-platform.md new file mode 100644 index 0000000..8303844 --- /dev/null +++ b/documentation/proxmox-platform.md @@ -0,0 +1,385 @@ +# Proxmox Platform Reference + +Authoritative, living reference for the Proxmox platform underneath `proxmox-controller`. +It records **facts about Proxmox and what we validated about it** — not Felhom design +decisions. Where a design choice exists, this doc points to the (future) controller +architecture document rather than making the choice here. + +**Evidence base** (raw, chronological spike logs — kept as the underlying record): +- [tests/phase0-findings.md](tests/phase0-findings.md) — VM-vs-LXC overhead, Docker-in-LXC viability +- [tests/phase1-2-findings.md](tests/phase1-2-findings.md) — privilege model, backup/restore round-trip +- [tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md](tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md) — **superseded** pre-spike reference (contains a known privsep error; do not cite as authoritative) + +Every nontrivial claim links to its evidence section. Validated on a single host +(`demo-felhom`, 192.168.0.162, 4 vCPU / 16 GB) on 2026-06-07; treat single-run timings and +measurements as indicative, not benchmarks. + +--- + +## 1. Platform baseline + +Validated stack [[phase0 §1](tests/phase0-findings.md)]: + +| Component | Version | +|---|---| +| Proxmox VE (`pve-manager`) | **9.2.2** (`b9984c6d90a4bd80`) | +| OS | Debian 13 (Trixie) | +| Kernel | proxmox-kernel **7.0.2-6-pve** | +| `pve-qemu-kvm` | 11.0.0-3 | +| `qemu-server` | 9.1.15 | +| `pve-container` | 6.1.10 | +| `lxc-pve` / `lxcfs` | 7.0.0-2 / 7.0.0-pve1 | +| `criu` | 4.1.1-1 | + +`pvesh get /version` → release 9.2. Always confirm the node name on the box +(`pvesh get /nodes`) rather than hard-coding it. + +### 1.1 Storage backends +Two backends were present and exercised [[phase0 §1](tests/phase0-findings.md), [phase1-2 §pre-flight](tests/phase1-2-findings.md)]: + +| Storage | Type | Path / VG | Content types | Holds | +|---|---|---|---|---| +| `local` | `dir` | `/var/lib/vz` | `iso, vztmpl, backup, import` | ISOs, CT templates, **vzdump archives** | +| `local-lvm` | `lvmthin` | VG `pve`, thinpool `data` | `rootdir, images` | guest disk volumes | + +**Why backups cannot live on LVM-thin:** LVM-thin is a *block* backend — it allocates +logical volumes for guest disks. Backup archives and templates are *files*, which require a +file-level backend (`dir`, NFS, CIFS, or PBS). A `vzdump` target must therefore be a +storage whose content types include `backup` (here, `local`); pointing `vzdump` at +`local-lvm` is not valid. [[phase1-2 §pre-flight / §2.1](tests/phase1-2-findings.md)] + +### 1.2 Repositories +PVE 9 uses **deb822** `.sources` files under `/etc/apt/sources.list.d/`. For a host +without a subscription, the enterprise repos (`pve-enterprise.sources`, +`ceph-*-enterprise.sources`) must be disabled (they return 401) and a no-subscription repo +enabled. *The spike host arrived with the no-subscription repo already configured and the +host updated [[phase0 baseline](tests/phase0-findings.md)]; the repo setup itself was not a +spike deliverable* — the canonical no-subscription `.sources` is the standard Proxmox 9 +procedure (`/etc/apt/sources.list.d/pve-no-subscription.sources` with +`Components: pve-no-subscription`). Treat the exact commands as standard setup, not +spike-validated. + +**Docker repository (validated):** Docker's official apt repo **has a `trixie` channel**; +no fallback to Debian's `docker.io` was needed. Installed Docker **29.5.3** from it in both +guest types. [[phase0 §1](tests/phase0-findings.md)] + +--- + +## 2. Guest model (LXC vs VM) — validated facts + +Both guest types ran the **identical** workload (Debian 13, Docker 29.5.3, a +postgres/redis/nginx compose stack) under identical resources (2 vCPU, 2048 MB, ~10 GB) +[[phase0](tests/phase0-findings.md)]. + +### 2.1 Isolation characteristic (fact, not recommendation) +- **LXC** is an OS-level container: it **shares the host kernel**. Docker-in-LXC needs the + container configured for nesting (see §2.3). +- **VM** runs its **own guest kernel** under KVM/QEMU, with full hardware-level isolation + and its own firmware. + +The trade-offs below follow directly from this difference. + +### 2.2 Resource overhead (measured) +Host RAM used = `MemTotal − MemAvailable`, deltas vs a both-stopped baseline of 1702 MB; +one guest measured at a time [[phase0 §2](tests/phase0-findings.md)]: + +| Metric | LXC | VM | Note | +|---|---|---|---| +| Idle host-RAM delta | **+211 MB** | **+2056 MB** | structural, see below | +| Under-load host-RAM delta | **+410 MB** | **+2084 MB** | | +| Per-guest attribution | cgroup `memory.current` 1961 MB¹ | KVM RSS ~2031 MB | | +| Idle host CPU used | ~0.3 % | ~6.0 % | VM has an emulation/guest-kernel floor | +| Under-load host CPU used | ~39.4 % | ~53.9 % | VM work shows as `%guest` (31.9 %) | +| pgbench throughput | 2211 tps | 1820 tps | identical load, 0 failed both | +| Disk used (host thin-LV) | ~2.67 GiB | ~2.94 GiB | of 10 GiB allocated | +| Provisioning (create→ready) | ~10–15 s | ~60–75 s | template-extract vs qcow2-import+boot | + +¹ `cgroup memory.current` counts reclaimable page cache shared with the host and +**overstates** the LXC's true incremental cost; the +211 MB host delta is the honest +number [[phase0 §4.4](tests/phase0-findings.md)]. + +**Why the RAM gap is structural** [[phase0 §4.3](tests/phase0-findings.md)]: LXC processes +share the host kernel and page cache, so only the working set counts against the host. A VM +with **no ballooning configured** has KVM back every guest-touched page (including the +guest's own page cache), so its host cost ≈ the full RAM allocation and is largely +load-independent. *Ballooning / KSM were not tested* and could change the VM figure. + +### 2.3 Docker-in-LXC viability (validated) +Docker ran **cleanly in an *unprivileged* LXC** configured with +`--features nesting=1,keyctl=1 --unprivileged 1` (PVE 9 syntax, accepted by `pct create`) +[[phase0 §3](tests/phase0-findings.md)]: + +- `docker run hello-world` → success; full 3-container stack healthy. +- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs` + fallback**. (Docker 29 names the overlay driver `overlayfs` via the containerd + snapshotter image store; same overlay technology as the legacy `overlay2`.) +- Named volume persisted writes; multi-container networking + published port worked + (`curl localhost:8080` → 200); 0 failed transactions under load. +- No privileged-container fallback was needed. + +### 2.4 Guest agent & app-consistency capability +- **VM:** `qemu-guest-agent` installs and reports (`agent: 1`), enabling + `guest-fsfreeze`-based app-consistent `snapshot` backups [[phase0 §4.8](tests/phase0-findings.md)]. + The Debian genericcloud image does **not** ship the agent — it must be installed + in-guest. +- **LXC:** no guest agent exists → **no fsfreeze** (see §4.2). + +--- + +## 3. API & access control + +### 3.1 Fundamentals +- **Base URL:** `https://:8006/api2/json`. Every `pve*` CLI is a thin wrapper over + this REST API. +- **Token auth header:** `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET`. The + secret is shown **once** at creation. Response envelope: `{"data": ...}`. +- **TLS reality:** the host serves the default **self-signed** certificate. `curl` without + `-k` fails `SSL certificate problem: unable to get local issuer certificate` + [[phase1-2 §1.5](tests/phase1-2-findings.md)]. Production trust (pin the PVE CA / install + a real cert) is a separate, not-yet-decided concern. + +### 3.2 RBAC model +An ACL entry is a triple **(path, principal, role)**; a role is a bundle of privileges, +assigned at the most specific path. Paths include `/`, `/vms/`, `/nodes/`, +`/storage/`, `/pool/`, `/access/...`. + +Introspection (**corrected for PVE 9**) [[phase1-2 §1.1](tests/phase1-2-findings.md)]: +- `pveum role list` — lists roles **with their privileges**. +- ⚠️ `pveum role info ` **does not exist in PVE 9** (the old reference used it). +- `pveum acl list`, `pveum user permissions --path `. + +### 3.3 Privilege-separated tokens — the intersection rule (corrected) +> **A privsep token's (`--privsep 1`) effective permissions are the *intersection* of (a) +> the backing user's permissions and (b) the token's own ACLs.** The role must therefore be +> granted on **BOTH the user AND the token** for the same path. Granting it on the token +> only yields an **empty intersection** and a **403 even on self-calls.** +> [[phase1-2 §1.2](tests/phase1-2-findings.md)] + +This corrects the superseded reference (§3 there grants the ACL to the token only). The +intersection is what keeps a privsep token ≤ its user while still being independently +scopeable to a narrow path. + +Working pattern (validated): +```bash +pveum role add -privs " ..." # NB: -privs is space-separated +pveum user add @pve +pveum user token add @pve --privsep 1 # capture SECRET (shown once) +pveum acl modify -user '@pve' -role # BOTH the user... +pveum acl modify -token '@pve!' -role # ...AND the token +``` +`pveum acl delete` **requires `--roles`** (a bare `-user`/`-token` path errors +`400 roles: property is missing`). Deleting the token/user/role auto-invalidates the +referencing ACLs. [[phase1-2 §5](tests/phase1-2-findings.md)] + +### 3.4 Validated minimal self-backup role +A token scoped to **one VMID + the backup datastore** can audit, snapshot, and back up +**only that guest**, and is denied on every other guest and on create/allocate +[[phase1-2 §1.3–1.4](tests/phase1-2-findings.md)]: + +> **Minimal role for self-audit + self-snapshot + both `snapshot`- and `stop`-mode +> self-backup:** +> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit` + +⚠️ **`VM.PowerMgmt` is NOT required for stop-mode backup** — `vzdump` performs the guest +shutdown/restart internally under `VM.Backup` (tested: stop-mode self-backup returned +`exitstatus OK` without it) [[phase1-2 §1.4](tests/phase1-2-findings.md)]. This corrects the +old reference's "likely yes" guess. + +Validated boundary (token scoped to `/vms/` + `/storage/local`): + +| Operation | Result | +|---|---| +| `GET /version` | 200 | +| `GET` self status, `POST` self snapshot, `POST` self vzdump | 200 / task `OK` | +| `GET`/`POST` against **another** guest's vmid | **403** (read) / task **403** (backup) | +| `POST /nodes//lxc` (create/allocate a guest) | **403** — create/allocate is operator-tier | + +### 3.5 Async tasks — trust `exitstatus`, not the POST +Long operations (`vzdump`, `snapshot`, clone, restore) return a **UPID**, not a result. +Poll `GET /nodes//tasks//status` until `status: stopped`, then read +`exitstatus` [[phase1-2 §1.3](tests/phase1-2-findings.md)]. + +> ⚠️ **Authorization can surface at task execution, not at the HTTP POST.** A `vzdump` +> against an unauthorized vmid returns **HTTP 200 + a UPID**, but the task then ends +> `exitstatus: "403 Permission check failed (/vms/, VM.Backup)"` and produces **no +> archive**. A caller that trusts the 200 would wrongly believe the backup ran. Always poll +> the task and check `exitstatus`. + +(The task owner — including a token — can read its own task status: 200.) + +### 3.6 Operator-tier agent role & root-vs-API boundary (validated) +The operator-tier **host agent** (`03-host-agent.md`) needs a far broader role than the +Phase-1 *guest self-backup* role (which is denied create/allocate — §3.4). The minimal role +that drives the full guest lifecycle via an API token, validated by paring +[[phase3 §B3](tests/phase3-findings.md)]: + +> **`FelhomAgent` (operator-tier, 16 privileges):** +> `VM.Allocate, VM.Audit, VM.Config.Disk, VM.Config.CPU, VM.Config.Memory, VM.Config.Network, +> VM.Config.Options, VM.PowerMgmt, VM.Snapshot, VM.Snapshot.Rollback, VM.Backup, +> Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit, Sys.Audit, SDN.Use` +> +> Paring proved: `SDN.Use` is **required** (PVE 9 gates bridge use; omitting it → `403 +> (/sdn/zones/localnetwork/vmbr0, SDN.Use)`); `Sys.Audit` required for host metrics +> (`GET /nodes//status`); `VM.Config.Network`/`VM.Config.Options` required for NIC/onboot +> config; `Datastore.AllocateTemplate` **not** needed (drop it). NB `VM.Config.CPUMemory` is +> not a real privilege — it is `VM.Config.CPU` + `VM.Config.Memory`. + +**Root-vs-API boundary** [[phase3 §B3](tests/phase3-findings.md)] — nearly the entire guest +lifecycle, **including restore**, is API-token-covered; the genuine OS-root residual is narrow: + +| Operation | Coverage | +|---|---| +| Create LXC (nesting-only), config, allocate, start/stop, snapshot/rollback, vzdump, **restore**, destroy, add storage definition, host metrics | **scoped API token** (the `FelhomAgent` role) | +| ⚠️ **Create LXC with `keyctl=1`** (Docker needs it — §2.3) | **OS root `root@pam` only** | +| USB physical mount-by-UUID / systemd mount unit / fstab; SMART/sensors | OS root / narrow sudoers | + +> ⚠️ **`keyctl=1` (and any feature flag except `nesting`) can be set only by an actual +> `root@pam` session** — `changing feature flags (except nesting) is only allowed for +> root@pam`. **No API token qualifies**, not even a non-privsep `root@pam` token (same 403). +> So *fresh provisioning* of a Docker-capable LXC needs `pct create` as OS root (or a narrow +> sudoers entry). **Restore is exempt:** a token-authorized `vzrestore` **preserves +> `keyctl=1`** from the archive — the DR path needs no root. + +--- + +## 4. Backup & restore (`vzdump` / `pct restore`) + +### 4.1 Modes +- **`stop`** — orderly guest shutdown → backup → restart. Highest consistency, defined + downtime. (For LXC the shutdown/restart is internal to `vzdump`; needs only `VM.Backup` — + §3.4.) +- **`snapshot`** — lowest downtime; copies blocks while running. Consistency depends on the + guest cooperating (§4.2). +- **`suspend`** — legacy/compat, not used. + +### 4.2 Consistency: crash-consistent vs quiesced, and no-fsfreeze-for-LXC +> ⚠️ **An LXC has no guest agent, so `snapshot`-mode `vzdump` does NOT fsfreeze.** A +> running-stack LXC backup is therefore **crash-consistent** (filesystem-level), not +> app-consistent. App-consistency for an LXC is the caller's job: quiesce in-guest first +> (stop the stack / flush DBs) or use `stop` mode. A **VM** with `qemu-guest-agent` gets +> `guest-fsfreeze` around the copy → near-free app-consistency. [[phase1-2 §2.1](tests/phase1-2-findings.md), [phase0 §4.8](tests/phase0-findings.md)] + +**Validated restore behaviour** (LXC, Postgres) [[phase1-2 §2.2](tests/phase1-2-findings.md)]: +- **Crash-consistent (running):** on first start Postgres ran **automatic WAL recovery** + (`database system was interrupted … not properly shut down; automatic recovery in + progress … redo done … ready to accept connections`) and the data was intact. +- **Quiesced (stack stopped):** clean start, no recovery, data intact. +- Both restored correctly here on an idle-at-backup DB; this is **not** a durability + guarantee under heavy write load (§6). + +### 4.3 What a backup captures +A single LXC `vzdump` captures the container rootfs **including the Docker named volumes** +(they live in the rootfs) — one backup = the whole guest and its data. Validated: a +sentinel row survived both variants [[phase1-2 §2.2](tests/phase1-2-findings.md)]. + +Sizes/timings (2.5 GiB source, zstd) [[phase1-2 §2.1–2.2](tests/phase1-2-findings.md)]: +backup ~934 MB (~2.7:1) in ~22–25 s; restore in ~11–12 s. + +### 4.4 Restore = recreate-from-archive (identity is preserved) +There is no single "restore" call — you recreate the guest from the archive into a **new +VMID**: +- **LXC:** `pct restore --storage ` +- **VM:** `qmrestore ` (or `POST /nodes//qemu` with `archive=`) + +> ⚠️ **`pct restore` preserves the source config — including the MAC address and +> hostname.** Restoring while the original still runs causes a **MAC/hostname collision** on +> the bridge; reset network identity (`pct set -net0 name=eth0,bridge=vmbr0,ip=dhcp` +> regenerates the MAC) before starting. [[phase1-2 §2.2](tests/phase1-2-findings.md)] + +**Restored config survives intact:** `unprivileged: 1` and `features: nesting=1,keyctl=1` +are preserved, so Docker runs in the restored CT [[phase1-2 §2.2](tests/phase1-2-findings.md)]. + +### 4.5 Snapshots +A **running, unprivileged LXC can be snapshotted on LVM-thin** with no stop required +(`exitstatus OK`; snapshot listed while the CT stays `running`) +[[phase1-2 §1.6](tests/phase1-2-findings.md)]. This is the mechanism available for a +snapshot-before-change rollback flow. + +### 4.6 PBS (Proxmox Backup Server) +**Not yet validated.** No PBS datastore was configured or tested in the spike. All backup +findings above are for `vzdump` to a `dir` storage. PBS (dedup, incremental, remote, dirty- +bitmap) is pending. + +### 4.7 vzdump scope by LXC mount type (validated) +A stop-mode `vzdump` includes/excludes each LXC mount point by **type and the `backup` flag** +[[phase3 §B2](tests/phase3-findings.md)]. Validated three ways (vzdump log, archive grep, +restore): + +| Location | `backup` flag | In the vzdump? | +|---|---|---| +| rootfs (and anything inside it) | — | **included** (always) | +| **Docker named volume** (default driver) | — | **included** — it lives in the rootfs (`/var/lib/docker/volumes//_data`) | +| volume mount point (`mpN`) | `backup=1` | included | +| volume mount point (`mpN`) | `backup=0` | **excluded** (vol recreated empty on restore) | +| bind mount point (`mpN: /host/path`) | n/a | **excluded** ("not a volume"); data is *not* in the archive | + +> ⚠️ **The `backup=` flag is honoured ONLY for *volume* mount points.** A **Docker +> named volume is in the rootfs and is always captured** — so a "bulk" volume left as a +> default named volume is silently swept into the whole-guest image. To keep bulk data **out**, +> realize it as a dedicated `backup=0` volume mount point (proven recipe: +> `pct set -mpN :,mp=/mnt/bulk,backup=0` then +> `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol`). +> A **bind mount's** data is excluded from the archive entirely; on same-host restore it +> reappears only because the bind config re-attaches the same host dir — on a *different* host +> (true DR) it is gone unless backed up separately. + +--- + +## 5. Gotchas & operational notes (quick reference) + +| Gotcha | Detail | Evidence | +|---|---|---| +| **deb822 repos** | PVE 9 repos are `.sources` files; disable enterprise, enable no-subscription | standard setup | +| **Privsep dual-grant** | privsep token needs the role on **both** user and token, else empty intersection → 403 | [phase1-2 §1.2](tests/phase1-2-findings.md) | +| **Async authz** | `vzdump` POST returns 200+UPID even when unauthorized; the 403 is in the task `exitstatus`; poll it | [phase1-2 §1.3](tests/phase1-2-findings.md) | +| **No fsfreeze for LXC** | running-LXC `snapshot` backup is crash-consistent only; quiesce or use `stop` for app-consistency | [phase1-2 §2.1](tests/phase1-2-findings.md) | +| **Restore identity collision** | `pct restore` keeps source MAC + hostname; reset before starting alongside the original | [phase1-2 §2.2](tests/phase1-2-findings.md) | +| **Restart policy for self-heal** | restored/rebooted containers come up `exited` with no restart policy; need a restart policy or an explicit `compose up -d` to return automatically | [phase1-2 §2.2/§3](tests/phase1-2-findings.md) | +| **Self-signed TLS** | host cert is self-signed; `curl` needs `-k` until trust is set up | [phase1-2 §1.5](tests/phase1-2-findings.md) | +| **`pveum role info` gone** | use `pveum role list` in PVE 9 | [phase1-2 §1.1](tests/phase1-2-findings.md) | +| **`pveum acl delete` needs `--roles`** | bare `-user`/`-token` path errors `400 roles: property is missing` | [phase1-2 §5](tests/phase1-2-findings.md) | +| **`VM.PowerMgmt` not needed** | stop-mode backup works under `VM.Backup` alone | [phase1-2 §1.4](tests/phase1-2-findings.md) | +| **`keyctl=1` is root-only** | feature flags except `nesting` need a `root@pam` session; no API token (even root's) can set them; restore preserves them | [phase3 §B3](tests/phase3-findings.md) | +| **`SDN.Use` gates bridge use** | PVE 9 needs `SDN.Use` to attach a NIC to `vmbr0`; omit it → 403 | [phase3 §B3](tests/phase3-findings.md) | +| **Docker named vol = always backed up** | named volumes live in rootfs; only *volume mountpoints* honour `backup=0`; bulk must be a dedicated `backup=0` mp | [phase3 §B2](tests/phase3-findings.md) | + +--- + +## 6. Validated vs open + +### Validated by the spike +| Fact | Evidence | +|---|---| +| PVE 9.2.2 / Debian 13 / kernel 7.0.2 baseline; `local` (dir) vs `local-lvm` (thin) roles | [phase0 §1](tests/phase0-findings.md), [phase1-2 pre-flight](tests/phase1-2-findings.md) | +| Docker runs in an **unprivileged** LXC (`nesting=1,keyctl=1`), driver `overlayfs`, cgroup v2 | [phase0 §3](tests/phase0-findings.md) | +| LXC vs VM overhead (idle host RAM +211 MB vs +2056 MB; CPU/throughput/provisioning) | [phase0 §2](tests/phase0-findings.md) | +| Privsep token = intersection of user ∩ token ACLs (dual-grant required) | [phase1-2 §1.2](tests/phase1-2-findings.md) | +| Minimal self-backup role; `VM.PowerMgmt` unnecessary | [phase1-2 §1.4](tests/phase1-2-findings.md) | +| Token scoped to one VMID: self-ops succeed, cross-guest + create/allocate denied | [phase1-2 §1.3](tests/phase1-2-findings.md) | +| Async UPID model; vzdump authz surfaces in `exitstatus`, not the POST | [phase1-2 §1.3](tests/phase1-2-findings.md) | +| Running, unprivileged LXC snapshots on LVM-thin (no stop) | [phase1-2 §1.6](tests/phase1-2-findings.md) | +| `vzdump` → `pct restore` round-trip; one backup captures Docker volumes; config survives | [phase1-2 §2](tests/phase1-2-findings.md) | +| Crash-consistent restore recovers via Postgres WAL; quiesced restores clean | [phase1-2 §2.2](tests/phase1-2-findings.md) | +| LXC vzdump scope by mount type; `backup=0` excludes volume mps; Docker named vols ride rootfs; proven bulk-exclusion recipe | [phase3 §B2](tests/phase3-findings.md) | +| Operator agent role (16 privs); guest lifecycle incl. restore is API-token-covered; `keyctl` create is `root@pam`-only | [phase3 §B3](tests/phase3-findings.md) | + +### Not yet validated (do not assume) +| Open item | Why it matters | +|---|---| +| **PBS** (dedup/incremental/remote backup) | the only backup path tested was `vzdump` to a `dir` | +| **The real controller running inside an LXC** reaching `host:8006` | spike used `curl`/CLI, not the actual Go controller | +| **App-consistency under heavy write load** | WAL recovery was validated only on an idle-at-backup DB | +| **Live migration / restore to a different host** | single-node spike only | +| **Ballooning / KSM** effect on VM RAM cost | VM RAM measured with neither configured | +| **Cluster / HA** behaviour | single node | +| **Production TLS trust** for the API | all calls used `-k` against a self-signed cert | +| **deb822 no-subscription repo setup** as a controlled step | host arrived pre-configured | + +--- + +## 7. Scope boundary + +This document holds **platform facts only.** Felhom design decisions — e.g. which guest +type is the default, whether to use privsep or non-privsep tokens, where PBS lives — are +**out of scope** and belong in the controller-architecture document. Where this reference +notes a decision exists, the decision itself is recorded there, not here. diff --git a/documentation/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md b/documentation/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md new file mode 100644 index 0000000..bc929c5 --- /dev/null +++ b/documentation/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md @@ -0,0 +1,176 @@ +> ⚠️ **SUPERSEDED — spike evidence only, not authoritative.** This is the *pre-spike* +> reference and contains at least one known error (the privsep/ACL mechanism in §3 — it +> grants the ACL to the token only, which yields an empty intersection and a 403 even on +> self-calls). For the corrected, validated facts read +> [`../proxmox-platform.md`](../proxmox-platform.md). Kept here unchanged as the record of +> what we believed going into the spike. + +# Proxmox Spike — API & Access-Control Reference + +Reference for the **controller-as-guest** architecture, synthesized from current +Proxmox VE 9.x documentation (June 2026). + +Items marked **[confirm on box]** should be verified once PVE is installed — +treat them as Phase 0/1 verification steps, not gospel. Every Proxmox CLI tool +is a thin wrapper over the same REST API, so anything below is reachable from Go. + +--- + +## 1. API fundamentals + +- **Base URL:** `https://192.168.0.162:8006/api2/json` +- **Auth (API token):** HTTP header + `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET` + The secret is shown **once** at creation — capture it immediately, it can't be + retrieved again. +- **Response shape:** `{ "data": ... }`; errors come back via HTTP status + body. +- **Discovery (do this live on the box instead of trusting any doc):** + - `pvesh get /version` + - `pvesh ls /nodes//qemu/` + - Full schema browser: `https://pve.proxmox.com/pve-docs/api-viewer/` + - "What call does the GUI make?" → perform the action in the web UI with + browser DevTools → Network open and read the request. Fastest way to find + the exact endpoint + params for anything. +- **Async tasks:** long operations (backup, restore, clone) return a **UPID** + (task id), not a result. Poll `GET /nodes//tasks//status` until + `status: stopped`, then check `exitstatus`. The controller must poll, not + block. **[confirm on box]** the exact polling/response shape. + +--- + +## 2. RBAC model — (path, principal, role) + +An ACL entry is a triple of **(path, user/group/token, role)**. A role is a +bundle of privileges, assigned at the most specific path possible. + +- **Paths:** `/`, `/vms/`, `/nodes/`, `/storage/`, + `/pool/`, `/access/...` +- **Predefined roles include:** `PVEAuditor` (read-only), `PVEVMUser`, + `PVEVMAdmin`, `PVEDatastoreUser`, `PVEAdmin`, `PVEUserAdmin`. +- **API tokens with privilege separation (`--privsep 1`):** the token's + effective permissions are the **intersection** of (a) the backing user's + permissions and (b) the token's own ACLs. A privsep token can therefore never + exceed its user, and you grant it a separate, minimal ACL. This is exactly the + property the in-guest controller needs. + +Introspection: +```bash +pveum role list +pveum role info PVEVMAdmin +pveum user permissions --path /vms/ +``` + +--- + +## 3. Two-tier privilege model (our architecture decision) + +**Tier A — in-guest controller (customer-facing, NARROW).** +Runs inside the customer's guest. Token scoped to *that guest's own VMID only*: +read its own status/config, snapshot itself, back itself up, write the backup to +the datastore. Cannot see or touch other guests. The LXC/VM's own privilege +level is irrelevant here — reaching `host:8006` is just an HTTPS call + token. + +**Tier B — operator (provisioning, BROAD).** +Creates/destroys guests, builds the golden template, attaches storage, wires PBS. +Lives operator-side (hub / tooling), never on the customer box. + +### Phase 1 runbook — minimal self-backup role + scoped token + +```bash +# 1. Custom least-privilege role: "back up / snapshot myself" +# [confirm on box: exact privilege names via `pveum role list` / api-viewer] +pveum role add FelhomSelfBackup \ + -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" + +# 2. Dedicated API-only user in the PVE realm (no login password) +pveum user add felhom-ctl@pve --comment "In-guest controller (self-backup)" + +# 3. Privsep token for that user (SECRET shown once) +pveum user token add felhom-ctl@pve ctl --privsep 1 + +# 4. Scope the TOKEN to one guest + the backup datastore only +pveum acl modify /vms/ -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup +pveum acl modify /storage/ -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup + +# 5. Test FROM INSIDE the guest +curl -k https://:8006/api2/json/version \ + -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=" + +curl -k -X POST https://:8006/api2/json/nodes//vzdump \ + -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=" \ + -d "vmid=&storage=&mode=snapshot" +``` + +**Pass criteria:** the token backs up its OWN vmid, and returns **403** on any +other vmid. That single result validates the whole controller-as-guest design. + +**Open question to settle here:** does Tier A also need `VM.PowerMgmt` so it can +stop/start its own guest for `stop`-mode backups? Likely yes — add it and re-test. + +--- + +## 4. Backup / restore (vzdump) + +**Modes:** +- **`stop`** — orderly guest shutdown → live backup → resume. Highest + consistency, short defined downtime. +- **`snapshot`** — lowest downtime; copies blocks while running. *Small + inconsistency risk* unless the guest cooperates (see below). +- **`suspend`** — legacy/compat, longer downtime, not recommended. + +**App-consistency — the concrete version of the earlier warning:** +- **VM:** install `qemu-guest-agent` in the guest and set `agent: 1`. + `snapshot`-mode vzdump then calls `guest-fsfreeze-freeze` / `-thaw` around the + copy → near-free filesystem consistency. **This is a real point in the VM's + favour over LXC.** +- **LXC:** no guest agent → no fsfreeze. App-consistency becomes the + *controller's* job: quiesce in-guest first (stop stacks / flush DBs) **then** + vzdump, or use `stop` mode. Same lesson as the restic work, moved to the guest + layer. + +**CLI / API:** +```bash +vzdump --mode snapshot --storage # CLI +# API (async → UPID): +POST /api2/json/nodes//vzdump params: vmid, storage, mode, ... +``` + +**Restore is NOT a single "restore" call** — you recreate the guest from the +archive: +- **VM:** `qmrestore ` / `POST /nodes//qemu` with `archive=...` +- **LXC:** `pct restore ` / `POST /nodes//lxc` with the archive as source + +Phase 2's real-restore test = restore to a **fresh vmid** and boot it. Do not +declare the backup "working" until a restored guest actually runs. + +--- + +## 5. Key REST endpoints (qemu shown; lxc is parallel under `/lxc`) + +``` +GET /nodes +GET /nodes//qemu list VMs +GET /nodes//qemu//status/current live status +GET /nodes//qemu//config config +POST /nodes//qemu//status/{start,stop,shutdown,reboot} +POST /nodes//qemu//snapshot (snapname, description) +GET /nodes//qemu//snapshot list snapshots +POST /nodes//qemu//snapshot//rollback +POST /nodes//vzdump backup (async, UPID) +GET /nodes//tasks//status poll async task +``` + +LXC: replace `/qemu/` with `/lxc/`. For **Docker-in-LXC** the container needs +`features nesting=1,keyctl=1` (`pct set -features nesting=1,keyctl=1`, or +the `features` property on `POST /nodes//lxc`) — **[confirm on box]**. + +--- + +## 6. Phase 0 confirm-on-box checklist + +- [ ] PVE 9.2 installed; storage = LVM-thin (leave free space to also test dir/qcow2) +- [ ] Exact privilege set for `FelhomSelfBackup` (`pveum role info`) +- [ ] UPID task-polling response shape +- [ ] Docker official apt repo has a `trixie` channel +- [ ] LXC `features nesting=1,keyctl=1` syntax + Docker actually runs inside an LXC +- [ ] Baseline idle + under-load RAM/CPU: one Debian VM vs one Debian LXC, identical resources \ No newline at end of file diff --git a/documentation/tests/phase0-findings.md b/documentation/tests/phase0-findings.md new file mode 100644 index 0000000..c8c2c2a --- /dev/null +++ b/documentation/tests/phase0-findings.md @@ -0,0 +1,331 @@ +# Phase 0 — VM vs LXC Overhead Spike: Findings + +**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, Debian 13 (Trixie), +kernel 7.0.2-6-pve, 4 vCPU, 16 GB RAM (15771 MB `MemTotal`). +**Date:** 2026-06-07. **Measured one guest at a time, the other fully stopped.** + +> This document presents **data and observations only**. No recommendation or verdict — +> the architecture decision is made elsewhere. + +--- + +## 1. Provenance + +### Platform +| Component | Version | +|---|---| +| pve-manager | 9.2.2 (`b9984c6d90a4bd80`) | +| kernel | proxmox-kernel 7.0.2-6-pve | +| pve-qemu-kvm | 11.0.0-3 | +| qemu-server | 9.1.15 | +| pve-container | 6.1.10 | +| lxc-pve / lxcfs | 7.0.0-2 / 7.0.0-pve1 | +| criu | 4.1.1-1 | + +`pvesh get /version` → release 9.2, version 9.2.2. + +### Guest images +| | LXC (9001) | VM (9000) | +|---|---|---| +| Source | `local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst` | `debian-13-genericcloud-amd64.qcow2` | +| Build | Debian 13.1 standard CT template (downloaded via `pveam`, checksum verified) | cloud build **20260601-2496**; in-guest reports Debian **13.5** after `apt update` | +| qcow2 | n/a | virtual 3 GiB, on-disk 323 MiB, compat 1.1/zlib | + +### Docker (identical in both guests) +| | LXC | VM | +|---|---|---| +| Source | Docker official apt repo, **`trixie` channel** (confirmed present) | same | +| Version | **29.5.3** build d1c06ef | **29.5.3** build d1c06ef | +| Storage Driver | **`overlayfs`** (not vfs) | **`overlayfs`** (not vfs) | +| Cgroup Version / Driver | **v2 / systemd** | **v2 / systemd** | +| `hello-world` | OK | OK | + +> Docker's official repo **does** have a `trixie` channel — no fallback to Debian's +> `docker.io` was needed. Docker 29 reports the driver as `overlayfs` (the containerd +> snapshotter image store) rather than the legacy name `overlay2`; this is the same +> overlay technology and is **not** a `vfs` fallback. + +--- + +## 2. Comparison table + +Baseline (both guests stopped): host RAM used **median 1702 MB** (range 1699–1703); +host CPU **~0.1 % used** (99.9 % idle). All RAM deltas below are vs this baseline. +Host RAM used = `MemTotal − MemAvailable`, 5 samples ~3 s apart (median reported). + +| Metric | LXC (9001) | VM (9000) | Δ (VM − LXC) | +|---|---|---|---| +| **Idle host-RAM delta** | **+211 MB** (1913) | **+2056 MB** (3758) | **+1845 MB** | +| **Under-load host-RAM delta** | **+410 MB** (2112) | **+2084 MB** (3786) | **+1674 MB** | +| **Per-guest mem attribution** | cgroup `memory.current` = **1961 MB**¹ | KVM process RSS = **2031 MB** (idle) / **2047 MB** (load) | — | +| **Idle host CPU used** | **~0.3 %** (0.20 usr + 0.10 sys) | **~6.0 %** (3.37 usr + 2.31 sys + 0.29 guest) | **+5.7 pp** | +| **Under-load host CPU used** | **~39.4 %** (17.1 usr + 7.5 sys + 14.5 iowait + 0.3 soft) | **~53.9 %** (31.9 guest + 16.4 iowait + 3.4 sys + 1.7 usr + 0.6 soft) | **+14.5 pp** | +| **pgbench throughput** | **2211.7 tps**, lat 1.809 ms, 132 710 tx/60 s, 0 failed | **1819.6 tps**, lat 2.198 ms, 163 764 tx/90 s, 0 failed² | **−392 tps** | +| **Disk allocated** | 10 GiB | 10 GiB | 0 | +| **Disk used (host thin-LV)** | 26.73 % ≈ **2.67 GiB** | 29.33 % ≈ **2.94 GiB** | +0.27 GiB | +| **Disk used (inside guest)** | 2.1 GiB / 9.7 GiB | 2.4 GiB / 9.7 GiB | +0.3 GiB | +| **Provisioning (rough, create→ready)** | ~10–15 s³ | ~60–75 s³ | — | + +¹ `memory.current` counts reclaimable page cache shared with the host and therefore +**overstates** the LXC's true incremental cost; the +211 MB host-RAM delta is the honest +number. ² VM 60 s runs gave 1739 & 1759 tps — consistent with the 90 s definitive run. +³ Guest-creation step only; see §4. Docker install + first image pull (~network-bound, +~identical for both) is excluded. + +### Inside-guest `free -m` (context only — not the decisive number) +| | total | used | buff/cache | available | +|---|---|---|---|---| +| LXC idle | 2048 | 125 | 1851 | 1922 | +| VM idle | 1974 | 509 | 1524 | 1464 | + +The VM sees **1974 MB** usable of 2048 allocated (firmware/kernel reservation). + +--- + +## 3. Docker-in-LXC viability + +**Worked cleanly in an *unprivileged* LXC with `--features nesting=1,keyctl=1`. No +privileged fallback was needed.** + +- `--features nesting=1,keyctl=1 --unprivileged 1` accepted by `pct create` (PVE 9 + syntax confirmed via `pct help create`). +- `docker run hello-world` → success. +- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs` + fallback**. +- Full 3-container stack (`postgres:17`, `redis:7`, `nginx:alpine`) came up healthy. +- Named volume `pgdata` persisted a write (`SELECT count` returned 1 after table + create/insert). +- Multi-container networking + published port worked: `curl localhost:8080` → **HTTP 200**. +- 60 s pgbench load: **0 failed transactions**. + +No errors, no `dmesg`/`journalctl` anomalies, no workarounds. The privileged-LXC +fallback path (step A5) was therefore **not exercised**. + +--- + +## 4. Observations & confounds + +1. **VM under-load CPU required a re-measurement (diagnosed, not hidden).** The first + VM-load sample showed host CPU ~5 % — identical to *idle* — while pgbench nonetheless + completed a full 60 s run (1739 tps). Root cause: the VM load was launched through a + **nested SSH + `nohup &`** layer (host→VM), which started pgbench *after* the sampling + window. The LXC path used local `pct exec` (no nested SSH) so its first sample was + valid. Re-running with pgbench held in the **foreground of a long-lived SSH channel** + (guaranteed active) and sampling during a confirmed window gave the true **53.9 %** + (`%guest`=31.9). **Confound:** the two guests' load was driven through different + plumbing (`pct exec` vs nested SSH); the *throughput* numbers are unaffected + (pgbench self-reports its own duration), but the CPU figures came from + methodologically asymmetric harnesses. +2. **Baseline drift from residual page cache.** After stopping each guest, host RAM did + not snap back to 1702 MB immediately (e.g. 1895 MB just after the LXC stopped; + 1965→1794 MB drifting down after the VM). This is reclaimable cache, not a leak. + Treat all RAM deltas as ±~100 MB. +3. **The headline RAM gap is structural, not incidental.** LXC processes share the host + kernel and page cache, so only the working set counts against the host (+211 MB idle). + The VM, with **no ballooning configured**, has KVM back every guest-touched page — + including the guest's own 1.5 GB page cache — so the host cost ≈ the full 2 GB + allocation (KVM RSS ≈ 2031 MB) and is **largely load-independent** (3758 idle → 3786 + load). Ballooning / KSM were not tested and could change this. +4. **`cgroup memory.current` ≠ host cost.** For the LXC it read 1961 MB (near the 2 GB + limit) because it includes reclaimable page cache; the real incremental host cost was + +211 MB. Per the protocol, `MemTotal − MemAvailable` is the decisive metric. +5. **VM idle CPU floor (~6 %) vs LXC (~0.3 %).** QEMU device emulation + a full guest + kernel's timer/housekeeping impose a small constant CPU cost even at rest. +6. **Throughput vs CPU trade.** The VM did slightly *less* work (1820 vs 2211 tps) for + *more* host CPU (53.9 vs 39.4 %). The extra cost surfaces as `%guest` (31.9 %) — the + actual DB work *plus* virtualization overhead — whereas in the LXC the same DB work + appears directly as host `%usr`/`%sys`. iowait was comparable (~15–16 %, WAL fsync). +7. **Workload fits in RAM.** pgbench scale `-s 10` (~150 MB) fits in cache in both + guests, so the test is commit/CPU-bound rather than disk-bound; a larger-than-RAM + dataset would stress the storage paths differently and is not covered here. +8. **qemu-guest-agent confirmed on the VM** (`qm guest cmd 9000 ping` → OK). This enables + `guest-fsfreeze`-based app-consistent `snapshot`-mode vzdump for the VM — a capability + the LXC has no equivalent for. The genericcloud image does **not** ship the agent; + it had to be installed in-guest (and the VM IP had to be found via `nmap`/MAC until + the agent was up). +9. **Provisioning asymmetry foreshadows cloning.** LXC create is template-extract-bound + (526 MiB at 387 MiB/s + SSH keygen, ~10–15 s). VM create is qcow2-import-bound (3 GiB + → LVM ≈ 30 s) plus a full firmware boot to SSH-ready (~30–45 s). Figures are rough, + single-run, and exclude the shared network-bound Docker install + first image pull. + +--- + +## 5. Raw command log (appendix) + +### 5.1 Provenance +``` +$ pveversion -v | grep ... +pve-manager: 9.2.2 (running version: 9.2.2/b9984c6d90a4bd80) +proxmox-kernel-7.0: 7.0.2-6 +criu: 4.1.1-1 +lxc-pve: 7.0.0-2 +lxcfs: 7.0.0-pve1 +pve-container: 6.1.10 +pve-qemu-kvm: 11.0.0-3 +qemu-server: 9.1.15 + +$ pvesm status +local dir active 98497780 4333576 89114656 4.40% +local-lvm lvmthin active 365760512 0 365760512 0.00% + +# Docker repo trixie channel: +$ curl -fsSL https://download.docker.com/linux/debian/dists/ | grep -oE 'trixie|bookworm|bullseye' +bookworm / bullseye / trixie # trixie present + +# Cloud image: +$ qemu-img info debian-13-genericcloud-amd64.qcow2 +virtual size: 3 GiB ; disk size: 323 MiB ; compat 1.1 ; build 20260601-2496 +``` + +### 5.2 Baseline (both guests stopped) +``` +$ for i in 1..5; awk MemTotal-MemAvailable /proc/meminfo ; sleep 3 +used=1699 MB / 1702 / 1702 / 1702 / 1703 MB (median 1702) + +$ mpstat 1 5 +Average: all 0.05 usr 0.05 sys ... 99.90 idle +``` + +### 5.3 LXC 9001 — create + Docker +``` +$ pct create 9001 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \ + --hostname spike-lxc --cores 2 --memory 2048 --rootfs local-lvm:10 \ + --net0 name=eth0,bridge=vmbr0,ip=dhcp --features nesting=1,keyctl=1 \ + --unprivileged 1 --start 1 + Logical volume "vm-9001-disk-0" created. + extracting archive ... Total bytes read: 551505920 (526MiB, 387MiB/s) + Creating SSH host key ... done +=== exit: 0 ; status: running +features: nesting=1,keyctl=1 ; unprivileged: 1 ; ip 192.168.0.115/24 + +# Docker install (official repo, trixie stable): DOCKER-INSTALL-OK +$ docker --version -> Docker version 29.5.3, build d1c06ef +$ docker run --rm hello-world -> Hello from Docker! +$ docker info | grep -iE 'Storage Driver|Cgroup' + Storage Driver: overlayfs + Cgroup Driver: systemd + Cgroup Version: 2 + Server Version: 29.5.3 ; Kernel: 7.0.2-6-pve ; OS: Debian GNU/Linux 13 (trixie) +``` + +### 5.4 LXC 9001 — stack health +``` +$ docker compose ps +spike-cache-1 running Up +spike-db-1 running Up +spike-web-1 running Up +$ curl -s -o /dev/null -w 'HTTP %{http_code}' localhost:8080 -> HTTP 200 +$ psql CREATE TABLE spike_persist; INSERT; SELECT count(*) -> 1 (volume persists) +``` + +### 5.5 LXC 9001 — idle measurement +``` +Host RAM used (5x3s): 1913 / 1914 / 1913 / 1914 / 1913 MB (median 1913, Δ +211) +cgroup memory.current: 2056036352 B = 1961 MB +inside free -m: total 2048 used 125 buff/cache 1851 available 1922 +mpstat 1 5 Average: 0.20 usr 0.10 sys ... 99.70 idle (~0.3% used) +pct df 9001: rootfs 9.7G size, 2.1G used, 21.6% +``` + +### 5.6 LXC 9001 — under-load measurement +``` +$ pgbench -i -s 10 -> done in 1.39 s +$ pgbench -T 60 -c 4 (run concurrently with sampling): +Host RAM used (5x3s): 2149 / 2143 / 2112 / 2086 / 2071 MB (median 2112, Δ +410) +cgroup memory.current: 2130382848 B = 2032 MB +mpstat 1 5 Average: 17.10 usr 7.50 sys 14.50 iowait 0.31 soft 60.59 idle (~39.4% used) +pgbench result: scaling 10, clients 4, 60 s + transactions: 132710 ; failed 0 (0.000%) + latency average = 1.809 ms ; tps = 2211.713864 +host thin LV vm-9001-disk-0: 10240 MB, Data% 26.73 (≈2.67 GiB) +``` + +### 5.7 VM 9000 — create + cloud-init +``` +$ qm create 9000 --name spike-vm --cores 2 --memory 2048 \ + --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --agent 1 +$ qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2 + transferred 3.0 GiB of 3.0 GiB (100.00%) + scsi0: successfully created disk 'local-lvm:vm-9000-disk-0,size=3G' +$ qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0 +$ qm disk resize 9000 scsi0 10G -> resized 3.00 -> 10.00 GiB +$ qm set 9000 --ciuser spike --cipassword spike --sshkeys /root/spike-pubkey.pub --ipconfig0 ip=dhcp + # pubkey file = the two real keys from the host's /etc/pve/priv/authorized_keys + # (incl. ssh-ed25519 ...kisfenyo@windows — the same workstation key) +$ qm start 9000 -> start-ok +``` + +### 5.8 VM 9000 — IP discovery + guest agent + Docker +``` +# genericcloud has no guest-agent at first boot -> qm guest cmd ping failed. +# IP found via MAC on the bridge: +$ nmap -sn 192.168.0.0/24 | grep -B2 BC:24:11:C7:41:87 + Nmap scan report for 192.168.0.155 ; MAC BC:24:11:C7:41:87 (Proxmox) +$ ssh -i /root/.ssh/id_rsa spike@192.168.0.155 'hostname; cat /etc/debian_version' + spike-vm ; 13.5 +# install qemu-guest-agent + Docker (official repo, trixie): VM-INSTALL-OK +$ qm guest cmd 9000 ping -> AGENT OK (fsfreeze available) +$ docker --version -> Docker version 29.5.3, build d1c06ef +$ docker run --rm hello-world -> Hello from Docker! +$ docker info | grep -iE 'Storage Driver|Cgroup' + Storage Driver: overlayfs ; Cgroup Driver: systemd ; Cgroup Version: 2 +``` + +### 5.9 VM 9000 — stack health +``` +$ docker compose ps -> spike-cache-1 / spike-db-1 / spike-web-1 all running +$ curl ... localhost:8080 -> HTTP 200 +$ psql ... SELECT count(*) -> 1 (volume persists) +``` + +### 5.10 VM 9000 — idle measurement +``` +Host RAM used (5x3s): 3758 / 3757 / 3754 / 3759 / 3758 MB (median 3758, Δ +2056) +KVM process RSS / VSZ: 2079988 / 3380896 KiB (RSS = 2031 MB) +inside free -m: total 1974 used 509 buff/cache 1524 available 1464 +mpstat 1 5 Average: 3.37 usr 2.31 sys 0.29 guest ... 94.04 idle (~6.0% used) +qm config: scsi0 local-lvm:vm-9000-disk-0,size=10G +host thin LV vm-9000-disk-0: 10240 MB, Data% 29.33 (≈2.94 GiB) +inside df -h /: 9.7G size, 2.4G used, 25% +``` + +### 5.11 VM 9000 — under-load measurement (definitive, load confirmed active) +``` +# First attempt (nested-ssh + nohup &) launched pgbench AFTER the sample window -> +# host CPU read a false ~5% (identical to idle). Diagnosed; re-run below holds +# pgbench in the foreground of a long-lived SSH channel and samples during it. + +$ pgbench -T 90 -c 4 (foreground, channel held): + transactions: 163764 ; failed 0 (0.000%) + latency average = 2.198 ms ; tps = 1819.602345 + (60 s confirmation runs: 1739 & 1759 tps) + +# Sampled 10 s into the confirmed-active load: +Host RAM used (5x3s): 3784 / 3786 / 3786 / 3786 / 3786 MB (median 3786, Δ +2084) +KVM process RSS / VSZ: 2096508 / 4495008 KiB (RSS = 2047 MB) +guest uptime: load average 1.71 (2 vCPU) -> vCPUs busy +mpstat 1 8 Average: + 1.70 usr 3.40 sys 16.35 iowait 0.58 soft 31.89 guest 46.08 idle (~53.9% used) +``` + +### 5.12 Teardown state +``` +$ qm list -> 9000 spike-vm stopped +$ pct list -> 9001 spike-lxc stopped +# both present, both stopped (numbers can be re-checked) +``` + +--- + +## 6. Teardown — destroy commands (NOT run) + +Both guests were left **stopped but present**. To remove them: + +```bash +qm destroy 9000 --purge # VM (also removes cloudinit + disks) +pct destroy 9001 --purge # LXC +# optional spike artifacts on the host: +rm -f /var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2 +rm -f /root/spike-pubkey.pub /root/vm-install.sh +# (Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst) +``` diff --git a/documentation/tests/phase1-2-findings.md b/documentation/tests/phase1-2-findings.md new file mode 100644 index 0000000..ff04080 --- /dev/null +++ b/documentation/tests/phase1-2-findings.md @@ -0,0 +1,315 @@ +# Phase 1 + 2 — Privilege Model & Backup/Restore Round-Trip: Findings + +**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via +`pvesh get /nodes` → `demo-felhom`. Storage: `local` (dir, content +`iso,vztmpl,backup,import`), `local-lvm` (LVM-thin, `rootdir,images`). +**Subject:** LXC `9001` (`spike-lxc`, unprivileged, `nesting=1,keyctl=1`, Docker + +postgres/redis/nginx stack). **Date:** 2026-06-07. + +> Data and observations only — **no recommendation or verdict**. + +## Hypotheses — verdicts at a glance +| | Hypothesis | Result | +|---|---|---| +| **H1** | Backup scopes to one VMID; restore/create needs node/pool allocate → denied to narrow token | **CONFIRMED** (create CT = 403) | +| **H2** | An LXC vzdump captures the Docker volumes (they live in the container rootfs) | **CONFIRMED** (sentinel survived both restores) | +| **H3** | Crash-consistent (running) *and* quiesced (stopped) backups both restore cleanly | **CONFIRMED** (A via WAL recovery, B clean start) | +| **H4** | Running unprivileged LXC snapshots on LVM-thin; restored CT keeps unprivileged+nesting/keyctl | **CONFIRMED** (live snapshot OK; config survived) | + +--- + +## 1. Phase 1 — Privilege model + +### 1.1 Setup (operator side, root) +``` +pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" +pveum user add felhom-ctl@pve --comment "spike in-guest controller" +pveum user token add felhom-ctl@pve ctl --privsep 1 # secret: b6547d9d-... (ephemeral, spike-only) +pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup +pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup +``` +Privilege names were verified against `PVEVMAdmin` / `PVEDatastoreUser` via +`pveum role list` first. **Note:** the reference doc's introspection command +`pveum role info ` **does not exist in PVE 9** — only `pveum role list` works. + +### 1.2 ⚠️ Privsep gotcha — the doc's runbook is incomplete +With `--privsep 1`, a token's effective rights are the **intersection of the backing +user's permissions AND the token's own ACLs**. The reference doc (§3) grants ACLs to the +**token only**. With the user `felhom-ctl@pve` holding **no** permissions, the +intersection was **empty** — the first self-audit call returned: +``` +HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"} +``` +**Fix applied:** also grant the user the role on the same paths +(`pveum acl modify /vms/9001 -user felhom-ctl@pve -role FelhomSelfBackup`, same for +`/storage/local`). After that the self-calls succeeded. **A privsep token needs the +permission present on *both* the user and the token** (the token ACL is what keeps the +token ≤ user / narrowly scoped). This must be reflected in the controller provisioning. + +### 1.3 Test matrix (every call run from **inside** the unprivileged LXC, `pct exec 9001`) +`H=192.168.0.162 N=demo-felhom AUTH="PVEAPIToken=felhom-ctl@pve!ctl="` + +| # | Call | Expected | **Actual** | Notes | +|---|---|---|---|---| +| 1 | `GET /version` | 200 | **200** | reachable + auth from inside LXC (no privilege needed) | +| 2 | `GET /nodes/$N/lxc/9001/status/current` | 200 | **200**¹ | self audit (after privsep fix) | +| 3 | `POST /nodes/$N/lxc/9001/snapshot snapname=spk1` | 200/UPID→OK | **200, task exitstatus OK** | **running-LXC self-snapshot (H4)** | +| 4 | `POST /nodes/$N/vzdump vmid=9001 storage=local mode=snapshot` | 200/UPID→OK | **200, task exitstatus OK** | self backup, archive produced | +| 5 | `GET /nodes/$N/qemu/9000/status/current` | 403 | **403** | `Permission check failed (/vms/9000, VM.Audit)` | +| 6 | `POST /nodes/$N/vzdump vmid=9000 storage=local` | 403 | **200 POST → task exitstatus 403**² | see note | +| 7 | `POST /nodes/$N/lxc` (create CT) | 403 | **403** | `Permission check failed` — **proves create/allocate is operator-tier (H1)** | + +¹ before the privsep fix this was 403; see §1.2. +² **Important nuance:** the `vzdump` endpoint accepts the POST and returns a UPID even for +an unauthorized vmid; the authorization failure surfaces at **task execution**, not at the +HTTP layer. Polled from root: +`exitstatus: "403 Permission check failed (/vms/9000, VM.Backup)"`, and **no 9000 archive +was created**. The boundary holds — but a controller must **poll the task exitstatus**, not +trust the POST's 200, to know a cross-guest backup was actually refused. + +**Pass criteria met:** self-ops (1–4) succeed; cross-guest read (5), cross-guest backup +(6, at task level), and create/allocate (7) are denied. The controller-as-guest boundary +and the two-tier split are validated. + +### 1.4 Final minimal role — `VM.PowerMgmt` **not** required +The doc's open question ("does Tier A need `VM.PowerMgmt` for stop-mode backups? Likely +yes"). **Tested and refuted:** a **stop-mode** self-vzdump submitted by the token +(`vmid=9001 mode=stop`) completed with **`exitstatus: OK`** using the role *without* +`VM.PowerMgmt`. `vzdump` performs the guest shutdown/restart internally under +`VM.Backup`; no separate power privilege is needed. + +> **Final minimal role (`FelhomSelfBackup`) — satisfies self-audit, self-snapshot, and +> both `snapshot`- and `stop`-mode self-backup:** +> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit` +> (`VM.PowerMgmt` deliberately omitted — confirmed unnecessary.) + +### 1.5 TLS observation +From inside the LXC, `curl` **without** `-k`: +``` +curl: (60) SSL certificate problem: unable to get local issuer certificate +``` +The host serves the default self-signed PVE cert; all tests used `-k`. Production trust +(pin the PVE CA / issue a proper cert) is a separate design decision, flagged here. + +### 1.6 Running-LXC snapshot (H4) +Call #3 snapshotted the **running** unprivileged LXC on LVM-thin (`exitstatus OK`). +`pct listsnapshot 9001` shows `spk1` with `pct status 9001 = running`. **No stop +required** — the snapshot-before-update rollback flow is viable on a live container. + +--- + +## 2. Phase 2 — Backup → real restore round-trip + +Sentinel written pre-flight into the `pgdata` volume: +`restore_check(42,'phase2-sentinel')` → clean read `42|phase2-sentinel`. + +### 2.1 Backups (operator/root side) +| Variant | Mode | Stack state | Task time | Wall | Archive | Size (zstd) | +|---|---|---|---|---|---|---| +| **A — crash-consistent** | `snapshot` | **running** | 00:00:24 | 25 s | `vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst` | **934 MB** (979,718,569 B) | +| **B — quiesced** | `snapshot` | **stopped** (`docker compose stop`) | 00:00:21 | 22 s | `vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst` | **934 MB** (979,671,582 B) | + +Both from a 2.5 GiB source; zstd → ~934 MB (~2.7:1). The stack was restarted after +Variant B. **LXC snapshot-mode vzdump does *not* fsfreeze** (no guest agent in an LXC — +consistent with the Phase 0 finding) → Variant A is genuinely crash-consistent. + +### 2.2 Restore → fresh VMID → boot → verify +| Check | 9002 (Variant A) | 9003 (Variant B) | +|---|---|---| +| Restore time (`pct restore … --storage local-lvm`) | **12 s** | **11 s** | +| `unprivileged: 1` survived | **yes** | **yes** | +| `features: nesting=1,keyctl=1` survived | **yes** | **yes** | +| Containers after boot | `exited` (no restart policy) → `docker compose up -d` | same | +| 3 containers healthy | **yes** | **yes** | +| `curl localhost:8080` | **HTTP 200** | **HTTP 200** | +| **Sentinel `(42,'phase2-sentinel')`** | **PRESENT** | **PRESENT** | +| Postgres first-start | **WAL crash recovery** (see below) | **clean start, no recovery** | + +> Restored CTs inherit 9001's fixed `hwaddr`. To avoid a MAC clash with the still-running +> 9001 on `vmbr0`, `net0` was reset to auto-generate a fresh MAC before boot. All +> verification (stack health, `curl localhost`, sentinel) is guest-internal and needs no +> external network — and the Docker images are inside the restored rootfs, so no pulls. + +**Variant A — Postgres automatic WAL recovery on 9002 (verbatim, post-restore boot):** +``` +LOG: database system was interrupted; last known up at 2026-06-07 18:13:21 UTC +LOG: database system was not properly shut down; automatic recovery in progress +LOG: redo starts at 0/CB12838 +LOG: invalid record length at 0/CB12870: expected at least 24, got 0 # normal end-of-WAL +LOG: redo done at 0/CB12838 ... +LOG: checkpoint starting: end-of-recovery immediate wait +LOG: database system is ready to accept connections +``` +**Variant B — clean start on 9003 (verbatim, post-restore boot):** +``` +LOG: database system was shut down at 2026-06-07 18:14:39 UTC +LOG: database system is ready to accept connections +``` + +**H2 confirmed:** one LXC vzdump captured the whole customer including the Docker named +volume — the sentinel data restored in both guests. **H3 confirmed:** both variants +restored to a bootable guest with intact data; the crash-consistent one recovered via WAL +with no manual intervention, the quiesced one started clean. **H4 confirmed:** restored +config preserved `unprivileged` + `nesting/keyctl`, so Docker ran in the restored CT. + +--- + +## 3. Observations & confounds +1. **Privsep token needs perms on user *and* token** (§1.2) — the single most important + correction to the reference runbook; without it every scoped call 403s. +2. **vzdump authorization is task-level, not POST-level** (§1.3 note ²) — a 200 + UPID + does **not** mean authorized. The controller must poll `exitstatus`. This is also the + general async-task lesson: every backup/snapshot/restore returns a UPID and the real + result is in the task status. +3. **`pveum role info` is gone in PVE 9** — use `pveum role list`. Minor doc drift. +4. **`VM.PowerMgmt` not needed for stop-mode backup** (§1.4) — narrower role than the doc + assumed. +5. **No fsfreeze for LXC** — Variant A relied on Postgres's own WAL crash recovery, which + worked here for an idle-at-backup DB. Under heavy write load, app-consistency for LXC + still rests on the controller quiescing first (or stop-mode), exactly as the reference + warned. This single test is not a durability guarantee under load. +6. **Restore MAC collision** (§2.2) — `pct restore` preserves the source `hwaddr`; + restoring while the original runs needs a MAC reset (or the original stopped). The + controller's restore flow must handle identity (MAC/hostname/IP) to avoid clashes. +7. **No restart policy on the compose services** — restored containers came up `exited`; + `docker compose up -d` (or a restart policy / systemd unit) is required for the stack + to return automatically after a restore or guest reboot. +8. **Restore is fast, backup dominated by I/O** — restores were 11–12 s (extract at + ~524 MiB/s); backups ~22–25 s (read 2.5 GiB at ~108–119 MiB/s + zstd). Single runs, + idle host, ~150 MB DB; not a throughput benchmark. +9. **Sequencing artifact:** a Phase-1 stop-mode self-backup ran before Phase 2 and + stopped/started 9001; the stack was brought back up and the sentinel re-verified + before the Variant A/B backups, so it does not affect the round-trip results. + +--- + +## 4. Raw command log (appendix) + +### 4.1 Pre-flight +``` +$ pvesh get /nodes -> node: demo-felhom +$ cat /etc/pve/storage.cfg +dir: local ... content iso,vztmpl,backup,import # 'backup' present +lvmthin: local-lvm ... content rootdir,images # no backup (expected) +$ pct start 9001 ; docker compose up -d -> 3 containers Started +$ curl localhost:8080 -> HTTP 200 +# sentinel: +CREATE TABLE ; INSERT 0 1 ; SELECT count -> 1 ; SELECT * -> 42 | phase2-sentinel +``` + +### 4.2 Phase 1 — role/user/token/ACL +``` +$ pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" -> role-ok +$ pveum user add felhom-ctl@pve --comment "spike in-guest controller" -> user-ok +$ pveum user token add felhom-ctl@pve ctl --privsep 1 + {"full-tokenid":"felhom-ctl@pve!ctl","info":{"privsep":"1"},"value":"b6547d9d-08ec-4f22-beb8-a551dc2cd69d"} +$ pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok +$ pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok +$ pveum role list | grep FelhomSelfBackup + FelhomSelfBackup | Datastore.AllocateSpace,Datastore.Audit,VM.Audit,VM.Backup,VM.Snapshot +$ pveum role info FelhomSelfBackup -> ERROR: unknown command 'pveum role info' # PVE9 has no 'role info' +``` + +### 4.3 Phase 1 — matrix (from inside LXC) +``` +# TLS without -k: +curl: (60) SSL certificate problem: unable to get local issuer certificate + +# BEFORE privsep fix: +#2 GET self status -> HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"} + +# privsep fix: +$ pveum acl modify /vms/9001 -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok +$ pveum acl modify /storage/local -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok + +# AFTER fix: +#1 GET /version -> HTTP 200 +#2 GET /nodes/.../lxc/9001/status/current -> HTTP 200 {"data":{...,"status":"running",...}} +#5 GET /nodes/.../qemu/9000/status/current -> HTTP 403 (/vms/9000, VM.Audit) +#6 POST vzdump vmid=9000 -> HTTP 200 {"data":"UPID:...vzdump:9000:felhom-ctl@pve!ctl:"} + root poll: exitstatus="403 Permission check failed (/vms/9000, VM.Backup)" + task log: TASK ERROR: 403 Permission check failed (/vms/9000, VM.Backup) + /var/lib/vz/dump: no 9000 archive created +#7 POST /nodes/.../lxc (create CT vmid=9009) -> HTTP 403 {"message":"Permission check failed\n"} + +#3 POST lxc/9001/snapshot snapname=spk1 -> HTTP 200 UPID:...vzsnapshot:9001... + root: exitstatus "OK" ; pct listsnapshot 9001 -> spk1 ; pct status 9001 -> running +#4 POST vzdump vmid=9001 storage=local mode=snapshot -> HTTP 200 UPID:...vzdump:9001... + root: exitstatus "OK" + token can read own task status: HTTP 200 {"...exitstatus":"OK"} # earlier poll TIMEOUTs were a shell-quoting bug in the helper, not a perms issue + +# stop-mode self-backup (VM.PowerMgmt test): +$ token POST vzdump vmid=9001 storage=local mode=stop -> HTTP 200 UPID:...vzdump:9001... + root poll: exitstatus "OK" # SUCCEEDED without VM.PowerMgmt in the role +``` + +### 4.4 Phase 2 — backups +``` +# Variant A (running): +$ vzdump 9001 --mode snapshot --storage local --compress zstd +INFO: Total bytes written: 2585589760 (2.5GiB, 108MiB/s) +INFO: archive file size: 934MB +INFO: Finished Backup of VM 9001 (00:00:24) ; WALL_SECONDS=25 +-> vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst (979718569 B) + +# Variant B (stopped): +$ docker compose stop (cache,db,web Stopped) +$ vzdump 9001 --mode snapshot --storage local --compress zstd +INFO: Total bytes written: 2585825280 (2.5GiB, 119MiB/s) +INFO: Finished Backup of VM 9001 (00:00:21) ; WALL_SECONDS=22 +-> vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst (979671582 B) +$ docker compose start (db,cache,web Started) +``` + +### 4.5 Phase 2 — restores + verification +``` +# A -> 9002: +$ pct restore 9002 .../20_13_43.tar.zst --storage local-lvm + Total bytes read: 2585589760 (2.5GiB, 524MiB/s) ; RESTORE_A_SECONDS=12 +$ pct config 9002 -> features: nesting=1,keyctl=1 ; unprivileged: 1 +$ pct set 9002 -net0 name=eth0,bridge=vmbr0,ip=dhcp # fresh MAC BC:24:11:E3:F4:64 +$ pct start 9002 ; docker compose up -d -> 3 running ; curl -> HTTP 200 +$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel + db log: "was interrupted ... not properly shut down; automatic recovery in progress + redo starts/redo done ... database system is ready to accept connections" + +# B -> 9003: +$ pct restore 9003 .../20_14_40.tar.zst --storage local-lvm + Total bytes read: 2585825280 (2.5GiB, 524MiB/s) ; RESTORE_B_SECONDS=11 +$ pct config 9003 -> features: nesting=1,keyctl=1 ; unprivileged: 1 +$ pct set 9003 -net0 ... (fresh MAC) ; pct start 9003 ; docker compose up -d -> 3 running ; curl 200 +$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel + db log: "database system was shut down at ... ; database system is ready to accept connections" # clean +``` + +--- + +## 5. Teardown (executed) +Restore targets destroyed; Phase 1 objects and spike artifacts removed; `9000`/`9001` +left **stopped-but-present**. Verified clean: `felhom-ctl@pve` deleted, no spike ACLs, +empty `dump/`, `spk1` removed. + +> **Correction:** `pveum acl delete` **requires `--roles`** (a bare `-user`/`-token` +> path errors `400 roles: property is missing`). In practice the explicit ACL deletes +> are unnecessary — deleting the token/user/role **auto-invalidates** the referencing +> ACLs (PVE logs `ignore invalid acl token …` and drops them). + +```bash +pct stop 9002 ; pct stop 9003 ; pct destroy 9002 --purge ; pct destroy 9003 --purge +# correct ACL-delete syntax (needs --roles), or just let user/role deletion clean them: +pveum acl delete /vms/9001 --roles FelhomSelfBackup --users 'felhom-ctl@pve' +pveum acl delete /vms/9001 --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl' +pveum acl delete /storage/local --roles FelhomSelfBackup --users 'felhom-ctl@pve' +pveum acl delete /storage/local --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl' +pveum user token remove felhom-ctl@pve ctl ; pveum user delete felhom-ctl@pve ; pveum role delete FelhomSelfBackup +pct delsnapshot 9001 spk1 +rm -f /var/lib/vz/dump/vzdump-lxc-9001-*.tar.zst /var/lib/vz/dump/vzdump-lxc-9001-*.log +pct stop 9001 # back to stopped-but-present +``` + +## 6. To destroy 9000/9001 later (NOT run — left stopped-but-present) +```bash +qm destroy 9000 --purge # VM (Phase 0 subject) +pct destroy 9001 --purge # LXC (Phase 0/1/2 subject) +# Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst +``` diff --git a/documentation/tests/phase3-findings.md b/documentation/tests/phase3-findings.md new file mode 100644 index 0000000..4e61b35 --- /dev/null +++ b/documentation/tests/phase3-findings.md @@ -0,0 +1,234 @@ +# Phase 3 — vzdump exclusion (B2) & agent operator role + root boundary (B3): Findings + +**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via +`pvesh get /nodes` → `demo-felhom`. **Date:** 2026-06-08. Throwaway resources (VMIDs +9010-9023, role/user `FelhomAgent`/`felhom-agent@pve`); all torn down (only the pre-existing +9000/9001 remain, stopped). Every Proxmox op polled to `task exitstatus` (not the POST +return). + +> Validates the two items the design review (`_design-review.md`) flagged as unvalidated: +> **B2** (what vzdump includes/excludes per LXC mount type + how to keep bulk out) and **B3** +> (the least-privilege operator role + the root-vs-API boundary). Data only. + +--- + +## B2 — vzdump inclusion/exclusion matrix + +**Setup:** one unprivileged LXC `9010` (`nesting=1,keyctl=1`, overlayfs), Docker 29.5.3 +installed, with five sentinel locations: + +| # | location | config | +|---|---|---| +| 1 | rootfs file `/SENTINEL_ROOTFS` | rootfs (`local-lvm:8`) | +| 2 | Docker **named** volume `b2vol` → `SENTINEL_DOCKERVOL` | default driver | +| 3 | `mp1` volume mount `/mnt/mp1` `SENTINEL_MP1` | `local-lvm:1,backup=1` | +| 4 | `mp2` volume mount `/mnt/mp2` `SENTINEL_MP2` | `local-lvm:1,backup=0` | +| 5 | `mp3` **bind** mount `/mnt/mp3` `SENTINEL_MP3` | host `/root/b2-bindsrc` | +| 6 | bulk Docker vol `bulkvol` bound onto mp2 → `SENTINEL_BULK` | `--driver local -o type=none -o o=bind -o device=/mnt/mp2` | + +**The "trap" confirmed at setup:** the Docker named volume's on-disk path is +`/var/lib/docker/volumes/b2vol/_data` — **inside the LXC rootfs**. + +### Result matrix (stop-mode vzdump → `local`, verified 3 ways: vzdump log, archive grep, restore to 9011) + +| Sentinel | location | flag | **in archive?** | restored 9011 | +|---|---|---|---|---| +| `SENTINEL_ROOTFS` | rootfs | — | **INCLUDED** | present | +| `SENTINEL_DOCKERVOL` | Docker named vol (in rootfs) | — | **INCLUDED** ⚠️ the trap | present | +| `SENTINEL_MP1` | volume mp | `backup=1` | **INCLUDED** | present | +| `SENTINEL_MP2` | volume mp | `backup=0` | **EXCLUDED** | absent (vol recreated empty) | +| `SENTINEL_MP3` | bind mount | n/a | **EXCLUDED** | reappears via re-bind only¹ | +| `SENTINEL_BULK` | Docker vol on mp2 | `backup=0` | **EXCLUDED** | absent | + +¹ The bind-mount **data is not in the archive** (archive grep shows no mp3 path). It +reappears in the restored 9011 only because `pct restore` preserves the bind config +`mp3: /root/b2-bindsrc` and re-attaches the **same host dir**. On a *different* host (true DR) +the bind data would be gone unless backed up separately — important for DR planning. + +**vzdump log (verbatim) — the authoritative per-mount decision:** +``` +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp1 ('/mnt/mp1') in backup +INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled) +INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume) +``` +**Archive contents (verbatim) — `tar --zstd -tf … | grep SENTINEL`:** +``` +./var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL +./SENTINEL_ROOTFS +./mnt/mp1/SENTINEL_MP1 +``` +**Restore verification (verbatim) — sentinels in restored 9011:** +``` +PRESENT : /SENTINEL_ROOTFS +PRESENT : /var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL +PRESENT : /mnt/mp1/SENTINEL_MP1 +ABSENT : /mnt/mp2/SENTINEL_MP2 +ABSENT : /mnt/mp2/SENTINEL_BULK +PRESENT : /mnt/mp3/SENTINEL_MP3 # via re-bind to same host dir, NOT from archive +``` + +### Proven bulk-exclusion recipe +A "bulk" Docker volume is kept out of the guest vzdump by binding it onto a **volume +mountpoint with `backup=0`**: +1. Attach a Proxmox volume mountpoint with the flag: + `pct set -mpN :,mp=/mnt/bulk,backup=0` +2. Realize the Docker volume on that path: + `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol` + (or a compose bind to `/mnt/bulk`). +3. Data written through `bulkvol` lands on the `backup=0` mountpoint → **excluded** from + vzdump, while rootfs/hot sentinels are **included**. Verified: `SENTINEL_BULK` absent from + archive and restore; `SENTINEL_ROOTFS` present. + +### The trap, stated for the placement component +`backup=` is **only honoured for volume mount points** (confirmed: pct manpage + +vzdump log "excluding volume mount point … (disabled)"). A Docker **named volume uses the +default driver and lands in the rootfs**, which is **always backed up** — so a "bulk" volume +left as an ordinary named volume is **silently swept into the whole-guest image**. The +per-volume placement component **must** realize every `bulk` volume as a dedicated `backup=0` +mountpoint (or external bind mount), never a default named volume. + +--- + +## B3 — agent operator role + root-vs-API boundary + +**Caveat applied (Phase 1):** privsep token needs the role on **both** user and token. Setup: +user `felhom-agent@pve` + privsep token `agent`, role `FelhomAgent`, dual-granted at `/`. +All ops driven **as the token** via the REST API; task `exitstatus` polled. + +> ⚠️ **Terminology:** the Phase-1 `FelhomSelfBackup` role is the discarded **guest-side +> self-backup** role (scoped to one guest, *denied* create/allocate). `FelhomAgent` here is +> its **operator-tier replacement** — a different, broader role. Do not conflate. + +### Op matrix (as the scoped token) + +| # | Operation | API call | Result | +|---|---|---|---| +| read | host status | `GET /nodes/$N/status` | **200** (needs `Sys.Audit`) | +| read | storage list | `GET /storage` | **200** (`Datastore.Audit`) | +| 1 | **create LXC, `nesting=1,keyctl=1`** | `POST /nodes/$N/lxc` | **403** — `changing feature flags (except nesting) is only allowed for root@pam` | +| 1′ | create LXC, **nesting-only** | `POST /nodes/$N/lxc` | **200 / OK** | +| 2 | set config (mem/cpu/options + mountpoint w/ `backup` flag) | `PUT /nodes/$N/lxc//config` | **200** | +| 3 | allocate volume | `POST /nodes/$N/storage/local-lvm/content` | **200** (`Datastore.AllocateSpace`) | +| 4 | start | `POST …/status/start` | **OK** (`VM.PowerMgmt`) | +| 5 | stop | `POST …/status/stop` | **OK** | +| 6a | snapshot | `POST …/snapshot` | **OK** (`VM.Snapshot`) | +| 6b | rollback | `POST …/snapshot/s1/rollback` | **OK** (`VM.Snapshot.Rollback`) | +| 7 | stop-mode backup | `POST /nodes/$N/vzdump mode=stop` | **OK** (`VM.Backup`) | +| 8 | restore → fresh vmid | `POST /nodes/$N/lxc restore=1` | **OK** — and **restored CT kept `features: nesting=1,keyctl=1`** | +| 9 | destroy CT | `DELETE /nodes/$N/lxc/?purge=1` | **OK** (`VM.Allocate`) | +| 9b | add storage definition (dir) | `POST /storage` | **200** (`Datastore.Allocate`, **no root**) | + +**The two headline results:** +1. **`keyctl=1` on create is `root@pam`-only.** Verbatim: + `Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)`. + Confirmed this is **not** token-fixable: a **non-privsep `root@pam` token** got the **same + 403**. Only an actual `root@pam` session (OS root / `pct create` as root) can set it. + `nesting` alone is allowed for a scoped token. +2. **Restore preserves `keyctl`.** A token-authorized `vzrestore` of a keyctl archive produced + `9021` with `features: nesting=1,keyctl=1, unprivileged: 1`. So the **DR/restore path is + fully token-covered**; only *fresh provisioning* needs root for the keyctl flag. + +### Paring (each drop shown to still pass, or proven needed) + +| Privilege | Verdict | Evidence | +|---|---|---| +| `Datastore.AllocateTemplate` | **DROP** (unnecessary) | create-from-template succeeded without it (200/OK) | +| `Sys.Audit` | **KEEP** | `GET /nodes/$N/status` → **403** without it (host metrics, `03` §5) | +| `VM.Config.Network` | **KEEP** | create with `net0` → **403 (/vms/…, VM.Config.Network)** without it | +| `VM.Config.Options` | **KEEP** | config `onboot=1` → **403 (/vms/…, VM.Config.Options)** without it | +| `SDN.Use` | **KEEP (added vs review sketch)** | create → **403 (/sdn/zones/localnetwork/vmbr0, SDN.Use)** without it | + +> Corrections to the review's candidate sketch: `VM.Config.CPUMemory` is **not a real +> privilege** — split into `VM.Config.CPU` + `VM.Config.Memory`. `SDN.Use` was **missing** and +> is **required** (PVE 9 gates bridge use behind it). `Datastore.AllocateTemplate` is **not +> needed**. + +### Final minimal `FelhomAgent` role (proven sufficient for ops 1′–9b) +``` +VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory +VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback +VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.Audit Sys.Audit SDN.Use +``` +(16 privileges. `Datastore.Allocate` is for the storage-definition add; drop it if the agent +never creates Proxmox storage entries via the API. `VM.PowerMgmt` is for start/stop lifecycle +— not for the backup itself, consistent with `proxmox-platform.md` §3.4.) + +### Root-vs-API boundary table (answers `03` §3) + +| Agent host operation | Coverage | Notes | +|---|---|---| +| Create unprivileged LXC, **nesting-only** | **API token** | `VM.Allocate`+`VM.Config.*`+`Datastore.AllocateSpace`+`SDN.Use` | +| **Create with `keyctl=1` (Docker needs it — Phase 0)** | **OS root `root@pam`** (`pct create` as root / sudoers) | no API token works, incl. a root@pam token | +| Set config (mem/cpu/net/options/mountpoint + `backup` flag) | API token | | +| Allocate guest volume | API token | `Datastore.AllocateSpace` | +| Start / stop / snapshot / rollback | API token | `VM.PowerMgmt` / `VM.Snapshot(.Rollback)` | +| vzdump backup (stop/snapshot mode) | API token | `VM.Backup` | +| **Restore from vzdump (preserves keyctl)** | **API token** | DR path needs no root | +| Destroy guest (scratch + compensating rollback, B1) | API token | `VM.Allocate` | +| Add Proxmox **storage definition** (dir/nfs/cifs/pbs) | API token | `Datastore.Allocate`; the *definition* only | +| Host status / metrics report | API token | `Sys.Audit` | +| **USB physical mount-by-UUID / systemd mount unit / fstab** | **OS root / narrow sudoers** | not a Proxmox API op (host-level mount; not tested here) | +| **SMART / hardware sensors** | OS root | not API-exposed | + +**Boundary summary:** nearly the entire guest lifecycle — including **restore** — is covered +by the scoped token. The genuine OS-root residual is narrow: **(1) fresh creation of a +Docker-capable LXC (the `keyctl` flag), (2) physical USB mount-by-UUID / systemd mount units / +fstab, (3) hardware/SMART.** This supports `03` §3's "non-root service + scoped token + narrow +sudoers" model — with the **specific** sudoers/root entries being: `pct create` (or just the +keyctl-setting step) and the host mount operations. + +--- + +## Raw command log (appendix) + +### B2 +``` +pct create 9010 ... --features nesting=1,keyctl=1 --unprivileged 1 # rootfs local-lvm:8 +pct set 9010 -mp1 local-lvm:1,mp=/mnt/mp1,backup=1 +pct set 9010 -mp2 local-lvm:1,mp=/mnt/mp2,backup=0 +pct set 9010 -mp3 /root/b2-bindsrc,mp=/mnt/mp3 +# docker named vol: docker volume inspect b2vol -> /var/lib/docker/volumes/b2vol/_data +# bulk: docker volume create --driver local -o type=none -o o=bind -o device=/mnt/mp2 bulkvol +vzdump 9010 --mode stop --storage local --compress zstd +# INFO: including mount point rootfs ('/') in backup +# INFO: including mount point mp1 ('/mnt/mp1') in backup +# INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled) +# INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume) +tar --zstd -tf | grep SENTINEL # -> rootfs, dockervol, mp1 only +pct restore 9011 --storage local-lvm # -> mp2/bulk absent, mp3 via re-bind +``` + +### B3 +``` +pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Sys.Audit" # candidate (pre-SDN) +pveum user add felhom-agent@pve ; pveum user token add felhom-agent@pve agent --privsep 1 +pveum acl modify / -user 'felhom-agent@pve' -role FelhomAgent +pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent + +# token create with keyctl: +POST /nodes/demo-felhom/lxc ... features=nesting=1,keyctl=1 + -> 403 "changing feature flags (except nesting) is only allowed for root@pam" +# + SDN.Use missing initially: + -> 403 "Permission check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)" +# root@pam non-privsep token, keyctl create: + -> 403 (same "only allowed for root@pam") # tokens never qualify + +# token nesting-only create / config(PUT) / start / stop / snapshot / rollback / +# vzdump(stop) / restore->9021 (kept keyctl) / destroy / POST /storage -> all 200/OK + +# paring: +GET /nodes/$N/status without Sys.Audit -> 403 (KEEP) +create net0 without VM.Config.Network -> 403 (KEEP) +config onboot=1 without VM.Config.Options -> 403 (KEEP) +create from template without Datastore.AllocateTemplate -> OK (DROP) +``` + +### Teardown +``` +pct destroy 9010 9011 9021 --purge # 9020/9022/9023 already destroyed during tests +pveum user token remove felhom-agent@pve agent ; pveum user delete felhom-agent@pve +pveum role delete FelhomAgent # ACLs at / auto-invalidated +rm -f /var/lib/vz/dump/vzdump-lxc-9010-* /var/lib/vz/dump/vzdump-lxc-9020-* +# verified: only 9000/9001 remain (stopped-but-present); no felhom-agent user/role; dump dir empty +``` diff --git a/documentation/tests/phase4-signing-findings.md b/documentation/tests/phase4-signing-findings.md new file mode 100644 index 0000000..9d0b0ca --- /dev/null +++ b/documentation/tests/phase4-signing-findings.md @@ -0,0 +1,257 @@ +# Phase 4 — Control-plane signing primitive (SSHSIG + Go verify): Findings + +**Where run:** build server `192.168.0.180` (Debian 13, **Go 1.24.4**, **OpenSSH 10.0p2**), +no Proxmox. **Date:** 2026-06-08. Throwaway key generated, used, and **deleted** — no private +key, passphrase, or `.sig` committed. + +> De-risks the signing primitive *before* it is written into `04-control-plane-authorization.md` +> or the agent's verify code. **Verdict up front: the approach works cleanly and is key-type- +> agnostic — no fallback needed.** Go verifies the armored `SSHSIG` format, every tamper/replay/ +> authorization case is rejected, and a synthetic FIDO2 `sk-ssh-ed25519` signature verifies +> through the **unchanged** code path (true hardware drop-in). + +--- + +## 0. Result at a glance — 14/14 checks pass + +``` +== Step 2: SSHSIG signature verification (key-type-agnostic path) == + PASS correct verified, op="guest_destroy" + PASS wrong key rejected: signer not in allowed set + PASS tampered blob rejected: signature invalid: ssh: signature did not verify + PASS wrong namespace rejected: namespace mismatch: got "felhom-op-wrong" want "felhom-op-v1" + +== Step 3: anti-replay / authorization (valid signature, still rejected) == + PASS first use verified, op="guest_destroy" + PASS replay (same nonce) rejected: replay: nonce a1b2c3d4...8f90 already seen + PASS expired rejected: expired (expires_at=2020-01-02 ..., now=2026-06-08 ...) + PASS not-yet-valid rejected: not yet valid (issued_at=2030-01-01 ...) + PASS retargeted host rejected: target mismatch: blob=demo-felhom/9001 this=other-host/9001 + PASS retargeted guest rejected: target mismatch: blob=demo-felhom/9001 this=demo-felhom/8888 + +== Step 4: key-type-agnosticism — FIDO2 sk-ssh-ed25519 (synthetic, no device) == + PASS parses sk pubkey type="sk-ssh-ed25519@openssh.com" + PASS authorized_keys form sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5... + PASS sk end-to-end verify verified, op="guest_destroy" +``` + +--- + +## 1. Software round-trip (baseline, CLI) + +- Key: `ssh-keygen -t ed25519 -f felhom-op -N '' -C felhom-operator`. + (Signing non-interactively used an `SSH_ASKPASS` helper + `setsid -w`; in production the + operator key lives behind an agent or a FIDO2 device, so the at-sign passphrase prompt is a + non-issue. The passphrase mechanics are **not** what this spike de-risks.) +- Sign with a **domain-separated namespace**: + `ssh-keygen -Y sign -f felhom-op -n felhom-op-v1 blob.json` → `blob.json.sig` + (armored `-----BEGIN SSH SIGNATURE-----`). +- Baseline verify (CLI sanity) with an allow-list: + ``` + allowed_signers: felhom-operator namespaces="felhom-op-v1" ssh-ed25519 AAAAC3... + $ ssh-keygen -Y verify -f allowed_signers -I felhom-operator -n felhom-op-v1 \ + -s blob.json.sig < blob.json + Good "felhom-op-v1" signature for felhom-operator with ED25519 key SHA256:y0Lj8dIYTM6... + ``` + +## 2. Canonical op blob spec (documented) + +The signature covers **these exact bytes**; the operator CLI (also Go) must reproduce them +byte-for-byte. **Canonical form: JSON, keys sorted lexicographically at every level, no +insignificant whitespace, no trailing newline, UTF-8.** + +```json +{"expires_at":"","issued_at":"","key_id":"","nonce":"<128-bit hex>","op":"","params":{...},"target":{"guest_id":"","host_id":""}} +``` + +| field | meaning | +|---|---| +| `op` | the operation, e.g. `guest_destroy`, `storage_detach`, `restore_overwrite` | +| `target.host_id` / `target.guest_id` | the box + guest the op is bound to (anti-retarget) | +| `params` | op-specific arguments (themselves canonical-sorted) | +| `nonce` | unique per op (anti-replay); ≥128-bit random | +| `issued_at` / `expires_at` | validity window (short — minutes) | +| `key_id` | which operator key (for rotation / audit) | + +Exact test blob (236 bytes): `{"expires_at":"2026-06-09T00:00:00Z","issued_at":"2026-06-08T00:00:00Z","key_id":"felhom-op-1","nonce":"a1b2c3d4e5f60718293a4b5c6d7e8f90","op":"guest_destroy","params":{"purge":true},"target":{"guest_id":"9001","host_id":"demo-felhom"}}` + +> Note: the SSHSIG **namespace** (`felhom-op-v1`) is the cryptographic domain separator and is +> a **fixed constant in the verifier**, never caller-supplied — a signature minted for any +> other namespace must not verify (proven: "wrong namespace" rejected). + +## 3. Go SSHSIG verify — approach + implementation cost + +**It is not a one-call verify, but it is clean — no hand-rolled crypto.** The only manual work +is SSHSIG *framing*; all crypto and key-type dispatch is the library's. Steps: + +1. `pem.Decode` the armor → `block.Type == "SSH SIGNATURE"`, `block.Bytes` is the binary SSHSIG. + *(Go's `encoding/pem` parses the armor directly — no manual base64/line handling.)* +2. Strip the literal 6-byte `SSHSIG` magic preamble (it is **not** length-prefixed). +3. `ssh.Unmarshal` the rest into a struct `{Version uint32; PublicKey, Namespace, Reserved, + HashAlgo, Signature string}` — library does the SSH wire parsing. +4. `ssh.ParsePublicKey([]byte(PublicKey))` → an `ssh.PublicKey`. +5. Recompute the signed data per spec: `"SSHSIG" || string(namespace) || string(reserved) || + string(hash_algorithm) || string(H(message))`, where `H` is the **named** hash + (`sha256`/`sha512`) — built with one `ssh.Marshal`. +6. `ssh.Unmarshal([]byte(Signature))` into `ssh.Signature`, then **`pub.Verify(signed, &sig)`** — + which **dispatches on the key's own algorithm** (this is what makes it key-agnostic). + +**Cost verdict:** ~40 lines of framing in one file, zero crypto implemented by us. Well within +the agent's budget; **no reason to fall back** to a different primitive. + +## 4. Anti-replay / authorization layer (on top of signature validity) + +Enforced in `VerifySignedOp` *after* the signature check, each proven to reject **even with a +valid signature** (Step 3 output above): + +- **replay** — nonce already recorded in the window → reject; +- **expired / not-yet-valid** — `now ∉ [issued_at, expires_at]` → reject (both sides shown); +- **retargeted** — `target.host_id`/`guest_id` ≠ this box/guest → reject (both shown). + +(Order matters: signature → namespace → allow-list → crypto verify → target → time → nonce, so +a replayed *but otherwise valid* op is still caught, and an invalid sig never consumes a nonce.) + +## 5. Key-type-agnosticism — **TRUE DROP-IN** (no box change for FIDO2 later) + +No FIDO2 device was used (by choice). Instead the spike **emulated the authenticator exactly**: + +- Synthesized a well-formed `sk-ssh-ed25519@openssh.com` public key; `ssh.ParsePublicKey` parses + it and `ssh.MarshalAuthorizedKey` round-trips it. +- Constructed a real `SSHSIG` whose inner signature follows the sk scheme (per OpenSSH + `PROTOCOL.u2f`): `ed25519` over `sha256(application) || flags || counter || sha256(signed_data)`, + with the blob `string(format) string(ed25519_sig) byte(flags) uint32(counter)` — i.e. exactly + what a FIDO2 key emits. +- Ran it through the **unchanged `VerifySignedOp`** → **verified** (`op="guest_destroy"`). + +**Verdict: true drop-in.** `pub.Verify` for `sk-ssh-ed25519` is implemented in +`golang.org/x/crypto/ssh` **v0.52.0** (it reconstructs `appDigest‖flags‖counter‖dataDigest` and +`ed25519.Verify`s it). Introducing a hardware operator key later is a **no-op on the boxes** — +the agent's verify code is identical; only the operator's signer key (and the allowed-signers +set entry) changes. No sk-specific handler is needed. + +> Because verification dispatches on the key type embedded in the signature, the same path also +> accepts `ssh-ed25519`, `rsa-sha2-*`, `ecdsa-sha2-*`, etc. — algorithm choice is the operator's, +> not the agent's. + +## 6. Fallback (not taken) and its cost + +A fallback would be a **raw Ed25519 detached signature** (or `minisign`): trivially one +`ed25519.Verify` call, no SSHSIG framing. **Rejected** because it **loses the clean FIDO2 path** — +a raw-Ed25519 verifier cannot consume an `sk-ssh-ed25519` signature (which carries flags+counter +and a different signed-data construction), so the future hardware swap would require **changing +the verifier on every box**. SSHSIG buys exactly the key-type-agnosticism (§5) that a raw scheme +forfeits, at a one-file framing cost (§3). **No fallback is warranted.** + +## 7. Reference verifier (seed of the agent's verify code) + +Verified working on Go 1.24.4 / `x/crypto` v0.52.0. (Test harness omitted; this is the verify +core + SSHSIG framing + anti-replay/authz.) + +```go +const Namespace = "felhom-op-v1" // FIXED domain separator, never caller-supplied +const sshsigMagic = "SSHSIG" + +type Target struct{ HostID, GuestID string } +type OpBlob struct { + Op string `json:"op"` + Target Target `json:"target"` + Params json.RawMessage `json:"params"` + Nonce string `json:"nonce"` + IssuedAt time.Time `json:"issued_at"` + ExpiresAt time.Time `json:"expires_at"` + KeyID string `json:"key_id"` +} +// (Target needs json tags host_id/guest_id in the real struct.) + +type NonceStore interface{ SeenOrRecord(nonce string, exp time.Time) bool } + +type sshsigBlob struct { + Version uint32 + PublicKey, Namespace, Reserved, HashAlgo, Signature string +} + +func hashByName(n string) (hash.Hash, error) { + switch n { + case "sha256": return sha256.New(), nil + case "sha512": return sha512.New(), nil + } + return nil, fmt.Errorf("unsupported SSHSIG hash %q", n) +} + +func parseArmoredSSHSIG(armored []byte) (*sshsigBlob, error) { + block, _ := pem.Decode(armored) + if block == nil || block.Type != "SSH SIGNATURE" { + return nil, errors.New("not an SSH SIGNATURE armor") + } + if len(block.Bytes) < 6 || string(block.Bytes[:6]) != sshsigMagic { + return nil, errors.New("missing SSHSIG magic") + } + var sb sshsigBlob + if err := ssh.Unmarshal(block.Bytes[6:], &sb); err != nil { return nil, err } + if sb.Version != 1 { return nil, fmt.Errorf("bad version %d", sb.Version) } + return &sb, nil +} + +func signedData(sb *sshsigBlob, msg []byte) ([]byte, error) { + h, err := hashByName(sb.HashAlgo); if err != nil { return nil, err } + h.Write(msg); md := h.Sum(nil) + body := ssh.Marshal(struct{ Namespace, Reserved, HashAlgo string; Hash []byte }{ + sb.Namespace, sb.Reserved, sb.HashAlgo, md}) + return append([]byte(sshsigMagic), body...), nil +} + +// VerifySignedOp: key-type-agnostic signature verify + anti-replay/authorization. +// allowedSigners is the trusted operator set (one key now; a quorum set later). +func VerifySignedOp(blob, sigArmored []byte, allowedSigners []ssh.PublicKey, + thisHostID, thisGuestID string, seenNonces NonceStore) (string, error) { + + sb, err := parseArmoredSSHSIG(sigArmored) + if err != nil { return "", err } + if sb.Namespace != Namespace { + return "", fmt.Errorf("namespace mismatch: got %q want %q", sb.Namespace, Namespace) + } + pub, err := ssh.ParsePublicKey([]byte(sb.PublicKey)) + if err != nil { return "", err } + allowed := false + for _, a := range allowedSigners { + if bytes.Equal(a.Marshal(), pub.Marshal()) { allowed = true; break } + } + if !allowed { return "", errors.New("signer not in allowed set") } + + signed, err := signedData(sb, blob) + if err != nil { return "", err } + var inner ssh.Signature + if err := ssh.Unmarshal([]byte(sb.Signature), &inner); err != nil { return "", err } + if err := pub.Verify(signed, &inner); err != nil { // dispatches on key algorithm + return "", fmt.Errorf("signature invalid: %w", err) + } + + var op OpBlob + if err := json.Unmarshal(blob, &op); err != nil { return "", err } + if op.Target.HostID != thisHostID || op.Target.GuestID != thisGuestID { + return "", fmt.Errorf("target mismatch") + } + now := time.Now().UTC() + if now.Before(op.IssuedAt) { return "", errors.New("not yet valid") } + if now.After(op.ExpiresAt) { return "", errors.New("expired") } + if seenNonces.SeenOrRecord(op.Nonce, op.ExpiresAt) { + return "", fmt.Errorf("replay: nonce %s already seen", op.Nonce) + } + return op.Op, nil +} +``` + +## 8. Inputs to the design doc (`04-control-plane-authorization.md`) + +- **Primitive confirmed:** SSHSIG (`ssh-keygen -Y sign` / armored `BEGIN SSH SIGNATURE`), + verified in Go via `pem.Decode` + `ssh.Unmarshal` + `ssh.ParsePublicKey` + `pub.Verify`. Low + implementation cost; no crypto hand-rolled. +- **Hub cannot forge:** the operator private key never touches the hub; the hub only queues the + opaque armored blob (matches `03` §4). +- **Key-type-agnostic / hardware-ready:** software `ed25519` now, FIDO2 `sk-ssh-ed25519` later is + a **box no-op** (proven end-to-end). The verifier hardcodes neither key type nor algorithm. +- **`allowedSigners` is a set:** single signer today; **threshold/quorum is just set sizing** plus + an N-of-M policy on top (out of scope here). +- **Anti-replay/authz are mandatory and cheap:** namespace (fixed), allow-list, then crypto, + then target-binding, time-window, nonce — all enforced and tested. +- **Canonical blob (§2)** is the shared contract between the operator CLI and the agent verifier.