From 4d842075728f8e377246b19da744b456ee422943 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 8 Jun 2026 13:58:41 +0200 Subject: [PATCH] moved docs --- docs/architecture/01-topology-and-trust.md | 224 ---------- docs/architecture/02-controller-module-map.md | 374 ----------------- docs/architecture/03-host-agent.md | 299 -------------- .../04-control-plane-authorization.md | 154 ------- docs/architecture/05-hub-architecture.md | 223 ---------- docs/architecture/_design-review.md | 260 ------------ docs/architecture/_hub-review.md | 221 ---------- docs/proxmox-platform.md | 385 ------------------ ..._Spike_-_API_&_Access-Control_Reference.md | 176 -------- docs/tests/phase0-findings.md | 331 --------------- docs/tests/phase1-2-findings.md | 315 -------------- docs/tests/phase3-findings.md | 234 ----------- docs/tests/phase4-signing-findings.md | 257 ------------ 13 files changed, 3453 deletions(-) delete mode 100644 docs/architecture/01-topology-and-trust.md delete mode 100644 docs/architecture/02-controller-module-map.md delete mode 100644 docs/architecture/03-host-agent.md delete mode 100644 docs/architecture/04-control-plane-authorization.md delete mode 100644 docs/architecture/05-hub-architecture.md delete mode 100644 docs/architecture/_design-review.md delete mode 100644 docs/architecture/_hub-review.md delete mode 100644 docs/proxmox-platform.md delete mode 100644 docs/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md delete mode 100644 docs/tests/phase0-findings.md delete mode 100644 docs/tests/phase1-2-findings.md delete mode 100644 docs/tests/phase3-findings.md delete mode 100644 docs/tests/phase4-signing-findings.md diff --git a/docs/architecture/01-topology-and-trust.md b/docs/architecture/01-topology-and-trust.md deleted file mode 100644 index 6dc8099..0000000 --- a/docs/architecture/01-topology-and-trust.md +++ /dev/null @@ -1,224 +0,0 @@ -# Felhom Controller Architecture — Part 1: Topology & Trust - -**Status:** draft (decisions from the topology/trust design sessions). -**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document -records *Felhom's decisions*, not Proxmox behaviour. - ---- - -## 1. Model at a glance - -Three components. **Control is always box-initiated** — the hub never connects *into* a -customer box. - -``` - operator side customer box (per Proxmox host) - ┌───────────────────┐ ┌───────────────────────────────────────────┐ - │ HUB │ │ Proxmox host │ - │ (dooplex.hu, k3s) │ │ ┌──────────────┐ │ - │ - report sink │◀──poll──┤ │ HOST AGENT │ operator-tier │ - │ - signed jobs │ signed │ │ (Proxmox │ • all Proxmox ops │ - │ - dashboard │ jobs │ │ token) │ • provision / restore │ - │ - customer record│ │ └──────┬───────┘ • storage mgmt │ - │ - PBS namespace │ │ │ local constrained API │ - └─────────▲─────────┘ │ ┌──────▼───────────────────────────────┐ │ - │ │ │ customer LXC (one per customer) │ │ - │ direct, app- │ │ ┌──────────────┐ Docker: │ │ - └───────────────────┼───┤ │ IN-GUEST │ [app] [app] ... │ │ - domain reports │ │ │ CONTROLLER │ (Docker containers)│ - │ │ │ (Docker-only)│ │ │ - │ │ └──────────────┘ │ │ - │ └───────────────────────────────────────┘ │ - └───────────────────────────────────────────┘ - PBS (offsite) ◀── outbound, client-side-encrypted backups ── customer box - end-users / customer ◀── Cloudflare Tunnel ── apps + controller UI -``` - ---- - -## 2. The customer node - -- One **Proxmox host** per box (PVE 9.2, Debian 13, LVM-thin). -- **Default workload topology:** one **customer LXC**, Docker inside it, each app a Docker - container/stack. Apps are isolated at the Docker layer (separate containers, networks, - volumes, cgroup limits); they share one LXC/kernel/Docker daemon. -- **Escape hatch:** promote an individual app to its own guest (LXC or VM) only for a - specific reason — a non-Linux/Windows app, a genuinely untrusted or exposed app needing - hard isolation, or a resource hog needing guarantees. -- **Multi-tenant:** one customer per host is the home default; multiple customer LXCs on - one host (a company environment) is **not precluded** — the agent manages a *set* of - guests. The only multi-tenant-specific work deferred to "if it becomes real" is resource - fairness (per-guest disk/RAM/CPU quotas). - ---- - -## 3. Components & responsibilities - -| | **Hub** | **Host agent** | **In-guest controller** | -|---|---|---|---| -| Runs on | dooplex.hu (k3s) | the Proxmox host | the customer LXC | -| Tier | operator backend | operator (high-privilege) | customer-facing (app) | -| Holds | customer records, signed-job source, PBS namespaces, escrowed keys | the **only** Proxmox API token; per-host operator identity | **no Proxmox creds**; its own hub API key + a local-API token to the agent | -| Does | reporting sink, dashboard, job queue, source of durable truth | all Proxmox ops (provision, restore, snapshot, backup, storage mgmt, LXC lifecycle); polls hub for signed jobs; exposes a constrained local API to the controller; **per-guest authorization gate** | Docker/app lifecycle, catalog deploy, customer UI, app-level (data-layer) backup; reports app-domain to the hub directly | -| Never does | initiate a connection *into* a box | — | touch the Proxmox API directly | - -**Key separation:** the controller manages Docker; the agent manages Proxmox. The controller's -only path to guest-level operations (snapshot-before-deploy, "grow my RAM") is a constrained -**local API call to the agent**, which the agent authorizes (scoped to that controller's own -guest) and executes with its operator-tier token. This consolidates all Proxmox access and -all per-guest authorization in one auditable place and leaves the guest with zero Proxmox -credentials. - ---- - -## 4. Control plane — box-initiated - -- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through - Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the - box exposes no control endpoint at all. -- The agent and the controller **poll** the hub; the hub never initiates inbound. -- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature - before executing, so a compromised hub database alone cannot forge commands. -- All operator-initiated actions are recorded in a **customer-visible audit log**. - ---- - -## 5. Trust boundaries - -| Boundary | What crosses | Mechanism | Blast radius if breached | -|---|---|---|---| -| end-user ↔ apps | app traffic | Cloudflare Tunnel → Traefik (Host routing) | that app | -| customer ↔ controller UI | management UI | Cloudflare Tunnel; UI auth (bcrypt) | the customer's own box | -| controller ↔ agent | snapshot/resize/backup requests | local constrained RPC; agent authorizes per-guest | the controller's own guest only | -| agent ↔ hub | reports + signed jobs | outbound poll; signed jobs | one box; signed jobs limit forgery | -| controller ↔ hub | app-domain reports/jobs (incl. geo desired-state) | outbound, own API key | app-domain of one customer | -| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) | -| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — | -| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF | - ---- - -## 6. Enrollment & identity - -- **Physical presence at provisioning** (on-site install, or pre-imaged-and-delivered). - This removes any zero-touch remote-enrollment problem. -- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful - config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure - case (fetch fails before anything is persisted). After the fetch, the code is irrelevant — - everything downstream runs on durable credentials, so retries don't need it. -- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped - operator-tier Proxmox token), then provisions the customer LXC from the golden template and - deploys the controller into it — injecting the controller's hub API key and its local-API - token. The controller is the agent's product, never the other way around. -- The **hub customer record is the durable source of truth**, and it survives box loss: - identity, domain, **Cloudflare tunnel token**, **PBS namespace**, **storage manifest**, a - **mirrored app inventory** (bottom-up reality, not operator-declared intent — apps themselves - restore from the PBS guest snapshot, never re-deployed from this record; see `05` §1/§9), and the - **escrowed (zero-knowledge) backup key**. This is what makes hardware replacement possible. - ---- - -## 7. Networking - -- **Cloudflare Tunnel** provides inbound access to apps and the controller UI (the CGNAT - solution). Tunnel token lives in the hub record → **reused on new hardware during DR**, so - DNS/routing stay intact through an outage. -- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control - endpoint exists in the chosen model. -- **Tunnel placement: host** (resolved, Part 3 §3/§5). `cloudflared` runs on the Proxmox host - as its own **agent-managed systemd service** — not inside the guest — so the data path - survives control-plane death by construction. Geo-restriction WAF is **hub-enforced** (the - hub holds the CF API token; the controller only reports geo desired-state). - ---- - -## 8. Storage & backup - -**Tiers** (escalating failure scope): - -| Layer | Mechanism | Survives | Note | -|---|---|---|---| -| Snapshot | LVM-thin snapshot (transient) | *logical* loss only | whole-LXC rollback; **not a backup** | -| Local — second storage | vzdump to `dir`/`nfs`/`cifs` | primary-disk failure (USB) / box death (NAS) | first *real* backup tier | -| Offsite — PBS | dedup'd, incremental, encrypted | site loss | the DR substrate; paid tier | - -- **Storage manifest** (hub-held, agent-reconciled): per target → type, durable identity - (UUID / `server:/export` / repo+fingerprint), **class** (fast/slow + rough IOPS, set once - at attach), role, encrypted credentials, schedule/retention. The agent creates the Proxmox - storages, continuously checks presence/reachability, and reports per-target status (a - disconnected target → actionable notification). -- **App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume - **hot** (DB/config/cache → fast storage, enforced) vs **bulk** (media/files → may be slow). - A photo app's DB stays on SSD while its blobs go to the USB. -- **Backup scoping:** hot data (LXC rootfs) rides the guest `vzdump` → tiers + PBS. Bulk data - on external mount points is **excluded** from the guest vzdump (per-mount `backup` flag) and - gets its own per-volume policy (file-level to a tier, slower cadence — or explicitly *not* - backed up for re-downloadable content, with the customer informed). -- **Tiers double as the DR restore-source priority:** restore from the fastest *surviving* - source (local if still attachable, PBS on true site loss). -- **Key custody (zero-knowledge default):** three tiers the customer chooses — - *customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows - the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery - code** the operator can't open; DR needs the customer's code. Access-notification is an - audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native - keyfile passphrase.) - ---- - -## 9. Disaster recovery - -- **Guest-loss (host + agent alive):** the agent restores the guest from the fastest - surviving tier, **resets identity** (MAC/hostname — see `proxmox-platform.md`), boots it, - controller returns. Validated mechanics: Phase 2. -- **Host / hardware-loss (agent gone):** re-provision (§6) in **restore mode** — the hub, - knowing the customer has PBS backups, hands the freshly-enrolled agent the existing identity - + PBS namespace + a restore directive instead of a clean-provision directive. The agent - restores from PBS; the controller returns on the same domain (tunnel reused from the hub - record). DR = provisioning + a restore mode, not a separate mechanism. -- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its - post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) - ---- - -## 10. How this embodies the product values - -- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it. -- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise - alone can't forge commands. -- **Customer-visible audit log** — every operator action is visible to the customer. -- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite, - support, new deployments); the customer's data and deployed apps remain recoverable by the - customer (recovery code), with nothing locked behind the operator. - ---- - -## 11. Open sub-decisions (carried into later parts) - -- **RTO/RPO targets** → drive the backup + offsite-replication schedule (§8). -- Offboarding / decommission (scenario 6) — not yet designed; must honour "never hold data - hostage" in credential revocation + data hand-off. -- Multi-tenant resource fairness — deferred until multi-tenant is real (§2). - ---- - -## Appendix — relationship to the spike - -- **Phase 0** → §2: LXC-default for the workload; overhead numbers. -- **Phase 1** → §3/§5: validated the privilege boundary (create/allocate is operator-tier). - The guest-side scoped-backup-token it proved possible is **not** used — we chose the - agent-mediated path — but it confirmed restore = operator-tier, which shapes the agent. -- **Phase 2** → §8/§9: backup→restore round-trip; identity reset on restore. - ---- - -## Changelog — design-review + Phase-3 fold-in (2026-06-08) - -- §5 trust boundaries: **added `hub ↔ Cloudflare API`** row (hub holds the CF token, enforces - geo→WAF); controller↔hub row notes it carries geo desired-state (S4). -- §7 networking: **tunnel placement resolved → host** (agent-managed systemd service); geo is - hub-enforced (S4/S5). -- §11 open items: removed the now-resolved **tunnel placement** and **self-update flow** entries - (S5; self-update designed in 03 §11). -- §6 durable record: **"declarative app inventory" → "mirrored app inventory"** — aligns the wording - with the locked two-driver model (`05` §1: apps are bottom-up mirror, never operator-declared; - `05` §9: apps restore from the PBS guest snapshot, not re-deployed from this record). \ No newline at end of file diff --git a/docs/architecture/02-controller-module-map.md b/docs/architecture/02-controller-module-map.md deleted file mode 100644 index 85d5f0a..0000000 --- a/docs/architecture/02-controller-module-map.md +++ /dev/null @@ -1,374 +0,0 @@ -# Felhom Controller Architecture — Part 2: Controller Module Map - -**Status:** audit (keep / port / delete / modify / add), grounded in the v0.33 source. -**Subject:** the v0.33 controller in `felhom-controller/controller/` (110 `.go` files, -~40 K LOC) audited against [01-topology-and-trust.md](01-topology-and-trust.md) and -[../proxmox-platform.md](../proxmox-platform.md). - -> This is a **planning map, not the port.** No controller code was changed. Source -> citations use `controller/internal/...:line` (a different repo, so links are not -> clickable). Classifications reflect the **target model**: the in-guest controller is -> **Docker-only and holds no Proxmox credentials**; everything host/disk/Proxmox moves to -> a new **host agent** (out of scope here); the controller reaches the agent through a -> constrained **local API**. - -## Classification scheme -**KEEP** (host-agnostic, ~unchanged) · **PORT** (survives, needs rework) · -**DELETE (→agent)** (responsibility moves to the host agent) · -**DELETE (obsolete)** (no longer needed) · **MODIFY** (stays, materially changes) · -**NEW** (no v0.33 equivalent). -Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-target with a keep/port target). - ---- - -## 0. Executive summary - -- The **app domain is largely intact and portable**: stack lifecycle (`stacks/`), catalog - git-sync (`sync/`), app-to-app integrations (`integrations/`), `.fab` export/import - (`appexport/`), the scheduler, crypto, asset sync, the hub report/notify *channels*, and - most of the web UI **KEEP/PORT cleanly**. -- The **disk/storage/host half deletes wholesale to the agent**: all of `storage/`, - `monitor/watchdog.go`, the restic/cross-drive/disk-layout/drive-mount parts of `backup/`, - `report/infra_backup*`+`infra_pull`, and the host-physical parts of `system/`. -- The **setup wizard (`setup/`) is obsolete** — the agent provisions the controller. -- **The single biggest hazard is `backup/`**: the keep side (DB dumps, Docker-volume - archive, per-app restore — needed by `appexport/` and the backup UI) and the delete side - (restic, cross-drive, drive-mount) are **interleaved inside the same files** - (`backup.go`, `restore.go`, `paths.go`), not cleanly file-separated. Extracting the - app-data-backup subset into a clean retained package is the critical refactor. -- **Intent-vs-reality corrections** (vs the task's provisional split): `monitor/pinger.go` - is already **dead** (legacy Healthchecks.io, "deprecated… now handled by Hub" per - `main.go`) → DELETE(obsolete), not keep. `backup.go`/`restore.go`/`paths.go` do **not** - split on file boundaries — they split *within* the file. `settings/` is **not** pure app - domain — it stores disk/disconnect/decommission state. `system/` is genuinely - mixed-per-function, not per-file. - ---- - -## 1. v0.33 module inventory (package → purpose, key deps) - -| Package | Purpose | Key internal deps | -|---|---|---| -| `cmd/controller/main.go` | Entry point; wires all subsystems; 6 adapters break import cycles; branches into setup mode | imports **every** package | -| `api/` | REST API (`router.go`) + geo endpoints (`geo.go`) | stacks, backup, metrics, notify, selfupdate, sync, system, assets, integrations, cloudflare, config, settings | -| `appexport/` | `.fab` app export/import (config+DB+volumes, AES-256-CTR+scrypt) | **backup** (DB dump), (provider iface → stacks) | -| `assets/` | Download/cache app assets from Hub API | — (HTTP only) | -| `backup/` | DB dumps, Docker-volume archive, **restic**, **cross-drive rsync**, per-app restore, **drive mount**, disk-layout, infra-backup metadata | config, monitor, settings, system, util | -| `cloudflare/` | Geo-restriction via Cloudflare WAF (zone/waf/geosync/countries) — **enforcement → hub** (S4) | settings | -| `config/` | `controller.yaml` schema + load | — | -| `crypto/` | AES-256-GCM for app.yaml secrets | — | -| `integrations/` | App-to-app (OnlyOffice→FileBrowser/Nextcloud) via docker exec / config patch | stacks, crypto, settings | -| `metrics/` | SQLite time-series: system + container metrics, log scan | system | -| `monitor/` | App health (`healthcheck`,`pinger`) + **storage/USB watchdog** | config, notify, settings, system | -| `notify/` | Hub event push (direct, own API key) | settings | -| `recovery/` | Generate `recovery-info.txt` (DR guide) | — | -| `report/` | Build+push hub report; **infra-backup payload**; **recovery pull** | backup, config, metrics, monitor, scheduler, settings, stacks, system | -| `scheduler/` | Cron/interval jobs, Budapest TZ | — | -| `selftest/` | Startup checks (docker/dirs/catalog/hub/**restic repos**/mountpoint) | backup, config, settings, system | -| `selfupdate/` | Self-update: pull image, edit compose, `up -d` | config | -| `settings/` | `settings.json` persistent state: **storage paths/disconnect/decommission**, cross-drive cfg, notif prefs, geo, integration state, DB-validation cache | — | -| `setup/` | **First-run wizard** (scan drives, hub-restore, manual config) | backup, config, report, settings, web | -| `stacks/` | Docker Compose lifecycle, deploy + memory validation, metadata (`.felhom.yml`), HDD-data delete | config, crypto, system | -| `storage/` | **Physical disk** scan/format/attach/mount/migrate/fstab/safety | backup, settings, util | -| `sync/` | Catalog git-sync (pull templates) | config | -| `system/` | Resource info: mem/cpu/load (guest) + **temp/disk-model/USB/mount topology (host)** | — | -| `util/` | String helper | — | -| `web/` | Hungarian dashboard: pages, auth, deploy, backup UI, **storage/disk UI**, DR restore UI, export UI, debug | appexport, backup, config, crypto, integrations, monitor, notify, scheduler, selfupdate, settings, stacks, storage, system | - ---- - -## 2. Classification table (per package/file) - -### `cmd/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `cmd/controller/main.go` | **MODIFY** | Wiring stays, but drop the setup-mode branch, the storage/watchdog/drive-migrator/restic/cross-drive/infra-backup wiring, and add the **agent local-API client**. 6 adapters shrink. | hazard | - -### `api/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `api/router.go` | **PORT/MODIFY** | Keep stacks/deploy/integrations/metrics/sync/assets/selfupdate routes; **remove `/api/storage/*` (disk)**; backup routes become **agent-coordinated guest-backup** requests; `config/apply` (hub-pushes-yaml) changes since the **agent** now injects config at provision. | needs-rework | -| `api/geo.go` | **PORT/MODIFY** | Keep the customer-facing geo **preference** endpoints (set/get global + per-app); **drop the Cloudflare-sync trigger** — enforcement → hub (S4). The controller reports geo desired-state up instead of calling the CF API. | needs-rework | - -### `appexport/` — KEEP/PORT (Docker-volume + DB level, no disk ops) -| File | Class | Reason | Risk | -|---|---|---|---| -| `crypto.go` | **KEEP** | Self-contained AES-256-CTR+HMAC+scrypt for `.fab`. | clean | -| `manifest.go`, `provider.go` | **KEEP** | Bundle metadata; provider interface (impl in main). | clean | -| `export.go` | **PORT** | Docker-volume `tar`, DB dump via `backup.DumpOne`, config copy. Depends on the **retained** app-data-backup subset of `backup/`; HDD-mount enumeration reworked to **per-volume placement**. | needs-rework | -| `restore.go` | **PORT** | `docker volume create`/`tar xf`, DB import, compose up. Same per-volume rework. | needs-rework | -| `estimate.go` | **PORT** | `du`/`df` on mounts → per-volume sizing. | clean | - -### `assets/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `syncer.go` | **KEEP** | Hub API download + checksum cache; already a direct hub channel. | clean | - -### `backup/` — THE SPLIT (delete side interleaved with keep side; see §3) -| File | Class | Reason | Risk | -|---|---|---|---| -| `dbdump.go` | **KEEP** | Pure `docker exec pg_dump`/`mariadb-dump` — app/DB data layer; the retained per-app backup. | clean | -| `appdata.go` | **PORT** | App-data discovery (stacks/volumes/DB containers, `du`). "HDD mount" concept → per-volume. | needs-rework | -| `backup.go` (1478 L) | **MODIFY (split)** | Mixes **keep** (`RunDBDumps`, `DumpAppVolumes(Safe)`, app restore) with **delete→agent** (`RunBackup`/`backupDrive`/restic snapshot/prune/check on per-drive repos). Must be torn in two. | hazard | -| `restore.go` (442 L) | **MODIFY (split)** | `RestoreApp` restic path → agent; Docker-volume + Tier-2 rsync restore (app layer) → keep. | hazard | -| `restore_app_linux.go`/`_other.go` | **PORT** | Per-app restore: compose pull/up, rsync app data, DB-dump restore. App layer; depends on backup location that changes. | needs-rework | -| `paths.go` | **MODIFY (split)** | `AppDBDumpPath`/`AppVolumeDumpPath` keep; `Primary/SecondaryResticRepoPath`, `InfraBackupDir` → agent. | needs-rework | -| `restic.go` | **DELETE (→agent)** | restic repos on drives = infra backup tier; agent does vzdump/PBS. | hazard | -| `crossdrive.go` | **DELETE (→agent)** | Tier-2 cross-drive rsync to secondary storage = storage-tier (agent + storage manifest). | hazard | -| `restore_drives_linux.go`/`_other.go` | **DELETE (→agent)** | `lsblk`/`blkid`/`mount`/fstab — pure host disk. | hazard | -| `disk_layout.go` | **DELETE (→agent)** | Disk topology for DR → agent. | clean | -| `local_infra.go` | **DELETE (→agent)** | Per-drive infra-backup metadata → agent. | clean | -| `restore_scan.go` | **DELETE (→agent)** | Scans drives to build a DR restore plan = agent-tier DR. | needs-rework | - -### `cloudflare/` — DELETE (→hub): CF-API enforcement moves to the hub (S4) -| File | Class | Reason | Risk | -|---|---|---|---| -| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework | - -### `config/`, `crypto/`, `util/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `config/config.go` | **MODIFY** | Drop `BackupConfig` (restic/retention), storage-drive keys, and `InfrastructureConfig.cf_api_token` (→hub, S4); keep customer/paths/web/git/stacks/monitoring/hub/assets/system; **add agent local-API endpoint+token**. | needs-rework | -| `crypto/crypto.go` | **KEEP** | App.yaml secret encryption. | clean | -| `util/strings.go` | **KEEP** | Trivial helper. | clean | - -### `integrations/` — all KEEP (pure app-domain) -| File | Class | Reason | Risk | -|---|---|---|---| -| `integrations.go`,`lifecycle.go`,`manager.go`,`onlyoffice_filebrowser.go`,`onlyoffice_nextcloud.go` | **KEEP** | App-to-app via `docker exec` / compose-config patch; no host ops. | clean | - -### `metrics/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `store.go`,`logscanner.go`,`telemetry.go`,`types.go` | **KEEP** | SQLite store, `docker logs` scan, container telemetry — app-domain. | clean | -| `collector.go` | **PORT** | Container metrics (`docker stats`) keep; host metrics via `system.GetInfo` (temp, physical disk) become **agent-provided or dropped**. | needs-rework | -| `sysinfo.go`/`sysinfo_other.go` | **MODIFY** | Reads `/host/etc`, `/proc/cpuinfo`, uptime — host static info; in-guest some is meaningful, hardware identity via agent. | needs-rework | - -### `monitor/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `healthcheck.go` | **PORT (split)** | Keep guest health (mem/cpu/docker/protected-containers); host health (temp, **physical disk**, storage-path mount status) becomes **agent-fed**. | needs-rework | -| `pinger.go` | **DELETE (obsolete)** | Legacy Healthchecks.io; `main.go` itself marks it "deprecated… now handled by Hub". *(Corrects the task's KEEP/PORT guess.)* | clean | -| `watchdog.go` (902 L) | **DELETE (→agent)** | Storage/USB disconnect monitoring: `umount -l`, `mount -T /host-fstab`, UUID probing, restic-lock cleanup — pure host storage. | hazard | - -### `notify/`, `recovery/`, `scheduler/`, `selftest/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `notify/notifier.go` | **KEEP/MODIFY** | Direct hub event channel (own API key) — keep; prune infra event types that move to the agent (`storage_disconnected`, `crossdrive_*`, `disaster_recovery_*`). | clean | -| `recovery/info.go` | **DELETE (obsolete)** | Generates a DR text guide (OS install, docker-setup.sh, hub restore UI); DR is now agent+hub provisioning. | clean | -| `scheduler/scheduler.go` | **KEEP** | Generic cron/interval, Budapest TZ. | clean | -| `selftest/selftest.go` | **PORT** | Keep docker/dirs/catalog/hub checks; drop restic-repo + system-data **mountpoint** checks (→agent). | needs-rework | - -### `report/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `pusher.go` | **KEEP** | Direct hub push (`/api/v1/report`, Bearer). | clean | -| `telemetry.go` | **KEEP** | Per-app telemetry section. | clean | -| `builder.go` (326 L) | **MODIFY** | Keep containers/telemetry/stacks/geo/app-health; drop/relocate host system info, physical storage, **restic backup status incl. restic password**. | hazard | -| `types.go` | **MODIFY** | Schema: drop infra fields (`restic password`, physical storage), keep app-domain. | needs-rework | -| `infra_backup.go`/`_linux.go`/`_other.go` | **DELETE (→agent)** | Builds infra-backup payload (disk layout, restic/enc passwords) for hub. | hazard | -| `infra_pull.go` | **DELETE (→agent)** | Pulls recovery config + infra backup from hub (setup-wizard DR). | needs-rework | - -### `selfupdate/` — controller is agent-managed (doc 03 §11) -| File | Class | Reason | Risk | -|---|---|---|---| -| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean | -| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean | -| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean | - -### `settings/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `settings/settings.go` (1101 L) | **MODIFY (split)** | Keep notif prefs, integration state, geo, DB-validation cache, cross-drive *intent*. The **storage-path registry** (`StoragePath` with `Disconnected`/`DisconnectedAt`/`StoppedStacks`/decommission) is disk-management state → reshape to **per-volume placement** fed by the agent's storage manifest; disconnect/decommission/migrate state leaves. (UUID is *not* a persisted field — runtime-derived from fstab.) | hazard | - -### `setup/` — all DELETE (obsolete); the agent provisions the controller -| File | Class | Reason | Risk | -|---|---|---|---| -| `handlers.go`,`setup.go`,`csrf.go`,`network.go` | **DELETE (obsolete)** | First-run wizard (hub-restore, manual config, LAN-IP detection). | needs-rework | -| `scanner.go` | **DELETE (→agent)** | Drive scan (`lsblk`+temp mounts) for backup discovery — host op; its capability informs the agent. | clean | - -### `stacks/` — core app domain (KEEP/PORT) -| File | Class | Reason | Risk | -|---|---|---|---| -| `manager.go` (1074 L) | **KEEP/PORT** | Docker Compose orchestration, scan/state/start/stop/logs — the heart. Minor port. | clean | -| `deploy.go` | **PORT** | Memory validation (`system.GetMemoryMB` — **guest** mem, fine in LXC), secret gen, encrypted app.yaml. **Add snapshot-before-deploy → agent** hook. | needs-rework | -| `healthprobe.go` | **KEEP** | TCP/HTTP app probes. | clean | -| `metadata.go` | **PORT** | `.felhom.yml` parse. **Add per-volume hot/bulk classification** (doc 01 §8). | needs-rework | -| `delete.go` | **PORT** | Stack delete + HDD-data `os.RemoveAll` on bind mounts → per-volume cleanup. | needs-rework | - -### `storage/` — entire package DELETE (→agent) -| File | Class | Reason | Risk | -|---|---|---|---| -| `scan*`,`format*`,`attach*`,`migrate*`,`migrate_drive*`,`safety*` | **DELETE (→agent)** | Physical disk: `lsblk`/`sfdisk`/`wipefs`/`mkfs.ext4`/`partprobe`/`mount`/`umount`/fstab/`blkid`/drive-rsync. The agent owns all of this (doc 01 §3, §8). | hazard | - -### `sync/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset, copy compose+`.felhom.yml`, never overwrite app.yaml). | clean | - -### `system/` — split per-function (not per-file) -| File | Class | Reason | Risk | -|---|---|---|---| -| `cpu_linux.go`/`cpu_other.go` | **KEEP** | `/proc/stat` works inside an LXC. | clean | -| `info.go`/`info_other.go` | **KEEP** | Structs/stubs. | clean | -| `info_linux.go` | **MODIFY (split)** | Keep mem (`/proc/meminfo`)/load/statfs (guest); **temp via `/host/sys`, hwmon → agent**. | needs-rework | -| `mounts_linux.go`/`mounts_other.go` | **DELETE (→agent)** mostly | Mount-point detection, USB, disk model, fstab, probe — host/disk. Guest-meaningful `statfs` disk-usage is the only keep-candidate → fold into the kept `info`. | hazard | - -### `web/` — split by UI surface -| File | Class | Reason | Risk | -|---|---|---|---| -| `auth.go`,`csrf.go`,`logbuffer.go`,`embed.go`,`templates.go` | **KEEP** | Session/CSRF, log ring buffer, embeds/logo. | clean | -| `funcmap.go` | **KEEP/PORT** | Template helpers; a few backup/state labels track the backup rework. | clean | -| `server.go` (559 L) | **MODIFY** | Routing/wiring; remove storage/DR-restore/watchdog wiring; keep app/deploy/backup/settings/export/debug. | needs-rework | -| `handlers.go` (1883 L) | **PORT/MODIFY** | Core pages keep; the embedded **storage-path management** (add/remove/label/schedulable, storage bars, FileBrowser mount sync) → per-volume / agent-fed. | hazard | -| `handler_export.go` | **KEEP/PORT** | `.fab` UI. | clean | -| `handler_debug.go` (823 L) | **PORT** | Drop storage-simulate/infra-push/DR debug; keep the rest. | needs-rework | -| `alerts.go` | **PORT/MODIFY** | Storage-disconnect alert now sourced from **agent** status; backup/update alerts keep. | needs-rework | -| `handler_restore.go` | **DELETE (→agent) / MODIFY** | DR restore-mode UI; DR is agent-tier — replace with an agent-status view or remove. | needs-rework | -| `storage_handlers.go` (1600 L) | **DELETE (→agent)** | Format/attach/mount/disconnect/migrate-drive/decommission disk UI. Any survivor is a **thin client calling the agent API** (e.g. per-volume placement requests). | hazard | -| `templates/` (HTML, non-Go) | **PORT** | Remove disk-wizard + DR pages; keep app/deploy/backup/settings pages. | needs-rework | - -### `scripts/` -| File | Class | Reason | Risk | -|---|---|---|---| -| `scripts/hashpass.go` | **KEEP** | Standalone bcrypt helper. | clean | - ---- - -## 3. Coupling hazards (delete-targets depended on by keep/port) - -1. **`backup/` is half-deleted but split *inside files*, not across them.** `backup.go` - contains both `RunDBDumps`/`DumpAppVolumesSafe`/app-restore (keep) and - `RunBackup`/`backupDrive` + restic (delete→agent); `restore.go` and `paths.go` are - likewise mixed. **Keep/port consumers reach into this same package:** - - `appexport/export.go:295` → `backup.DiscoverDatabases`/`DumpOne` (DB dump is app-layer — must survive) - - `report/builder.go:buildBackupReport` → backup status (MODIFY) - - `web/handlers.go` (backups page, `buildAppBackupRows`), `web/funcmap.go`, `web/alerts.go`, `web/handler_restore.go`, `web/handler_debug.go` - - `selftest/selftest.go:217` → `checkResticRepos` (restic path — delete) - - `main.go` scheduler chain `RunFullBackup` (DB→volume→restic→infra-push) interleaves both sides. - **Action:** extract the app-data-backup subset (DB dump, volume archive, per-app - restore) into a clean retained package *before* deleting the restic/cross-drive code, - or every keep consumer breaks. - -2. **`backup/crossdrive.go` (delete→agent) is wired as `crossDriveRunner` into** - `main.go`, `api/router.go`, `web/server.go`, and surfaced by `report/builder.go` and the - backups page. Removing it requires reworking the backup UI/report to the agent's - guest-backup status. - -3. **`storage/` (delete→agent) depended on by keep/port UI:** `web/storage_handlers.go` - (delete) and `web/server.go`/`web/handlers.go` (port) — the latter renders storage - labels/bars and runs **FileBrowser mount sync** off the storage-path registry. - `storage/migrate*.go` also imports `backup` (also being split). Untangle the per-volume - placement UI from the disk-management UI. - -4. **`monitor/watchdog.go` (delete→agent) depended on by** `web/alerts.go` (port), - `web/server.go`, `web/handler_debug.go`, `main.go`. The disconnect **alert** must instead - consume agent-reported storage status. - -5. **`system/` mixed-per-function, consumed by both sides.** Keep consumers — - `stacks/deploy.go` (`GetMemoryMB`, guest), `metrics/collector.go` (container) — must not - drag in the host-disk/temp/USB code that goes to the agent (`mounts_linux.go`, - `info_linux.go` temp). Also consumed by `report/builder.go` (MODIFY), `monitor/healthcheck.go` - (PORT), `selftest`, `crossdrive` (delete). **Split `system/` cleanly into guest-info vs - host-info first.** - -6. **`settings/StoragePath` carries disk state into an app-domain store.** Disk fields - (`Disconnected`,`DisconnectedAt`,`StoppedStacks`, decommission — UUID is *not* persisted, it's runtime-derived from fstab via `system.ParseFstabUUID`/`watchdog.go`) are written by - `watchdog.go`/`storage_handlers.go`/`crossdrive.go` (all delete) but the same struct is - read by `stacks`/`web` for labels and **placement** (keep). Reshape `StoragePath` to a - placement record fed by the agent manifest. - -7. **`report/builder.go` imports almost everything** (backup, monitor, scheduler, stacks, - system, metrics, settings, config). Its MODIFY must land *after* the backup and system - splits, or it pulls deleted code along. - -8. **`backup/paths.go` shared both ways** — `appexport` + `selftest` + the kept DB-dump - flow use the app-dump path helpers; the same file holds the restic/secondary helpers - that leave. - -9. **DR/provisioning chain is cross-cut:** `setup/` (obsolete) → `report/infra_pull` + - `recovery/info` + `backup.MountDrivesFromLayout` + `backup.ReadLocalInfraBackup`. All - obsolete/→agent, but `main.go`'s setup branch and `web/handler_restore.go` reference - them; remove together. - ---- - -## 4. Moves to the host agent (consolidated — feeds the future agent design) - -> Reporting only; **not** designing the agent here. - -- **All physical-disk management** — `storage/` in full: scan/classify, format - (`wipefs`/`sfdisk`/`mkfs.ext4`/`partprobe`), attach (raw mount + bind + fstab), per-app - and full-drive migration (rsync), safety checks (system-disk detection). -- **Storage/USB watchdog** — `monitor/watchdog.go`: disconnect/reconnect detection, - `umount -l`, `mount -T /host-fstab`, UUID-by-id probing, safe-disconnect, restic-lock - cleanup. -- **Infra/disk backup tier** — `backup/restic.go`, `crossdrive.go`, - `restore_drives_*`, `disk_layout.go`, `local_infra.go`, `restore_scan.go`, plus the - restic-snapshot half of `backup.go`, the restic-restore half of `restore.go`, and the - restic/secondary path helpers in `paths.go`. (Maps to the agent's `vzdump`→tiers→PBS in - doc 01 §8.) -- **Infra-backup payload + recovery pull** — `report/infra_backup*`, `report/infra_pull`. -- **Host-physical telemetry** — `system/mounts_linux.go` (mount topology, USB, disk - model), the temp/hwmon parts of `system/info_linux.go`, and the host-hardware parts of - `metrics/sysinfo.go`. -- **Drive scanning for provisioning/DR** — `setup/scanner.go`. -- **Self-restore-test execution** — the agent performs the restore-to-scratch-guest; the - controller only orchestrates/validates (see §5). - ---- - -## 5. New components to build (no v0.33 equivalent) - -1. **Agent local-API client** — the controller's only path to guest-level Proxmox - operations (doc 01 §3, §5): `snapshot-before-deploy` + rollback, "grow my RAM", request - guest backup/restore, read the storage manifest / mount placement, query per-target - storage status. Replaces the deleted direct host/disk code with constrained RPC. The - controller holds **no Proxmox creds** — only a local-API token. -2. **Per-volume storage placement** (doc 01 §8) — `.felhom.yml` `hot`/`bulk` volume - classification (extend `stacks/metadata.go`), enforcement at deploy (extend - `stacks/deploy.go`), and a placement record in `settings`. Replaces the per-app - HDD-path + cross-drive model. A `bulk` volume must be realized as a `backup=0` mount point, - **never** a rootfs Docker named volume (validated recipe: `phase3-findings.md` B2 / doc 03 §7). -3. **Self-restore-test status display** (read-only) — the **agent owns orchestration** (it - holds the PBS key and creates the scratch guest — operator-tier, doc 03 §8); the controller - only surfaces `GET /restore-test/status` in its UI. (Round-trip validated: Phase 2, - [../proxmox-platform.md](../proxmox-platform.md) §4.) -4. **Snapshot-before-deploy/rollback flow** in the deploy path — wraps the existing - compose deploy with agent snapshot → health check → agent rollback-on-failure - (doc 01 §9). New behaviour on top of `stacks/deploy.go` + `stacks/healthprobe.go`. -5. **Agent-provisioning bootstrap receiver** — the controller accepts its injected hub API - key + local-API token from the agent at provision time (doc 01 §6), replacing the - deleted `setup/` wizard. - ---- - -## 6. Open / blocked items - -- **Geo — resolved (S4):** CF-API **enforcement moves to the hub** (it holds the CF token and - reconciles geo → WAF); the controller keeps the geo **preference UI/data** and reports - desired-state up. Tunnel placement is settled (host, agent-managed, doc 03 §3/§5). The - `cloudflare/` package + `api/geo.go`'s CF-sync are DELETE-from-controller → hub. -- **Self-update — resolved (doc 03 §11):** the controller is agent-managed; its self-update - path is removed. -- **`settings`/`stacks` per-volume reshape** — depends on the storage-manifest contract - between hub ↔ agent ↔ controller (doc 01 §8), not yet specified. -- **Backup UI/report surface** — depends on the agent's guest-backup status API shape - (what the controller can see about vzdump/PBS state) — undefined. -- **Notification event taxonomy** — which infra events (`storage_disconnected`, - `crossdrive_*`, `disaster_recovery_*`) the **agent** emits vs the controller, once those - responsibilities move. - ---- - -## Changelog — design-review + Phase-3 fold-in (2026-06-08) - -- **M1:** removed `UUID` from the `settings.StoragePath` field lists (§ settings, hazard #6) — - it is runtime-derived from fstab, not persisted. -- **S4 (geo):** `cloudflare/` reclassified **PORT(blocked) → DELETE(→hub)** (CF-API enforcement - moves to the hub); `api/geo.go` → **PORT/MODIFY** (keep geo *preference* endpoints, drop the - CF-sync trigger); `config/config.go` also drops `cf_api_token`. §6 + §1 updated. -- **S5:** cloudflare/geo no longer "blocked on tunnel placement" (resolved). -- **S6:** §5(3) self-restore-test → **status-display only**; the agent owns orchestration. -- **Self-update resolved (03 §11):** `updater.go` → **DELETE(→agent)**, `state.go` → - DELETE(obsolete), `version.go` KEEP; §6 + §5(2) updated (bulk = `backup=0` mountpoint recipe). diff --git a/docs/architecture/03-host-agent.md b/docs/architecture/03-host-agent.md deleted file mode 100644 index 644864d..0000000 --- a/docs/architecture/03-host-agent.md +++ /dev/null @@ -1,299 +0,0 @@ -# Architecture Part 3 — The Host Agent - -> Status: design draft (decision content). To be grounded by Claude Code against -> `docs/proxmox-platform.md` and `docs/architecture/02-controller-module-map.md`, -> then placed at `docs/architecture/03-host-agent.md`. -> -> Builds on Part 1 (`01-topology-and-trust.md`) and Part 2 (`02-controller-module-map.md`). -> Where this doc and the locked decisions disagree, the locked decisions win and this -> draft is wrong — flag it. - -## 1. Purpose & scope - -The **host agent** is the operator-tier component that runs on each Proxmox host and -owns *all* Proxmox interaction. It is the trusted host actor: it provisions and restores -guests, manages host storage, orchestrates backups and restore-tests, watches the host -and the tunnel, talks to the hub, and exposes a narrow local API to the in-guest -controllers it deploys. - -It is the privileged tier. The controller deliberately holds **no** Proxmox credentials -(Part 1) — the privilege the controller shed by losing `storage/` did not disappear, it -**moved here**. That makes the agent's hardening and blast-radius discipline the most -security-sensitive part of the platform. - -The agent manages a **set** of guests on its host (usually one customer = one guest, but -the multi-tenant/company case is not precluded — the agent's data model is per-host, -N-guests, never "the guest"). - -## 2. Responsibilities (and explicit non-responsibilities) - -Owns: - -1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable. -2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests. -3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**. -4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. -5. **Provisioning** — provision a guest **by restoring the golden base image** (§9), deploy the controller into it, hand it its bootstrap config; also **build and refresh the golden base image** itself. -6. **Hub control loop** — poll for desired state + signed jobs, reconcile, execute, report, heartbeat. -7. **Local API** — the per-guest authorization gate the controller calls. -8. **Self-update** — update itself (carefully — it is a host service) and update the controllers it owns. - -Explicitly does **not**: - -- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat. -- Hold or proxy customer application data. -- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them. -- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules. - -## 3. Process model & host integration - -- **Native Go binary, systemd service** on the host: boot-start, `Restart=always`, systemd watchdog (kill+restart on hang), journald logging, resource limits. -- **Root-minimized (boundary settled — Phase 3 B3).** The agent runs as a **non-root** service user with the scoped `FelhomAgent` token for all API-covered work + a **narrow `sudoers` allowlist** for true host ops. Per Phase 3 (B3) the boundary is settled: the entire per-customer guest lifecycle — provision (by restore, §9), config, start/stop, snapshot, backup, **restore**, destroy — is token-covered. Genuine OS-root is confined to: (1) building/refreshing the **golden base image** (`keyctl` create is `root@pam`-only — one-time at enrollment + a maintenance cadence, §9); (2) **host mounts** (USB mount-by-UUID, systemd mount units / fstab); (3) **SMART / hardware sensors**. Root therefore never sits on the per-customer path. See `proxmox-platform.md` §3.6 for the role + boundary table. -- **`cloudflared` is a separate systemd service**, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process. - -## 4. Control model — reconcile + signed destructive ops - -Two channels, split by **reversibility**, not by transport. - -**(a) Desired-state reconciliation — steady state.** -The hub holds desired state for the host: which guests should exist (and at what spec), -the storage manifest, backup/retention policies, controller image versions. The agent -runs a reconcile loop converging actual Proxmox state → desired: idempotent, self-healing, -and tolerant of missed polls (drift is corrected on the next loop). Provisioning retries, -re-attach of a flapping USB target, redeploy of a crashed controller — all fall out of -reconciliation for free. - -**(b) Signed one-shot jobs — operator actions.** -Restore-now, decommission, force-backup, break-glass-enable. Discrete, run-once -(idempotency key), written to the customer-visible audit log, and **outside** the reconcile -loop — they are point-in-time and often destructive, and a reconciler must never re-run a -restore because it "sees drift." A one-shot job names a **target** ("restore guest X from -snapshot S"), not a procedure; the agent owns the *how*. - -**The reversibility gate (security-critical).** -"Signed jobs resist hub compromise" only holds if the agent also distrusts hub-supplied -*desired state* for destructive changes. The gate is by **provenance + data-bearing-ness, not -by verb**: - -- **The reconciler MAY act without an operator signature** when: (a) creating/starting/restarting; (b) destroying resources it created earlier **within the same journaled transaction** (compensating rollback, §10); (c) destroying resources it **tagged ephemeral/scratch** (e.g. restore-test scratch guests, §8). The ephemeral/scratch tag is **agent-internal provenance and is never accepted from the hub** — else a compromised hub could relabel a data-bearing guest as scratch to walk the gate. -- **An operator signature is always required** to destroy/overwrite any resource holding the only/primary copy of customer data — live-guest destroy, storage detach/wipe, restore-overwrite, decommission — *regardless of whether it arrives as a job or as a desired-state delta*. A compromised hub cannot forge them because the signing key is **not held by the hub** (it lives with the operator / a separate signing path; the hub only queues opaque signed blobs). -- **Healing a crashed controller is non-destructive by construction:** it is reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. (v0.33 precedent: `watchdog.go` restarts stopped stacks, it never destroys the guest.) - -Signed payloads carry a **nonce + expiry** (anti-replay: a captured "restore" job cannot be -re-injected later) and a target binding (host + guest id) so a signature can't be retargeted. -Notification-on-destructive-op is an **audit signal, never the guard** — a compromised hub -could both issue and suppress the notice, which is exactly why the *signature* (not the -notification) is the control. - -## 5. Hub ↔ agent protocol (host domain) - -**Box-initiated poll.** The hub never connects inbound. Each poll cycle exchanges: - -- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health, agent + controller versions, audit-log tail. -- **Down:** the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes). - -**Dead-man's-switch (essential, not optional).** In a box-initiated model the heartbeat -*is* the liveness signal — a box that stops checking in is otherwise invisible. The hub -alerts the operator when an agent misses its expected check-in window. This is the worst -failure mode for a managed service, so it gets first-class treatment hub-side. - -**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged -(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit, -**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via -the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed, -logged operation; it auto-expires. - -## 6. Agent ↔ controller local API - -The controller (in its LXC) reaches the agent (on the host) over the local bridge. - -- **Transport:** HTTPS to the host's bridge IP on a fixed port. -- **Auth:** a per-guest local token, minted by the agent when it deploys the controller and written into the guest's bootstrap config. The agent maps token → guest and **authorizes per guest**: a controller can only act on *its own* guest. This is the agent acting as the per-guest authorization gate from Part 1. -- **Surface (minimal, all scoped to the caller's own guest):** - - `GET /storage` — mounts available to this guest and their **class** (fast/slow), so the controller can place hot vs bulk volumes per `.felhom.yml`. (The agent owns the actual mounts; the controller just binds to the paths it's given.) - - `POST /snapshot` — snapshot *this* guest (the snapshot-before-deploy primitive). - - `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery). - - `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive). - - `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8). - - `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. - -Note what is *absent*: nothing here lets a controller touch another guest, the host, storage -attachment, or restore-overwrite. Destructive/cross-guest power stays operator-signed (§4). - -A controller can only `POST /rollback` (or snapshot/backup) **its own** guest — the agent maps -token → guest and authorizes per guest, so a compromised controller's blast radius is -**self-scoped and bounded** to its own guest. - -## 7. Storage manifest & reconciliation - -The manifest is the load-bearing contract. It absorbs the **persisted** disk-state fields that -`settings.StoragePath` carries today **and adds** `durable_id`/UUID — today the controller -re-derives the UUID from fstab each boot (Part 2 / Phase-3), so persisting it is an -improvement. Held in the hub, reconciled by the agent. - -Per target: - -| field | meaning | -|---|---| -| `type` | `local-dir` / `usb` / `nfs` / `cifs` / `pbs` | -| `durable_id` | UUID (USB), `server:export` (NFS/CIFS), `repo+fingerprint` (PBS) — survives box loss | -| `class` | `fast` or `slow`, set **once at attach**, with an IOPS marker; no runtime speed-test | -| `role` | `primary` / `vzdump-target` / `pbs-offsite` / `bulk-data` | -| `creds` | encrypted (NFS/CIFS/PBS); USB has none | -| `policy` | schedule + retention for this target | -| `state` | `attached` / `disconnected` / `decommissioned` | - -Reconciliation: ensure each `attached` target is mounted (USB-by-UUID via the sudoers -allowlist), each Proxmox storage entry matches, and `disconnected` targets are surfaced to -the hub (the storage watchdog — detect a USB drop in seconds, not at the next health cycle). - -**Placement is per-volume, not per-app.** Hot volumes (DB/config) → a `fast` target, -**enforced**; bulk volumes (media) → may live on `slow`, declared in `.felhom.yml`. - -A `bulk` volume **MUST** be realized as a `backup=0` **volume mount point** (or an external -bind mount) — **never** a Docker named volume in rootfs, which `vzdump` always captures -(verified, `phase3-findings.md` B2). Proven recipe: attach -`-mpN :,mp=/mnt/bulk,backup=0`, then -`docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk ` (or a -compose bind). The per-volume placement component (Part 2 §5(2)) enforces this at deploy. The -**DR consequence** of excluding bulk is covered in §8. - -**Field re-homing (from `settings.StoragePath`, Part 2):** `Label` → manifest (canonical); -`IsDefault`/`Schedulable` → manifest `policy`; `MigratedTo` + decommission → manifest `state`; -`StoppedStacks` → the **controller's `settings`** (app-domain: which apps to restart on -reconnect, not a host concern). - -## 8. Backup/restore orchestration - -Tiers double as backup *and* restore-source priority (fastest surviving source first), -per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a backup) → -**local second storage** (vzdump to dir/NFS/CIFS) → **PBS offsite** (the DR substrate). - -- **Quiescing (controller-driven for app-consistency):** an LXC has no fsfreeze - (`proxmox-platform.md` §4.2), so app-consistency is the controller's job: it learns a backup - is due (`GET /backup/due`, §6, or via its hub channel) → **quiesces** the app stack → - `POST /backup` → polls `GET /backup/status` → unquiesces. **An agent-initiated vzdump is - crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5). - Every Proxmox op is async → the agent polls `task exitstatus`, never trusts the POST return. -- **Bulk volumes have no DR coverage from the guest vzdump** — they are excluded (§7). Every - `bulk` volume needs an explicit own-backup decision: its own backup target per the manifest - `policy`, **or deliberately none** when the data is re-downloadable (customer informed). On - host-loss, un-backed-up bulk is gone; a **bind-mounted** bulk volume re-attaches only on the - *same* host, so cross-host DR needs the separate backup. A deliberate per-volume choice, - never a silent loss. -- **Key custody (PBS):** the **live** PBS key sits on the box so the agent can both back up - *and* run restore-tests. The hub holds only the **recovery-code-wrapped escrow** copy it - cannot open (zero-knowledge default). So: the box can restore-test; the operator cannot - read the data; the customer's offsite recovery code is the irreducible residual. -- **Self-restore-test:** the closing of the "tested restore is the critical gap" theme. The - agent periodically restores a backup into a **throwaway scratch guest**, boots it, runs - health checks, reports pass/fail, and tears it down. Zero-knowledge backups can *only* be - restore-tested by the box (the operator lacks the key) — so this lives in the agent by - necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often - as the lighter check. - -## 9. Provisioning & DR flows - -**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs -the `keyctl=1` feature flag, which Proxmox permits only for `root@pam` (Phase 3, B3) — not the -scoped token. But a token-authorized **restore preserves `keyctl`** (Phase 3, B3), so the agent -provisions **by restoring a golden base image**, never by `pct create` on the per-customer path: - -- A **golden base archive** — minimal Debian + Docker, `nesting=1,keyctl=1`, overlayfs — is - built once as `root@pam` **at enrollment** (when the agent legitimately holds root to mint its - Proxmox token) and refreshed on a maintenance cadence. This is the one place `keyctl`/root - provisioning lives — off the per-customer path. -- To provision guest G: restore the golden archive → new VMID (token-covered: `VM.Allocate` + - `Datastore.AllocateSpace`; `keyctl` preserved) → reset identity (MAC/hostname) → size the guest - (CPU/mem config + `pct resize` rootfs, token-covered) → attach storage mounts per the manifest - → deploy the controller → hand it bootstrap config. A mid-flight failure is journaled and - compensating-rolled-back (destroy the just-restored guest — allowed without a signature per §4, - same-transaction provenance). - -**Unified bring-up primitive.** Provisioning and DR-restore share the same token-covered front -half — *restore an archive → reset identity* — and differ only in the archive and the back half: -provisioning restores the **golden base** then deploys a fresh controller; DR-restore restores -the **customer's backup** (already containing controller + data), brings it up, and reattaches -external storage. One code path, exercised by every restore-test (§8). - -**Guest loss.** Agent restores G from the fastest surviving tier and resets identity -(MAC/hostname) so the restored guest rejoins cleanly — this *is* the unified restore primitive -above (customer-backup archive, DR back half). - -**Host/hardware loss.** Re-enroll the new host in **restore mode**; the hub — the durable -source of truth that survives box death — hands the new agent the existing identity, PBS -namespace, tunnel token, storage manifest, and a restore directive. Tunnel is reused from -the hub record, so DNS stays intact. - -## 10. Concurrency, crash-safety, idempotency - -- **Per-guest serialization.** Reconcile, one-shot jobs, and local-API calls all feed a - work queue that serializes mutations **per guest** (Proxmox dislikes concurrent conflicting - ops on the same guest). Independent guests proceed in parallel. -- **Operation journaling.** Multi-step async ops (provision, restore, controller-update, agent - self-update) are journaled with their in-flight Proxmox task ids. On agent restart, the - journal is replayed: resume-or-rollback, so a crash mid-restore never leaves a corrupt or - half-built guest. -- **Idempotency keys** on one-shot jobs (run-once across retries and restarts). - -## 11. Self-update - -- **Agent (the hard case — a host service, no snapshot-rollback).** **A/B layout:** download → - verify signature → stage as the inactive slot → flip a `current → good|new` symlink → restart. - **Revert authority lives outside the swapped binary** — `Restart=always` alone just - crash-loops a bad binary — so a **separate health-gate** (a systemd oneshot `ExecStartPost` - probe, or a tiny supervisor unit) flips `current` back to last-good and restarts on a failed - health window. The new version is **committed as "good" only after a clean health window**. - Triggered by a hub signed job within the update window; manual always allowed. Journaled (§10). -- **Controller (the easy case — it's a guest).** The agent owns the controller's lifecycle, - so the **agent updates the controller**: snapshot-before-update (free rollback, because the - controller *is* a snapshottable guest) → pull new image → redeploy → health-check → rollback - on failure. This resolves the Part-2 `selfupdate/` open: the controller is **agent-managed**, - not self-updating; the controller's old self-update path is removed. - -## 12. Secrets at rest on the host - -The agent holds, root-only on the host fs: the scoped Proxmox token, the hub API key, the -operator's **public** verify key (for §4 signatures — public, low-risk), the Cloudflare -tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**. The privilege -and the secret footprint that left the controller now concentrate here — which is the whole -argument for §3's root-minimization and a small, auditable agent. - -## 13. Open items / what this unblocks - -Resolved here: tunnel placement (host, agent-managed, own systemd service), the -reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update -ownership, the local-API surface, the storage-manifest schema, **provision-by-restore**, and -the **root-vs-API boundary** (Phase 3, B3). - -Still open: - -- Multi-tenant **resource fairness** on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass. -- Operator-side **signing tooling** — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub"). -- Hub-side **desired-state editing UX** and the host-domain report schema details — belong to the hub architecture doc. -- **Golden base image** refresh cadence + fleet versioning — who triggers a rebuild, how the per-host image version is tracked (operational detail, not blocking; §9). - -This doc hands the implementation three contracts it was waiting on: - -1. the **local-API surface** (§6) → the controller's NEW local-API client, snapshot-before-deploy, and self-restore-test wiring (Part 2); -2. the **storage-manifest schema** (§7) → the `settings.StoragePath` reshape and per-volume hot/bulk placement (Part 2); -3. the **backup contract** (§7–8) → the destination for the app-data-backup package extracted in the Part-2 refactor. - ---- - -## Changelog — design-review + Phase-3 fold-in (2026-06-08) - -- **NEW provision-by-restore** (§9): the agent provisions by **restoring a golden base image** - (token-covered, preserves `keyctl`), never `pct create` on the per-customer path; one unified - restore primitive shared with DR. §2 responsibility + §3 boundary updated. -- **B3** (§2/§3): replaced "Phase-1 minimal role" with the validated **`FelhomAgent`** operator - role; root-vs-API boundary **settled** (root only for golden-image build, host mounts, SMART). -- **B1** (§4): reversibility gate rewritten as **provenance + data-bearing** (scratch tag is - agent-internal, never hub-supplied; crashed-controller heal is non-destructive in-place). -- **B2** (§7/§8): validated bulk-as-`backup=0`-mountpoint recipe + the **bulk-DR consequence** - (excluded bulk needs its own backup decision). -- **S1** (§6/§8): `GET /backup/due` added; controller-driven quiescing; agent vzdump is - crash-consistent only. **S2** (§10/§11): A/B self-update with external revert authority; - controller-update + agent self-update journaled. **S3** (§7): `StoragePath` field re-homing. - **S4:** geo non-responsibility added (§2). **M2** (§7): manifest "absorbs + adds durable_id". - **§6:** rollback is self-scoped/bounded. **§13:** golden-image refresh cadence added as open. \ No newline at end of file diff --git a/docs/architecture/04-control-plane-authorization.md b/docs/architecture/04-control-plane-authorization.md deleted file mode 100644 index c9abbbb..0000000 --- a/docs/architecture/04-control-plane-authorization.md +++ /dev/null @@ -1,154 +0,0 @@ -# Architecture Part 4 — Control-plane authorization (operator signing) - -> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`. -> To be reviewed by Claude Code against that spike + `03` §4, then placed at -> `docs/architecture/04-control-plane-authorization.md`. -> -> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate). -> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge." - -## 1. Purpose & scope - -`03` §4 gates **destructive/irreversible** operations behind an operator signature the hub cannot -forge. That gate is only real if signing is real. This doc defines the signing mechanism: the -primitive, the keys, rotation, the three components' roles, and the operator workflow. The -*policy* (what needs a signature) lives in `03` §4; this is the *how*. - -**Recap of what needs a signature** (from `03` §4, by reversibility, not by verb): destroying or -overwriting any resource holding the only/primary copy of customer data — live-guest destroy, -storage detach/wipe, restore-overwrite, decommission — **regardless of whether it arrives as a job -or a desired-state delta**. Benign convergence (deploy a guest, attach storage, restore to a *new* -guest, bump a version) runs on normal hub auth, unsigned. Most recovery is therefore unsigned; -signed ops are rare and deliberate. - -## 2. Primitive — SSH signatures (SSHSIG) - -Confirmed by Phase 4: destructive ops carry an **SSH signature** (`ssh-keygen -Y sign`, the armored -`SSHSIG` format), verified by the agent in Go (`golang.org/x/crypto/ssh`) — `pem.Decode` → -`ssh.Unmarshal` → `ssh.ParsePublicKey` → `pub.Verify`. ~40 lines of framing, no hand-rolled crypto. - -**Why SSHSIG and not raw Ed25519 / minisign:** SSHSIG verification dispatches on the key type -embedded in the signature, so the **same verifier accepts a software key (`ssh-ed25519`) today and -a FIDO2 hardware key (`sk-ssh-ed25519@openssh.com`) later** — which is exactly the hardware-ready -foundation we want (§7). A raw-Ed25519 verifier cannot consume an sk signature (flags+counter, -different signed-data), so it would force a verifier change on every box at hardware-adoption time. -SSHSIG buys key-type-agnosticism for a one-file framing cost (Phase 4 §5–6). - -### 2.1 The signed object — canonical op blob -The signature covers an op blob (Phase 4 §2): - -``` -{ op, target:{host_id, guest_id}, params, nonce, issued_at, expires_at, key_id } -``` - -- **Canonical form is a *signer-side* requirement** — JSON, keys sorted at every level, no - insignificant whitespace, UTF-8 — so the blob is deterministic and human-auditable. The - **verifier trusts the exact bytes it receives** (it verifies the signature over the raw bytes and - parses those same bytes for fields), so there is no canonicalization-mismatch risk on the verify - side. The canonical form is the shared contract between the operator CLI and the agent (both Go). -- `nonce` ≥128-bit random; `issued_at`/`expires_at` a short window (minutes); `key_id` identifies - the signing key (rotation/audit). - -### 2.2 Domain separation — the namespace -The SSHSIG **namespace** `felhom-op-v1` is a **fixed constant in the verifier**, never -caller-supplied. A signature minted for any other namespace must not verify (proven). This stops a -signature made for one purpose being reused for another. - -### 2.3 Verify pipeline (order is load-bearing) -`namespace → allow-list → crypto verify → target binding → time window → nonce`. The **nonce is -recorded last**, only after everything else passes, so an invalid signature can never consume a -nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4): -- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box - A cannot be replayed at box B); -- **time window** — `now ∈ [issued_at, expires_at]`; -- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts** - and expiry-pruned; a non-persistent store reopens the replay window after a restart). - -The Phase-4 reference verifier (`VerifySignedOp`) is the seed of the agent's implementation. - -## 3. The keys — two-key model, software now - -Both software (SSH-format) keys today; both are also valid FIDO2-resident keys later with no box -change (§7). - -- **Operational signing key** — the "master stamp" for destructive ops. A **dedicated** key (NOT - the operator's daily SSH login key), passphrase-protected, on the operator workstation. Used only - for destructive ops — rare, so its exposure is low. -- **Cold recovery key** — generated once, kept **offline** (password manager / a USB held back / - printed). Never used for ordinary ops; its sole power is to authorize rotating the operational key - if that key is lost or compromised. - -Both **public** keys are pinned onto the agent at enrollment (the allowed-signers set). The -operational key is authorized for ops; the recovery key is authorized **only** for key-rotation -instructions. - -**Allowed-signers is a set** → single signer today; **quorum (N-of-M) for the highest-blast ops is -just set sizing + a threshold policy**, addable later without a redesign (Phase 4 §8). Out of scope -now. - -## 4. Rotation & compromise recovery - -The agents pin the operator public keys. The danger: rotation must **not** flow as plain hub config, -or a compromised hub re-pins its own key and forges everything. So **every re-pin is itself a signed -op the agent verifies** (same pipeline, §2.3) — never unauthenticated config. - -- **Planned rotation:** the *current* operational key signs a "new operational public key = X" op; - the agent accepts it because it's signed by the trusted current key (key-signs-key). -- **Operational key lost/compromised:** the **cold recovery key** signs the re-pin; the agent accepts - it because the recovery key is pinned and authorized for rotation. The compromised key is removed - from the allowed set in the same signed op. -- **Both keys gone:** on-site physical re-enrollment (last resort — re-establishes the trust root the - way initial enrollment did). - -## 5. Component roles - -- **Operator tooling (the workstation).** A signing CLI behind a thin **`Signer` interface** - (`Sign(blob) → signature`). The backend today is a **file key**; a **FIDO2/PIV** backend drops in - later (§7) with no change to the blob format, the hub, or the agent. Holds the operational private - key (passphrase-protected); can reach the cold recovery key when rotation is needed. -- **Hub.** Queues the **opaque** signed blobs and surfaces pending destructive ops + their signature - status in the operator UI. Holds **no** private key and cannot sign — a compromised hub can only - queue blobs the agent rejects. (Matches `03` §4 / box-initiated poll.) -- **Agent (each box).** Pins the allowed-signers set (operational + recovery) at enrollment; runs the - verify pipeline (§2.3) on any destructive op before executing; writes every signed op to the - customer-visible **audit log**. Notification-on-destructive-op is an audit signal, never the guard - (a compromised hub could issue *and* suppress notice — the signature is the control). -- **Enrollment.** Pins the initial operational + recovery public keys onto the agent during the - physical-presence provisioning step (the trust root is established on-site, not via the hub). - -## 6. Operator workflow - -- **Routine work** (deploy, monitor, attach storage, restore to a *new* guest): no signing, zero - overhead. -- **A destructive op** (rare): the operator runs the signing CLI on their workstation — which builds - the canonical blob, signs it (passphrase, or later a hardware touch), and posts it to the hub - queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the - desk. **Never** a site visit. - -## 7. Hardware readiness (Viktor's "build the foundation now") - -Software `ssh-ed25519` now; a FIDO2 `sk-ssh-ed25519@openssh.com` key later is a **no-op on the -boxes** — proven end-to-end against the OpenSSH spec in Phase 4 §5 (the unchanged verifier accepts a -spec-faithful sk signature). At hardware adoption the operator generates an sk-key, points the -`Signer` backend at it, and updates the allowed-signers entry; nothing on the boxes changes. - -Two honest notes: -- **Confirm with a real device at adoption.** §5 was validated to spec, not against live hardware — - a 5-minute real-key round-trip should confirm it (no surprise expected; signer/library/device all - follow the same spec). -- **Optional future hardening:** require the FIDO2 **user-presence (touch) flag**. The verifier is - crypto-only today (correct for software keys); enforcing the flag is a small later option once - hardware is in use. - -## 8. Open items -- **Quorum policy** (N-of-M per op-class, e.g. two signatures for decommission) — deferred; the - allowed-signers-set foundation supports it. -- **Signing-key passphrase UX** on the workstation (ssh-agent / askpass) — minor operator-tooling - detail. -- **Hub-side pending-op UI** (showing ops awaiting signature + audit) — belongs to the hub doc. - -## 9. What this unblocks -Closes the `03` §4 "undesigned signing path." Hands the implementation: the **canonical blob spec** -(§2.1) + the **`VerifySignedOp` reference** (Phase 4 §7) for the agent's verify path, the -**`Signer` interface** for the operator CLI, and the **allowed-signers pinning** step for enrollment. -The hub's signed-job queue + pending-op UI carry into the hub architecture doc. \ No newline at end of file diff --git a/docs/architecture/05-hub-architecture.md b/docs/architecture/05-hub-architecture.md deleted file mode 100644 index 61bb238..0000000 --- a/docs/architecture/05-hub-architecture.md +++ /dev/null @@ -1,223 +0,0 @@ -# Architecture Part 5 — The Hub - -> Status: design draft (decision content). To be validated by Claude Code against the **actual -> felhom-hub source** (`felhom.eu` repo, `hub/`) + Parts 01–04, then placed at -> `docs/architecture/05-hub-architecture.md`. -> -> The hub is **not** greenfield — it's a mature service (felhom-hub v0.6.3, Go + SQLite on k3s, -> `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus the new -> data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing). - -## 1. Source-of-truth model — two drivers, two directions - -The single most important framing, and the one that governs everything below: the hub is **not** a -monolithic source of truth. State flows in two directions with opposite drivers. - -- **Operator-driven *intent* — hub authors, agent reconciles (top-down).** Which guests should - exist and their spec, storage *policy* (a target's role/class/backup schedule), controller + - golden-image versions, identity, tunnel. The operator sets these in the hub; the agent converges - toward them. Here the hub *is* the source of truth. -- **Box/customer-driven *reality* — box authors, pushes up, hub mirrors (bottom-up).** Which USB - drive is *physically* attached (and its `durable_id`), what apps are deployed and where, the - customer's controller configs/settings, host/guest health, latest PBS snapshot pointers. The - customer or the physical world drives these; the box reports them; the hub stays an up-to-date - **mirror** but is **never** the driver. - -They meet at a **handshake**, not a tug-of-war. Storage is the clearest case: the customer plugs in -a drive → the agent *detects* it and reports `durable_id X attached` (reality) → the operator -assigns `role=bulk, class=slow, backup=weekly` (policy, intent) → the agent reconciles that policy -*onto the detected drive*. **Apps never enter the reconcile loop** — app deployment is the -controller's domain (customer- or operator-driven, inside the guest); the hub only mirrors the -resulting inventory. **Reconciliation applies to infrastructure; the app/customer layer is mirrored.** - -## 2. Data model (Part 1 decision (b): customer-anchored) - -A customer's deployment is one **Host** (its agent) plus one-or-more **Guests** (its controllers). -1 customer = 1 host + N guests; the shared-host multi-tenant case is deferred (not precluded — the -`hosts` table is the seam it would use). - -- **`customer_configs`** (existing) — the Customer anchor: identity, domain, email, - `retrieval_password`, status, config_json. Unchanged role. -- **`hosts`** (new) — `host_id PK, customer_id, api_key` (the agent's hub key), `agent_version`, - desired-state intent (storage manifest + policies + golden-image version, as JSON), a per-host - **`desired_generation`** counter, the slim DR record (§9), timestamps. -- **`guests`** (new) — `guest_id PK, customer_id, host_id, api_key` (the controller's hub key), - `display_name, controller_version`, per-guest **`desired_spec_json`** (CPU/mem/disk, versions), - timestamps. - -**Per-reporter keys:** today's per-customer `customer_configs.api_key` becomes per-reporter — -`hosts.api_key` (agent) and `guests.api_key` (controller). The hub resolves a presented Bearer key → -host or guest → customer; `customer_configs.api_key` goes unused once auth resolves via the new keys. -**Clean cutover:** no dual-model support; the demo re-enrolls fresh into `host + guests`. - -## 3. Report ingest — two domains - -The single controller report splits. The de-privileged controller no longer sees host disks/storage/ -backup, so its report **slims** (it loses System/Storage/Backup, keeps app-domain). - -- **`POST /api/v1/host-report`** (new, agent) → **`host_reports`**: host CPU/RAM/disk, per-guest - up/down + spec, storage-target status (attached drives + `durable_id` + reachability), last backup - + restore-test per target, latest PBS snapshot pointers, `cloudflared` health, agent + controller - versions. Denormalized columns for the dashboard; full `report_json`. Index `(host_id, received_at - DESC)` + `(customer_id, received_at DESC)`. -- **`POST /api/v1/report`** (existing, slimmed controller) → the renamed **`guest_reports`**: it - gains `guest_id` + `host_id`; its `cpu/memory` denorm now means *guest-level*; `backup_last_snapshot` - goes quiet (backup status lives in `host_reports`). App telemetry / log issues stay. - -These two streams are the bottom-up mirror of §1 — they keep the hub current without a separate push. - -## 4. Liveness / dead-man's-switch - -Evolves the existing staleness checker (60s **cadence**, 30m/1h **thresholds** — OK <30m, down at -2× = >1h; today: controller-report recency → `node_stale`/`down`/`recovered`): - -- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's - liveness signal; a silent agent = the box is gone (the critical alert). -- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster - than waiting for a guest report to go stale. -- **Guest-report recency = secondary** app-level signal. - -**Backup-deadline checker:** today it is *event-based* — it scans for `backup_completed`/`backup_failed` -events since local midnight and alerts if none. Two changes: (1) **mechanism** — move it to a field -check on `host_reports`' last-backup-per-target (cleaner now that backup state arrives in the host -report); (2) **emitter** — the de-privileged controller no longer runs backups, so the **agent** is the -source of the last-backup status (Part 3 §8). Without re-homing the source, the deadline check would go -silent after the controller stops backing up. - -## 5. Desired-state serving - -The operator's **intent** (§1 top-down) lives as JSON on `hosts`/`guests` (storage manifest + -policies + golden version on the host; per-guest spec + versions on the guest) with a per-host -`desired_generation`. The agent pulls its host's desired state on poll (with the generation, so it -reconciles only on change and reports which generation it has converged to). - -- **Benign convergence** (create a guest, attach storage per policy, bump a version, adjust a - non-destructive policy) → the agent reconciles freely. -- **Destructive convergence** (guest removal = destroy, storage detach/wipe, data-losing resize) → - the agent requires a **matching signed op** (§6) before executing that delta; absent/invalid → it - refuses and reports `pending_signature`. - -**Geo is *not* in the agent's desired state** — it's customer→hub→Cloudflare (§7); the agent never -touches WAF. - -## 6. Authorization — signed-op queue + editing flow - -Implements Part 4's gate on the hub side. The hub holds **no signing key**. - -- **`signed_ops`** (new): `op_id, customer_id, host_id, target_guest, op_type, op_blob (canonical - JSON), signature (armored SSHSIG), status (pending_signature → signed → delivered → executed / - failed / expired / rejected), nonce, issued_at, expires_at, executed_at, result`. -- **Editing flow:** the operator edits a customer's desired state, reusing the existing config-form + - diff UX. Note the **transport inverts**: today's "Push" is a hub→box *inbound* POST (forbidden by the - box-initiated model); here "publish" means **write to desired state, delivered on the next agent/ - controller poll**. The form and diff carry over; the push transport does not. The hub diffs vs current - and **classifies each delta** (B1 rule): - - **benign** → published straight to desired state; - - **destructive** → the hub generates the canonical op blob and routes it through signing. -- **Signing hand-off (Part 4 option (b)):** a local operator CLI (`felhom-sign --pending`) fetches - the pending blob from the hub, signs it on the workstation with the dedicated key, and posts the - signature back into `signed_ops`. The hub never sees the key. -- The agent polls `signed_ops` for its host alongside desired state, verifies (Part 4 pipeline), - executes, and reports status → the hub logs to the existing **`events`** audit trail. -- **Classification lives in both places, with different jobs:** the hub classifies at *edit time* - for UX (prompt to sign); the **agent's classification is the authoritative guard** (a compromised - hub could skip the prompt, but the agent still enforces the signature). -- A **pending-ops view** per customer shows the lifecycle (awaiting signature → awaiting agent → - executed). - -## 7. Geo enforcement (Part-2 S4) - -The hub already holds the CF API token and already has a remove-all path -(`internal/web/configs.go` `handleGeoDisable` → `cloudflare.RemoveGeoRules`). **But the token is -dual-purpose today** — DNS-01/ACME *and* WAF/geo — and `configgen.Generate` deep-merges it (via -`config_json`) into the generated `controller.yaml`, so it currently ships **down to the box**. Two -things follow: - -- **ACME assumption (must be stated, not skipped):** in the Cloudflare-Tunnel-default model the edge - terminates TLS, so the box needs no public certificate and the **DNS-01/ACME use of the token goes - away**. Granting that, the token comes fully off the box and lives hub-only. (If any box still does - DNS-01, the token cannot fully come off — so this assumption is load-bearing.) -- **`configgen` must stop emitting `cf_api_token`** into `controller.yaml` (drop it from the merge / - relocate it to a hub-only field). - -The delta: the **customer sets geo in the controller UI → the controller reports the geo desired-state -up → the hub reconciles it into the Cloudflare WAF** (rather than the box calling the CF API). The hub -keeps the remove-all override for self-lockout. The controller no longer calls the CF API. - -## 8. Enrollment (evolution of the existing retrieval-password/config-gen flow) - -Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase) returns a deep-merged -`controller.yaml`. New: - -- Enrollment mints the **agent identity first** (the agent then provisions controllers), pins the - **operator signing public keys** (Part 4 — operational + cold recovery) onto the agent, and the - agent mints each controller's bootstrap (its hub guest key + local-API token). -- A **restore-mode** re-enrollment (§9) hands an existing identity to a fresh agent. - -The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the -agent-first + key-pinning + restore-mode steps. - -## 9. DR model - -The headline: the **old heavy infra-backup push retires** — not because the hub authors everything -(§1 says it doesn't), but because (a) the box-driven mirror already arrives via the §3 report streams, -and (b) the actual app **data + configs live inside the PBS guest snapshot**. So a separate -config+secrets+restic-password infra-backup blob is redundant. - -What remains: -- the **report streams** keep the hub's mirror current (storage layout + `durable_id`s, app inventory, - snapshot pointers) — but this mirror is **convenience, not the DR source of record** (reports are - pruned by age); -- the agent **escrows the recovery-code-wrapped PBS key** to the hub (the one artifact only the box - can produce — zero-knowledge: the hub stores it, cannot open it); -- a **slim DR record** on the `hosts` row (PBS namespace + repo fingerprint + the wrapped escrow key). - These last two are *box-reported* columns on an otherwise operator-intent row — labelled as such so - the §1 two-driver split stays legible per column. - -Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers -hit it) **and** `infra_backups` (the deprecated legacy mirror). The slim DR record folds onto `hosts` -instead. The **controller's infra-backup push is removed** (it's de-privileged). - -**Recovery (host loss):** the new agent re-enrolls in **restore mode**; the hub hands it the durable -record — and DR reads from the **durable sources, not the prunable report mirror**: operator intent -(desired-state on `hosts`/`guests` — identity, tunnel token, storage manifest), the slim DR record -(PBS namespace + repo fingerprint), the **wrapped escrow key**, and **PBS's own snapshot enumeration** -(the agent lists snapshots once it has the namespace + unwrapped key). Guest inventory + app data come -from **inside the PBS guest snapshots**, not from a retained `host_report`, so recovery doesn't degrade -when the last report has aged out. The **customer provides their recovery code at the agent**, which -unwraps the PBS key locally (never sent to the hub); the agent restores guests from PBS, resets -identity, reuses the tunnel. The customer recovery code is the irreducible residual (the premium -operator-managed custody tier avoids it, at the cost of the operator holding the key). The old -controller-targeted `GET /recovery/{id}` is replaced by this agent restore-mode flow. - -## 10. What persists from today (unchanged or lightly adapted) - -The Customer record (`customer_configs`); config generation/retrieval (`configgen`); the two-tier -notification system (operator English / customer Hungarian, Resend, cooldowns); `events` + audit; -`app_telemetry` / `app_log_issues`; customer lifecycle actions (block/unblock, trigger-update, -delete); the asset manager; and the dashboard — adapted to render the **host + guests** view per -customer instead of a single controller. - -## 11. Schema deltas (grounded in store.go's idempotent style; clean cutover) - -- **NEW:** `hosts`, `guests`, `host_reports`, `signed_ops`. -- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data - migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean - guest-level; `backup_last_snapshot` goes quiet. -- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the - slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`. -- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR - record replaces them on `hosts`. -- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`, - `app_telemetry`, `app_log_issues`. -- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth - rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup - push; any that carry over should scope to the resolved host/guest → customer under §2. - -## 12. Open items -- Operator signing-key operational mechanics (Part 4 §8) — the hub-side pending-op UI is here; the - key custody/rotation tooling is Part 4's. -- Multi-tenant resource fairness (deferred shared-host case). -- Hub-side desired-state **editing UX** specifics (form/diff wiring) — to be grounded against - `hub/internal/web/configs.go` at implementation. -- Golden-image refresh cadence / fleet versioning (carried from Part 3 §13). \ No newline at end of file diff --git a/docs/architecture/_design-review.md b/docs/architecture/_design-review.md deleted file mode 100644 index 4f2c7ea..0000000 --- a/docs/architecture/_design-review.md +++ /dev/null @@ -1,260 +0,0 @@ -# Critical design review — Proxmox re-platform doc set - -> ✅ **RESOLVED (2026-06-08).** All findings folded into 01/02/03 + `proxmox-platform.md` -> (Phase-3 spike run for B2/B3 → `tests/phase3-findings.md`). **Folded:** B1 (03 §4), B2 -> (03 §7/§8 + platform §4.7), B3 (03 §2/§3 + platform §3.6), S1 (03 §6/§8), S2 (03 §10/§11), -> S3 (03 §7), S4 (01 §5/§7 + 02 + 03 §2), S5 (01 §7/§11 + 02 §6), S6 (02 §5), M1 (02 §3), -> M2 (03 §7), M3 (03 §10), §6-residual (03 §6). Plus the two Phase-3 design updates: -> provision-by-restore (03 §9) and the settled root-vs-API boundary (03 §3). **Deferred/none:** -> no finding was deferred; the pre-existing open items (operator signing-key mechanics, -> multi-tenant fairness, hub-side desired-state UX, golden-image refresh cadence) remain -> flagged in 03 §13. This artifact can be deleted once confirmed. - -Working artifact. Review pass over `01-topology-and-trust.md`, `02-controller-module-map.md`, -`03-host-agent.md`, `proxmox-platform.md`, and the Phase 0 / Phase 1-2 findings, grounded -against the v0.33 source (`felhom-controller/controller/`). Every finding cites a -file+line or a doc section. Severity: **blocking** / **should-fix** / **minor**. - -Two findings are self-corrections of my own earlier work (`02` and `proxmox-platform.md`) — -flagged as such. - ---- - -## Ranked summary - -| # | Severity | Finding | Where | -|---|---|---|---| -| B1 | **blocking** | Reversibility gate contradicts the self-heal reconcile loop — crashed-guest healing can require a signature-gated destroy → reconcile stalls | `03` §4 vs §4(a) | -| B2 | **blocking** | vzdump bulk-exclusion only works for **volume** mount points; Docker **named volumes live in the LXC rootfs and ARE captured** → naive placement silently backs up the 1 TB media drive. Unvalidated by spike. | `03` §7 vs `proxmox-platform.md` §4.3 + pct manpage | -| B3 | **blocking** | Agent's Proxmox role is called "the minimal role from Phase 1" — but that role is the *narrow self-backup* role that Phase 1 proved is **denied** create/allocate/restore. The agent's operator-tier role is undefined. | `03` §2/§3 vs `phase1-2` §1.3-1.4, `01` appendix | -| S1 | should-fix | Quiescing for agent/hub-scheduled backups has **no agent→controller channel** — the local API is controller→agent only | `03` §6, §8 | -| S2 | should-fix | Agent self-update revert authority unspecified — if the new binary won't boot, nothing outside it can flip back | `03` §11 | -| S3 | should-fix | Storage manifest drops fields `settings.StoragePath` carries today (Label, Schedulable/default, StoppedStacks, MigratedTo) with no re-homing stated | `03` §7 vs `settings.go:90-103` | -| S4 | should-fix | Geo-restriction WAF ownership + Cloudflare **API token** placement unspecified after tunnel placement was locked; zone-wide token in a guest is a blast-radius concern | `03` (absent), `01` §3, `config.go` InfrastructureConfig | -| S5 | should-fix | Cross-doc staleness: `01` §11 still lists tunnel placement OPEN; `02` §6 lists geo "blocked on tunnel placement" — both resolved by `03` §13 | `01` §11, `02` §6 vs `03` §13 | -| S6 | should-fix (self-correct) | `02` put self-restore-test **orchestration** in the controller; `03` correctly makes it agent-owned (controller only reads status) | `02` §5(3) vs `03` §6/§8 | -| M1 | minor (self-correct) | `02` §3 lists `UUID` as a `settings.StoragePath` field — it isn't; UUID is derived from fstab at runtime | `02` §3 vs `settings.go:91-103` | -| M2 | minor | `03` §7 says the manifest "absorbs the disk-state fields StoragePath carries today" incl. UUID — UUID isn't persisted today, so the manifest *adds* it (an improvement, not absorption) | `03` §7 | -| M3 | minor | controller-update is not in `03` §10's journaled-ops list, though it's a multi-step async op | `03` §10 vs §11 | - -**Values check: clean.** No DR/key-custody/offboarding path leaves a customer locked out. -Zero-knowledge DR (`03` §8, `01` §8) correctly makes the customer recovery code the -irreducible residual; the operator cannot read data and the box can still restore-test. -No hostage path found. - -**Locked premises:** reviewed for soundness/consistency only; not relitigated. - ---- - -## Blocking findings - -### B1 — The reversibility gate stalls the self-healing reconcile loop -**Where:** `03` §4(a) vs the gate in §4. -**What:** §4(a) lists "redeploy of a crashed controller" as benign convergence that "falls -out of reconciliation for free." The gate then lists **guest destroy** among the -irreversible ops that require an operator signature "*regardless of whether they arrive as a -job or as a desired-state delta*." These collide: if healing a wedged guest requires -destroy+recreate (corrupt rootfs, failed in-place restart, half-built guest from an -interrupted provision), the reconciler hits a signature-gated op and **cannot proceed -without an operator** — the loop either stalls or silently gives up, defeating "self-healing -… tolerant of missed polls." -**Why it matters:** This is the security-critical control model. A fuzzy benign/destructive -line is unimplementable: either the reconciler can destroy (and a compromised hub's desired -state can wipe guests — the exact threat §4 exists to stop), or it can't (and self-heal is a -fiction for the crashed-guest case). -**Grounding:** `03` §4 self-describes the gate as "security-critical"; §9/§10 already rely on -the reconciler rolling back "a half-built guest" — which *is* a destroy of a customer-id-bound -resource, contradicting the blanket "guest destroy needs a signature." -**Suggested fix (crisp, implementable rule):** Scope the reconciler's destructive verbs by -*provenance and data-bearing-ness*, not by verb: -- The reconciler MAY, without a signature: (a) create/start/restart; (b) destroy resources it - **created earlier in the same journaled transaction** (compensating rollback, §10); (c) - destroy resources **tagged ephemeral/scratch** (restore-test scratch guests, §8). -- Destroying or overwriting any resource that **holds the only/primary copy of customer data** - always needs an operator signature. -- **Healing a crashed controller is non-destructive by construction:** the controller is - reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the - LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. State - this explicitly so the two clauses stop colliding. (The v0.33 self-heal precedent is already - in-place restart: `watchdog.go` restarts stopped stacks, it never destroys the guest.) - -### B2 — vzdump bulk-exclusion: the rootfs-Docker-volume trap -**Where:** `03` §7 ("Bulk external mounts are excluded from the guest's vzdump (a per-mount -backup flag)"). -**What:** Two grounded problems: -1. The flag is real but narrow. The pct manpage (verified): `backup=` — - *"Whether to include the mount point in backups (**only used for volume mount points**)."* - It does **not** apply to bind mounts / device mounts (those are handled separately). -2. The trap: `proxmox-platform.md` §4.3 (validated in `phase1-2` §2.2) proved that **Docker - named volumes live inside the LXC rootfs and ARE captured by vzdump** — a sentinel in - `pgdata` survived. The default Felhom app uses Docker named volumes. So unless bulk data is - deliberately placed on a **dedicated Proxmox volume mount point** (backup=0) or a bind - mount, a "bulk" volume will be an ordinary named volume in rootfs and will be **silently - swept into the whole-guest image** — exactly the 1 TB-media-in-every-backup outcome §7 says - it prevents. -**Why it matters:** Backup size/cost and RPO blow up silently; the failure is invisible until -a media drive fills the vzdump target. This is load-bearing for the §8 tier model. -**Grounding:** pct manpage (fetched 2026); `proxmox-platform.md` §4.3; `phase1-2` §2.2. -Not covered by any spike — `proxmox-platform.md` §6 "not yet validated" should gain this row. -**Suggested fix:** Make the placement contract explicit: a `bulk` volume **must** be realized -as a dedicated LXC mount point (volume mountpoint with `backup=0`, or an external bind mount), -**never** a Docker named volume in rootfs. The per-volume placement component (`02` §5(2)) -must enforce this at deploy. Add a Phase-3 spike: create an LXC with a `backup=0` volume -mountpoint + a bind mount, vzdump it, confirm both are excluded and the rootfs+`backup=1` -volume are included. - -### B3 — The agent's Proxmox role is mis-grounded as "the Phase-1 minimal role" -**Where:** `03` §2 ("scoped Proxmox API token (minimal role from Phase 1)"), §3 ("the -Phase-1 minimal role is the API floor"). -**What:** Phase 1's minimal role (`FelhomSelfBackup` = `VM.Audit, VM.Snapshot, VM.Backup, -Datastore.AllocateSpace, Datastore.Audit`) is the **narrow self-backup** role scoped to one -guest, and Phase 1 explicitly proved it is **denied (403)** on create/allocate -(`phase1-2` §1.3 call #7) — i.e. exactly the operator-tier ops the agent's whole job consists -of (provision, restore, storage allocation). Worse, `01` appendix states that guest-side role -"**is not used** — we chose the agent-mediated path." So `03` cites, as the agent's role -floor, a role that (a) the architecture discarded and (b) is provably insufficient for the -agent. -**Why it matters:** The agent's actual operator-tier role is **undefined**. Provisioning, -restore, and storage management cannot be built or hardened against an undefined privilege -set, and §3's root-minimization argument ("the Phase-1 minimal role is the API floor") -collapses because that floor can't create a guest. -**Grounding:** `phase1-2` §1.3 (create CT = 403), §1.4 (role = self-backup only); `01` -appendix ("not used … confirmed restore = operator-tier"); `proxmox-platform.md` §3.4. -**Suggested fix:** Replace the Phase-1 reference with a **new agent operator role** to be -defined and least-privilege-tested in a Phase-3 spike — minimally `VM.Allocate`, `VM.Config.*`, -`VM.PowerMgmt`, `VM.Snapshot(.Rollback)`, `VM.Backup`, `VM.Audit`, `Datastore.Allocate(Space)`, -`Datastore.Audit`, plus whatever storage-attach needs (see S4/root-boundary below). Keep §3's -"API token, not root, where the API suffices" principle — that part is sound — but stop -calling it the Phase-1 role. - ---- - -## Should-fix findings - -### S1 — No agent→controller channel for backup quiescing -**Where:** `03` §6 (local API is controller→agent only) vs §8 ("the controller stops the app -stack … before a guest vzdump where app-consistency matters"). -**What:** App-consistent LXC backup requires the controller to quiesce (no fsfreeze for LXC — -`proxmox-platform.md` §4.2, `phase1-2` §2.1). But the §6 surface is entirely controller→agent; -the box-initiated model forbids the hub calling in, and there is no agent→controller call -defined. For a **hub/agent-scheduled** backup (schedule lives in the manifest `policy`, §7), -the agent has no way to tell the controller "quiesce now." -**Why it matters:** Either scheduled backups silently fall back to crash-consistent (relying -on WAL recovery, which `phase1-2` §3 warns is unvalidated under write load), or the feature -can't be built as drawn. -**Suggested fix:** Make backups **controller-driven for app-consistency**: the controller -learns due/policy via its own hub channel (or a `GET /backup/due` on the local API), quiesces, -calls the existing `POST /backup`, then unquiesces on completion. Document that agent-initiated -vzdump is crash-consistent only. (No inbound-to-guest channel needed — preserves §3/§5.) - -### S2 — Agent self-update revert authority unspecified -**Where:** `03` §11 ("a watchdog reverts to last-good if the new binary fails to come up -healthy"). -**What:** The agent is a single host systemd service with `Restart=always` (§3). If the new -binary crashes on startup, systemd just restarts the **same bad binary** in a loop. "Revert -to last-good" cannot be done *by* the thing that won't boot. §11 doesn't name the actor. -**Why it matters:** A bad self-update can brick the crown-jewel host agent — the one component -that recovers everything else — with no automatic recovery, requiring break-glass. -**Suggested fix:** Put revert authority **outside** the swapped binary: e.g. an A/B symlink -(`current → good|new`) where a separate systemd oneshot health-gate (`ExecStartPost` probe; on -failure flip the symlink back and restart), or a tiny supervisor unit. Boot-into-last-good + -explicit "commit" after a clean health window is the robust pattern. Add agent-update to the -§10 journal so an interrupted swap is resumable. - -### S3 — Manifest schema omits live `StoragePath` fields without re-homing them -**Where:** `03` §7 table vs `settings.go:90-103`. -**What:** Today's `StoragePath` carries `Label`, `IsDefault`, `Schedulable`, `StoppedStacks`, -`Decommissioned`/`DecommissionedAt`/`MigratedTo`. The manifest covers state (attached/ -disconnected/decommissioned) and durable_id, but drops: **Label** (human name, e.g. "Külső -HDD 1TB" — UI), **Schedulable/IsDefault** (default placement target for new apps), -**StoppedStacks** (which apps to restart on reconnect — app-domain), **MigratedTo** (decommission -target pointer). -**Why it matters:** `02` named this manifest as the contract that the `settings.StoragePath` -reshape depends on. Silently dropped fields become lost behavior (no default-drive choice, no -restart-after-reconnect list, no friendly labels). -**Suggested fix:** Either add Label + a placement-default marker to the manifest, or explicitly -state which fields re-home to the controller's `settings` (StoppedStacks and Label are -plausibly controller-side; default/schedulable placement must live wherever placement decisions -are made). Make the split explicit so neither side assumes the other owns it. - -### S4 — Geo-WAF ownership + Cloudflare API token placement unspecified -**Where:** `03` covers `cloudflared` (tunnel) health but is silent on geo-restriction WAF; `02` -§6 had `cloudflare/`+`geo` "blocked on tunnel placement"; `01` §3 lists the controller's creds -as "hub API key + local-API token" only. -**What:** Now that tunnel placement is locked (host), the **geo-restriction WAF** management -(`cloudflare/` package: zone/waf/geosync) still has no home. It requires a Cloudflare **API -token** (`config.go` InfrastructureConfig.cf_api_token) with zone-wide WAF edit rights. If geo -stays in the controller (app-domain, per `02`), a **zone-wide Cloudflare token sits inside the -customer guest** — a real blast-radius concern (compromise → edit/disable WAF for the whole -zone, potentially other customers on the same zone). -**Why it matters:** Trust-boundary gap. `01` §5's boundary table has no row for controller↔ -Cloudflare-API. Unspecified ownership blocks the `02` geo classification from being unblocked. -**Suggested fix:** Decide geo-WAF ownership explicitly and add it to `01` §5. Options: (a) move -WAF management to the **agent/hub** (operator-tier, token off the customer box); (b) keep it in -the controller but scope the CF token per-zone/per-customer if the account model allows. Note -this is now *unblocked* by the tunnel decision and should leave `02` §6's "blocked" state. - -### S5 — Cross-doc staleness on the now-locked tunnel placement -**Where:** `01` §11 ("Cloudflare Tunnel placement: host vs guest (§7)") and `02` §6 -("`cloudflare/` + `api/geo.go` — blocked on tunnel placement") vs `03` §13 ("Resolved here: -tunnel placement (host, agent-managed)") and the LOCKED list. -**What:** `01` and `02` still present as OPEN/blocked a decision `03` and the locked set have -resolved. -**Why it matters:** A dev reading `01`/`02` would treat a settled decision as open (or a -classification as blocked when only geo-ownership, S4, actually remains). -**Suggested fix:** When folding this review in: update `01` §7/§11 to record tunnel=host -(agent-managed systemd service); update `02` §6 to reduce the cloudflare item from "blocked on -tunnel placement" to the narrower "blocked on geo-WAF ownership (S4)." - -### S6 — (self-correction) self-restore-test orchestration belongs to the agent, not the controller -**Where:** `02` §5(3) said "Self-restore-test orchestration — *controller* asks the agent to -restore to scratch guest, validates, reports." `03` §8 makes the **agent** drive it -autonomously; §6 gives the controller only `GET /restore-test/status` (read-only). -**What:** `03` is right and `02` overreached. Zero-knowledge means only the box/agent holds the -PBS key (`03` §8); creating a scratch guest is operator-tier (create/allocate — `phase1-2` -§1.3 #7); the controller cannot do either. The controller's only piece is surfacing status. -**Why it matters:** Keeps the NEW-component list honest — this is not a controller component to -build beyond a status read. -**Suggested fix:** Amend `02` §5(3) to "self-restore-test **status display** (read-only); the -agent owns orchestration." - ---- - -## Minor findings - -- **M1 (self-correction):** `02` §3 lists `UUID` among `settings.StoragePath` fields. It is - **not** there (`settings.go:91-103`: Path, Label, IsDefault, Schedulable, AddedAt, - Disconnected/At, StoppedStacks, Decommissioned/At, MigratedTo). UUID is derived at runtime - from fstab / `/host-dev/disk/by-uuid` by `system.ParseFstabUUID` and `watchdog.go`. The - classification (settings = MODIFY/split) is unaffected; the field list was wrong. -- **M2:** Consequently `03` §7's "absorbs the disk-state fields `settings.StoragePath` carries - today" overstates: `durable_id`/UUID is *not* carried today, so the manifest **adds** durable - identity (a genuine improvement — today the controller re-derives UUID from fstab each boot, - which is fragile). Reword "absorbs" → "absorbs + adds durable_id." -- **M3:** `03` §10 journals "provision, restore" but not **controller-update** (§11), which is - also a multi-step async op (snapshot→pull→redeploy→health→rollback). Add it so an agent crash - mid-controller-update is resume-or-rollback like the others. - ---- - -## Verified-correct (no action) — grounding that held up - -- LXC flags `nesting=1,keyctl=1` + overlayfs (`03` §9) match `proxmox-platform.md` §2.3 / - `phase0` §3. ✓ -- async `task exitstatus`, not POST return (`03` §8) matches `proxmox-platform.md` §3.5. ✓ -- stop-mode backup not requiring `VM.PowerMgmt` (`03` §8 "per Phase 1") matches - `proxmox-platform.md` §3.4. ✓ (applies to the agent role too.) -- running-LXC snapshot on LVM-thin (`03` §6/§8/§11) matches `proxmox-platform.md` §4.5 / - `phase1-2` §1.6. ✓ -- `monitor/pinger.go` deprecation (`02` DELETE-obsolete) confirmed in `main.go:168,175` - ("legacy, will be removed" / "no longer used — monitoring is now handled by the Hub"). ✓ -- backup keep/delete **intra-file tear** (`02` hazard) confirmed: `backup.go` holds both - `RunDBDumps`/`DumpAppVolumes(Safe)` (keep) and `RunBackup`/`RunFullBackup` (restic, delete); - `restore.go` holds `RestoreApp` (restic) + `RestoreAppFromTier2` (app). The §7-8 backup - contract gives the extracted app-data-backup package a coherent destination. ✓ -- Control-plane-not-data-plane (`03` §2/§43): apps keep serving if the agent dies — consistent - with Docker-in-LXC running independently (`phase0` §3). ✓ -- §6 per-guest local-API authorization (token→guest map): sound; a leaked token acts only on - its own guest. Residual: a compromised controller can `POST /rollback` its **own** guest - (blast radius = self) — acceptable per design; worth a one-line note that rollback is - self-scoped and bounded. diff --git a/docs/architecture/_hub-review.md b/docs/architecture/_hub-review.md deleted file mode 100644 index b8e91cf..0000000 --- a/docs/architecture/_hub-review.md +++ /dev/null @@ -1,221 +0,0 @@ -# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04) - -Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every -cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong / -breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**. - -The two highest-value catches (doc assumes something the code contradicts) are **S1** and **S2**. - ---- - -## Ranked summary - -| # | What | Where (doc → code) | Severity | -|---|---|---|---| -| S1 | §9/§11 name the **wrong infra-backup table as current** — `infra_backup_versions` is the live/primary one; `infra_backups` is the deprecated write-only mirror | 05 §9/§11 → `store.go:198-217,541-578` | should-fix (code-contradiction) | -| S2 | §7 treats the CF token as **geo-only**; it is **dual-purpose (DNS-01/ACME + WAF)** and is injected into the generated `controller.yaml` | 05 §7 → `config_form.html:76-80`, `controller.yaml.default:26`, `configgen.go:28-37`, `configs.go:1041` | should-fix (code-contradiction / unverified assumption) | -| S3 | §6 leans on the existing **"Push"**, but that is a hub→box **inbound** POST — forbidden by the box-initiated model; transport must invert to poll | 05 §6 → `configs.go:569-570,1148-1150`; Part 1 §4/§5/§11; Part 3 §5 | should-fix | -| S4 | Part 1 §6 calls app inventory **"declarative"**; 05 §1 (LOCKED) says apps are mirrored, never declared/reconciled, restored from PBS | Part 1 §6 ↔ 05 §1/§9 | should-fix (cross-doc) | -| S5 | §9 hands "guest inventory + snapshots" **from the prunable report mirror**; DR soundness actually rests on durable sources | 05 §9/§3 → `store.go:809-816` | should-fix (DR robustness) | -| S6 | §4 says backup-deadline checker "maps onto host_reports' last-backup field"; today it is **event-based** and controller-emitted | 05 §4 → `deadline.go:31-86` | should-fix (mechanism) | -| M1 | "60s staleness checker" conflates the 60s **cadence** with the 30m/1h **threshold** | 05 §4 → `main.go:207-217,99-102`, `staleness.go:33-37` | minor | -| M2 | §2 `customer_configs` field list omits `api_key` — the very field the per-reporter plan retires | 05 §2 → `store.go:102-112` | minor | -| M3 | §11 `reports`→`guest_reports` "rename" is really drop+create under the locked clean cutover | 05 §11 → `store.go:55-119` | minor | -| M4 | Pre-existing weak authz on infra-backup GET / `/notify` (any valid key, not customer-scoped) | handler.go:407,536,568,596 | minor | - -No **blocking** findings — the data model and the two-driver framing are sound, and the LOCKED clean -cutover absorbs most schema risk. The items below are gaps/contradictions worth fixing before the doc -drives work. - ---- - -## Highest-value: doc assumes something the code contradicts - -### S1 — `infra_backups` vs `infra_backup_versions` is inverted (should-fix, code-contradiction) -05 §9: *"`infra_backup_versions` retires; `infra_backups` is repurposed into the slim DR record."* -§11 repeats: *"RETIRE `infra_backup_versions`; repurpose `infra_backups`."* - -The code is the other way round: -- `infra_backup_versions` (added v0.7.0, `store.go:198-211`) is the **live/primary** table. **Every read - path hits it**: `GetInfraBackup` (`store.go:565-578`), `GetInfraBackupByID` (`store.go:581-593`), - `GetInfraBackupMeta` (`store.go:604`), `ListInfraBackupVersions` (`store.go:640`), and the recovery - endpoint (`handler.go:670-686`). -- `infra_backups` (original single-row, `store.go:96-100`) is **deprecated**. It is now **written only - as a legacy mirror** ("for backward compatibility during rollback window", `store.go:552-558`) and is - **never read** except as the one-time migration *source* (`store.go:214-217`). - -So the doc proposes retiring the current table and repurposing the dead one. Under the LOCKED clean -cutover both are discarded anyway, so blast radius is low — but an implementer following §9/§11 -literally would point the DR record at the wrong table. -**Fix:** take §11's own alternative — *fold the slim DR record onto `hosts`* and **drop both** -infra-backup tables. If a standalone table is kept, base it on `infra_backup_versions` (the one with the -data/readers), and correct the "which is current" framing. - -### S2 — the CF API token is **not** geo-only; it is the ACME token too, and ships into `controller.yaml` (should-fix, code-contradiction) -05 §7: *"The hub already holds the CF API token (the config form notes Zone WAF:Edit)… rather than -pushing the token down to the controller… The controller no longer calls the CF API."* - -Grounding confirms the hub **does** hold the token and **does** have a remove-all path: -`config_json → infrastructure.cf_api_token` (`configs.go:714-715,1041-1042,1089-1096`) → -`cfClient.RemoveGeoRules(cfToken, cfg.Domain, …)` in `handleGeoDisable` (`configs.go:1112`), route -`/customers/{id}/geo/disable` (`server.go:201-205`). ✓ The §7 framing of geo-enforcement-moves-to-hub -is also consistent with Part 1 §5/§7 and Part 3 §2/§46. - -**But the doc's assumption that the token is *for geo* is contradicted by the code:** the same -`cf_api_token` is **dual-purpose** — -- the config-form hint says **"Zone DNS:Edit (ACME), Zone WAF:Edit (geo)"** (`config_form.html:80`), -- `controller.yaml.default:26` documents it as the **"Cloudflare API token (DNS-01 challenge)"**, -- and it is **deep-merged into the generated `controller.yaml`** via `configgen.Generate` (config_json - overrides, `configgen.go:28-37`), i.e. **today it is shipped down to the box** and served at - `/config/{id}` and `/recovery/{id}`. - -Consequences §7 must address: -1. **"Token off the controller" is incomplete** if the box still does DNS-01/ACME. In the CF-Tunnel - model the box may no longer need a public cert at all (edge-terminated), making the ACME use moot — - but that is an assumption the doc must state, not skip. Either confirm ACME is gone, or the CF token - cannot fully come off the box. -2. **`configgen` must stop emitting `cf_api_token` into `controller.yaml`** (or relocate it to a - hub-only field). As written, the generated config still carries it. - ---- - -## Should-fix - -### S3 — §6 "Push" is an inbound-to-box mechanism the new model forbids -05 §6: *"the operator edits a customer's desired state (building on the existing config-form + -Push/Pull/Diff)."* The form + diff/pull/push handlers exist — `handlePushConfig` (`configs.go:569`), -`handlePullConfig` (`configs.go:952`), `handleConfigDiff` (`configs.go:861`), routes at -`server.go:209-229`. ✓ So the UI base is real. - -The wrinkle: **"Push" today is a hub→controller outbound POST** (`handlePushConfig` "sends the generated -YAML config to the controller", `configs.go:569-570`), as is the geo-disable notify -(`notifyControllerGeoDisable` → `POST controllerURL/api/geo/settings`, `configs.go:1148-1153`). Both are -the hub **connecting into the box** — explicitly disallowed by the box-initiated model (Part 1 §4 -"the hub never initiates inbound"; §5 row `agent↔hub`/`controller↔hub` = outbound poll; Part 3 §5 "The -hub never connects inbound"). 05's own §5 already resolves this (desired state is **pulled** on poll -with a `desired_generation`). So the doc is internally consistent in *mechanism* but loose in *wording*: -**make §6 explicit that "Push" becomes "publish to desired state, delivered on the next agent/controller -poll," not a reuse of the inbound push transport.** The form/diff UX carries over; the transport inverts. -(Same applies to the geo-disable controller-notify path.) - -### S4 — "declarative app inventory" (Part 1 §6) vs "apps are mirrored, never reconciled" (05 §1) -Part 1 §6 lists the durable record as including a **"declarative app inventory"** that survives box loss -— wording that implies an operator-authored, re-deployable spec. 05 §1 (LOCKED two-driver model) is -explicit the opposite way: *"Apps never enter the reconcile loop… the hub only mirrors the resulting -inventory… the app/customer layer is mirrored,"* and 05 §9 restores apps **from the PBS guest snapshot**, -not by re-deploying a declared inventory. These are reconcilable (the mirror *is* durable last-known -truth) but the word "declarative" contradicts the locked framing and the §9 restore-from-snapshot path. -**Fix (align the older doc to the locked model):** in Part 1 §6 change "declarative app inventory" → -"mirrored / last-reported app inventory," and note apps are recovered from the guest snapshot, not -re-declared. (Flagging an internal inconsistency, not relitigating the locked premise.) - -### S5 — §9 reads DR inputs from a prunable mirror; soundness rests on durable sources -05 §9 hands the recovering agent *"identity, tunnel token, storage manifest, PBS namespace, guest -inventory + snapshots."* §3 places "guest inventory" and "latest PBS snapshot pointers" in -`host_reports` — the bottom-up mirror. But reports are **pruned** (`Prune` deletes rows older than -`maxDays`, `store.go:809-816`; the doc keeps this), so after a long pre-DR outage the last `host_report` -can be gone or stale. The actually-durable DR inputs are: desired-state on `hosts`/`guests` (§5), the -slim DR record (PBS namespace + repo fingerprint + wrapped escrow key, §9/§11), and **PBS's own snapshot -enumeration** (the agent lists snapshots once it has the namespace + unwrapped key). The mirrored -inventory/pointers are convenience, not the source of record. -**Fix:** state in §9 that DR reads from the durable sources (desired-state + DR record + PBS), **not** -from prunable `host_reports`, so recovery doesn't degrade when the last report has aged out. This also -keeps §1's two-driver discipline clean: DR must not depend on bottom-up mirror rows being retained. -(Note: the `hosts` row legitimately mixes top-down intent columns with a few box-reported columns — -repo fingerprint, wrapped escrow key. That is fine; just label them as box-reported so the §1 split -stays legible at the column level.) - -### S6 — backup-deadline checker: doc says field-based, code is event-based (and re-emitter changes) -05 §4: *"The existing backup-deadline checker maps onto `host_reports`' last-backup-per-target."* The -existing checker is **event-based**, not field-based: `CheckBackupDeadlines` looks for -`backup_completed` / `backup_failed` (and `db_dump_*`) **events** since Budapest midnight and emits -`expected_backup_missed` if neither is present (`deadline.go:31-86`). Two changes the doc should make -explicit: -1. **Mechanism:** either keep it event-based (someone emits `backup_completed`) or genuinely move it to - a `host_reports.last_backup_per_target` field check — the doc says the latter but the impl is the - former. -2. **Emitter:** today the **controller** emits backup events; in the de-privileged model the **agent** - owns backup/PBS (Part 3 §8), so the agent must now emit `backup_completed`/`backup_failed` (or the - host report carries last-backup-per-target). Without re-homing the emitter, the deadline check goes - silent after the controller stops doing backups. - ---- - -## Minor - -- **M1 — "60s staleness checker" (§4).** 60s is the **check cadence** (`main.go:207-217`, - `ticker := time.NewTicker(60 * time.Second)`); the **staleness threshold** is 30m (default, - `main.go:99-102`) with down at 2× = 60m (`staleness.go:33-37`; CLAUDE.md "OK <30m, DOWN >1h"). The - event-transition mechanism (`node_stale`/`node_down`/`node_recovered`) is described correctly - (`staleness.go:155-185`). Reword to "the staleness checker (60s cadence, 30m/1h thresholds)." -- **M2 — `customer_configs` fields (§2).** The list ("identity, domain, email, retrieval_password, - status, config_json") omits **`api_key`** (`store.go:108`) — the field §2's per-reporter plan - actually retires. Worth noting `customer_configs.api_key` becomes unused once auth resolves via - `hosts.api_key` / `guests.api_key`. -- **M3 — rename under clean cutover (§11).** `migrate()` is all `CREATE TABLE IF NOT EXISTS` + - idempotent `ALTER` (`store.go:55-119,146-149`). §11's claim "grounded in store.go's idempotent style" - is accurate. But a `reports`→`guest_reports` **rename** isn't part of that style; under the LOCKED - clean cutover (demo re-enrolls fresh, §2) it is really **drop `reports` + create `guest_reports`** - with no data migration. Name it as such to avoid implying an in-place rename + backfill. -- **M4 — pre-existing weak authz.** `handleInfraBackupGet`/`Versions` and `handleNotify`/ - `handleSavePreferences`/`handleInfraBackupPush` use `checkAuth` (global **or any** customer key, - `handler.go:63-66`), not customer-scoped `checkAuthCustomer`. Most retire with the infra-backup push - (§9); for any that carry over, the per-reporter model (§2) should scope them to the resolved - host/guest→customer. Not a regression the doc introduces — a cleanup the cutover enables. - ---- - -## Confirmed accurate (grounding that holds — so the rest of the doc can be trusted) - -- **§10 KEEP list** matches the schema exactly: `customer_configs`, `events`, `customer_notifications`, - `notification_log`, `app_telemetry`, `app_log_issues` all present (`store.go:74-189,102-135`). The - asset manager exists (`handler.go:57,834-867`). ✓ -- **§10 two-tier notifications** (operator English / customer Hungarian, Resend, cooldowns) match - `notify/dispatcher.go`: `processOperator` (1h cooldown, `FormatOperatorEmail`, gated by `operatorOn`, - `dispatcher.go:91-114`) + `processCustomer` (prefs-driven, default 6h, `FormatCustomerEmail`, - `dispatcher.go:116-158`); wired in `main.go:134`. ✓ -- **§8 enrollment / §11 configgen** — deep-merge + Hungarian passphrase base is real: - `configgen.deepMerge` (`configgen.go:76-91`), programmatic overrides + `hub.api_key = cfg.APIKey` - (`configgen.go:40-47`), retrieval-password gate (`handler.go:709-753`). The evolution to agent-first + - per-guest keys + key-pinning is a clean extension. ✓ -- **§2 auth extension** (Bearer → reporter → customer) is clean against today's - `checkAuthCustomer` (global key, else `GetCustomerConfigByAPIKey`, `handler.go:72-90`, - `store.go:913-935`); adding host/guest key lookups slots straight in. ✓ -- **§11 "idempotent style"** is accurate (`store.go:55-119`). New tables/columns (`hosts`, `guests`, - `host_reports`, `signed_ops`, `desired_generation`, `desired_spec_json`) follow the existing - `CREATE IF NOT EXISTS` / `ALTER … ` pattern cleanly. -- **§9 escrow/custody** is consistent with Part 1 §8 (three-tier custody, zero-knowledge default, - recovery-code-wrapped PBS keyfile, operator can't open) and Part 3 §8 (live PBS key on the box for - backup + restore-test; hub holds only the wrapped escrow). The "customer recovery code is the - irreducible residual; operator-managed tier avoids it" matches Part 1 §8 verbatim in spirit. ✓ -- **§4 dead-man's-switch** (host-report recency = primary liveness) is consistent with Part 3 §5 - ("the heartbeat *is* the liveness signal… first-class treatment hub-side"). ✓ -- **§5/§6 signed-op + desired-state** are consistent with Part 4 and Part 3 §4: - hub holds **no** signing key and queues opaque blobs (Part 4 §5; 05 §6 "The hub holds no signing - key"); agent runs the verify pipeline and is the authoritative guard (Part 4 §2.3, Part 3 §4; 05 §6 - "the agent's classification is the authoritative guard"); hub classifies at edit-time for UX only. - 05 §6's `signed_ops` columns are a consistent superset of Part 4 §2.1's blob - `{op, target:{host_id,guest_id}, params, nonce, issued_at, expires_at, key_id}` (05 adds hub-side - lifecycle states `delivered`/`rejected` — fine). The local-CLI hand-off (`felhom-sign --pending`) - matches Part 4 §5–6's `Signer`-on-the-workstation model. ✓ - -## Two-driver soundness (axis 3) — holds -No place in 05 has the hub **drive** box/customer-owned state. Desired-state (§5) is all infrastructure -intent (guests, storage *policy*, versions, identity, tunnel) — top-down and legitimate. Apps are -explicitly excluded from reconcile (§1, §5) and mirrored only. Storage is the handshake (detect → -assign policy → reconcile policy onto the detected drive), matching Part 3 §7. The one nuance (S5): the -`hosts` row holds both top-down intent and a few box-reported columns (repo fingerprint, wrapped escrow -key) — acceptable, just label provenance per column. Reconcile (§5) never collides with app/storage -reality because the reality columns (`durable_id` attached, snapshot pointers, app inventory) are -mirror-only and never serve as desired state. - -## DR completeness (axis 4) — safe to retire the heavy push, with S5's clarification -Retiring the controller's infra-backup push is safe **given** that DR reads from durable sources, not -the prunable mirror (S5). What the old push carried — `deployed_stacks` + `disk_layout.mounts` -(`store.go:768-795`, surfaced by `handleRecovery`, `handler.go:620-705`) — is reconstructible: -storage layout/`durable_id`s from the storage manifest (desired-state, durable) + host-report mirror; -app inventory from the guest **inside the PBS snapshot** (so it need not be separately stored); snapshot -list from PBS itself. The one artifact only the box can produce — the recovery-code-wrapped PBS key — is -explicitly escrowed (§9), zero-knowledge, consistent with Part 1 §8 / Part 3 §8. So nothing -DR-essential is lost by removing the push **provided** §9 is amended per S5 to name durable sources and -not lean on `host_reports` retention. diff --git a/docs/proxmox-platform.md b/docs/proxmox-platform.md deleted file mode 100644 index cfea66a..0000000 --- a/docs/proxmox-platform.md +++ /dev/null @@ -1,385 +0,0 @@ -# Proxmox Platform Reference - -Authoritative, living reference for the Proxmox platform underneath `felhom-agent`. -It records **facts about Proxmox and what we validated about it** — not Felhom design -decisions. Where a design choice exists, this doc points to the (future) controller -architecture document rather than making the choice here. - -**Evidence base** (raw, chronological spike logs — kept as the underlying record): -- [tests/phase0-findings.md](tests/phase0-findings.md) — VM-vs-LXC overhead, Docker-in-LXC viability -- [tests/phase1-2-findings.md](tests/phase1-2-findings.md) — privilege model, backup/restore round-trip -- [tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md](tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md) — **superseded** pre-spike reference (contains a known privsep error; do not cite as authoritative) - -Every nontrivial claim links to its evidence section. Validated on a single host -(`demo-felhom`, 192.168.0.162, 4 vCPU / 16 GB) on 2026-06-07; treat single-run timings and -measurements as indicative, not benchmarks. - ---- - -## 1. Platform baseline - -Validated stack [[phase0 §1](tests/phase0-findings.md)]: - -| Component | Version | -|---|---| -| Proxmox VE (`pve-manager`) | **9.2.2** (`b9984c6d90a4bd80`) | -| OS | Debian 13 (Trixie) | -| Kernel | proxmox-kernel **7.0.2-6-pve** | -| `pve-qemu-kvm` | 11.0.0-3 | -| `qemu-server` | 9.1.15 | -| `pve-container` | 6.1.10 | -| `lxc-pve` / `lxcfs` | 7.0.0-2 / 7.0.0-pve1 | -| `criu` | 4.1.1-1 | - -`pvesh get /version` → release 9.2. Always confirm the node name on the box -(`pvesh get /nodes`) rather than hard-coding it. - -### 1.1 Storage backends -Two backends were present and exercised [[phase0 §1](tests/phase0-findings.md), [phase1-2 §pre-flight](tests/phase1-2-findings.md)]: - -| Storage | Type | Path / VG | Content types | Holds | -|---|---|---|---|---| -| `local` | `dir` | `/var/lib/vz` | `iso, vztmpl, backup, import` | ISOs, CT templates, **vzdump archives** | -| `local-lvm` | `lvmthin` | VG `pve`, thinpool `data` | `rootdir, images` | guest disk volumes | - -**Why backups cannot live on LVM-thin:** LVM-thin is a *block* backend — it allocates -logical volumes for guest disks. Backup archives and templates are *files*, which require a -file-level backend (`dir`, NFS, CIFS, or PBS). A `vzdump` target must therefore be a -storage whose content types include `backup` (here, `local`); pointing `vzdump` at -`local-lvm` is not valid. [[phase1-2 §pre-flight / §2.1](tests/phase1-2-findings.md)] - -### 1.2 Repositories -PVE 9 uses **deb822** `.sources` files under `/etc/apt/sources.list.d/`. For a host -without a subscription, the enterprise repos (`pve-enterprise.sources`, -`ceph-*-enterprise.sources`) must be disabled (they return 401) and a no-subscription repo -enabled. *The spike host arrived with the no-subscription repo already configured and the -host updated [[phase0 baseline](tests/phase0-findings.md)]; the repo setup itself was not a -spike deliverable* — the canonical no-subscription `.sources` is the standard Proxmox 9 -procedure (`/etc/apt/sources.list.d/pve-no-subscription.sources` with -`Components: pve-no-subscription`). Treat the exact commands as standard setup, not -spike-validated. - -**Docker repository (validated):** Docker's official apt repo **has a `trixie` channel**; -no fallback to Debian's `docker.io` was needed. Installed Docker **29.5.3** from it in both -guest types. [[phase0 §1](tests/phase0-findings.md)] - ---- - -## 2. Guest model (LXC vs VM) — validated facts - -Both guest types ran the **identical** workload (Debian 13, Docker 29.5.3, a -postgres/redis/nginx compose stack) under identical resources (2 vCPU, 2048 MB, ~10 GB) -[[phase0](tests/phase0-findings.md)]. - -### 2.1 Isolation characteristic (fact, not recommendation) -- **LXC** is an OS-level container: it **shares the host kernel**. Docker-in-LXC needs the - container configured for nesting (see §2.3). -- **VM** runs its **own guest kernel** under KVM/QEMU, with full hardware-level isolation - and its own firmware. - -The trade-offs below follow directly from this difference. - -### 2.2 Resource overhead (measured) -Host RAM used = `MemTotal − MemAvailable`, deltas vs a both-stopped baseline of 1702 MB; -one guest measured at a time [[phase0 §2](tests/phase0-findings.md)]: - -| Metric | LXC | VM | Note | -|---|---|---|---| -| Idle host-RAM delta | **+211 MB** | **+2056 MB** | structural, see below | -| Under-load host-RAM delta | **+410 MB** | **+2084 MB** | | -| Per-guest attribution | cgroup `memory.current` 1961 MB¹ | KVM RSS ~2031 MB | | -| Idle host CPU used | ~0.3 % | ~6.0 % | VM has an emulation/guest-kernel floor | -| Under-load host CPU used | ~39.4 % | ~53.9 % | VM work shows as `%guest` (31.9 %) | -| pgbench throughput | 2211 tps | 1820 tps | identical load, 0 failed both | -| Disk used (host thin-LV) | ~2.67 GiB | ~2.94 GiB | of 10 GiB allocated | -| Provisioning (create→ready) | ~10–15 s | ~60–75 s | template-extract vs qcow2-import+boot | - -¹ `cgroup memory.current` counts reclaimable page cache shared with the host and -**overstates** the LXC's true incremental cost; the +211 MB host delta is the honest -number [[phase0 §4.4](tests/phase0-findings.md)]. - -**Why the RAM gap is structural** [[phase0 §4.3](tests/phase0-findings.md)]: LXC processes -share the host kernel and page cache, so only the working set counts against the host. A VM -with **no ballooning configured** has KVM back every guest-touched page (including the -guest's own page cache), so its host cost ≈ the full RAM allocation and is largely -load-independent. *Ballooning / KSM were not tested* and could change the VM figure. - -### 2.3 Docker-in-LXC viability (validated) -Docker ran **cleanly in an *unprivileged* LXC** configured with -`--features nesting=1,keyctl=1 --unprivileged 1` (PVE 9 syntax, accepted by `pct create`) -[[phase0 §3](tests/phase0-findings.md)]: - -- `docker run hello-world` → success; full 3-container stack healthy. -- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs` - fallback**. (Docker 29 names the overlay driver `overlayfs` via the containerd - snapshotter image store; same overlay technology as the legacy `overlay2`.) -- Named volume persisted writes; multi-container networking + published port worked - (`curl localhost:8080` → 200); 0 failed transactions under load. -- No privileged-container fallback was needed. - -### 2.4 Guest agent & app-consistency capability -- **VM:** `qemu-guest-agent` installs and reports (`agent: 1`), enabling - `guest-fsfreeze`-based app-consistent `snapshot` backups [[phase0 §4.8](tests/phase0-findings.md)]. - The Debian genericcloud image does **not** ship the agent — it must be installed - in-guest. -- **LXC:** no guest agent exists → **no fsfreeze** (see §4.2). - ---- - -## 3. API & access control - -### 3.1 Fundamentals -- **Base URL:** `https://:8006/api2/json`. Every `pve*` CLI is a thin wrapper over - this REST API. -- **Token auth header:** `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET`. The - secret is shown **once** at creation. Response envelope: `{"data": ...}`. -- **TLS reality:** the host serves the default **self-signed** certificate. `curl` without - `-k` fails `SSL certificate problem: unable to get local issuer certificate` - [[phase1-2 §1.5](tests/phase1-2-findings.md)]. Production trust (pin the PVE CA / install - a real cert) is a separate, not-yet-decided concern. - -### 3.2 RBAC model -An ACL entry is a triple **(path, principal, role)**; a role is a bundle of privileges, -assigned at the most specific path. Paths include `/`, `/vms/`, `/nodes/`, -`/storage/`, `/pool/`, `/access/...`. - -Introspection (**corrected for PVE 9**) [[phase1-2 §1.1](tests/phase1-2-findings.md)]: -- `pveum role list` — lists roles **with their privileges**. -- ⚠️ `pveum role info ` **does not exist in PVE 9** (the old reference used it). -- `pveum acl list`, `pveum user permissions --path `. - -### 3.3 Privilege-separated tokens — the intersection rule (corrected) -> **A privsep token's (`--privsep 1`) effective permissions are the *intersection* of (a) -> the backing user's permissions and (b) the token's own ACLs.** The role must therefore be -> granted on **BOTH the user AND the token** for the same path. Granting it on the token -> only yields an **empty intersection** and a **403 even on self-calls.** -> [[phase1-2 §1.2](tests/phase1-2-findings.md)] - -This corrects the superseded reference (§3 there grants the ACL to the token only). The -intersection is what keeps a privsep token ≤ its user while still being independently -scopeable to a narrow path. - -Working pattern (validated): -```bash -pveum role add -privs " ..." # NB: -privs is space-separated -pveum user add @pve -pveum user token add @pve --privsep 1 # capture SECRET (shown once) -pveum acl modify -user '@pve' -role # BOTH the user... -pveum acl modify -token '@pve!' -role # ...AND the token -``` -`pveum acl delete` **requires `--roles`** (a bare `-user`/`-token` path errors -`400 roles: property is missing`). Deleting the token/user/role auto-invalidates the -referencing ACLs. [[phase1-2 §5](tests/phase1-2-findings.md)] - -### 3.4 Validated minimal self-backup role -A token scoped to **one VMID + the backup datastore** can audit, snapshot, and back up -**only that guest**, and is denied on every other guest and on create/allocate -[[phase1-2 §1.3–1.4](tests/phase1-2-findings.md)]: - -> **Minimal role for self-audit + self-snapshot + both `snapshot`- and `stop`-mode -> self-backup:** -> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit` - -⚠️ **`VM.PowerMgmt` is NOT required for stop-mode backup** — `vzdump` performs the guest -shutdown/restart internally under `VM.Backup` (tested: stop-mode self-backup returned -`exitstatus OK` without it) [[phase1-2 §1.4](tests/phase1-2-findings.md)]. This corrects the -old reference's "likely yes" guess. - -Validated boundary (token scoped to `/vms/` + `/storage/local`): - -| Operation | Result | -|---|---| -| `GET /version` | 200 | -| `GET` self status, `POST` self snapshot, `POST` self vzdump | 200 / task `OK` | -| `GET`/`POST` against **another** guest's vmid | **403** (read) / task **403** (backup) | -| `POST /nodes//lxc` (create/allocate a guest) | **403** — create/allocate is operator-tier | - -### 3.5 Async tasks — trust `exitstatus`, not the POST -Long operations (`vzdump`, `snapshot`, clone, restore) return a **UPID**, not a result. -Poll `GET /nodes//tasks//status` until `status: stopped`, then read -`exitstatus` [[phase1-2 §1.3](tests/phase1-2-findings.md)]. - -> ⚠️ **Authorization can surface at task execution, not at the HTTP POST.** A `vzdump` -> against an unauthorized vmid returns **HTTP 200 + a UPID**, but the task then ends -> `exitstatus: "403 Permission check failed (/vms/, VM.Backup)"` and produces **no -> archive**. A caller that trusts the 200 would wrongly believe the backup ran. Always poll -> the task and check `exitstatus`. - -(The task owner — including a token — can read its own task status: 200.) - -### 3.6 Operator-tier agent role & root-vs-API boundary (validated) -The operator-tier **host agent** (`03-host-agent.md`) needs a far broader role than the -Phase-1 *guest self-backup* role (which is denied create/allocate — §3.4). The minimal role -that drives the full guest lifecycle via an API token, validated by paring -[[phase3 §B3](tests/phase3-findings.md)]: - -> **`FelhomAgent` (operator-tier, 16 privileges):** -> `VM.Allocate, VM.Audit, VM.Config.Disk, VM.Config.CPU, VM.Config.Memory, VM.Config.Network, -> VM.Config.Options, VM.PowerMgmt, VM.Snapshot, VM.Snapshot.Rollback, VM.Backup, -> Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit, Sys.Audit, SDN.Use` -> -> Paring proved: `SDN.Use` is **required** (PVE 9 gates bridge use; omitting it → `403 -> (/sdn/zones/localnetwork/vmbr0, SDN.Use)`); `Sys.Audit` required for host metrics -> (`GET /nodes//status`); `VM.Config.Network`/`VM.Config.Options` required for NIC/onboot -> config; `Datastore.AllocateTemplate` **not** needed (drop it). NB `VM.Config.CPUMemory` is -> not a real privilege — it is `VM.Config.CPU` + `VM.Config.Memory`. - -**Root-vs-API boundary** [[phase3 §B3](tests/phase3-findings.md)] — nearly the entire guest -lifecycle, **including restore**, is API-token-covered; the genuine OS-root residual is narrow: - -| Operation | Coverage | -|---|---| -| Create LXC (nesting-only), config, allocate, start/stop, snapshot/rollback, vzdump, **restore**, destroy, add storage definition, host metrics | **scoped API token** (the `FelhomAgent` role) | -| ⚠️ **Create LXC with `keyctl=1`** (Docker needs it — §2.3) | **OS root `root@pam` only** | -| USB physical mount-by-UUID / systemd mount unit / fstab; SMART/sensors | OS root / narrow sudoers | - -> ⚠️ **`keyctl=1` (and any feature flag except `nesting`) can be set only by an actual -> `root@pam` session** — `changing feature flags (except nesting) is only allowed for -> root@pam`. **No API token qualifies**, not even a non-privsep `root@pam` token (same 403). -> So *fresh provisioning* of a Docker-capable LXC needs `pct create` as OS root (or a narrow -> sudoers entry). **Restore is exempt:** a token-authorized `vzrestore` **preserves -> `keyctl=1`** from the archive — the DR path needs no root. - ---- - -## 4. Backup & restore (`vzdump` / `pct restore`) - -### 4.1 Modes -- **`stop`** — orderly guest shutdown → backup → restart. Highest consistency, defined - downtime. (For LXC the shutdown/restart is internal to `vzdump`; needs only `VM.Backup` — - §3.4.) -- **`snapshot`** — lowest downtime; copies blocks while running. Consistency depends on the - guest cooperating (§4.2). -- **`suspend`** — legacy/compat, not used. - -### 4.2 Consistency: crash-consistent vs quiesced, and no-fsfreeze-for-LXC -> ⚠️ **An LXC has no guest agent, so `snapshot`-mode `vzdump` does NOT fsfreeze.** A -> running-stack LXC backup is therefore **crash-consistent** (filesystem-level), not -> app-consistent. App-consistency for an LXC is the caller's job: quiesce in-guest first -> (stop the stack / flush DBs) or use `stop` mode. A **VM** with `qemu-guest-agent` gets -> `guest-fsfreeze` around the copy → near-free app-consistency. [[phase1-2 §2.1](tests/phase1-2-findings.md), [phase0 §4.8](tests/phase0-findings.md)] - -**Validated restore behaviour** (LXC, Postgres) [[phase1-2 §2.2](tests/phase1-2-findings.md)]: -- **Crash-consistent (running):** on first start Postgres ran **automatic WAL recovery** - (`database system was interrupted … not properly shut down; automatic recovery in - progress … redo done … ready to accept connections`) and the data was intact. -- **Quiesced (stack stopped):** clean start, no recovery, data intact. -- Both restored correctly here on an idle-at-backup DB; this is **not** a durability - guarantee under heavy write load (§6). - -### 4.3 What a backup captures -A single LXC `vzdump` captures the container rootfs **including the Docker named volumes** -(they live in the rootfs) — one backup = the whole guest and its data. Validated: a -sentinel row survived both variants [[phase1-2 §2.2](tests/phase1-2-findings.md)]. - -Sizes/timings (2.5 GiB source, zstd) [[phase1-2 §2.1–2.2](tests/phase1-2-findings.md)]: -backup ~934 MB (~2.7:1) in ~22–25 s; restore in ~11–12 s. - -### 4.4 Restore = recreate-from-archive (identity is preserved) -There is no single "restore" call — you recreate the guest from the archive into a **new -VMID**: -- **LXC:** `pct restore --storage ` -- **VM:** `qmrestore ` (or `POST /nodes//qemu` with `archive=`) - -> ⚠️ **`pct restore` preserves the source config — including the MAC address and -> hostname.** Restoring while the original still runs causes a **MAC/hostname collision** on -> the bridge; reset network identity (`pct set -net0 name=eth0,bridge=vmbr0,ip=dhcp` -> regenerates the MAC) before starting. [[phase1-2 §2.2](tests/phase1-2-findings.md)] - -**Restored config survives intact:** `unprivileged: 1` and `features: nesting=1,keyctl=1` -are preserved, so Docker runs in the restored CT [[phase1-2 §2.2](tests/phase1-2-findings.md)]. - -### 4.5 Snapshots -A **running, unprivileged LXC can be snapshotted on LVM-thin** with no stop required -(`exitstatus OK`; snapshot listed while the CT stays `running`) -[[phase1-2 §1.6](tests/phase1-2-findings.md)]. This is the mechanism available for a -snapshot-before-change rollback flow. - -### 4.6 PBS (Proxmox Backup Server) -**Not yet validated.** No PBS datastore was configured or tested in the spike. All backup -findings above are for `vzdump` to a `dir` storage. PBS (dedup, incremental, remote, dirty- -bitmap) is pending. - -### 4.7 vzdump scope by LXC mount type (validated) -A stop-mode `vzdump` includes/excludes each LXC mount point by **type and the `backup` flag** -[[phase3 §B2](tests/phase3-findings.md)]. Validated three ways (vzdump log, archive grep, -restore): - -| Location | `backup` flag | In the vzdump? | -|---|---|---| -| rootfs (and anything inside it) | — | **included** (always) | -| **Docker named volume** (default driver) | — | **included** — it lives in the rootfs (`/var/lib/docker/volumes//_data`) | -| volume mount point (`mpN`) | `backup=1` | included | -| volume mount point (`mpN`) | `backup=0` | **excluded** (vol recreated empty on restore) | -| bind mount point (`mpN: /host/path`) | n/a | **excluded** ("not a volume"); data is *not* in the archive | - -> ⚠️ **The `backup=` flag is honoured ONLY for *volume* mount points.** A **Docker -> named volume is in the rootfs and is always captured** — so a "bulk" volume left as a -> default named volume is silently swept into the whole-guest image. To keep bulk data **out**, -> realize it as a dedicated `backup=0` volume mount point (proven recipe: -> `pct set -mpN :,mp=/mnt/bulk,backup=0` then -> `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol`). -> A **bind mount's** data is excluded from the archive entirely; on same-host restore it -> reappears only because the bind config re-attaches the same host dir — on a *different* host -> (true DR) it is gone unless backed up separately. - ---- - -## 5. Gotchas & operational notes (quick reference) - -| Gotcha | Detail | Evidence | -|---|---|---| -| **deb822 repos** | PVE 9 repos are `.sources` files; disable enterprise, enable no-subscription | standard setup | -| **Privsep dual-grant** | privsep token needs the role on **both** user and token, else empty intersection → 403 | [phase1-2 §1.2](tests/phase1-2-findings.md) | -| **Async authz** | `vzdump` POST returns 200+UPID even when unauthorized; the 403 is in the task `exitstatus`; poll it | [phase1-2 §1.3](tests/phase1-2-findings.md) | -| **No fsfreeze for LXC** | running-LXC `snapshot` backup is crash-consistent only; quiesce or use `stop` for app-consistency | [phase1-2 §2.1](tests/phase1-2-findings.md) | -| **Restore identity collision** | `pct restore` keeps source MAC + hostname; reset before starting alongside the original | [phase1-2 §2.2](tests/phase1-2-findings.md) | -| **Restart policy for self-heal** | restored/rebooted containers come up `exited` with no restart policy; need a restart policy or an explicit `compose up -d` to return automatically | [phase1-2 §2.2/§3](tests/phase1-2-findings.md) | -| **Self-signed TLS** | host cert is self-signed; `curl` needs `-k` until trust is set up | [phase1-2 §1.5](tests/phase1-2-findings.md) | -| **`pveum role info` gone** | use `pveum role list` in PVE 9 | [phase1-2 §1.1](tests/phase1-2-findings.md) | -| **`pveum acl delete` needs `--roles`** | bare `-user`/`-token` path errors `400 roles: property is missing` | [phase1-2 §5](tests/phase1-2-findings.md) | -| **`VM.PowerMgmt` not needed** | stop-mode backup works under `VM.Backup` alone | [phase1-2 §1.4](tests/phase1-2-findings.md) | -| **`keyctl=1` is root-only** | feature flags except `nesting` need a `root@pam` session; no API token (even root's) can set them; restore preserves them | [phase3 §B3](tests/phase3-findings.md) | -| **`SDN.Use` gates bridge use** | PVE 9 needs `SDN.Use` to attach a NIC to `vmbr0`; omit it → 403 | [phase3 §B3](tests/phase3-findings.md) | -| **Docker named vol = always backed up** | named volumes live in rootfs; only *volume mountpoints* honour `backup=0`; bulk must be a dedicated `backup=0` mp | [phase3 §B2](tests/phase3-findings.md) | - ---- - -## 6. Validated vs open - -### Validated by the spike -| Fact | Evidence | -|---|---| -| PVE 9.2.2 / Debian 13 / kernel 7.0.2 baseline; `local` (dir) vs `local-lvm` (thin) roles | [phase0 §1](tests/phase0-findings.md), [phase1-2 pre-flight](tests/phase1-2-findings.md) | -| Docker runs in an **unprivileged** LXC (`nesting=1,keyctl=1`), driver `overlayfs`, cgroup v2 | [phase0 §3](tests/phase0-findings.md) | -| LXC vs VM overhead (idle host RAM +211 MB vs +2056 MB; CPU/throughput/provisioning) | [phase0 §2](tests/phase0-findings.md) | -| Privsep token = intersection of user ∩ token ACLs (dual-grant required) | [phase1-2 §1.2](tests/phase1-2-findings.md) | -| Minimal self-backup role; `VM.PowerMgmt` unnecessary | [phase1-2 §1.4](tests/phase1-2-findings.md) | -| Token scoped to one VMID: self-ops succeed, cross-guest + create/allocate denied | [phase1-2 §1.3](tests/phase1-2-findings.md) | -| Async UPID model; vzdump authz surfaces in `exitstatus`, not the POST | [phase1-2 §1.3](tests/phase1-2-findings.md) | -| Running, unprivileged LXC snapshots on LVM-thin (no stop) | [phase1-2 §1.6](tests/phase1-2-findings.md) | -| `vzdump` → `pct restore` round-trip; one backup captures Docker volumes; config survives | [phase1-2 §2](tests/phase1-2-findings.md) | -| Crash-consistent restore recovers via Postgres WAL; quiesced restores clean | [phase1-2 §2.2](tests/phase1-2-findings.md) | -| LXC vzdump scope by mount type; `backup=0` excludes volume mps; Docker named vols ride rootfs; proven bulk-exclusion recipe | [phase3 §B2](tests/phase3-findings.md) | -| Operator agent role (16 privs); guest lifecycle incl. restore is API-token-covered; `keyctl` create is `root@pam`-only | [phase3 §B3](tests/phase3-findings.md) | - -### Not yet validated (do not assume) -| Open item | Why it matters | -|---|---| -| **PBS** (dedup/incremental/remote backup) | the only backup path tested was `vzdump` to a `dir` | -| **The real controller running inside an LXC** reaching `host:8006` | spike used `curl`/CLI, not the actual Go controller | -| **App-consistency under heavy write load** | WAL recovery was validated only on an idle-at-backup DB | -| **Live migration / restore to a different host** | single-node spike only | -| **Ballooning / KSM** effect on VM RAM cost | VM RAM measured with neither configured | -| **Cluster / HA** behaviour | single node | -| **Production TLS trust** for the API | all calls used `-k` against a self-signed cert | -| **deb822 no-subscription repo setup** as a controlled step | host arrived pre-configured | - ---- - -## 7. Scope boundary - -This document holds **platform facts only.** Felhom design decisions — e.g. which guest -type is the default, whether to use privsep or non-privsep tokens, where PBS lives — are -**out of scope** and belong in the controller-architecture document. Where this reference -notes a decision exists, the decision itself is recorded there, not here. diff --git a/docs/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md b/docs/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md deleted file mode 100644 index bc929c5..0000000 --- a/docs/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md +++ /dev/null @@ -1,176 +0,0 @@ -> ⚠️ **SUPERSEDED — spike evidence only, not authoritative.** This is the *pre-spike* -> reference and contains at least one known error (the privsep/ACL mechanism in §3 — it -> grants the ACL to the token only, which yields an empty intersection and a 403 even on -> self-calls). For the corrected, validated facts read -> [`../proxmox-platform.md`](../proxmox-platform.md). Kept here unchanged as the record of -> what we believed going into the spike. - -# Proxmox Spike — API & Access-Control Reference - -Reference for the **controller-as-guest** architecture, synthesized from current -Proxmox VE 9.x documentation (June 2026). - -Items marked **[confirm on box]** should be verified once PVE is installed — -treat them as Phase 0/1 verification steps, not gospel. Every Proxmox CLI tool -is a thin wrapper over the same REST API, so anything below is reachable from Go. - ---- - -## 1. API fundamentals - -- **Base URL:** `https://192.168.0.162:8006/api2/json` -- **Auth (API token):** HTTP header - `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET` - The secret is shown **once** at creation — capture it immediately, it can't be - retrieved again. -- **Response shape:** `{ "data": ... }`; errors come back via HTTP status + body. -- **Discovery (do this live on the box instead of trusting any doc):** - - `pvesh get /version` - - `pvesh ls /nodes//qemu/` - - Full schema browser: `https://pve.proxmox.com/pve-docs/api-viewer/` - - "What call does the GUI make?" → perform the action in the web UI with - browser DevTools → Network open and read the request. Fastest way to find - the exact endpoint + params for anything. -- **Async tasks:** long operations (backup, restore, clone) return a **UPID** - (task id), not a result. Poll `GET /nodes//tasks//status` until - `status: stopped`, then check `exitstatus`. The controller must poll, not - block. **[confirm on box]** the exact polling/response shape. - ---- - -## 2. RBAC model — (path, principal, role) - -An ACL entry is a triple of **(path, user/group/token, role)**. A role is a -bundle of privileges, assigned at the most specific path possible. - -- **Paths:** `/`, `/vms/`, `/nodes/`, `/storage/`, - `/pool/`, `/access/...` -- **Predefined roles include:** `PVEAuditor` (read-only), `PVEVMUser`, - `PVEVMAdmin`, `PVEDatastoreUser`, `PVEAdmin`, `PVEUserAdmin`. -- **API tokens with privilege separation (`--privsep 1`):** the token's - effective permissions are the **intersection** of (a) the backing user's - permissions and (b) the token's own ACLs. A privsep token can therefore never - exceed its user, and you grant it a separate, minimal ACL. This is exactly the - property the in-guest controller needs. - -Introspection: -```bash -pveum role list -pveum role info PVEVMAdmin -pveum user permissions --path /vms/ -``` - ---- - -## 3. Two-tier privilege model (our architecture decision) - -**Tier A — in-guest controller (customer-facing, NARROW).** -Runs inside the customer's guest. Token scoped to *that guest's own VMID only*: -read its own status/config, snapshot itself, back itself up, write the backup to -the datastore. Cannot see or touch other guests. The LXC/VM's own privilege -level is irrelevant here — reaching `host:8006` is just an HTTPS call + token. - -**Tier B — operator (provisioning, BROAD).** -Creates/destroys guests, builds the golden template, attaches storage, wires PBS. -Lives operator-side (hub / tooling), never on the customer box. - -### Phase 1 runbook — minimal self-backup role + scoped token - -```bash -# 1. Custom least-privilege role: "back up / snapshot myself" -# [confirm on box: exact privilege names via `pveum role list` / api-viewer] -pveum role add FelhomSelfBackup \ - -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" - -# 2. Dedicated API-only user in the PVE realm (no login password) -pveum user add felhom-ctl@pve --comment "In-guest controller (self-backup)" - -# 3. Privsep token for that user (SECRET shown once) -pveum user token add felhom-ctl@pve ctl --privsep 1 - -# 4. Scope the TOKEN to one guest + the backup datastore only -pveum acl modify /vms/ -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -pveum acl modify /storage/ -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup - -# 5. Test FROM INSIDE the guest -curl -k https://:8006/api2/json/version \ - -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=" - -curl -k -X POST https://:8006/api2/json/nodes//vzdump \ - -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=" \ - -d "vmid=&storage=&mode=snapshot" -``` - -**Pass criteria:** the token backs up its OWN vmid, and returns **403** on any -other vmid. That single result validates the whole controller-as-guest design. - -**Open question to settle here:** does Tier A also need `VM.PowerMgmt` so it can -stop/start its own guest for `stop`-mode backups? Likely yes — add it and re-test. - ---- - -## 4. Backup / restore (vzdump) - -**Modes:** -- **`stop`** — orderly guest shutdown → live backup → resume. Highest - consistency, short defined downtime. -- **`snapshot`** — lowest downtime; copies blocks while running. *Small - inconsistency risk* unless the guest cooperates (see below). -- **`suspend`** — legacy/compat, longer downtime, not recommended. - -**App-consistency — the concrete version of the earlier warning:** -- **VM:** install `qemu-guest-agent` in the guest and set `agent: 1`. - `snapshot`-mode vzdump then calls `guest-fsfreeze-freeze` / `-thaw` around the - copy → near-free filesystem consistency. **This is a real point in the VM's - favour over LXC.** -- **LXC:** no guest agent → no fsfreeze. App-consistency becomes the - *controller's* job: quiesce in-guest first (stop stacks / flush DBs) **then** - vzdump, or use `stop` mode. Same lesson as the restic work, moved to the guest - layer. - -**CLI / API:** -```bash -vzdump --mode snapshot --storage # CLI -# API (async → UPID): -POST /api2/json/nodes//vzdump params: vmid, storage, mode, ... -``` - -**Restore is NOT a single "restore" call** — you recreate the guest from the -archive: -- **VM:** `qmrestore ` / `POST /nodes//qemu` with `archive=...` -- **LXC:** `pct restore ` / `POST /nodes//lxc` with the archive as source - -Phase 2's real-restore test = restore to a **fresh vmid** and boot it. Do not -declare the backup "working" until a restored guest actually runs. - ---- - -## 5. Key REST endpoints (qemu shown; lxc is parallel under `/lxc`) - -``` -GET /nodes -GET /nodes//qemu list VMs -GET /nodes//qemu//status/current live status -GET /nodes//qemu//config config -POST /nodes//qemu//status/{start,stop,shutdown,reboot} -POST /nodes//qemu//snapshot (snapname, description) -GET /nodes//qemu//snapshot list snapshots -POST /nodes//qemu//snapshot//rollback -POST /nodes//vzdump backup (async, UPID) -GET /nodes//tasks//status poll async task -``` - -LXC: replace `/qemu/` with `/lxc/`. For **Docker-in-LXC** the container needs -`features nesting=1,keyctl=1` (`pct set -features nesting=1,keyctl=1`, or -the `features` property on `POST /nodes//lxc`) — **[confirm on box]**. - ---- - -## 6. Phase 0 confirm-on-box checklist - -- [ ] PVE 9.2 installed; storage = LVM-thin (leave free space to also test dir/qcow2) -- [ ] Exact privilege set for `FelhomSelfBackup` (`pveum role info`) -- [ ] UPID task-polling response shape -- [ ] Docker official apt repo has a `trixie` channel -- [ ] LXC `features nesting=1,keyctl=1` syntax + Docker actually runs inside an LXC -- [ ] Baseline idle + under-load RAM/CPU: one Debian VM vs one Debian LXC, identical resources \ No newline at end of file diff --git a/docs/tests/phase0-findings.md b/docs/tests/phase0-findings.md deleted file mode 100644 index c8c2c2a..0000000 --- a/docs/tests/phase0-findings.md +++ /dev/null @@ -1,331 +0,0 @@ -# Phase 0 — VM vs LXC Overhead Spike: Findings - -**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, Debian 13 (Trixie), -kernel 7.0.2-6-pve, 4 vCPU, 16 GB RAM (15771 MB `MemTotal`). -**Date:** 2026-06-07. **Measured one guest at a time, the other fully stopped.** - -> This document presents **data and observations only**. No recommendation or verdict — -> the architecture decision is made elsewhere. - ---- - -## 1. Provenance - -### Platform -| Component | Version | -|---|---| -| pve-manager | 9.2.2 (`b9984c6d90a4bd80`) | -| kernel | proxmox-kernel 7.0.2-6-pve | -| pve-qemu-kvm | 11.0.0-3 | -| qemu-server | 9.1.15 | -| pve-container | 6.1.10 | -| lxc-pve / lxcfs | 7.0.0-2 / 7.0.0-pve1 | -| criu | 4.1.1-1 | - -`pvesh get /version` → release 9.2, version 9.2.2. - -### Guest images -| | LXC (9001) | VM (9000) | -|---|---|---| -| Source | `local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst` | `debian-13-genericcloud-amd64.qcow2` | -| Build | Debian 13.1 standard CT template (downloaded via `pveam`, checksum verified) | cloud build **20260601-2496**; in-guest reports Debian **13.5** after `apt update` | -| qcow2 | n/a | virtual 3 GiB, on-disk 323 MiB, compat 1.1/zlib | - -### Docker (identical in both guests) -| | LXC | VM | -|---|---|---| -| Source | Docker official apt repo, **`trixie` channel** (confirmed present) | same | -| Version | **29.5.3** build d1c06ef | **29.5.3** build d1c06ef | -| Storage Driver | **`overlayfs`** (not vfs) | **`overlayfs`** (not vfs) | -| Cgroup Version / Driver | **v2 / systemd** | **v2 / systemd** | -| `hello-world` | OK | OK | - -> Docker's official repo **does** have a `trixie` channel — no fallback to Debian's -> `docker.io` was needed. Docker 29 reports the driver as `overlayfs` (the containerd -> snapshotter image store) rather than the legacy name `overlay2`; this is the same -> overlay technology and is **not** a `vfs` fallback. - ---- - -## 2. Comparison table - -Baseline (both guests stopped): host RAM used **median 1702 MB** (range 1699–1703); -host CPU **~0.1 % used** (99.9 % idle). All RAM deltas below are vs this baseline. -Host RAM used = `MemTotal − MemAvailable`, 5 samples ~3 s apart (median reported). - -| Metric | LXC (9001) | VM (9000) | Δ (VM − LXC) | -|---|---|---|---| -| **Idle host-RAM delta** | **+211 MB** (1913) | **+2056 MB** (3758) | **+1845 MB** | -| **Under-load host-RAM delta** | **+410 MB** (2112) | **+2084 MB** (3786) | **+1674 MB** | -| **Per-guest mem attribution** | cgroup `memory.current` = **1961 MB**¹ | KVM process RSS = **2031 MB** (idle) / **2047 MB** (load) | — | -| **Idle host CPU used** | **~0.3 %** (0.20 usr + 0.10 sys) | **~6.0 %** (3.37 usr + 2.31 sys + 0.29 guest) | **+5.7 pp** | -| **Under-load host CPU used** | **~39.4 %** (17.1 usr + 7.5 sys + 14.5 iowait + 0.3 soft) | **~53.9 %** (31.9 guest + 16.4 iowait + 3.4 sys + 1.7 usr + 0.6 soft) | **+14.5 pp** | -| **pgbench throughput** | **2211.7 tps**, lat 1.809 ms, 132 710 tx/60 s, 0 failed | **1819.6 tps**, lat 2.198 ms, 163 764 tx/90 s, 0 failed² | **−392 tps** | -| **Disk allocated** | 10 GiB | 10 GiB | 0 | -| **Disk used (host thin-LV)** | 26.73 % ≈ **2.67 GiB** | 29.33 % ≈ **2.94 GiB** | +0.27 GiB | -| **Disk used (inside guest)** | 2.1 GiB / 9.7 GiB | 2.4 GiB / 9.7 GiB | +0.3 GiB | -| **Provisioning (rough, create→ready)** | ~10–15 s³ | ~60–75 s³ | — | - -¹ `memory.current` counts reclaimable page cache shared with the host and therefore -**overstates** the LXC's true incremental cost; the +211 MB host-RAM delta is the honest -number. ² VM 60 s runs gave 1739 & 1759 tps — consistent with the 90 s definitive run. -³ Guest-creation step only; see §4. Docker install + first image pull (~network-bound, -~identical for both) is excluded. - -### Inside-guest `free -m` (context only — not the decisive number) -| | total | used | buff/cache | available | -|---|---|---|---|---| -| LXC idle | 2048 | 125 | 1851 | 1922 | -| VM idle | 1974 | 509 | 1524 | 1464 | - -The VM sees **1974 MB** usable of 2048 allocated (firmware/kernel reservation). - ---- - -## 3. Docker-in-LXC viability - -**Worked cleanly in an *unprivileged* LXC with `--features nesting=1,keyctl=1`. No -privileged fallback was needed.** - -- `--features nesting=1,keyctl=1 --unprivileged 1` accepted by `pct create` (PVE 9 - syntax confirmed via `pct help create`). -- `docker run hello-world` → success. -- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs` - fallback**. -- Full 3-container stack (`postgres:17`, `redis:7`, `nginx:alpine`) came up healthy. -- Named volume `pgdata` persisted a write (`SELECT count` returned 1 after table - create/insert). -- Multi-container networking + published port worked: `curl localhost:8080` → **HTTP 200**. -- 60 s pgbench load: **0 failed transactions**. - -No errors, no `dmesg`/`journalctl` anomalies, no workarounds. The privileged-LXC -fallback path (step A5) was therefore **not exercised**. - ---- - -## 4. Observations & confounds - -1. **VM under-load CPU required a re-measurement (diagnosed, not hidden).** The first - VM-load sample showed host CPU ~5 % — identical to *idle* — while pgbench nonetheless - completed a full 60 s run (1739 tps). Root cause: the VM load was launched through a - **nested SSH + `nohup &`** layer (host→VM), which started pgbench *after* the sampling - window. The LXC path used local `pct exec` (no nested SSH) so its first sample was - valid. Re-running with pgbench held in the **foreground of a long-lived SSH channel** - (guaranteed active) and sampling during a confirmed window gave the true **53.9 %** - (`%guest`=31.9). **Confound:** the two guests' load was driven through different - plumbing (`pct exec` vs nested SSH); the *throughput* numbers are unaffected - (pgbench self-reports its own duration), but the CPU figures came from - methodologically asymmetric harnesses. -2. **Baseline drift from residual page cache.** After stopping each guest, host RAM did - not snap back to 1702 MB immediately (e.g. 1895 MB just after the LXC stopped; - 1965→1794 MB drifting down after the VM). This is reclaimable cache, not a leak. - Treat all RAM deltas as ±~100 MB. -3. **The headline RAM gap is structural, not incidental.** LXC processes share the host - kernel and page cache, so only the working set counts against the host (+211 MB idle). - The VM, with **no ballooning configured**, has KVM back every guest-touched page — - including the guest's own 1.5 GB page cache — so the host cost ≈ the full 2 GB - allocation (KVM RSS ≈ 2031 MB) and is **largely load-independent** (3758 idle → 3786 - load). Ballooning / KSM were not tested and could change this. -4. **`cgroup memory.current` ≠ host cost.** For the LXC it read 1961 MB (near the 2 GB - limit) because it includes reclaimable page cache; the real incremental host cost was - +211 MB. Per the protocol, `MemTotal − MemAvailable` is the decisive metric. -5. **VM idle CPU floor (~6 %) vs LXC (~0.3 %).** QEMU device emulation + a full guest - kernel's timer/housekeeping impose a small constant CPU cost even at rest. -6. **Throughput vs CPU trade.** The VM did slightly *less* work (1820 vs 2211 tps) for - *more* host CPU (53.9 vs 39.4 %). The extra cost surfaces as `%guest` (31.9 %) — the - actual DB work *plus* virtualization overhead — whereas in the LXC the same DB work - appears directly as host `%usr`/`%sys`. iowait was comparable (~15–16 %, WAL fsync). -7. **Workload fits in RAM.** pgbench scale `-s 10` (~150 MB) fits in cache in both - guests, so the test is commit/CPU-bound rather than disk-bound; a larger-than-RAM - dataset would stress the storage paths differently and is not covered here. -8. **qemu-guest-agent confirmed on the VM** (`qm guest cmd 9000 ping` → OK). This enables - `guest-fsfreeze`-based app-consistent `snapshot`-mode vzdump for the VM — a capability - the LXC has no equivalent for. The genericcloud image does **not** ship the agent; - it had to be installed in-guest (and the VM IP had to be found via `nmap`/MAC until - the agent was up). -9. **Provisioning asymmetry foreshadows cloning.** LXC create is template-extract-bound - (526 MiB at 387 MiB/s + SSH keygen, ~10–15 s). VM create is qcow2-import-bound (3 GiB - → LVM ≈ 30 s) plus a full firmware boot to SSH-ready (~30–45 s). Figures are rough, - single-run, and exclude the shared network-bound Docker install + first image pull. - ---- - -## 5. Raw command log (appendix) - -### 5.1 Provenance -``` -$ pveversion -v | grep ... -pve-manager: 9.2.2 (running version: 9.2.2/b9984c6d90a4bd80) -proxmox-kernel-7.0: 7.0.2-6 -criu: 4.1.1-1 -lxc-pve: 7.0.0-2 -lxcfs: 7.0.0-pve1 -pve-container: 6.1.10 -pve-qemu-kvm: 11.0.0-3 -qemu-server: 9.1.15 - -$ pvesm status -local dir active 98497780 4333576 89114656 4.40% -local-lvm lvmthin active 365760512 0 365760512 0.00% - -# Docker repo trixie channel: -$ curl -fsSL https://download.docker.com/linux/debian/dists/ | grep -oE 'trixie|bookworm|bullseye' -bookworm / bullseye / trixie # trixie present - -# Cloud image: -$ qemu-img info debian-13-genericcloud-amd64.qcow2 -virtual size: 3 GiB ; disk size: 323 MiB ; compat 1.1 ; build 20260601-2496 -``` - -### 5.2 Baseline (both guests stopped) -``` -$ for i in 1..5; awk MemTotal-MemAvailable /proc/meminfo ; sleep 3 -used=1699 MB / 1702 / 1702 / 1702 / 1703 MB (median 1702) - -$ mpstat 1 5 -Average: all 0.05 usr 0.05 sys ... 99.90 idle -``` - -### 5.3 LXC 9001 — create + Docker -``` -$ pct create 9001 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \ - --hostname spike-lxc --cores 2 --memory 2048 --rootfs local-lvm:10 \ - --net0 name=eth0,bridge=vmbr0,ip=dhcp --features nesting=1,keyctl=1 \ - --unprivileged 1 --start 1 - Logical volume "vm-9001-disk-0" created. - extracting archive ... Total bytes read: 551505920 (526MiB, 387MiB/s) - Creating SSH host key ... done -=== exit: 0 ; status: running -features: nesting=1,keyctl=1 ; unprivileged: 1 ; ip 192.168.0.115/24 - -# Docker install (official repo, trixie stable): DOCKER-INSTALL-OK -$ docker --version -> Docker version 29.5.3, build d1c06ef -$ docker run --rm hello-world -> Hello from Docker! -$ docker info | grep -iE 'Storage Driver|Cgroup' - Storage Driver: overlayfs - Cgroup Driver: systemd - Cgroup Version: 2 - Server Version: 29.5.3 ; Kernel: 7.0.2-6-pve ; OS: Debian GNU/Linux 13 (trixie) -``` - -### 5.4 LXC 9001 — stack health -``` -$ docker compose ps -spike-cache-1 running Up -spike-db-1 running Up -spike-web-1 running Up -$ curl -s -o /dev/null -w 'HTTP %{http_code}' localhost:8080 -> HTTP 200 -$ psql CREATE TABLE spike_persist; INSERT; SELECT count(*) -> 1 (volume persists) -``` - -### 5.5 LXC 9001 — idle measurement -``` -Host RAM used (5x3s): 1913 / 1914 / 1913 / 1914 / 1913 MB (median 1913, Δ +211) -cgroup memory.current: 2056036352 B = 1961 MB -inside free -m: total 2048 used 125 buff/cache 1851 available 1922 -mpstat 1 5 Average: 0.20 usr 0.10 sys ... 99.70 idle (~0.3% used) -pct df 9001: rootfs 9.7G size, 2.1G used, 21.6% -``` - -### 5.6 LXC 9001 — under-load measurement -``` -$ pgbench -i -s 10 -> done in 1.39 s -$ pgbench -T 60 -c 4 (run concurrently with sampling): -Host RAM used (5x3s): 2149 / 2143 / 2112 / 2086 / 2071 MB (median 2112, Δ +410) -cgroup memory.current: 2130382848 B = 2032 MB -mpstat 1 5 Average: 17.10 usr 7.50 sys 14.50 iowait 0.31 soft 60.59 idle (~39.4% used) -pgbench result: scaling 10, clients 4, 60 s - transactions: 132710 ; failed 0 (0.000%) - latency average = 1.809 ms ; tps = 2211.713864 -host thin LV vm-9001-disk-0: 10240 MB, Data% 26.73 (≈2.67 GiB) -``` - -### 5.7 VM 9000 — create + cloud-init -``` -$ qm create 9000 --name spike-vm --cores 2 --memory 2048 \ - --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --agent 1 -$ qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2 - transferred 3.0 GiB of 3.0 GiB (100.00%) - scsi0: successfully created disk 'local-lvm:vm-9000-disk-0,size=3G' -$ qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0 -$ qm disk resize 9000 scsi0 10G -> resized 3.00 -> 10.00 GiB -$ qm set 9000 --ciuser spike --cipassword spike --sshkeys /root/spike-pubkey.pub --ipconfig0 ip=dhcp - # pubkey file = the two real keys from the host's /etc/pve/priv/authorized_keys - # (incl. ssh-ed25519 ...kisfenyo@windows — the same workstation key) -$ qm start 9000 -> start-ok -``` - -### 5.8 VM 9000 — IP discovery + guest agent + Docker -``` -# genericcloud has no guest-agent at first boot -> qm guest cmd ping failed. -# IP found via MAC on the bridge: -$ nmap -sn 192.168.0.0/24 | grep -B2 BC:24:11:C7:41:87 - Nmap scan report for 192.168.0.155 ; MAC BC:24:11:C7:41:87 (Proxmox) -$ ssh -i /root/.ssh/id_rsa spike@192.168.0.155 'hostname; cat /etc/debian_version' - spike-vm ; 13.5 -# install qemu-guest-agent + Docker (official repo, trixie): VM-INSTALL-OK -$ qm guest cmd 9000 ping -> AGENT OK (fsfreeze available) -$ docker --version -> Docker version 29.5.3, build d1c06ef -$ docker run --rm hello-world -> Hello from Docker! -$ docker info | grep -iE 'Storage Driver|Cgroup' - Storage Driver: overlayfs ; Cgroup Driver: systemd ; Cgroup Version: 2 -``` - -### 5.9 VM 9000 — stack health -``` -$ docker compose ps -> spike-cache-1 / spike-db-1 / spike-web-1 all running -$ curl ... localhost:8080 -> HTTP 200 -$ psql ... SELECT count(*) -> 1 (volume persists) -``` - -### 5.10 VM 9000 — idle measurement -``` -Host RAM used (5x3s): 3758 / 3757 / 3754 / 3759 / 3758 MB (median 3758, Δ +2056) -KVM process RSS / VSZ: 2079988 / 3380896 KiB (RSS = 2031 MB) -inside free -m: total 1974 used 509 buff/cache 1524 available 1464 -mpstat 1 5 Average: 3.37 usr 2.31 sys 0.29 guest ... 94.04 idle (~6.0% used) -qm config: scsi0 local-lvm:vm-9000-disk-0,size=10G -host thin LV vm-9000-disk-0: 10240 MB, Data% 29.33 (≈2.94 GiB) -inside df -h /: 9.7G size, 2.4G used, 25% -``` - -### 5.11 VM 9000 — under-load measurement (definitive, load confirmed active) -``` -# First attempt (nested-ssh + nohup &) launched pgbench AFTER the sample window -> -# host CPU read a false ~5% (identical to idle). Diagnosed; re-run below holds -# pgbench in the foreground of a long-lived SSH channel and samples during it. - -$ pgbench -T 90 -c 4 (foreground, channel held): - transactions: 163764 ; failed 0 (0.000%) - latency average = 2.198 ms ; tps = 1819.602345 - (60 s confirmation runs: 1739 & 1759 tps) - -# Sampled 10 s into the confirmed-active load: -Host RAM used (5x3s): 3784 / 3786 / 3786 / 3786 / 3786 MB (median 3786, Δ +2084) -KVM process RSS / VSZ: 2096508 / 4495008 KiB (RSS = 2047 MB) -guest uptime: load average 1.71 (2 vCPU) -> vCPUs busy -mpstat 1 8 Average: - 1.70 usr 3.40 sys 16.35 iowait 0.58 soft 31.89 guest 46.08 idle (~53.9% used) -``` - -### 5.12 Teardown state -``` -$ qm list -> 9000 spike-vm stopped -$ pct list -> 9001 spike-lxc stopped -# both present, both stopped (numbers can be re-checked) -``` - ---- - -## 6. Teardown — destroy commands (NOT run) - -Both guests were left **stopped but present**. To remove them: - -```bash -qm destroy 9000 --purge # VM (also removes cloudinit + disks) -pct destroy 9001 --purge # LXC -# optional spike artifacts on the host: -rm -f /var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2 -rm -f /root/spike-pubkey.pub /root/vm-install.sh -# (Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst) -``` diff --git a/docs/tests/phase1-2-findings.md b/docs/tests/phase1-2-findings.md deleted file mode 100644 index ff04080..0000000 --- a/docs/tests/phase1-2-findings.md +++ /dev/null @@ -1,315 +0,0 @@ -# Phase 1 + 2 — Privilege Model & Backup/Restore Round-Trip: Findings - -**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via -`pvesh get /nodes` → `demo-felhom`. Storage: `local` (dir, content -`iso,vztmpl,backup,import`), `local-lvm` (LVM-thin, `rootdir,images`). -**Subject:** LXC `9001` (`spike-lxc`, unprivileged, `nesting=1,keyctl=1`, Docker + -postgres/redis/nginx stack). **Date:** 2026-06-07. - -> Data and observations only — **no recommendation or verdict**. - -## Hypotheses — verdicts at a glance -| | Hypothesis | Result | -|---|---|---| -| **H1** | Backup scopes to one VMID; restore/create needs node/pool allocate → denied to narrow token | **CONFIRMED** (create CT = 403) | -| **H2** | An LXC vzdump captures the Docker volumes (they live in the container rootfs) | **CONFIRMED** (sentinel survived both restores) | -| **H3** | Crash-consistent (running) *and* quiesced (stopped) backups both restore cleanly | **CONFIRMED** (A via WAL recovery, B clean start) | -| **H4** | Running unprivileged LXC snapshots on LVM-thin; restored CT keeps unprivileged+nesting/keyctl | **CONFIRMED** (live snapshot OK; config survived) | - ---- - -## 1. Phase 1 — Privilege model - -### 1.1 Setup (operator side, root) -``` -pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" -pveum user add felhom-ctl@pve --comment "spike in-guest controller" -pveum user token add felhom-ctl@pve ctl --privsep 1 # secret: b6547d9d-... (ephemeral, spike-only) -pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -``` -Privilege names were verified against `PVEVMAdmin` / `PVEDatastoreUser` via -`pveum role list` first. **Note:** the reference doc's introspection command -`pveum role info ` **does not exist in PVE 9** — only `pveum role list` works. - -### 1.2 ⚠️ Privsep gotcha — the doc's runbook is incomplete -With `--privsep 1`, a token's effective rights are the **intersection of the backing -user's permissions AND the token's own ACLs**. The reference doc (§3) grants ACLs to the -**token only**. With the user `felhom-ctl@pve` holding **no** permissions, the -intersection was **empty** — the first self-audit call returned: -``` -HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"} -``` -**Fix applied:** also grant the user the role on the same paths -(`pveum acl modify /vms/9001 -user felhom-ctl@pve -role FelhomSelfBackup`, same for -`/storage/local`). After that the self-calls succeeded. **A privsep token needs the -permission present on *both* the user and the token** (the token ACL is what keeps the -token ≤ user / narrowly scoped). This must be reflected in the controller provisioning. - -### 1.3 Test matrix (every call run from **inside** the unprivileged LXC, `pct exec 9001`) -`H=192.168.0.162 N=demo-felhom AUTH="PVEAPIToken=felhom-ctl@pve!ctl="` - -| # | Call | Expected | **Actual** | Notes | -|---|---|---|---|---| -| 1 | `GET /version` | 200 | **200** | reachable + auth from inside LXC (no privilege needed) | -| 2 | `GET /nodes/$N/lxc/9001/status/current` | 200 | **200**¹ | self audit (after privsep fix) | -| 3 | `POST /nodes/$N/lxc/9001/snapshot snapname=spk1` | 200/UPID→OK | **200, task exitstatus OK** | **running-LXC self-snapshot (H4)** | -| 4 | `POST /nodes/$N/vzdump vmid=9001 storage=local mode=snapshot` | 200/UPID→OK | **200, task exitstatus OK** | self backup, archive produced | -| 5 | `GET /nodes/$N/qemu/9000/status/current` | 403 | **403** | `Permission check failed (/vms/9000, VM.Audit)` | -| 6 | `POST /nodes/$N/vzdump vmid=9000 storage=local` | 403 | **200 POST → task exitstatus 403**² | see note | -| 7 | `POST /nodes/$N/lxc` (create CT) | 403 | **403** | `Permission check failed` — **proves create/allocate is operator-tier (H1)** | - -¹ before the privsep fix this was 403; see §1.2. -² **Important nuance:** the `vzdump` endpoint accepts the POST and returns a UPID even for -an unauthorized vmid; the authorization failure surfaces at **task execution**, not at the -HTTP layer. Polled from root: -`exitstatus: "403 Permission check failed (/vms/9000, VM.Backup)"`, and **no 9000 archive -was created**. The boundary holds — but a controller must **poll the task exitstatus**, not -trust the POST's 200, to know a cross-guest backup was actually refused. - -**Pass criteria met:** self-ops (1–4) succeed; cross-guest read (5), cross-guest backup -(6, at task level), and create/allocate (7) are denied. The controller-as-guest boundary -and the two-tier split are validated. - -### 1.4 Final minimal role — `VM.PowerMgmt` **not** required -The doc's open question ("does Tier A need `VM.PowerMgmt` for stop-mode backups? Likely -yes"). **Tested and refuted:** a **stop-mode** self-vzdump submitted by the token -(`vmid=9001 mode=stop`) completed with **`exitstatus: OK`** using the role *without* -`VM.PowerMgmt`. `vzdump` performs the guest shutdown/restart internally under -`VM.Backup`; no separate power privilege is needed. - -> **Final minimal role (`FelhomSelfBackup`) — satisfies self-audit, self-snapshot, and -> both `snapshot`- and `stop`-mode self-backup:** -> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit` -> (`VM.PowerMgmt` deliberately omitted — confirmed unnecessary.) - -### 1.5 TLS observation -From inside the LXC, `curl` **without** `-k`: -``` -curl: (60) SSL certificate problem: unable to get local issuer certificate -``` -The host serves the default self-signed PVE cert; all tests used `-k`. Production trust -(pin the PVE CA / issue a proper cert) is a separate design decision, flagged here. - -### 1.6 Running-LXC snapshot (H4) -Call #3 snapshotted the **running** unprivileged LXC on LVM-thin (`exitstatus OK`). -`pct listsnapshot 9001` shows `spk1` with `pct status 9001 = running`. **No stop -required** — the snapshot-before-update rollback flow is viable on a live container. - ---- - -## 2. Phase 2 — Backup → real restore round-trip - -Sentinel written pre-flight into the `pgdata` volume: -`restore_check(42,'phase2-sentinel')` → clean read `42|phase2-sentinel`. - -### 2.1 Backups (operator/root side) -| Variant | Mode | Stack state | Task time | Wall | Archive | Size (zstd) | -|---|---|---|---|---|---|---| -| **A — crash-consistent** | `snapshot` | **running** | 00:00:24 | 25 s | `vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst` | **934 MB** (979,718,569 B) | -| **B — quiesced** | `snapshot` | **stopped** (`docker compose stop`) | 00:00:21 | 22 s | `vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst` | **934 MB** (979,671,582 B) | - -Both from a 2.5 GiB source; zstd → ~934 MB (~2.7:1). The stack was restarted after -Variant B. **LXC snapshot-mode vzdump does *not* fsfreeze** (no guest agent in an LXC — -consistent with the Phase 0 finding) → Variant A is genuinely crash-consistent. - -### 2.2 Restore → fresh VMID → boot → verify -| Check | 9002 (Variant A) | 9003 (Variant B) | -|---|---|---| -| Restore time (`pct restore … --storage local-lvm`) | **12 s** | **11 s** | -| `unprivileged: 1` survived | **yes** | **yes** | -| `features: nesting=1,keyctl=1` survived | **yes** | **yes** | -| Containers after boot | `exited` (no restart policy) → `docker compose up -d` | same | -| 3 containers healthy | **yes** | **yes** | -| `curl localhost:8080` | **HTTP 200** | **HTTP 200** | -| **Sentinel `(42,'phase2-sentinel')`** | **PRESENT** | **PRESENT** | -| Postgres first-start | **WAL crash recovery** (see below) | **clean start, no recovery** | - -> Restored CTs inherit 9001's fixed `hwaddr`. To avoid a MAC clash with the still-running -> 9001 on `vmbr0`, `net0` was reset to auto-generate a fresh MAC before boot. All -> verification (stack health, `curl localhost`, sentinel) is guest-internal and needs no -> external network — and the Docker images are inside the restored rootfs, so no pulls. - -**Variant A — Postgres automatic WAL recovery on 9002 (verbatim, post-restore boot):** -``` -LOG: database system was interrupted; last known up at 2026-06-07 18:13:21 UTC -LOG: database system was not properly shut down; automatic recovery in progress -LOG: redo starts at 0/CB12838 -LOG: invalid record length at 0/CB12870: expected at least 24, got 0 # normal end-of-WAL -LOG: redo done at 0/CB12838 ... -LOG: checkpoint starting: end-of-recovery immediate wait -LOG: database system is ready to accept connections -``` -**Variant B — clean start on 9003 (verbatim, post-restore boot):** -``` -LOG: database system was shut down at 2026-06-07 18:14:39 UTC -LOG: database system is ready to accept connections -``` - -**H2 confirmed:** one LXC vzdump captured the whole customer including the Docker named -volume — the sentinel data restored in both guests. **H3 confirmed:** both variants -restored to a bootable guest with intact data; the crash-consistent one recovered via WAL -with no manual intervention, the quiesced one started clean. **H4 confirmed:** restored -config preserved `unprivileged` + `nesting/keyctl`, so Docker ran in the restored CT. - ---- - -## 3. Observations & confounds -1. **Privsep token needs perms on user *and* token** (§1.2) — the single most important - correction to the reference runbook; without it every scoped call 403s. -2. **vzdump authorization is task-level, not POST-level** (§1.3 note ²) — a 200 + UPID - does **not** mean authorized. The controller must poll `exitstatus`. This is also the - general async-task lesson: every backup/snapshot/restore returns a UPID and the real - result is in the task status. -3. **`pveum role info` is gone in PVE 9** — use `pveum role list`. Minor doc drift. -4. **`VM.PowerMgmt` not needed for stop-mode backup** (§1.4) — narrower role than the doc - assumed. -5. **No fsfreeze for LXC** — Variant A relied on Postgres's own WAL crash recovery, which - worked here for an idle-at-backup DB. Under heavy write load, app-consistency for LXC - still rests on the controller quiescing first (or stop-mode), exactly as the reference - warned. This single test is not a durability guarantee under load. -6. **Restore MAC collision** (§2.2) — `pct restore` preserves the source `hwaddr`; - restoring while the original runs needs a MAC reset (or the original stopped). The - controller's restore flow must handle identity (MAC/hostname/IP) to avoid clashes. -7. **No restart policy on the compose services** — restored containers came up `exited`; - `docker compose up -d` (or a restart policy / systemd unit) is required for the stack - to return automatically after a restore or guest reboot. -8. **Restore is fast, backup dominated by I/O** — restores were 11–12 s (extract at - ~524 MiB/s); backups ~22–25 s (read 2.5 GiB at ~108–119 MiB/s + zstd). Single runs, - idle host, ~150 MB DB; not a throughput benchmark. -9. **Sequencing artifact:** a Phase-1 stop-mode self-backup ran before Phase 2 and - stopped/started 9001; the stack was brought back up and the sentinel re-verified - before the Variant A/B backups, so it does not affect the round-trip results. - ---- - -## 4. Raw command log (appendix) - -### 4.1 Pre-flight -``` -$ pvesh get /nodes -> node: demo-felhom -$ cat /etc/pve/storage.cfg -dir: local ... content iso,vztmpl,backup,import # 'backup' present -lvmthin: local-lvm ... content rootdir,images # no backup (expected) -$ pct start 9001 ; docker compose up -d -> 3 containers Started -$ curl localhost:8080 -> HTTP 200 -# sentinel: -CREATE TABLE ; INSERT 0 1 ; SELECT count -> 1 ; SELECT * -> 42 | phase2-sentinel -``` - -### 4.2 Phase 1 — role/user/token/ACL -``` -$ pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit" -> role-ok -$ pveum user add felhom-ctl@pve --comment "spike in-guest controller" -> user-ok -$ pveum user token add felhom-ctl@pve ctl --privsep 1 - {"full-tokenid":"felhom-ctl@pve!ctl","info":{"privsep":"1"},"value":"b6547d9d-08ec-4f22-beb8-a551dc2cd69d"} -$ pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok -$ pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok -$ pveum role list | grep FelhomSelfBackup - FelhomSelfBackup | Datastore.AllocateSpace,Datastore.Audit,VM.Audit,VM.Backup,VM.Snapshot -$ pveum role info FelhomSelfBackup -> ERROR: unknown command 'pveum role info' # PVE9 has no 'role info' -``` - -### 4.3 Phase 1 — matrix (from inside LXC) -``` -# TLS without -k: -curl: (60) SSL certificate problem: unable to get local issuer certificate - -# BEFORE privsep fix: -#2 GET self status -> HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"} - -# privsep fix: -$ pveum acl modify /vms/9001 -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok -$ pveum acl modify /storage/local -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok - -# AFTER fix: -#1 GET /version -> HTTP 200 -#2 GET /nodes/.../lxc/9001/status/current -> HTTP 200 {"data":{...,"status":"running",...}} -#5 GET /nodes/.../qemu/9000/status/current -> HTTP 403 (/vms/9000, VM.Audit) -#6 POST vzdump vmid=9000 -> HTTP 200 {"data":"UPID:...vzdump:9000:felhom-ctl@pve!ctl:"} - root poll: exitstatus="403 Permission check failed (/vms/9000, VM.Backup)" - task log: TASK ERROR: 403 Permission check failed (/vms/9000, VM.Backup) - /var/lib/vz/dump: no 9000 archive created -#7 POST /nodes/.../lxc (create CT vmid=9009) -> HTTP 403 {"message":"Permission check failed\n"} - -#3 POST lxc/9001/snapshot snapname=spk1 -> HTTP 200 UPID:...vzsnapshot:9001... - root: exitstatus "OK" ; pct listsnapshot 9001 -> spk1 ; pct status 9001 -> running -#4 POST vzdump vmid=9001 storage=local mode=snapshot -> HTTP 200 UPID:...vzdump:9001... - root: exitstatus "OK" - token can read own task status: HTTP 200 {"...exitstatus":"OK"} # earlier poll TIMEOUTs were a shell-quoting bug in the helper, not a perms issue - -# stop-mode self-backup (VM.PowerMgmt test): -$ token POST vzdump vmid=9001 storage=local mode=stop -> HTTP 200 UPID:...vzdump:9001... - root poll: exitstatus "OK" # SUCCEEDED without VM.PowerMgmt in the role -``` - -### 4.4 Phase 2 — backups -``` -# Variant A (running): -$ vzdump 9001 --mode snapshot --storage local --compress zstd -INFO: Total bytes written: 2585589760 (2.5GiB, 108MiB/s) -INFO: archive file size: 934MB -INFO: Finished Backup of VM 9001 (00:00:24) ; WALL_SECONDS=25 --> vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst (979718569 B) - -# Variant B (stopped): -$ docker compose stop (cache,db,web Stopped) -$ vzdump 9001 --mode snapshot --storage local --compress zstd -INFO: Total bytes written: 2585825280 (2.5GiB, 119MiB/s) -INFO: Finished Backup of VM 9001 (00:00:21) ; WALL_SECONDS=22 --> vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst (979671582 B) -$ docker compose start (db,cache,web Started) -``` - -### 4.5 Phase 2 — restores + verification -``` -# A -> 9002: -$ pct restore 9002 .../20_13_43.tar.zst --storage local-lvm - Total bytes read: 2585589760 (2.5GiB, 524MiB/s) ; RESTORE_A_SECONDS=12 -$ pct config 9002 -> features: nesting=1,keyctl=1 ; unprivileged: 1 -$ pct set 9002 -net0 name=eth0,bridge=vmbr0,ip=dhcp # fresh MAC BC:24:11:E3:F4:64 -$ pct start 9002 ; docker compose up -d -> 3 running ; curl -> HTTP 200 -$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel - db log: "was interrupted ... not properly shut down; automatic recovery in progress - redo starts/redo done ... database system is ready to accept connections" - -# B -> 9003: -$ pct restore 9003 .../20_14_40.tar.zst --storage local-lvm - Total bytes read: 2585825280 (2.5GiB, 524MiB/s) ; RESTORE_B_SECONDS=11 -$ pct config 9003 -> features: nesting=1,keyctl=1 ; unprivileged: 1 -$ pct set 9003 -net0 ... (fresh MAC) ; pct start 9003 ; docker compose up -d -> 3 running ; curl 200 -$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel - db log: "database system was shut down at ... ; database system is ready to accept connections" # clean -``` - ---- - -## 5. Teardown (executed) -Restore targets destroyed; Phase 1 objects and spike artifacts removed; `9000`/`9001` -left **stopped-but-present**. Verified clean: `felhom-ctl@pve` deleted, no spike ACLs, -empty `dump/`, `spk1` removed. - -> **Correction:** `pveum acl delete` **requires `--roles`** (a bare `-user`/`-token` -> path errors `400 roles: property is missing`). In practice the explicit ACL deletes -> are unnecessary — deleting the token/user/role **auto-invalidates** the referencing -> ACLs (PVE logs `ignore invalid acl token …` and drops them). - -```bash -pct stop 9002 ; pct stop 9003 ; pct destroy 9002 --purge ; pct destroy 9003 --purge -# correct ACL-delete syntax (needs --roles), or just let user/role deletion clean them: -pveum acl delete /vms/9001 --roles FelhomSelfBackup --users 'felhom-ctl@pve' -pveum acl delete /vms/9001 --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl' -pveum acl delete /storage/local --roles FelhomSelfBackup --users 'felhom-ctl@pve' -pveum acl delete /storage/local --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl' -pveum user token remove felhom-ctl@pve ctl ; pveum user delete felhom-ctl@pve ; pveum role delete FelhomSelfBackup -pct delsnapshot 9001 spk1 -rm -f /var/lib/vz/dump/vzdump-lxc-9001-*.tar.zst /var/lib/vz/dump/vzdump-lxc-9001-*.log -pct stop 9001 # back to stopped-but-present -``` - -## 6. To destroy 9000/9001 later (NOT run — left stopped-but-present) -```bash -qm destroy 9000 --purge # VM (Phase 0 subject) -pct destroy 9001 --purge # LXC (Phase 0/1/2 subject) -# Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst -``` diff --git a/docs/tests/phase3-findings.md b/docs/tests/phase3-findings.md deleted file mode 100644 index 4e61b35..0000000 --- a/docs/tests/phase3-findings.md +++ /dev/null @@ -1,234 +0,0 @@ -# Phase 3 — vzdump exclusion (B2) & agent operator role + root boundary (B3): Findings - -**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via -`pvesh get /nodes` → `demo-felhom`. **Date:** 2026-06-08. Throwaway resources (VMIDs -9010-9023, role/user `FelhomAgent`/`felhom-agent@pve`); all torn down (only the pre-existing -9000/9001 remain, stopped). Every Proxmox op polled to `task exitstatus` (not the POST -return). - -> Validates the two items the design review (`_design-review.md`) flagged as unvalidated: -> **B2** (what vzdump includes/excludes per LXC mount type + how to keep bulk out) and **B3** -> (the least-privilege operator role + the root-vs-API boundary). Data only. - ---- - -## B2 — vzdump inclusion/exclusion matrix - -**Setup:** one unprivileged LXC `9010` (`nesting=1,keyctl=1`, overlayfs), Docker 29.5.3 -installed, with five sentinel locations: - -| # | location | config | -|---|---|---| -| 1 | rootfs file `/SENTINEL_ROOTFS` | rootfs (`local-lvm:8`) | -| 2 | Docker **named** volume `b2vol` → `SENTINEL_DOCKERVOL` | default driver | -| 3 | `mp1` volume mount `/mnt/mp1` `SENTINEL_MP1` | `local-lvm:1,backup=1` | -| 4 | `mp2` volume mount `/mnt/mp2` `SENTINEL_MP2` | `local-lvm:1,backup=0` | -| 5 | `mp3` **bind** mount `/mnt/mp3` `SENTINEL_MP3` | host `/root/b2-bindsrc` | -| 6 | bulk Docker vol `bulkvol` bound onto mp2 → `SENTINEL_BULK` | `--driver local -o type=none -o o=bind -o device=/mnt/mp2` | - -**The "trap" confirmed at setup:** the Docker named volume's on-disk path is -`/var/lib/docker/volumes/b2vol/_data` — **inside the LXC rootfs**. - -### Result matrix (stop-mode vzdump → `local`, verified 3 ways: vzdump log, archive grep, restore to 9011) - -| Sentinel | location | flag | **in archive?** | restored 9011 | -|---|---|---|---|---| -| `SENTINEL_ROOTFS` | rootfs | — | **INCLUDED** | present | -| `SENTINEL_DOCKERVOL` | Docker named vol (in rootfs) | — | **INCLUDED** ⚠️ the trap | present | -| `SENTINEL_MP1` | volume mp | `backup=1` | **INCLUDED** | present | -| `SENTINEL_MP2` | volume mp | `backup=0` | **EXCLUDED** | absent (vol recreated empty) | -| `SENTINEL_MP3` | bind mount | n/a | **EXCLUDED** | reappears via re-bind only¹ | -| `SENTINEL_BULK` | Docker vol on mp2 | `backup=0` | **EXCLUDED** | absent | - -¹ The bind-mount **data is not in the archive** (archive grep shows no mp3 path). It -reappears in the restored 9011 only because `pct restore` preserves the bind config -`mp3: /root/b2-bindsrc` and re-attaches the **same host dir**. On a *different* host (true DR) -the bind data would be gone unless backed up separately — important for DR planning. - -**vzdump log (verbatim) — the authoritative per-mount decision:** -``` -INFO: including mount point rootfs ('/') in backup -INFO: including mount point mp1 ('/mnt/mp1') in backup -INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled) -INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume) -``` -**Archive contents (verbatim) — `tar --zstd -tf … | grep SENTINEL`:** -``` -./var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL -./SENTINEL_ROOTFS -./mnt/mp1/SENTINEL_MP1 -``` -**Restore verification (verbatim) — sentinels in restored 9011:** -``` -PRESENT : /SENTINEL_ROOTFS -PRESENT : /var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL -PRESENT : /mnt/mp1/SENTINEL_MP1 -ABSENT : /mnt/mp2/SENTINEL_MP2 -ABSENT : /mnt/mp2/SENTINEL_BULK -PRESENT : /mnt/mp3/SENTINEL_MP3 # via re-bind to same host dir, NOT from archive -``` - -### Proven bulk-exclusion recipe -A "bulk" Docker volume is kept out of the guest vzdump by binding it onto a **volume -mountpoint with `backup=0`**: -1. Attach a Proxmox volume mountpoint with the flag: - `pct set -mpN :,mp=/mnt/bulk,backup=0` -2. Realize the Docker volume on that path: - `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol` - (or a compose bind to `/mnt/bulk`). -3. Data written through `bulkvol` lands on the `backup=0` mountpoint → **excluded** from - vzdump, while rootfs/hot sentinels are **included**. Verified: `SENTINEL_BULK` absent from - archive and restore; `SENTINEL_ROOTFS` present. - -### The trap, stated for the placement component -`backup=` is **only honoured for volume mount points** (confirmed: pct manpage + -vzdump log "excluding volume mount point … (disabled)"). A Docker **named volume uses the -default driver and lands in the rootfs**, which is **always backed up** — so a "bulk" volume -left as an ordinary named volume is **silently swept into the whole-guest image**. The -per-volume placement component **must** realize every `bulk` volume as a dedicated `backup=0` -mountpoint (or external bind mount), never a default named volume. - ---- - -## B3 — agent operator role + root-vs-API boundary - -**Caveat applied (Phase 1):** privsep token needs the role on **both** user and token. Setup: -user `felhom-agent@pve` + privsep token `agent`, role `FelhomAgent`, dual-granted at `/`. -All ops driven **as the token** via the REST API; task `exitstatus` polled. - -> ⚠️ **Terminology:** the Phase-1 `FelhomSelfBackup` role is the discarded **guest-side -> self-backup** role (scoped to one guest, *denied* create/allocate). `FelhomAgent` here is -> its **operator-tier replacement** — a different, broader role. Do not conflate. - -### Op matrix (as the scoped token) - -| # | Operation | API call | Result | -|---|---|---|---| -| read | host status | `GET /nodes/$N/status` | **200** (needs `Sys.Audit`) | -| read | storage list | `GET /storage` | **200** (`Datastore.Audit`) | -| 1 | **create LXC, `nesting=1,keyctl=1`** | `POST /nodes/$N/lxc` | **403** — `changing feature flags (except nesting) is only allowed for root@pam` | -| 1′ | create LXC, **nesting-only** | `POST /nodes/$N/lxc` | **200 / OK** | -| 2 | set config (mem/cpu/options + mountpoint w/ `backup` flag) | `PUT /nodes/$N/lxc//config` | **200** | -| 3 | allocate volume | `POST /nodes/$N/storage/local-lvm/content` | **200** (`Datastore.AllocateSpace`) | -| 4 | start | `POST …/status/start` | **OK** (`VM.PowerMgmt`) | -| 5 | stop | `POST …/status/stop` | **OK** | -| 6a | snapshot | `POST …/snapshot` | **OK** (`VM.Snapshot`) | -| 6b | rollback | `POST …/snapshot/s1/rollback` | **OK** (`VM.Snapshot.Rollback`) | -| 7 | stop-mode backup | `POST /nodes/$N/vzdump mode=stop` | **OK** (`VM.Backup`) | -| 8 | restore → fresh vmid | `POST /nodes/$N/lxc restore=1` | **OK** — and **restored CT kept `features: nesting=1,keyctl=1`** | -| 9 | destroy CT | `DELETE /nodes/$N/lxc/?purge=1` | **OK** (`VM.Allocate`) | -| 9b | add storage definition (dir) | `POST /storage` | **200** (`Datastore.Allocate`, **no root**) | - -**The two headline results:** -1. **`keyctl=1` on create is `root@pam`-only.** Verbatim: - `Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)`. - Confirmed this is **not** token-fixable: a **non-privsep `root@pam` token** got the **same - 403**. Only an actual `root@pam` session (OS root / `pct create` as root) can set it. - `nesting` alone is allowed for a scoped token. -2. **Restore preserves `keyctl`.** A token-authorized `vzrestore` of a keyctl archive produced - `9021` with `features: nesting=1,keyctl=1, unprivileged: 1`. So the **DR/restore path is - fully token-covered**; only *fresh provisioning* needs root for the keyctl flag. - -### Paring (each drop shown to still pass, or proven needed) - -| Privilege | Verdict | Evidence | -|---|---|---| -| `Datastore.AllocateTemplate` | **DROP** (unnecessary) | create-from-template succeeded without it (200/OK) | -| `Sys.Audit` | **KEEP** | `GET /nodes/$N/status` → **403** without it (host metrics, `03` §5) | -| `VM.Config.Network` | **KEEP** | create with `net0` → **403 (/vms/…, VM.Config.Network)** without it | -| `VM.Config.Options` | **KEEP** | config `onboot=1` → **403 (/vms/…, VM.Config.Options)** without it | -| `SDN.Use` | **KEEP (added vs review sketch)** | create → **403 (/sdn/zones/localnetwork/vmbr0, SDN.Use)** without it | - -> Corrections to the review's candidate sketch: `VM.Config.CPUMemory` is **not a real -> privilege** — split into `VM.Config.CPU` + `VM.Config.Memory`. `SDN.Use` was **missing** and -> is **required** (PVE 9 gates bridge use behind it). `Datastore.AllocateTemplate` is **not -> needed**. - -### Final minimal `FelhomAgent` role (proven sufficient for ops 1′–9b) -``` -VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory -VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback -VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.Audit Sys.Audit SDN.Use -``` -(16 privileges. `Datastore.Allocate` is for the storage-definition add; drop it if the agent -never creates Proxmox storage entries via the API. `VM.PowerMgmt` is for start/stop lifecycle -— not for the backup itself, consistent with `proxmox-platform.md` §3.4.) - -### Root-vs-API boundary table (answers `03` §3) - -| Agent host operation | Coverage | Notes | -|---|---|---| -| Create unprivileged LXC, **nesting-only** | **API token** | `VM.Allocate`+`VM.Config.*`+`Datastore.AllocateSpace`+`SDN.Use` | -| **Create with `keyctl=1` (Docker needs it — Phase 0)** | **OS root `root@pam`** (`pct create` as root / sudoers) | no API token works, incl. a root@pam token | -| Set config (mem/cpu/net/options/mountpoint + `backup` flag) | API token | | -| Allocate guest volume | API token | `Datastore.AllocateSpace` | -| Start / stop / snapshot / rollback | API token | `VM.PowerMgmt` / `VM.Snapshot(.Rollback)` | -| vzdump backup (stop/snapshot mode) | API token | `VM.Backup` | -| **Restore from vzdump (preserves keyctl)** | **API token** | DR path needs no root | -| Destroy guest (scratch + compensating rollback, B1) | API token | `VM.Allocate` | -| Add Proxmox **storage definition** (dir/nfs/cifs/pbs) | API token | `Datastore.Allocate`; the *definition* only | -| Host status / metrics report | API token | `Sys.Audit` | -| **USB physical mount-by-UUID / systemd mount unit / fstab** | **OS root / narrow sudoers** | not a Proxmox API op (host-level mount; not tested here) | -| **SMART / hardware sensors** | OS root | not API-exposed | - -**Boundary summary:** nearly the entire guest lifecycle — including **restore** — is covered -by the scoped token. The genuine OS-root residual is narrow: **(1) fresh creation of a -Docker-capable LXC (the `keyctl` flag), (2) physical USB mount-by-UUID / systemd mount units / -fstab, (3) hardware/SMART.** This supports `03` §3's "non-root service + scoped token + narrow -sudoers" model — with the **specific** sudoers/root entries being: `pct create` (or just the -keyctl-setting step) and the host mount operations. - ---- - -## Raw command log (appendix) - -### B2 -``` -pct create 9010 ... --features nesting=1,keyctl=1 --unprivileged 1 # rootfs local-lvm:8 -pct set 9010 -mp1 local-lvm:1,mp=/mnt/mp1,backup=1 -pct set 9010 -mp2 local-lvm:1,mp=/mnt/mp2,backup=0 -pct set 9010 -mp3 /root/b2-bindsrc,mp=/mnt/mp3 -# docker named vol: docker volume inspect b2vol -> /var/lib/docker/volumes/b2vol/_data -# bulk: docker volume create --driver local -o type=none -o o=bind -o device=/mnt/mp2 bulkvol -vzdump 9010 --mode stop --storage local --compress zstd -# INFO: including mount point rootfs ('/') in backup -# INFO: including mount point mp1 ('/mnt/mp1') in backup -# INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled) -# INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume) -tar --zstd -tf | grep SENTINEL # -> rootfs, dockervol, mp1 only -pct restore 9011 --storage local-lvm # -> mp2/bulk absent, mp3 via re-bind -``` - -### B3 -``` -pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Sys.Audit" # candidate (pre-SDN) -pveum user add felhom-agent@pve ; pveum user token add felhom-agent@pve agent --privsep 1 -pveum acl modify / -user 'felhom-agent@pve' -role FelhomAgent -pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent - -# token create with keyctl: -POST /nodes/demo-felhom/lxc ... features=nesting=1,keyctl=1 - -> 403 "changing feature flags (except nesting) is only allowed for root@pam" -# + SDN.Use missing initially: - -> 403 "Permission check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)" -# root@pam non-privsep token, keyctl create: - -> 403 (same "only allowed for root@pam") # tokens never qualify - -# token nesting-only create / config(PUT) / start / stop / snapshot / rollback / -# vzdump(stop) / restore->9021 (kept keyctl) / destroy / POST /storage -> all 200/OK - -# paring: -GET /nodes/$N/status without Sys.Audit -> 403 (KEEP) -create net0 without VM.Config.Network -> 403 (KEEP) -config onboot=1 without VM.Config.Options -> 403 (KEEP) -create from template without Datastore.AllocateTemplate -> OK (DROP) -``` - -### Teardown -``` -pct destroy 9010 9011 9021 --purge # 9020/9022/9023 already destroyed during tests -pveum user token remove felhom-agent@pve agent ; pveum user delete felhom-agent@pve -pveum role delete FelhomAgent # ACLs at / auto-invalidated -rm -f /var/lib/vz/dump/vzdump-lxc-9010-* /var/lib/vz/dump/vzdump-lxc-9020-* -# verified: only 9000/9001 remain (stopped-but-present); no felhom-agent user/role; dump dir empty -``` diff --git a/docs/tests/phase4-signing-findings.md b/docs/tests/phase4-signing-findings.md deleted file mode 100644 index 9d0b0ca..0000000 --- a/docs/tests/phase4-signing-findings.md +++ /dev/null @@ -1,257 +0,0 @@ -# Phase 4 — Control-plane signing primitive (SSHSIG + Go verify): Findings - -**Where run:** build server `192.168.0.180` (Debian 13, **Go 1.24.4**, **OpenSSH 10.0p2**), -no Proxmox. **Date:** 2026-06-08. Throwaway key generated, used, and **deleted** — no private -key, passphrase, or `.sig` committed. - -> De-risks the signing primitive *before* it is written into `04-control-plane-authorization.md` -> or the agent's verify code. **Verdict up front: the approach works cleanly and is key-type- -> agnostic — no fallback needed.** Go verifies the armored `SSHSIG` format, every tamper/replay/ -> authorization case is rejected, and a synthetic FIDO2 `sk-ssh-ed25519` signature verifies -> through the **unchanged** code path (true hardware drop-in). - ---- - -## 0. Result at a glance — 14/14 checks pass - -``` -== Step 2: SSHSIG signature verification (key-type-agnostic path) == - PASS correct verified, op="guest_destroy" - PASS wrong key rejected: signer not in allowed set - PASS tampered blob rejected: signature invalid: ssh: signature did not verify - PASS wrong namespace rejected: namespace mismatch: got "felhom-op-wrong" want "felhom-op-v1" - -== Step 3: anti-replay / authorization (valid signature, still rejected) == - PASS first use verified, op="guest_destroy" - PASS replay (same nonce) rejected: replay: nonce a1b2c3d4...8f90 already seen - PASS expired rejected: expired (expires_at=2020-01-02 ..., now=2026-06-08 ...) - PASS not-yet-valid rejected: not yet valid (issued_at=2030-01-01 ...) - PASS retargeted host rejected: target mismatch: blob=demo-felhom/9001 this=other-host/9001 - PASS retargeted guest rejected: target mismatch: blob=demo-felhom/9001 this=demo-felhom/8888 - -== Step 4: key-type-agnosticism — FIDO2 sk-ssh-ed25519 (synthetic, no device) == - PASS parses sk pubkey type="sk-ssh-ed25519@openssh.com" - PASS authorized_keys form sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5... - PASS sk end-to-end verify verified, op="guest_destroy" -``` - ---- - -## 1. Software round-trip (baseline, CLI) - -- Key: `ssh-keygen -t ed25519 -f felhom-op -N '' -C felhom-operator`. - (Signing non-interactively used an `SSH_ASKPASS` helper + `setsid -w`; in production the - operator key lives behind an agent or a FIDO2 device, so the at-sign passphrase prompt is a - non-issue. The passphrase mechanics are **not** what this spike de-risks.) -- Sign with a **domain-separated namespace**: - `ssh-keygen -Y sign -f felhom-op -n felhom-op-v1 blob.json` → `blob.json.sig` - (armored `-----BEGIN SSH SIGNATURE-----`). -- Baseline verify (CLI sanity) with an allow-list: - ``` - allowed_signers: felhom-operator namespaces="felhom-op-v1" ssh-ed25519 AAAAC3... - $ ssh-keygen -Y verify -f allowed_signers -I felhom-operator -n felhom-op-v1 \ - -s blob.json.sig < blob.json - Good "felhom-op-v1" signature for felhom-operator with ED25519 key SHA256:y0Lj8dIYTM6... - ``` - -## 2. Canonical op blob spec (documented) - -The signature covers **these exact bytes**; the operator CLI (also Go) must reproduce them -byte-for-byte. **Canonical form: JSON, keys sorted lexicographically at every level, no -insignificant whitespace, no trailing newline, UTF-8.** - -```json -{"expires_at":"","issued_at":"","key_id":"","nonce":"<128-bit hex>","op":"","params":{...},"target":{"guest_id":"","host_id":""}} -``` - -| field | meaning | -|---|---| -| `op` | the operation, e.g. `guest_destroy`, `storage_detach`, `restore_overwrite` | -| `target.host_id` / `target.guest_id` | the box + guest the op is bound to (anti-retarget) | -| `params` | op-specific arguments (themselves canonical-sorted) | -| `nonce` | unique per op (anti-replay); ≥128-bit random | -| `issued_at` / `expires_at` | validity window (short — minutes) | -| `key_id` | which operator key (for rotation / audit) | - -Exact test blob (236 bytes): `{"expires_at":"2026-06-09T00:00:00Z","issued_at":"2026-06-08T00:00:00Z","key_id":"felhom-op-1","nonce":"a1b2c3d4e5f60718293a4b5c6d7e8f90","op":"guest_destroy","params":{"purge":true},"target":{"guest_id":"9001","host_id":"demo-felhom"}}` - -> Note: the SSHSIG **namespace** (`felhom-op-v1`) is the cryptographic domain separator and is -> a **fixed constant in the verifier**, never caller-supplied — a signature minted for any -> other namespace must not verify (proven: "wrong namespace" rejected). - -## 3. Go SSHSIG verify — approach + implementation cost - -**It is not a one-call verify, but it is clean — no hand-rolled crypto.** The only manual work -is SSHSIG *framing*; all crypto and key-type dispatch is the library's. Steps: - -1. `pem.Decode` the armor → `block.Type == "SSH SIGNATURE"`, `block.Bytes` is the binary SSHSIG. - *(Go's `encoding/pem` parses the armor directly — no manual base64/line handling.)* -2. Strip the literal 6-byte `SSHSIG` magic preamble (it is **not** length-prefixed). -3. `ssh.Unmarshal` the rest into a struct `{Version uint32; PublicKey, Namespace, Reserved, - HashAlgo, Signature string}` — library does the SSH wire parsing. -4. `ssh.ParsePublicKey([]byte(PublicKey))` → an `ssh.PublicKey`. -5. Recompute the signed data per spec: `"SSHSIG" || string(namespace) || string(reserved) || - string(hash_algorithm) || string(H(message))`, where `H` is the **named** hash - (`sha256`/`sha512`) — built with one `ssh.Marshal`. -6. `ssh.Unmarshal([]byte(Signature))` into `ssh.Signature`, then **`pub.Verify(signed, &sig)`** — - which **dispatches on the key's own algorithm** (this is what makes it key-agnostic). - -**Cost verdict:** ~40 lines of framing in one file, zero crypto implemented by us. Well within -the agent's budget; **no reason to fall back** to a different primitive. - -## 4. Anti-replay / authorization layer (on top of signature validity) - -Enforced in `VerifySignedOp` *after* the signature check, each proven to reject **even with a -valid signature** (Step 3 output above): - -- **replay** — nonce already recorded in the window → reject; -- **expired / not-yet-valid** — `now ∉ [issued_at, expires_at]` → reject (both sides shown); -- **retargeted** — `target.host_id`/`guest_id` ≠ this box/guest → reject (both shown). - -(Order matters: signature → namespace → allow-list → crypto verify → target → time → nonce, so -a replayed *but otherwise valid* op is still caught, and an invalid sig never consumes a nonce.) - -## 5. Key-type-agnosticism — **TRUE DROP-IN** (no box change for FIDO2 later) - -No FIDO2 device was used (by choice). Instead the spike **emulated the authenticator exactly**: - -- Synthesized a well-formed `sk-ssh-ed25519@openssh.com` public key; `ssh.ParsePublicKey` parses - it and `ssh.MarshalAuthorizedKey` round-trips it. -- Constructed a real `SSHSIG` whose inner signature follows the sk scheme (per OpenSSH - `PROTOCOL.u2f`): `ed25519` over `sha256(application) || flags || counter || sha256(signed_data)`, - with the blob `string(format) string(ed25519_sig) byte(flags) uint32(counter)` — i.e. exactly - what a FIDO2 key emits. -- Ran it through the **unchanged `VerifySignedOp`** → **verified** (`op="guest_destroy"`). - -**Verdict: true drop-in.** `pub.Verify` for `sk-ssh-ed25519` is implemented in -`golang.org/x/crypto/ssh` **v0.52.0** (it reconstructs `appDigest‖flags‖counter‖dataDigest` and -`ed25519.Verify`s it). Introducing a hardware operator key later is a **no-op on the boxes** — -the agent's verify code is identical; only the operator's signer key (and the allowed-signers -set entry) changes. No sk-specific handler is needed. - -> Because verification dispatches on the key type embedded in the signature, the same path also -> accepts `ssh-ed25519`, `rsa-sha2-*`, `ecdsa-sha2-*`, etc. — algorithm choice is the operator's, -> not the agent's. - -## 6. Fallback (not taken) and its cost - -A fallback would be a **raw Ed25519 detached signature** (or `minisign`): trivially one -`ed25519.Verify` call, no SSHSIG framing. **Rejected** because it **loses the clean FIDO2 path** — -a raw-Ed25519 verifier cannot consume an `sk-ssh-ed25519` signature (which carries flags+counter -and a different signed-data construction), so the future hardware swap would require **changing -the verifier on every box**. SSHSIG buys exactly the key-type-agnosticism (§5) that a raw scheme -forfeits, at a one-file framing cost (§3). **No fallback is warranted.** - -## 7. Reference verifier (seed of the agent's verify code) - -Verified working on Go 1.24.4 / `x/crypto` v0.52.0. (Test harness omitted; this is the verify -core + SSHSIG framing + anti-replay/authz.) - -```go -const Namespace = "felhom-op-v1" // FIXED domain separator, never caller-supplied -const sshsigMagic = "SSHSIG" - -type Target struct{ HostID, GuestID string } -type OpBlob struct { - Op string `json:"op"` - Target Target `json:"target"` - Params json.RawMessage `json:"params"` - Nonce string `json:"nonce"` - IssuedAt time.Time `json:"issued_at"` - ExpiresAt time.Time `json:"expires_at"` - KeyID string `json:"key_id"` -} -// (Target needs json tags host_id/guest_id in the real struct.) - -type NonceStore interface{ SeenOrRecord(nonce string, exp time.Time) bool } - -type sshsigBlob struct { - Version uint32 - PublicKey, Namespace, Reserved, HashAlgo, Signature string -} - -func hashByName(n string) (hash.Hash, error) { - switch n { - case "sha256": return sha256.New(), nil - case "sha512": return sha512.New(), nil - } - return nil, fmt.Errorf("unsupported SSHSIG hash %q", n) -} - -func parseArmoredSSHSIG(armored []byte) (*sshsigBlob, error) { - block, _ := pem.Decode(armored) - if block == nil || block.Type != "SSH SIGNATURE" { - return nil, errors.New("not an SSH SIGNATURE armor") - } - if len(block.Bytes) < 6 || string(block.Bytes[:6]) != sshsigMagic { - return nil, errors.New("missing SSHSIG magic") - } - var sb sshsigBlob - if err := ssh.Unmarshal(block.Bytes[6:], &sb); err != nil { return nil, err } - if sb.Version != 1 { return nil, fmt.Errorf("bad version %d", sb.Version) } - return &sb, nil -} - -func signedData(sb *sshsigBlob, msg []byte) ([]byte, error) { - h, err := hashByName(sb.HashAlgo); if err != nil { return nil, err } - h.Write(msg); md := h.Sum(nil) - body := ssh.Marshal(struct{ Namespace, Reserved, HashAlgo string; Hash []byte }{ - sb.Namespace, sb.Reserved, sb.HashAlgo, md}) - return append([]byte(sshsigMagic), body...), nil -} - -// VerifySignedOp: key-type-agnostic signature verify + anti-replay/authorization. -// allowedSigners is the trusted operator set (one key now; a quorum set later). -func VerifySignedOp(blob, sigArmored []byte, allowedSigners []ssh.PublicKey, - thisHostID, thisGuestID string, seenNonces NonceStore) (string, error) { - - sb, err := parseArmoredSSHSIG(sigArmored) - if err != nil { return "", err } - if sb.Namespace != Namespace { - return "", fmt.Errorf("namespace mismatch: got %q want %q", sb.Namespace, Namespace) - } - pub, err := ssh.ParsePublicKey([]byte(sb.PublicKey)) - if err != nil { return "", err } - allowed := false - for _, a := range allowedSigners { - if bytes.Equal(a.Marshal(), pub.Marshal()) { allowed = true; break } - } - if !allowed { return "", errors.New("signer not in allowed set") } - - signed, err := signedData(sb, blob) - if err != nil { return "", err } - var inner ssh.Signature - if err := ssh.Unmarshal([]byte(sb.Signature), &inner); err != nil { return "", err } - if err := pub.Verify(signed, &inner); err != nil { // dispatches on key algorithm - return "", fmt.Errorf("signature invalid: %w", err) - } - - var op OpBlob - if err := json.Unmarshal(blob, &op); err != nil { return "", err } - if op.Target.HostID != thisHostID || op.Target.GuestID != thisGuestID { - return "", fmt.Errorf("target mismatch") - } - now := time.Now().UTC() - if now.Before(op.IssuedAt) { return "", errors.New("not yet valid") } - if now.After(op.ExpiresAt) { return "", errors.New("expired") } - if seenNonces.SeenOrRecord(op.Nonce, op.ExpiresAt) { - return "", fmt.Errorf("replay: nonce %s already seen", op.Nonce) - } - return op.Op, nil -} -``` - -## 8. Inputs to the design doc (`04-control-plane-authorization.md`) - -- **Primitive confirmed:** SSHSIG (`ssh-keygen -Y sign` / armored `BEGIN SSH SIGNATURE`), - verified in Go via `pem.Decode` + `ssh.Unmarshal` + `ssh.ParsePublicKey` + `pub.Verify`. Low - implementation cost; no crypto hand-rolled. -- **Hub cannot forge:** the operator private key never touches the hub; the hub only queues the - opaque armored blob (matches `03` §4). -- **Key-type-agnostic / hardware-ready:** software `ed25519` now, FIDO2 `sk-ssh-ed25519` later is - a **box no-op** (proven end-to-end). The verifier hardcodes neither key type nor algorithm. -- **`allowedSigners` is a set:** single signer today; **threshold/quorum is just set sizing** plus - an N-of-M policy on top (out of scope here). -- **Anti-replay/authz are mandatory and cheap:** namespace (fixed), allow-list, then crypto, - then target-binding, time-window, nonce — all enforced and tested. -- **Canonical blob (§2)** is the shared contract between the operator CLI and the agent verifier.