diff --git a/documentation/runbooks/RUNBOOK-appliance-deployment.md b/documentation/runbooks/RUNBOOK-appliance-deployment.md new file mode 100644 index 0000000..0c628da --- /dev/null +++ b/documentation/runbooks/RUNBOOK-appliance-deployment.md @@ -0,0 +1,84 @@ +# RUNBOOK — Appliance deployment (Felhom-owned box, "bare" hardware) + + + +## 0. Prerequisites (operator, before touching the hardware) +1. **Hub customer exists** (Customers → create): customer-id, display name, domain, email. Note the + **retrieval passphrase** (shown once at creation — this authenticates the day-0 artifact/config + fetches; you'll type it at a no-echo prompt during install). +2. **Cloudflare**: tunnel token + API token on the customer's config (Infrastructure section) so the + generated controller.yaml carries them — the dashboard goes public on the customer domain at first boot. +3. **Day-0 artifacts current** (Configuration page): agent + golden vouched at the intended versions, + global floor set. (The install fetches ANONYMOUSLY and verifies the hub-vouched sha256 — no Gitea + credential is needed on the box; registry pulls are likewise anonymous since controller v0.112.0.) +4. **Operator keys file** at hand (`/path/to/operator-keys` pubkeys) — appliance boxes should arm + operator-signed agent self-update from day 0 (pass `--operator-pubkey-file`); this is what lets you + ship agent updates later without touching the box. +5. If the box will run the PBS DR tier (recommended for appliances): plan the PBS storage entry — note + the ACL gotcha in Step 3. + +## 1. Proxmox VE install (the "no OS" step) +- Boot the official Proxmox VE ISO (USB), install with defaults: **ext4/LVM on the system disk** → this + yields the `local` + `local-lvm` storages the Felhom defaults expect. Set the root password (it will be + ROTATED by the installer to a vaulted recovery credential — don't invest in it), hostname, static IP or + DHCP reservation. +- Post-install: ensure internet egress + DNS work (`ping felhom.eu`). No other preparation — do NOT + pre-create users/storage; the installer owns that. + +## 2. Preflight (always, before the real run) +Generate the command on the hub customer page (Setup Command → **APPLIANCE** mode) with +`--preflight-only` ticked, run it as root on the box: +``` +curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \ + && sudo bash felhom-host-install.sh --customer-id --mode appliance --preflight-only [--acl-storages "local local-lvm"] +``` +- Appliance mode takes the whole box — no `--cores/--memory` caps (those are the BYO containment knobs). +- **ACL gotcha:** the default ACL storage list includes `felhom-pbs`; on a fresh box that storage doesn't + exist yet → preflight fails by design. Either pass `--acl-storages "local local-lvm"` now and rescope + later when PBS is added (`--rescope-acl`), or configure the PBS storage first. +- Preflight is READ-ONLY (installs nothing) and dies with the exact remedy on any finding. + +## 3. Install +Same command without `--preflight-only`. The retrieval passphrase is asked at a no-echo prompt. +What it does (the full host-mutation list is printed and requires typed consent): creates the non-root +agent user + narrow sudoers; PVE user/token/roles/ACLs (write only on the named storages; VM privileges +only in the `/pool/felhom` pool) + the pool; **rotates root@pam to a strong generated recovery password +and vaults it to the hub** (never logged — retrievable from the customer page's Credentials card); +fetches + sha-verifies the agent and the golden; provisions the LXC guest (default vmid 9201) from the +golden; installs the guest hook; the in-guest controller pulls its own controller.yaml. `--dry-run` +prints every mutating command without executing, if you want the paper trail first. + +## 4. Verify (all remote, hub-side) +Within one report cycle: the host appears under Hosts/Customers with the guest; controller version = the +floor; Cloudflare tunnel Healthy; the customer dashboard serves on the domain. The Credentials card holds +the vaulted root recovery password. Agent capabilities: expect 56/56 (a `wg-handshake-read` degradation +is normal until/unless the WG offsite tier is configured). + +## 5. Post-install tiers (each its own flow, in order) +1. **Offsite (restic → Storage Box):** enable on the customer config (shared/dedicated + quota) — the + controller self-applies (verify-pin → consume-once → key install) → `pending`. +2. **Escrow ceremony** → see RUNBOOK-escrow-ceremony (auto-confirm flips `pending → escrowed`; offsite + runs start). +3. **PBS DR tier (optional but recommended on appliances):** PBS storage entry over the WG tunnel + (hub-assigned peer, agent-reconciled `wg-felhom` unit) — see documentation/runbooks/offsite-endpoint.md; + then `--rescope-acl` to include the PBS storage. +4. Drive enrollment / NAS shares per the customer's data plan. + +## 6. Rollback +`sudo bash felhom-host-install.sh --uninstall --customer-id ` — destroys the Felhom guest, removes +the agent + config (+ `.bak` siblings), optional `--remove-golden`. The box returns to a plain PVE. + +## Roadmap sketch — the true zero-touch ISO (NOT BUILT) +The productized appliance image = the official PVE ISO repacked with `proxmox-auto-install-assistant`: +an `answer.toml` (disk layout, network via DHCP, a throwaway root password) + a **first-boot systemd +unit** that runs the host-install in appliance mode with a per-box provisioning token (the customer-id + +retrieval passphrase must NOT be baked into a generic image — the unit should fetch its assignment from +the hub using a one-time enrollment code printed on the box/label, which is a small hub feature). +Deliverables when we build it: the repack script in `felhom.eu/scripts/`, the first-boot unit, the hub +one-time enrollment-code endpoint, and a burn-in checklist. Until then: Steps 1–3 above are ~20 minutes +of operator time per box, which is acceptable at pilot scale. diff --git a/documentation/runbooks/RUNBOOK-byo-deployment.md b/documentation/runbooks/RUNBOOK-byo-deployment.md new file mode 100644 index 0000000..f923e59 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-byo-deployment.md @@ -0,0 +1,89 @@ +# RUNBOOK — BYO deployment (a host Felhom does NOT own) + + + +## 0. Prerequisites +1. **Hub customer** created; **retrieval passphrase** delivered to the box owner over a secure channel + (typed at a no-echo prompt — never on the command line). +2. Cloudflare tunnel + API tokens on the customer config (the dashboard publishes on their domain). +3. (Optional, recommended) **Pre-enable offsite** on the customer config — the descriptor + one-time + password wait harmlessly, and day-0 wires offsite hands-off, answering the customer-egress + reachability question in the first report cycles. +4. Owner-side facts to collect up front (one command for them: `pvesm status`): + - **storage names** for `--acl-storages` (which storages may Felhom WRITE: minimum the archive storage + — `local` — and the guest-restore storage); + - is there an **LVM-thin** (`local-lvm`)? The guest restore target defaults to `local-lvm` and there is + currently NO flag to change it — a ZFS-only box needs the `--restore-storage` installer follow-up + first (open item); + - **cluster?** → node name (the installer REFUSES multi-node without an explicit `--node` — the + wrong-node footgun guard, verified live); + - agreed guest slice (cores/MiB — REQUIRED in byo mode) and vmid (default 9201; check `pct list`+ + `qm list` for collisions — the owner may have 100+ guests). + +## 1. The trust conversation (what the owner will ask — verified answers) +- **"Nothing changes on my system?"** Additive-only, and provable: (a) `--dry-run` prints every mutating + command with zero execution; (b) the real install prints the full host-mutation list and requires typed + consent; (c) write containment: `Datastore.Allocate/AllocateSpace` ONLY on the storages they name + (verifiable: `pveum user permissions felhom-agent@pve --path /storage/`), read-only Audit + elsewhere (the monitor needs to SEE all storages — by design); (d) VM privileges exist only at + `/pool/felhom` — their existing guests are permission-unreachable; (e) no network/bridge/sysctl/HA/ + storage.cfg changes; (f) `--uninstall` reverts fully. +- **Root password:** the installer ROTATES root@pam to a generated recovery credential vaulted to the hub + — tell the owner BEFORE the install, not after they're locked out of the PVE UI. +- The agent runs non-root behind a fixed-argument sudoers allowlist; disk formatting exists only for + drives the owner explicitly enrolls (type-to-confirm; data-bearing devices refused without an + operator signature). + +## 2. Preflight (read-only — installs nothing, not even the agent) +``` +curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \ + && sudo bash felhom-host-install.sh --customer-id --mode byo \ + --cores --memory --node --acl-storages "local [...]" --preflight-only +``` +Reads PVE version, storages, RAM, guest count; DIES with the exact remedy on findings (missing ACL +storage names, foreign DNS resolver on :53 — refuse-over-mutate). Multi-storage syntax: ONE quoted, +space-separated argument. + +## 3. Install +Same command minus `--preflight-only`; passphrase at the prompt; typed consent after the mutation list. +If it fails mid-way: fix the finding and re-run with **`--resume`** (idempotent; keep the full corrected +argument list, e.g. the widened `--acl-storages`). Permission 403s at the restore step = the ACL list is +missing the storage named in the error → `--rescope-acl --acl-storages ""`, then `--resume` +(live-verified recovery path). + +## 4. Cluster rules (verified live — hand these to the owner in writing) +- The Felhom guest is **node-local by design**: agent, guest hook (agent-installed under the node's + `local:snippets`), and physical/NAS drive binds all live on the install node. +- **Never add the guest to HA resources; never migrate it.** Four node-locality layers break on a foreign + node: binds (empty), agent (unreachable → controller shows the WRONG node's storage data), guest hook + (missing → the guest fail-closed refuses to start — a feature), and the controller's fixed + agent-endpoint. Everything self-heals on migrating back. +- The cluster itself is untouched: pveum entries replicate via pmxcfs (inert), corosync/HA/quorum are + never modified. + +## 5. Verify (remote, hub-side — no SSH exists to a BYO box, by design) +Host + guest appear on the hub within a cycle; controller at the floor version (updates are floor-driven +and, since v0.112.0, credential-free — the Git Sync fields are ONLY for private catalogs); tunnel +Healthy; dashboard live on the customer domain. If offsite was pre-enabled: report shows +`offsite: pending` — proof the bridge consumed the one-time password and installed its key from the +CUSTOMER's egress. Then → RUNBOOK-escrow-ceremony (note the K prerequisite there: a BYO box without the +PBS tier needs agent ≥ v0.80.0 identity-only mode). + +## 6. Data storage for apps (BYO owners rarely hand over disks) +Preferred: **NAS share (Hálózati tárhely)** — NFS or SMB, mounted host-side under `/mnt/felhom-drives/`, +role-gated bulk-userdata, automount. NFS recipe (live-verified): a DEDICATED export (never reuse their +backup share — `all_squash` would break its other clients), options +`rw,no_subtree_check,insecure,all_squash,anonuid=101000,anongid=101000`, directory owned `101000:101000` +(container uid 1000 + the unprivileged-LXC +100000 offset — the agent source documents this as the +export's job). **NFSv4 path = pseudo-root-relative** (`/export/foo` on the server → enter `/foo`). +UI uid field: the app's container uid (1000). Physical drive enrollment stays available if they dedicate +an empty disk. + +## 7. Rollback +`--uninstall` (guest destroyed, agent + config + .bak purged, optional `--remove-golden`) — the promise +from Step 1, kept. diff --git a/documentation/runbooks/RUNBOOK-escrow-ceremony.md b/documentation/runbooks/RUNBOOK-escrow-ceremony.md new file mode 100644 index 0000000..8f6806b --- /dev/null +++ b/documentation/runbooks/RUNBOOK-escrow-ceremony.md @@ -0,0 +1,73 @@ +# RUNBOOK — Escrow ceremony (customer recovery-code R) + + + +## When a ceremony runs +1. **Onboarding** (with the PBS tier): seals K + the identity secrets. +2. **Offsite enable** (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any + offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds + `pending` until the hub-verified auto-confirm. +3. **Any new unrecoverable secret / suspected R exposure**: re-ceremony (R cannot be retro-fitted — + it is never retained). + +## Prerequisites (check BEFORE scheduling with the customer) +- **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob + auto-confirm can never match). **No-PBS hosts** (BYO without the PBS tier): ≥ **v0.80.0** + (identity-only mode; below that the ceremony hard-requires the PBS key and refuses). +- **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present + (`cfg.Backup.PBSEncKeyPath()`). +- **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged + password file present on the agent (`escrow-stage/restic_repo_password` — pushed automatically at + enable; re-saving the offsite config re-pushes it if missing). +- The customer is present/reachable: R is displayed ONCE and must land in their hands. + +## The ceremony (ONE command, on the host, as the agent's operator context) +```bash +felhom-agent --selftest=escrow-create --upload +``` +- `--storage ` only if `escrow.pbs_storage_id` isn't configured. No-PBS hosts (agent + ≥0.80.0): omit — identity-only engages automatically. +- What it does, in order: generates a fresh **R** (EFF-wordlist passphrase; entropy printed) → seals K + (if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic + password auto-injected; the live WG key auto-captured if present) → **self-verifies by recovering its + own blob** ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob + + `restic_pw_sha256` to the hub → **wipes the staged secret**. +- Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the + command is safe to run over a screen-share EXCEPT the R moment. + +## R handling — the rules to say out loud to the customer +1. R is displayed **once**; the system keeps no copy anywhere. Write it down / password manager / + safe — NOT on the server it protects. +2. **A new ceremony's R SUPERSEDES the old one.** Replace the stored code; the old blob remains valid + only for pre-existing history. +3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data + while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is + the zero-knowledge trade, stated honestly. + +## After the ceremony — hands off (auto-confirm) +Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches +`sha256(local repo password)` and flips `pending → escrowed` on its own, wiping the agent-staged copy. +**Verify remotely:** the customer page / report shows `escrow_state: "escrowed"` with zero manual clicks; +the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob +does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it. +(The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.) + +## Recovery (the other half — operator-assisted, sketch) +Total loss → the customer's R + the hub blob: +`felhom-agent --selftest=identity-consume --blob --keydest <0600 scratch>` with R → recovers the +bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues +SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject +the recovered password on the fresh controller (`POST /backup/offbox/inject-password`, honored by the +secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered +password opened the production repo). + +## Do NOT +- Run a ceremony unattended or pipe its output to a log (R would land in it). +- Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems. +- Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R). diff --git a/documentation/runbooks/TASK-identity-only-escrow.md b/documentation/runbooks/TASK-identity-only-escrow.md new file mode 100644 index 0000000..fe608f8 --- /dev/null +++ b/documentation/runbooks/TASK-identity-only-escrow.md @@ -0,0 +1,52 @@ +# TASK — agent: identity-only escrow ceremony (K-optional) — unblocks no-PBS customers + + + +## 0. Scope & baseline +felhom-agent v0.79.0 (re-confirm live head) → **v0.80.0**. No controller/hub change: the upload wire +shape already carries `restic_pw_sha256` + the blob; auto-confirm matches the hash — K is irrelevant to it. + +## 1. Design +- `escrow.CreateOptions.KeyPath == ""` becomes VALID **iff** `IdentityBundle != nil` → identity-only mode: + skip `KeyFingerprint`, skip the K `Wrap`, skip paperkey; generate R exactly as today; `WrapIdentityBundle` + under R; **self-verify by unwrapping the identity blob** (the "an escrow you haven't recovered isn't an + escrow" rule applies to the identity path too — today only K is self-verified; identity-only mode MUST + self-verify the identity blob); upload with `KeyFingerprint` empty/omitted. +- `runSelftestEscrowCreate`: when no `-storage` AND `escrow.pbs_storage_id` empty → do NOT die; log + "no PBS storage — identity-only escrow" and proceed iff there is anything to escrow (staged restic + password and/or bundle file and/or WG key); if truly nothing to wrap → clear error ("nothing to escrow"). +- K present (demo & PBS customers) → behavior byte-identical to today (K + identity, both self-verified). +- Hub: verify `SaveHostEscrow`/upload handler tolerate an empty key fingerprint (read the handler; if it + rejects empty, a one-line relaxation on the hub side joins this task — re-confirm at source, do not assume). + +## 2. R semantics (unchanged, restate in the doc) +Fresh R per ceremony, displayed once, never retained; supersedes the customer's previous R; the old blob +stays valid for old history only. + +## 3. Tests (non-hollow + red-proofs) +- Identity-only: no KeyPath + bundle with `restic_repo_password` → blob created, **self-verify unwrap + byte-exact**, upload payload has the blob + `restic_pw_sha256`, no K ops attempted (fake records). + **Red-proof:** restore the KeyPath-required guard → identity-only test fails. +- Self-verify red-proof: corrupt the blob before verify → ceremony FAILS, nothing uploaded. +- Nothing-to-escrow: no K, no staged pw, no bundle, no WG key → clear refuse. +- K path regression: with KeyPath → identical call sequence to v0.79.0 (golden assertions). + +## 4. Deploy / live +Build v0.80.0 → felhom-pve (demo agent) → healthy, 56/56 caps. Publish 0.80.0 to Gitea + bump the hub +Day-0 agent manifest (the publish-train pattern; the operator-sign step is Viktor's 🛑 as per GL-1). +**Peti's agent update path:** the agent self-update is operator-signed + pinned — confirm from the go-live +record how a BYO agent updates (self-update channel armed at his install? operator pubkey file was NOT +passed on his install form) — if his box cannot self-update the agent, REPORT must say so and the ceremony +waits for the next Peti-touch window (he runs one update command). Do not improvise a new update path. + +## NOT to do +Do not weaken the K path or its self-verify; do not skip the identity self-verify; do not change the R +generation/display; do not touch controller/hub auto-confirm logic; do not run a ceremony on any real box +from this task (unit + demo-agent deploy only).