added runbooks

documentation/runbooks/RUNBOOK-appliance-deployment.md

@@ -0,0 +1,84 @@
+# RUNBOOK — Appliance deployment (Felhom-owned box, "bare" hardware)
+
+<!--
+  The path for hardware FELHOM owns and ships (e.g. N100-class mini-PCs). Honest status: today this is a
+  TWO-STEP flow — (1) Proxmox VE from the official ISO, (2) one host-install command in appliance mode.
+  A true zero-touch "one ISO, no OS needed" image is NOT BUILT YET — a design sketch is included at the
+  end because it is the natural productization. Everything in Steps 0–6 is executable today.
+-->
+
+## 0. Prerequisites (operator, before touching the hardware)
+1. **Hub customer exists** (Customers → create): customer-id, display name, domain, email. Note the
+   **retrieval passphrase** (shown once at creation — this authenticates the day-0 artifact/config
+   fetches; you'll type it at a no-echo prompt during install).
+2. **Cloudflare**: tunnel token + API token on the customer's config (Infrastructure section) so the
+   generated controller.yaml carries them — the dashboard goes public on the customer domain at first boot.
+3. **Day-0 artifacts current** (Configuration page): agent + golden vouched at the intended versions,
+   global floor set. (The install fetches ANONYMOUSLY and verifies the hub-vouched sha256 — no Gitea
+   credential is needed on the box; registry pulls are likewise anonymous since controller v0.112.0.)
+4. **Operator keys file** at hand (`/path/to/operator-keys` pubkeys) — appliance boxes should arm
+   operator-signed agent self-update from day 0 (pass `--operator-pubkey-file`); this is what lets you
+   ship agent updates later without touching the box.
+5. If the box will run the PBS DR tier (recommended for appliances): plan the PBS storage entry — note
+   the ACL gotcha in Step 3.
+
+## 1. Proxmox VE install (the "no OS" step)
+- Boot the official Proxmox VE ISO (USB), install with defaults: **ext4/LVM on the system disk** → this
+  yields the `local` + `local-lvm` storages the Felhom defaults expect. Set the root password (it will be
+  ROTATED by the installer to a vaulted recovery credential — don't invest in it), hostname, static IP or
+  DHCP reservation.
+- Post-install: ensure internet egress + DNS work (`ping felhom.eu`). No other preparation — do NOT
+  pre-create users/storage; the installer owns that.
+
+## 2. Preflight (always, before the real run)
+Generate the command on the hub customer page (Setup Command → **APPLIANCE** mode) with
+`--preflight-only` ticked, run it as root on the box:
+```
+curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
+  && sudo bash felhom-host-install.sh --customer-id <ID> --mode appliance --preflight-only [--acl-storages "local local-lvm"]
+```
+- Appliance mode takes the whole box — no `--cores/--memory` caps (those are the BYO containment knobs).
+- **ACL gotcha:** the default ACL storage list includes `felhom-pbs`; on a fresh box that storage doesn't
+  exist yet → preflight fails by design. Either pass `--acl-storages "local local-lvm"` now and rescope
+  later when PBS is added (`--rescope-acl`), or configure the PBS storage first.
+- Preflight is READ-ONLY (installs nothing) and dies with the exact remedy on any finding.
+
+## 3. Install
+Same command without `--preflight-only`. The retrieval passphrase is asked at a no-echo prompt.
+What it does (the full host-mutation list is printed and requires typed consent): creates the non-root
+agent user + narrow sudoers; PVE user/token/roles/ACLs (write only on the named storages; VM privileges
+only in the `/pool/felhom` pool) + the pool; **rotates root@pam to a strong generated recovery password
+and vaults it to the hub** (never logged — retrievable from the customer page's Credentials card);
+fetches + sha-verifies the agent and the golden; provisions the LXC guest (default vmid 9201) from the
+golden; installs the guest hook; the in-guest controller pulls its own controller.yaml. `--dry-run`
+prints every mutating command without executing, if you want the paper trail first.
+
+## 4. Verify (all remote, hub-side)
+Within one report cycle: the host appears under Hosts/Customers with the guest; controller version = the
+floor; Cloudflare tunnel Healthy; the customer dashboard serves on the domain. The Credentials card holds
+the vaulted root recovery password. Agent capabilities: expect 56/56 (a `wg-handshake-read` degradation
+is normal until/unless the WG offsite tier is configured).
+
+## 5. Post-install tiers (each its own flow, in order)
+1. **Offsite (restic → Storage Box):** enable on the customer config (shared/dedicated + quota) — the
+   controller self-applies (verify-pin → consume-once → key install) → `pending`.
+2. **Escrow ceremony** → see RUNBOOK-escrow-ceremony (auto-confirm flips `pending → escrowed`; offsite
+   runs start).
+3. **PBS DR tier (optional but recommended on appliances):** PBS storage entry over the WG tunnel
+   (hub-assigned peer, agent-reconciled `wg-felhom` unit) — see documentation/runbooks/offsite-endpoint.md;
+   then `--rescope-acl` to include the PBS storage.
+4. Drive enrollment / NAS shares per the customer's data plan.
+
+## 6. Rollback
+`sudo bash felhom-host-install.sh --uninstall --customer-id <ID>` — destroys the Felhom guest, removes
+the agent + config (+ `.bak` siblings), optional `--remove-golden`. The box returns to a plain PVE.
+
+## Roadmap sketch — the true zero-touch ISO (NOT BUILT)
+The productized appliance image = the official PVE ISO repacked with `proxmox-auto-install-assistant`:
+an `answer.toml` (disk layout, network via DHCP, a throwaway root password) + a **first-boot systemd
+unit** that runs the host-install in appliance mode with a per-box provisioning token (the customer-id +
+retrieval passphrase must NOT be baked into a generic image — the unit should fetch its assignment from
+the hub using a one-time enrollment code printed on the box/label, which is a small hub feature).
+Deliverables when we build it: the repack script in `felhom.eu/scripts/`, the first-boot unit, the hub
+one-time enrollment-code endpoint, and a burn-in checklist. Until then: Steps 1–3 above are ~20 minutes
+of operator time per box, which is acceptable at pilot scale.

documentation/runbooks/RUNBOOK-byo-deployment.md

@@ -0,0 +1,89 @@
+# RUNBOOK — BYO deployment (a host Felhom does NOT own)
+
+<!--
+  The path for a customer's own Proxmox host. Distilled from the first live BYO onboarding (Peti,
+  2026-07-10: 2-node HA cluster, 26 existing guests) — every gotcha below was hit or verified live.
+  The defining property of BYO: STRICT CONTAINMENT — capped guest resources, write access only on
+  storages the owner names, VM privileges only inside the felhom pool, full reversibility.
+-->
+
+## 0. Prerequisites
+1. **Hub customer** created; **retrieval passphrase** delivered to the box owner over a secure channel
+   (typed at a no-echo prompt — never on the command line).
+2. Cloudflare tunnel + API tokens on the customer config (the dashboard publishes on their domain).
+3. (Optional, recommended) **Pre-enable offsite** on the customer config — the descriptor + one-time
+   password wait harmlessly, and day-0 wires offsite hands-off, answering the customer-egress
+   reachability question in the first report cycles.
+4. Owner-side facts to collect up front (one command for them: `pvesm status`):
+   - **storage names** for `--acl-storages` (which storages may Felhom WRITE: minimum the archive storage
+     — `local` — and the guest-restore storage);
+   - is there an **LVM-thin** (`local-lvm`)? The guest restore target defaults to `local-lvm` and there is
+     currently NO flag to change it — a ZFS-only box needs the `--restore-storage` installer follow-up
+     first (open item);
+   - **cluster?** → node name (the installer REFUSES multi-node without an explicit `--node` — the
+     wrong-node footgun guard, verified live);
+   - agreed guest slice (cores/MiB — REQUIRED in byo mode) and vmid (default 9201; check `pct list`+
+     `qm list` for collisions — the owner may have 100+ guests).
+
+## 1. The trust conversation (what the owner will ask — verified answers)
+- **"Nothing changes on my system?"** Additive-only, and provable: (a) `--dry-run` prints every mutating
+  command with zero execution; (b) the real install prints the full host-mutation list and requires typed
+  consent; (c) write containment: `Datastore.Allocate/AllocateSpace` ONLY on the storages they name
+  (verifiable: `pveum user permissions felhom-agent@pve --path /storage/<any>`), read-only Audit
+  elsewhere (the monitor needs to SEE all storages — by design); (d) VM privileges exist only at
+  `/pool/felhom` — their existing guests are permission-unreachable; (e) no network/bridge/sysctl/HA/
+  storage.cfg changes; (f) `--uninstall` reverts fully.
+- **Root password:** the installer ROTATES root@pam to a generated recovery credential vaulted to the hub
+  — tell the owner BEFORE the install, not after they're locked out of the PVE UI.
+- The agent runs non-root behind a fixed-argument sudoers allowlist; disk formatting exists only for
+  drives the owner explicitly enrolls (type-to-confirm; data-bearing devices refused without an
+  operator signature).
+
+## 2. Preflight (read-only — installs nothing, not even the agent)
+```
+curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
+  && sudo bash felhom-host-install.sh --customer-id <ID> --mode byo \
+     --cores <N> --memory <MiB> --node <node> --acl-storages "local <restore-storage> [...]" --preflight-only
+```
+Reads PVE version, storages, RAM, guest count; DIES with the exact remedy on findings (missing ACL
+storage names, foreign DNS resolver on :53 — refuse-over-mutate). Multi-storage syntax: ONE quoted,
+space-separated argument.
+
+## 3. Install
+Same command minus `--preflight-only`; passphrase at the prompt; typed consent after the mutation list.
+If it fails mid-way: fix the finding and re-run with **`--resume`** (idempotent; keep the full corrected
+argument list, e.g. the widened `--acl-storages`). Permission 403s at the restore step = the ACL list is
+missing the storage named in the error → `--rescope-acl --acl-storages "<corrected list>"`, then `--resume`
+(live-verified recovery path).
+
+## 4. Cluster rules (verified live — hand these to the owner in writing)
+- The Felhom guest is **node-local by design**: agent, guest hook (agent-installed under the node's
+  `local:snippets`), and physical/NAS drive binds all live on the install node.
+- **Never add the guest to HA resources; never migrate it.** Four node-locality layers break on a foreign
+  node: binds (empty), agent (unreachable → controller shows the WRONG node's storage data), guest hook
+  (missing → the guest fail-closed refuses to start — a feature), and the controller's fixed
+  agent-endpoint. Everything self-heals on migrating back.
+- The cluster itself is untouched: pveum entries replicate via pmxcfs (inert), corosync/HA/quorum are
+  never modified.
+
+## 5. Verify (remote, hub-side — no SSH exists to a BYO box, by design)
+Host + guest appear on the hub within a cycle; controller at the floor version (updates are floor-driven
+and, since v0.112.0, credential-free — the Git Sync fields are ONLY for private catalogs); tunnel
+Healthy; dashboard live on the customer domain. If offsite was pre-enabled: report shows
+`offsite: pending` — proof the bridge consumed the one-time password and installed its key from the
+CUSTOMER's egress. Then → RUNBOOK-escrow-ceremony (note the K prerequisite there: a BYO box without the
+PBS tier needs agent ≥ v0.80.0 identity-only mode).
+
+## 6. Data storage for apps (BYO owners rarely hand over disks)
+Preferred: **NAS share (Hálózati tárhely)** — NFS or SMB, mounted host-side under `/mnt/felhom-drives/`,
+role-gated bulk-userdata, automount. NFS recipe (live-verified): a DEDICATED export (never reuse their
+backup share — `all_squash` would break its other clients), options
+`rw,no_subtree_check,insecure,all_squash,anonuid=101000,anongid=101000`, directory owned `101000:101000`
+(container uid 1000 + the unprivileged-LXC +100000 offset — the agent source documents this as the
+export's job). **NFSv4 path = pseudo-root-relative** (`/export/foo` on the server → enter `/foo`).
+UI uid field: the app's container uid (1000). Physical drive enrollment stays available if they dedicate
+an empty disk.
+
+## 7. Rollback
+`--uninstall` (guest destroyed, agent + config + .bak purged, optional `--remove-golden`) — the promise
+from Step 1, kept.

documentation/runbooks/RUNBOOK-escrow-ceremony.md

@@ -0,0 +1,73 @@
+# RUNBOOK — Escrow ceremony (customer recovery-code R)
+
+<!--
+  The zero-knowledge custody ceremony: seals the customer's unrecoverable secrets (PBS key K where
+  present; the IdentityBundle — restic offsite password, tunnel/PBS tokens, WG key) under a fresh
+  customer-held recovery code R. The hub stores only the opaque blob + a non-reversible password hash;
+  neither Felhom nor a hub compromise can open it. Cross-validated live: fork-4 validation (blob →
+  recover → decrypted the real repo) and the offsite arc (auto-confirm on hash match).
+-->
+
+## When a ceremony runs
+1. **Onboarding** (with the PBS tier): seals K + the identity secrets.
+2. **Offsite enable** (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any
+   offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds
+   `pending` until the hub-verified auto-confirm.
+3. **Any new unrecoverable secret / suspected R exposure**: re-ceremony (R cannot be retro-fitted —
+   it is never retained).
+
+## Prerequisites (check BEFORE scheduling with the customer)
+- **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob
+  auto-confirm can never match). **No-PBS hosts** (BYO without the PBS tier): ≥ **v0.80.0**
+  (identity-only mode; below that the ceremony hard-requires the PBS key and refuses).
+- **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present
+  (`cfg.Backup.PBSEncKeyPath(<id>)`).
+- **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged
+  password file present on the agent (`escrow-stage/restic_repo_password` — pushed automatically at
+  enable; re-saving the offsite config re-pushes it if missing).
+- The customer is present/reachable: R is displayed ONCE and must land in their hands.
+
+## The ceremony (ONE command, on the host, as the agent's operator context)
+```bash
+felhom-agent --selftest=escrow-create --upload
+```
+- `--storage <pbs-storage-id>` only if `escrow.pbs_storage_id` isn't configured. No-PBS hosts (agent
+  ≥0.80.0): omit — identity-only engages automatically.
+- What it does, in order: generates a fresh **R** (EFF-wordlist passphrase; entropy printed) → seals K
+  (if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic
+  password auto-injected; the live WG key auto-captured if present) → **self-verifies by recovering its
+  own blob** ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob +
+  `restic_pw_sha256` to the hub → **wipes the staged secret**.
+- Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the
+  command is safe to run over a screen-share EXCEPT the R moment.
+
+## R handling — the rules to say out loud to the customer
+1. R is displayed **once**; the system keeps no copy anywhere. Write it down / password manager /
+   safe — NOT on the server it protects.
+2. **A new ceremony's R SUPERSEDES the old one.** Replace the stored code; the old blob remains valid
+   only for pre-existing history.
+3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data
+   while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is
+   the zero-knowledge trade, stated honestly.
+
+## After the ceremony — hands off (auto-confirm)
+Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches
+`sha256(local repo password)` and flips `pending → escrowed` on its own, wiping the agent-staged copy.
+**Verify remotely:** the customer page / report shows `escrow_state: "escrowed"` with zero manual clicks;
+the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob
+does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it.
+(The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)
+
+## Recovery (the other half — operator-assisted, sketch)
+Total loss → the customer's R + the hub blob:
+`felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch>` with R → recovers the
+bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues
+SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject
+the recovered password on the fresh controller (`POST /backup/offbox/inject-password`, honored by the
+secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered
+password opened the production repo).
+
+## Do NOT
+- Run a ceremony unattended or pipe its output to a log (R would land in it).
+- Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
+- Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).

documentation/runbooks/TASK-identity-only-escrow.md

@@ -0,0 +1,52 @@
+# TASK — agent: identity-only escrow ceremony (K-optional) — unblocks no-PBS customers
+
+<!--
+  CLASS: Implementation (agent only, small). BLOCKER it removes: escrow.Create hard-requires the PBS key K
+  (escrow.go:64 "KeyPath ... is required") and runSelftestEscrowCreate dies without a PBS storage id + key
+  file — so a customer WITHOUT the PBS tier (Peti: BYO, PBS deliberately skipped) cannot run the ceremony
+  at all, which blocks fork-4 custody + auto-confirm for exactly the customer class most likely to skip
+  PBS. Fix: an identity-only mode — wrap the IdentityBundle (restic password, tunnel/pbs tokens, WG key)
+  under a fresh R with age, self-verify, upload; skip every K step. WRITTEN BY: project-Claude. BY: CC.
+-->
+
+## 0. Scope & baseline
+felhom-agent v0.79.0 (re-confirm live head) → **v0.80.0**. No controller/hub change: the upload wire
+shape already carries `restic_pw_sha256` + the blob; auto-confirm matches the hash — K is irrelevant to it.
+
+## 1. Design
+- `escrow.CreateOptions.KeyPath == ""` becomes VALID **iff** `IdentityBundle != nil` → identity-only mode:
+  skip `KeyFingerprint`, skip the K `Wrap`, skip paperkey; generate R exactly as today; `WrapIdentityBundle`
+  under R; **self-verify by unwrapping the identity blob** (the "an escrow you haven't recovered isn't an
+  escrow" rule applies to the identity path too — today only K is self-verified; identity-only mode MUST
+  self-verify the identity blob); upload with `KeyFingerprint` empty/omitted.
+- `runSelftestEscrowCreate`: when no `-storage` AND `escrow.pbs_storage_id` empty → do NOT die; log
+  "no PBS storage — identity-only escrow" and proceed iff there is anything to escrow (staged restic
+  password and/or bundle file and/or WG key); if truly nothing to wrap → clear error ("nothing to escrow").
+- K present (demo & PBS customers) → behavior byte-identical to today (K + identity, both self-verified).
+- Hub: verify `SaveHostEscrow`/upload handler tolerate an empty key fingerprint (read the handler; if it
+  rejects empty, a one-line relaxation on the hub side joins this task — re-confirm at source, do not assume).
+
+## 2. R semantics (unchanged, restate in the doc)
+Fresh R per ceremony, displayed once, never retained; supersedes the customer's previous R; the old blob
+stays valid for old history only.
+
+## 3. Tests (non-hollow + red-proofs)
+- Identity-only: no KeyPath + bundle with `restic_repo_password` → blob created, **self-verify unwrap
+  byte-exact**, upload payload has the blob + `restic_pw_sha256`, no K ops attempted (fake records).
+  **Red-proof:** restore the KeyPath-required guard → identity-only test fails.
+- Self-verify red-proof: corrupt the blob before verify → ceremony FAILS, nothing uploaded.
+- Nothing-to-escrow: no K, no staged pw, no bundle, no WG key → clear refuse.
+- K path regression: with KeyPath → identical call sequence to v0.79.0 (golden assertions).
+
+## 4. Deploy / live
+Build v0.80.0 → felhom-pve (demo agent) → healthy, 56/56 caps. Publish 0.80.0 to Gitea + bump the hub
+Day-0 agent manifest (the publish-train pattern; the operator-sign step is Viktor's 🛑 as per GL-1).
+**Peti's agent update path:** the agent self-update is operator-signed + pinned — confirm from the go-live
+record how a BYO agent updates (self-update channel armed at his install? operator pubkey file was NOT
+passed on his install form) — if his box cannot self-update the agent, REPORT must say so and the ceremony
+waits for the next Peti-touch window (he runs one update command). Do not improvise a new update path.
+
+## NOT to do
+Do not weaken the K path or its self-verify; do not skip the identity self-verify; do not change the R
+generation/display; do not touch controller/hub auto-confirm logic; do not run a ceremony on any real box
+from this task (unit + demo-agent deploy only).