felhom.eu/documentation/runbooks/RUNBOOK-appliance-deployment.md

RUNBOOK — Appliance deployment (Felhom-owned box, "bare" hardware)

0. Prerequisites (operator, before touching the hardware)

1. Hub customer exists (Customers → create): customer-id, display name, domain, email. Note the retrieval passphrase (shown once at creation — this authenticates the day-0 artifact/config fetches; you'll type it at a no-echo prompt during install).
2. Cloudflare: tunnel token + API token on the customer's config (Infrastructure section) so the generated controller.yaml carries them — the dashboard goes public on the customer domain at first boot.
3. Day-0 artifacts current (Configuration page): agent + golden vouched at the intended versions, global floor set. (The install fetches ANONYMOUSLY and verifies the hub-vouched sha256 — no Gitea credential is needed on the box; registry pulls are likewise anonymous since controller v0.112.0.)
4. Operator keys file at hand (/path/to/operator-keys pubkeys) — appliance boxes should arm operator-signed agent self-update from day 0 (pass --operator-pubkey-file); this is what lets you ship agent updates later without touching the box.
5. If the box will run the PBS DR tier (recommended for appliances): plan the PBS storage entry — note the ACL gotcha in Step 3.

1. Proxmox VE install (the "no OS" step)

• Boot the official Proxmox VE ISO (USB), install with defaults: ext4/LVM on the system disk → this yields the local + local-lvm storages the Felhom defaults expect. Set the root password (it will be ROTATED by the installer to a vaulted recovery credential — don't invest in it), hostname, static IP or DHCP reservation.
• Post-install: ensure internet egress + DNS work (ping felhom.eu). No other preparation — do NOT pre-create users/storage; the installer owns that.

2. Preflight (always, before the real run)

Generate the command on the hub customer page (Setup Command → APPLIANCE mode) with --preflight-only ticked, run it as root on the box:

curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
  && sudo bash felhom-host-install.sh --customer-id <ID> --mode appliance --preflight-only [--acl-storages "local local-lvm"]

• Appliance mode takes the whole box — no --cores/--memory caps (those are the BYO containment knobs).
• ACL gotcha: the default ACL storage list includes felhom-pbs; on a fresh box that storage doesn't exist yet → preflight fails by design. Either pass --acl-storages "local local-lvm" now and rescope later when PBS is added (--rescope-acl), or configure the PBS storage first.
• Preflight is READ-ONLY (installs nothing) and dies with the exact remedy on any finding.

3. Install

Same command without --preflight-only. The retrieval passphrase is asked at a no-echo prompt. What it does (the full host-mutation list is printed and requires typed consent): creates the non-root agent user + narrow sudoers; PVE user/token/roles/ACLs (write only on the named storages; VM privileges only in the /pool/felhom pool) + the pool; rotates root@pam to a strong generated recovery password and vaults it to the hub (never logged — retrievable from the customer page's Credentials card); fetches + sha-verifies the agent and the golden; provisions the LXC guest (default vmid 9201) from the golden; installs the guest hook; the in-guest controller pulls its own controller.yaml. --dry-run prints every mutating command without executing, if you want the paper trail first.

4. Verify (all remote, hub-side)

Within one report cycle: the host appears under Hosts/Customers with the guest; controller version = the floor; Cloudflare tunnel Healthy; the customer dashboard serves on the domain. The Credentials card holds the vaulted root recovery password. Agent capabilities: expect 56/56 (a wg-handshake-read degradation is normal until/unless the WG offsite tier is configured).

5. Post-install tiers (each its own flow, in order)

1. Offsite (restic → Storage Box): enable on the customer config (shared/dedicated + quota) — the controller self-applies (verify-pin → consume-once → key install) → pending.
2. Escrow ceremony → see RUNBOOK-escrow-ceremony (auto-confirm flips pending → escrowed; offsite runs start).
3. PBS DR tier (optional but recommended on appliances): PBS storage entry over the WG tunnel (hub-assigned peer, agent-reconciled wg-felhom unit) — see documentation/runbooks/offsite-endpoint.md; then --rescope-acl to include the PBS storage.
4. Drive enrollment / NAS shares per the customer's data plan.

6. Rollback

sudo bash felhom-host-install.sh --uninstall --customer-id <ID> — destroys the Felhom guest, removes the agent + config (+ .bak siblings), optional --remove-golden. The box returns to a plain PVE.

Roadmap sketch — the true zero-touch ISO (NOT BUILT)

The productized appliance image = the official PVE ISO repacked with proxmox-auto-install-assistant: an answer.toml (disk layout, network via DHCP, a throwaway root password) + a first-boot systemd unit that runs the host-install in appliance mode with a per-box provisioning token (the customer-id + retrieval passphrase must NOT be baked into a generic image — the unit should fetch its assignment from the hub using a one-time enrollment code printed on the box/label, which is a small hub feature). Deliverables when we build it: the repack script in felhom.eu/scripts/, the first-boot unit, the hub one-time enrollment-code endpoint, and a burn-in checklist. Until then: Steps 1–3 above are ~20 minutes of operator time per box, which is acceptable at pilot scale.