felhom.eu/documentation/runbooks/RUNBOOK-appliance-deployment.md

# RUNBOOK — Appliance deployment (Felhom-owned box, "bare" hardware)

<!--
  The path for hardware FELHOM owns and ships (e.g. N100-class mini-PCs). Honest status: today this is a
  TWO-STEP flow — (1) Proxmox VE from the official ISO, (2) one host-install command in appliance mode.
  A true zero-touch "one ISO, no OS needed" image is NOT BUILT YET — a design sketch is included at the
  end because it is the natural productization. Everything in Steps 0–6 is executable today.
-->

## 0. Prerequisites (operator, before touching the hardware)
1. **Hub customer exists** (Customers → create): customer-id, display name, domain, email. Note the
   **retrieval passphrase** (shown once at creation — this authenticates the day-0 artifact/config
   fetches; you'll type it at a no-echo prompt during install).
2. **Cloudflare**: tunnel token + API token on the customer's config (Infrastructure section) so the
   generated controller.yaml carries them — the dashboard goes public on the customer domain at first boot.
3. **Day-0 artifacts current** (Configuration page): agent + golden vouched at the intended versions,
   global floor set. (The install fetches ANONYMOUSLY and verifies the hub-vouched sha256 — no Gitea
   credential is needed on the box; registry pulls are likewise anonymous since controller v0.112.0.)
4. **Operator keys file** at hand (`/path/to/operator-keys` pubkeys) — appliance boxes should arm
   operator-signed agent self-update from day 0 (pass `--operator-pubkey-file`); this is what lets you
   ship agent updates later without touching the box.
5. If the box will run the PBS DR tier (recommended for appliances): plan the PBS storage entry — note
   the ACL gotcha in Step 3.

## 1. Proxmox VE install (the "no OS" step)
- Boot the official Proxmox VE ISO (USB), install with defaults: **ext4/LVM on the system disk** → this
  yields the `local` + `local-lvm` storages the Felhom defaults expect. Set the root password (it will be
  ROTATED by the installer to a vaulted recovery credential — don't invest in it), hostname, static IP or
  DHCP reservation.
- Post-install: ensure internet egress + DNS work (`ping felhom.eu`). No other preparation — do NOT
  pre-create users/storage; the installer owns that.

## 2. Preflight (always, before the real run)
Generate the command on the hub customer page (Setup Command → **APPLIANCE** mode) with
`--preflight-only` ticked, run it as root on the box:
```
curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
  && sudo bash felhom-host-install.sh --customer-id <ID> --mode appliance --preflight-only [--acl-storages "local local-lvm"]
```
- Appliance mode takes the whole box — no `--cores/--memory` caps (those are the BYO containment knobs).
- **ACL gotcha:** the default ACL storage list includes `felhom-pbs`; on a fresh box that storage doesn't
  exist yet → preflight fails by design. Either pass `--acl-storages "local local-lvm"` now and rescope
  later when PBS is added (`--rescope-acl`), or configure the PBS storage first.
- Preflight is READ-ONLY (installs nothing) and dies with the exact remedy on any finding.

## 3. Install
Same command without `--preflight-only`. The retrieval passphrase is asked at a no-echo prompt.
What it does (the full host-mutation list is printed and requires typed consent): creates the non-root
agent user + narrow sudoers; PVE user/token/roles/ACLs (write only on the named storages; VM privileges
only in the `/pool/felhom` pool) + the pool; **rotates root@pam to a strong generated recovery password
and vaults it to the hub** (never logged — retrievable from the customer page's Credentials card);
fetches + sha-verifies the agent and the golden; provisions the LXC guest (default vmid 9201) from the
golden; installs the guest hook; the in-guest controller pulls its own controller.yaml. `--dry-run`
prints every mutating command without executing, if you want the paper trail first.

## 4. Verify (all remote, hub-side)
Within one report cycle: the host appears under Hosts/Customers with the guest; controller version = the
floor; Cloudflare tunnel Healthy; the customer dashboard serves on the domain. The Credentials card holds
the vaulted root recovery password. Agent capabilities: expect 56/56 (a `wg-handshake-read` degradation
is normal until/unless the WG offsite tier is configured).

## 5. Post-install tiers (each its own flow, in order)
1. **Offsite (restic → Storage Box):** enable on the customer config (shared/dedicated + quota) — the
   controller self-applies (verify-pin → consume-once → key install) → `pending`.
2. **Escrow ceremony** → see RUNBOOK-escrow-ceremony (auto-confirm flips `pending → escrowed`; offsite
   runs start).
3. **PBS DR tier (optional but recommended on appliances):** PBS storage entry over the WG tunnel
   (hub-assigned peer, agent-reconciled `wg-felhom` unit) — see documentation/runbooks/offsite-endpoint.md;
   then `--rescope-acl` to include the PBS storage.
4. Drive enrollment / NAS shares per the customer's data plan.

## 6. Rollback
`sudo bash felhom-host-install.sh --uninstall --customer-id <ID>` — destroys the Felhom guest, removes
the agent + config (+ `.bak` siblings), optional `--remove-golden`. The box returns to a plain PVE.

## Roadmap sketch — the true zero-touch ISO (NOT BUILT)
The productized appliance image = the official PVE ISO repacked with `proxmox-auto-install-assistant`:
an `answer.toml` (disk layout, network via DHCP, a throwaway root password) + a **first-boot systemd
unit** that runs the host-install in appliance mode with a per-box provisioning token (the customer-id +
retrieval passphrase must NOT be baked into a generic image — the unit should fetch its assignment from
the hub using a one-time enrollment code printed on the box/label, which is a small hub feature).
Deliverables when we build it: the repack script in `felhom.eu/scripts/`, the first-boot unit, the hub
one-time enrollment-code endpoint, and a burn-in checklist. Until then: Steps 1–3 above are ~20 minutes
of operator time per box, which is acceptable at pilot scale.