added runbooks
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
# RUNBOOK — Appliance deployment (Felhom-owned box, "bare" hardware)
|
||||
|
||||
<!--
|
||||
The path for hardware FELHOM owns and ships (e.g. N100-class mini-PCs). Honest status: today this is a
|
||||
TWO-STEP flow — (1) Proxmox VE from the official ISO, (2) one host-install command in appliance mode.
|
||||
A true zero-touch "one ISO, no OS needed" image is NOT BUILT YET — a design sketch is included at the
|
||||
end because it is the natural productization. Everything in Steps 0–6 is executable today.
|
||||
-->
|
||||
|
||||
## 0. Prerequisites (operator, before touching the hardware)
|
||||
1. **Hub customer exists** (Customers → create): customer-id, display name, domain, email. Note the
|
||||
**retrieval passphrase** (shown once at creation — this authenticates the day-0 artifact/config
|
||||
fetches; you'll type it at a no-echo prompt during install).
|
||||
2. **Cloudflare**: tunnel token + API token on the customer's config (Infrastructure section) so the
|
||||
generated controller.yaml carries them — the dashboard goes public on the customer domain at first boot.
|
||||
3. **Day-0 artifacts current** (Configuration page): agent + golden vouched at the intended versions,
|
||||
global floor set. (The install fetches ANONYMOUSLY and verifies the hub-vouched sha256 — no Gitea
|
||||
credential is needed on the box; registry pulls are likewise anonymous since controller v0.112.0.)
|
||||
4. **Operator keys file** at hand (`/path/to/operator-keys` pubkeys) — appliance boxes should arm
|
||||
operator-signed agent self-update from day 0 (pass `--operator-pubkey-file`); this is what lets you
|
||||
ship agent updates later without touching the box.
|
||||
5. If the box will run the PBS DR tier (recommended for appliances): plan the PBS storage entry — note
|
||||
the ACL gotcha in Step 3.
|
||||
|
||||
## 1. Proxmox VE install (the "no OS" step)
|
||||
- Boot the official Proxmox VE ISO (USB), install with defaults: **ext4/LVM on the system disk** → this
|
||||
yields the `local` + `local-lvm` storages the Felhom defaults expect. Set the root password (it will be
|
||||
ROTATED by the installer to a vaulted recovery credential — don't invest in it), hostname, static IP or
|
||||
DHCP reservation.
|
||||
- Post-install: ensure internet egress + DNS work (`ping felhom.eu`). No other preparation — do NOT
|
||||
pre-create users/storage; the installer owns that.
|
||||
|
||||
## 2. Preflight (always, before the real run)
|
||||
Generate the command on the hub customer page (Setup Command → **APPLIANCE** mode) with
|
||||
`--preflight-only` ticked, run it as root on the box:
|
||||
```
|
||||
curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
|
||||
&& sudo bash felhom-host-install.sh --customer-id <ID> --mode appliance --preflight-only [--acl-storages "local local-lvm"]
|
||||
```
|
||||
- Appliance mode takes the whole box — no `--cores/--memory` caps (those are the BYO containment knobs).
|
||||
- **ACL gotcha:** the default ACL storage list includes `felhom-pbs`; on a fresh box that storage doesn't
|
||||
exist yet → preflight fails by design. Either pass `--acl-storages "local local-lvm"` now and rescope
|
||||
later when PBS is added (`--rescope-acl`), or configure the PBS storage first.
|
||||
- Preflight is READ-ONLY (installs nothing) and dies with the exact remedy on any finding.
|
||||
|
||||
## 3. Install
|
||||
Same command without `--preflight-only`. The retrieval passphrase is asked at a no-echo prompt.
|
||||
What it does (the full host-mutation list is printed and requires typed consent): creates the non-root
|
||||
agent user + narrow sudoers; PVE user/token/roles/ACLs (write only on the named storages; VM privileges
|
||||
only in the `/pool/felhom` pool) + the pool; **rotates root@pam to a strong generated recovery password
|
||||
and vaults it to the hub** (never logged — retrievable from the customer page's Credentials card);
|
||||
fetches + sha-verifies the agent and the golden; provisions the LXC guest (default vmid 9201) from the
|
||||
golden; installs the guest hook; the in-guest controller pulls its own controller.yaml. `--dry-run`
|
||||
prints every mutating command without executing, if you want the paper trail first.
|
||||
|
||||
## 4. Verify (all remote, hub-side)
|
||||
Within one report cycle: the host appears under Hosts/Customers with the guest; controller version = the
|
||||
floor; Cloudflare tunnel Healthy; the customer dashboard serves on the domain. The Credentials card holds
|
||||
the vaulted root recovery password. Agent capabilities: expect 56/56 (a `wg-handshake-read` degradation
|
||||
is normal until/unless the WG offsite tier is configured).
|
||||
|
||||
## 5. Post-install tiers (each its own flow, in order)
|
||||
1. **Offsite (restic → Storage Box):** enable on the customer config (shared/dedicated + quota) — the
|
||||
controller self-applies (verify-pin → consume-once → key install) → `pending`.
|
||||
2. **Escrow ceremony** → see RUNBOOK-escrow-ceremony (auto-confirm flips `pending → escrowed`; offsite
|
||||
runs start).
|
||||
3. **PBS DR tier (optional but recommended on appliances):** PBS storage entry over the WG tunnel
|
||||
(hub-assigned peer, agent-reconciled `wg-felhom` unit) — see documentation/runbooks/offsite-endpoint.md;
|
||||
then `--rescope-acl` to include the PBS storage.
|
||||
4. Drive enrollment / NAS shares per the customer's data plan.
|
||||
|
||||
## 6. Rollback
|
||||
`sudo bash felhom-host-install.sh --uninstall --customer-id <ID>` — destroys the Felhom guest, removes
|
||||
the agent + config (+ `.bak` siblings), optional `--remove-golden`. The box returns to a plain PVE.
|
||||
|
||||
## Roadmap sketch — the true zero-touch ISO (NOT BUILT)
|
||||
The productized appliance image = the official PVE ISO repacked with `proxmox-auto-install-assistant`:
|
||||
an `answer.toml` (disk layout, network via DHCP, a throwaway root password) + a **first-boot systemd
|
||||
unit** that runs the host-install in appliance mode with a per-box provisioning token (the customer-id +
|
||||
retrieval passphrase must NOT be baked into a generic image — the unit should fetch its assignment from
|
||||
the hub using a one-time enrollment code printed on the box/label, which is a small hub feature).
|
||||
Deliverables when we build it: the repack script in `felhom.eu/scripts/`, the first-boot unit, the hub
|
||||
one-time enrollment-code endpoint, and a burn-in checklist. Until then: Steps 1–3 above are ~20 minutes
|
||||
of operator time per box, which is acceptable at pilot scale.
|
||||
Reference in New Issue
Block a user