admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Go-live package: Day-0 install runbook + clean-room drill + host-install v1.9.1

Browse Source 
- documentation/runbooks/day0-install.md (NEW): operator Day-0 guide, every
  command drill-executed verbatim (Parts A-F incl. the mandatory one-time
  controller update D.1b and the OQ-3 prereq checklist).
- documentation/audits/DRILL-day0-cleanroom-2026-07-03.md (NEW): nested-PVE
  clean-room drill evidence D0-D6, residue tables, OQ-1/2/3 answers, findings
  ledger (B1-B5), go/no-go for Peti (GO).
- scripts/felhom-host-install.sh v1.9.1: residue-free uninstall (agent config,
  shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq
  snippets; zero-residue diff proven) + post-provision guest reboot (R6) +
  header/version sync.
- CONTEXT.md, REPORT.md, scripts/CHANGELOG.md updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-03 15:14:47 +02:00
parent 996526273a
commit 0fa79cd90d
6 changed files with 690 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/runbooks/day0-install.md
+328
View File
@@ -0,0 +1,328 @@
# Runbook — Day-0 install: first-time deployment of a Felhom host
> **The complete, step-by-step first-time deployment guide** for putting Felhom on a customer's
> Proxmox box — hub-side onboarding, box prerequisites, the install run, post-install verification,
> first app deploy, and rollback/uninstall. Every command in this guide was executed verbatim during
> the clean-room drill (see the header line below); placeholders are marked `<LIKE-THIS>`.
>
> **Drill-validated: 2026-07-03** on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build
> server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed
> verbatim. Versions: script **v1.9.1**, agent **v0.62.0**, golden **v0.85.1** (controller updated
> to 0.98.3 via D.1b). Evidence: `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
>
> Scope: a **shared** Proxmox box (the colleague's-box model) — a box that already runs, or will run,
> non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.
## Who does what
| Actor | Where | Does |
|---|---|---|
| **Operator** | HQ (hub UI + Cloudflare) | Part A (customer onboarding), go/no-go, on call during install |
| **Operator (or trusted colleague with root SSH)** | the customer's Proxmox box | Parts B–E |
| **Customer** | — | confirms the Part B prerequisites, receives the dashboard URL |
One secret crosses to the box: the customer's **retrieval passphrase** (5 Hungarian words, generated
by the hub in Part A). Everything else the box needs, it fetches and verifies itself.
---
## Part A — hub-side customer onboarding (operator, at HQ)
### A.1 Create the Cloudflare tunnel (per customer domain)
The customer's apps are published through a Cloudflare Tunnel; the in-guest controller runs
`cloudflared` with a per-customer token. In the Cloudflare dashboard (Zero Trust → Networks →
Tunnels):
1. Create a tunnel named after the customer (e.g. `<customer-id>`). Connector type: Cloudflared.
2. Copy the **tunnel token** (the long base64 string from the `cloudflared service install <token>`
command) — this goes into the hub customer form in A.2.
3. In the customer domain's DNS zone, the controller manages per-app hostnames itself via the
tunnel; make sure the domain is on Cloudflare (nameservers active).
Also have ready (optional but recommended): a **Cloudflare API token** with Zone edit rights for the
customer's zone — the hub uses it for geo-restriction management.
### A.2 Create the customer in the hub
Hub UI (`https://hub.felhom.eu`, operator password) → **Customers → New**:
| Field | Value | Notes |
|---|---|---|
| Customer ID | e.g. `sajatfelhom` | letters, numbers, dots, hyphens only; **immutable** — it keys everything |
| Customer name | display name | |
| Domain | e.g. `sajatfelhom.hu` | the customer's app domain (on Cloudflare) |
| Email | customer's email | used for customer-tier notifications (Hungarian) |
| CF tunnel token | from A.1 | → `infrastructure.cf_tunnel_token` |
| CF API token | from A.1 (optional) | → `infrastructure.cf_api_token` (geo rules) |
| Git username | Gitea read account | → `git.username` — **required for Day-0** |
| Git token | Gitea read token | → `git.token` — **required for Day-0**: the install script and the controller fetch artifacts from Gitea with this credential; without it the install dies at step 5/8 |
On save the hub generates two credentials:
- **Retrieval passphrase** (5 Hungarian words) — the ONE secret the installer carries to the box.
Dictate or hand it to whoever runs Part C. Treat it like a password.
- **Customer API key** — internal (baked into the generated controller.yaml); never handled manually.
### A.3 Verify the Day-0 artifact manifest
Hub UI → **Configuration → Day-0 artifacts**. The manifest must vouch an **agent version** and a
**golden version** (with their sha256s — auto-resolved from Gitea). The install script refuses to
install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here
first (dropdowns; the sha is fetched authoritatively).
### A.4 Check the global controller-version floor
Hub UI → **Configuration**. Know the current `min_controller_version` floor, but understand its
limit (drill-proven, OQ-2): a fresh box lands on the golden's baked-in controller (**v0.85.1**,
which predates the floor-honoring code), so **the floor does NOT pull a fresh install up** — the
one-time update in D.1b is the required step. Raising the floor still matters for keeping the box
current afterwards. The structural fix — rebuilding + re-vouching the golden with a ≥ 0.86.0
controller — is a recorded operator follow-up.
---
## Part B — box prerequisites (confirm BEFORE install day)
Collect these from the customer / the box ahead of time. Every line is checkable in two minutes; a
"no" on any of them turns install day into debugging day.
```text
[ ] Proxmox VE 9.x installed and reachable over SSH as root
pveversion # want pve-manager/9.x
[ ] Single node or cluster?
pvesh get /nodes --output-format json | python3 -c 'import json,sys; ns=json.load(sys.stdin); print(len(ns), [n["node"] for n in ns])'
→ more than one node: the install command MUST carry --node <name>
[ ] local-lvm free space ≥ 120 GiB
lvs /dev/pve/data -o lv_size,data_percent
[ ] Free vmid chosen from the FULL guest list (never from the hub view)
pct list; qm list # pick a vmid not in either list; avoid 9999 and 990000-990009
[ ] Host cores + RAM known (to size the appliance cap)
nproc; free -m
→ shared box: cap the guest so the other guests keep breathing (Part C uses --cores/--memory)
[ ] Outbound HTTPS reachability from the box:
curl -fsSo /dev/null -w '%{http_code}\n' https://hub.felhom.eu # want 302 (login redirect)
curl -fsSo /dev/null -w '%{http_code}\n' https://gitea.dooplex.hu # want 200
curl -fsSo /dev/null -w '%{http_code}\n' https://felhom.eu # want 200
[ ] python3 present (PVE 9 default: yes)
command -v python3
[ ] The operator has the retrieval passphrase from Part A at hand
```
Notes:
- The box needs **no inbound** reachability — hub, Gitea and the Cloudflare tunnel are all
outbound-only.
- vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
`pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only until the
Headscale/WireGuard work lands. Go-live standard = **local backups only** (that is what the
Part C command configures); offsite DR is retrofitted later via `--rescope-acl` /
`--acl-storages` without reinstalling.
---
## Part C — the install run (on the box, as root)
### C.1 Fetch the installer
```bash
curl -fsSO https://felhom.eu/scripts/felhom-host-install.sh
chmod +x felhom-host-install.sh
./felhom-host-install.sh -h | head -3 # sanity: must print v1.9.1 (or newer) — the version this guide was drilled against
```
### C.2 Preview (recommended)
`--dry-run` prints every mutating command without executing anything (it still contacts the hub
read-only and asks for the passphrase):
```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --vmid <VMID> \
--cores 2 --memory 4096 \
--force-gitea-golden --acl-storages "local local-lvm" \
--dry-run
```
### C.3 The canonical shared-box install command
```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --vmid <VMID> \
--cores 2 --memory 4096 \
--force-gitea-golden --acl-storages "local local-lvm"
```
Placeholders and flags — what and why:
| Flag | Value | Why |
|---|---|---|
| `--customer-id` | the Part A customer ID | keys every hub call |
| `--vmid` | the free vmid from Part B | ALWAYS pass it explicitly — deterministic, and recorded in the install state for a later `--uninstall` |
| `--cores 2 --memory 4096` | the appliance cap | protects the other guests on a shared box; size to roughly half the host if the box is small (cap ≤ host resources) |
| `--force-gitea-golden` | — | a fresh box has no local golden archive; fetch it from Gitea and verify its sha256 against the hub manifest (this is the normal customer path) |
| `--acl-storages "local local-lvm"` | — | grants the agent's token write access on exactly these storages. The default set includes `felhom-pbs` (offsite), which does not exist on a customer box — granting it is harmless but misleading; scope to what is real. |
| `--node <NAME>` | only on a **cluster** | the script refuses to auto-pick a node on a multi-node cluster |
When prompted, type the **retrieval passphrase** (no echo; never on the command line). Alternative
for remote/scripted runs: put it in a 0600 file and pass `--passphrase-file <path>`.
### C.4 What the script does (so you can follow the output)
Eight steps, all idempotent/resumable (`--resume` skips completed steps after a mid-run failure):
1. **pre-flight** — root/PVE/hub checks, cluster refusal without `--node`, LVM/RAM floors,
passphrase validated read-only against the hub, vmid guard.
2. **Proxmox API token** — `felhom-agent@pve!agent` minted (privsep), the `felhom` pool created, and
the pool-scoped 3-role ACL applied (blast-radius containment: the token can only touch
Felhom-pool guests + the granted storages).
3. **volume grows** — computed from free space (Docker-data/user-data sizing).
4. **host enrollment** — `POST /host-enroll` (passphrase-authed, idempotent) mints/returns the
per-host hub credential.
5. **agent install** — binary fetched from Gitea, **sha256-verified against the hub manifest**,
installed with the non-root service user, sudoers (visudo-validated), guarded-mkfs wrapper,
systemd unit.
6. **agent config + service** — config written 0600, read-only selftest must pass, daemon started.
7. **golden archive** — fetched from Gitea + sha256-verified (because `--force-gitea-golden`).
8. **provision** — the agent restores the golden into vmid `<VMID>` (into the `felhom` pool),
grows volumes, renders `bootstrap.json`, sets onboot, starts the guest. Then a verify pass.
Expected duration: minutes; dominated by the two Gitea downloads (agent ~20 MB, golden ~1-2 GB) on
the customer's uplink.
If a step fails: read the error (they are specific), fix the cause, re-run the SAME command with
`--resume` appended.
---
## Part D — post-install verification (on the box + hub)
Run through ALL of these before calling it done.
### D.1 On the box
```bash
# 1. The script's own final verdict said: "Day-0 provision SUCCESS — vmid=… host_id=…"
# 2. Agent service up, running non-root
systemctl is-active felhom-agent # active
systemctl show felhom-agent -p User --value # felhom-agent
# 3. Agent selftest — all capabilities, including the pool read (stale-lock reaper)
felhom-agent --config /etc/felhom-agent/agent.json --selftest 2>&1 | tail -25
# want: no FAILs; the "pve:pool-read" line ok (agent >= v0.62.0)
# 4. Guest up + onboot
pct status <VMID> # status: running
pct config <VMID> | grep -E '^(onboot|rootfs|mp0|mp1)'
# 5. Controller container healthy in-guest
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Status}} {{.Image}}'
# 6. Dashboard answers in-guest (the controller sits BEHIND traefik — probe with the Host header;
# the controller's own :8080 is internal to the Docker network and not directly reachable)
pct exec <VMID> -- curl -ksSo /dev/null -w '%{http_code}\n' \
-H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/ # want 200
```
### D.1b One-time controller update (REQUIRED — do not skip)
The golden bakes controller **v0.85.1**, which predates the v0.86.0 floor-honoring code — **a fresh
install does NOT auto-update, regardless of the hub floor**. Bring it current once; from then on the
floor mechanism manages it:
```bash
# 1. Restart the agent first — a fresh install's local API 401s the swap until the agent restarts
# (known open issue; see troubleshooting)
systemctl restart felhom-agent
# 2. In the dashboard: Beállítások → "Frissítés" (update to latest), or equivalently:
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/update
# 3. Wait ~1 min, confirm the new version is running (image pull happens first, then an agent swap):
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'
```
### D.2 In the hub UI
- **Hosts**: the new host reports (status OK, fresh last-report timestamp).
- The host's guest list shows `<VMID>` running.
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
related capability is expected degraded — that is the documented go-live standard, not a fault.
- The controller version shown for the guest matches what D.1b brought it to (the golden's baked
0.85.1 does NOT self-converge — D.1b is the required manual step until the golden is rebuilt with
a ≥ 0.86.0 controller).
### D.3 Customer-visible
- `https://felhom.<customer-domain>` loads the dashboard through the tunnel (cloudflared up
in-guest: `pct exec <VMID> -- docker ps --filter name=cloudflared`).
### D.4 First app deploy (smoke)
In the dashboard, deploy one small catalog app end-to-end (pick something dependency-free). The app
reaches "running" and its URL answers. This proves the full customer-visible chain: catalog →
controller → Docker → tunnel → browser.
---
## Part E — rollback / uninstall
`--uninstall` cleanly reverts an install — no hub contact, no passphrase needed:
```bash
./felhom-host-install.sh --uninstall --vmid <VMID>
```
Guardrails (all verified in the drill):
- **Ours-check**: refuses to destroy a guest without the `/etc/felhom-bootstrap` mount (i.e. not
Felhom-provisioned) unless `--force`.
- **Typed confirmation**: you must type the vmid back before anything is destroyed.
- **Shared-box safety**: if OTHER Felhom guests remain on the host, only the named guest is
removed; the agent/token/ACL stay (guest-only mode). `--force` overrides.
- The `felhom` pool is deleted only if empty.
- `--vmid` is optional when the install state file recorded the provisioned vmid (a fresh
v1.9.x install records it); passing it explicitly is still the recommended, deterministic form.
What it removes (v1.9.1, residue-diff-proven — a full-filesystem diff against the pre-install
baseline showed **zero Felhom-named leftovers**): the guest, the agent (unit/sudoers/binary/
state dir/config/service user), the shared-parent unit + wrapper + `/mnt/felhom-drives`, the
guarded-mkfs wrapper, the guest-hook snippet, the lan-resolver dnsmasq snippets, the pveum
roles/ACL/token/user, the pool (if empty), the install state file.
**Expected remnants** (documented, not residue):
- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
removing it is a direct-DB operator action (recorded follow-up).
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.
---
## Part F — troubleshooting (from the drill + live history)
| Symptom | Cause | Fix |
|---|---|---|
| step 1 dies: "this is a N-node cluster" | multi-node cluster | re-run with `--node <name>` |
| step 5 dies: "hub artifact manifest has no agent version" | Day-0 manifest unset/incomplete | Part A.3 — set it in the operator UI |
| step 5 dies: "no git token in controller.yaml" | customer created without git credentials | Part A.2 — add `git.username`/`git.token`, regenerate config |
| step 1: passphrase REJECTED (401) | typo / wrong customer | re-check with the hub UI's printed curl command |
| step 8 fails: "CT <vmid> already exists" | vmid collision with a hub-invisible guest | pick from `pct list` + `qm list` (Part B); the agent destroys nothing on collision — re-run with a free vmid and `--resume` |
| controller container missing in-guest after provision (docker ps empty) | pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach | `pct reboot <VMID>` — the unit runs on the next boot (v1.9.1 does this reboot itself) |
| self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401) | fresh-install local-API stale token map — known open issue | `systemctl restart felhom-agent`, then retry (this is why D.1b restarts the agent first) |
| `local-lvm free < 120 GiB` warning | small/shared box | proceed only if you sized the grows deliberately; otherwise free space first |
| cloudflared restarting in-guest | bad/missing tunnel token | re-check A.1/A.2 token, regenerate config, `docker restart cloudflared` in-guest |
(Extended during the drill with every real hiccup encountered.)
---
## Related
- vmid picking + reserved bands: `documentation/runbooks/provisioning.md`
- The Day-0 trust chain (why the box can verify what it fetches):
`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`
- Pool-scoped ACL model: `documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`
- Secrets handling: `documentation/runbooks/secrets.md`