felhom.eu/documentation/runbooks/day0-install.md

# Runbook — Day-0 install: first-time deployment of a Felhom host

> **The complete, step-by-step first-time deployment guide** for putting Felhom on a customer's
> Proxmox box — hub-side onboarding, box prerequisites, the install run, post-install verification,
> first app deploy, and rollback/uninstall. Every command in this guide was executed verbatim during
> the clean-room drill (see the header line below); placeholders are marked `<LIKE-THIS>`.
>
> **Drill-validated: 2026-07-03** on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build
> server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed
> verbatim. Versions: script **v1.9.1**, agent **v0.62.0**, golden **v0.85.1** (controller updated
> to 0.98.3 via D.1b). Evidence: `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
>
> Scope: a **shared** Proxmox box (the colleague's-box model) — a box that already runs, or will run,
> non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.

## Who does what

| Actor | Where | Does |
|---|---|---|
| **Operator** | HQ (hub UI + Cloudflare) | Part A (customer onboarding), go/no-go, on call during install |
| **Operator (or trusted colleague with root SSH)** | the customer's Proxmox box | Parts B–E |
| **Customer** | — | confirms the Part B prerequisites, receives the dashboard URL |

One secret crosses to the box: the customer's **retrieval passphrase** (5 Hungarian words, generated
by the hub in Part A). Everything else the box needs, it fetches and verifies itself.

---

## Part A — hub-side customer onboarding (operator, at HQ)

### A.1 Create the Cloudflare tunnel (per customer domain)

The customer's apps are published through a Cloudflare Tunnel; the in-guest controller runs
`cloudflared` with a per-customer token. In the Cloudflare dashboard (Zero Trust → Networks →
Tunnels):

1. Create a tunnel named after the customer (e.g. `<customer-id>`). Connector type: Cloudflared.
2. Copy the **tunnel token** (the long base64 string from the `cloudflared service install <token>`
   command) — this goes into the hub customer form in A.2.
3. In the customer domain's DNS zone, the controller manages per-app hostnames itself via the
   tunnel; make sure the domain is on Cloudflare (nameservers active).

Also have ready (optional but recommended): a **Cloudflare API token** with Zone edit rights for the
customer's zone — the hub uses it for geo-restriction management.

### A.2 Create the customer in the hub

Hub UI (`https://hub.felhom.eu`, operator password) → **Customers → New**:

| Field | Value | Notes |
|---|---|---|
| Customer ID | e.g. `sajatfelhom` | letters, numbers, dots, hyphens only; **immutable** — it keys everything |
| Customer name | display name | |
| Domain | e.g. `sajatfelhom.hu` | the customer's app domain (on Cloudflare) |
| Email | customer's email | used for customer-tier notifications (Hungarian) |
| CF tunnel token | from A.1 | → `infrastructure.cf_tunnel_token` |
| CF API token | from A.1 (optional) | → `infrastructure.cf_api_token` (geo rules) |
| Git username | Gitea read account | → `git.username` — **required for Day-0** |
| Git token | Gitea read token | → `git.token` — **required for Day-0**: the install script and the controller fetch artifacts from Gitea with this credential; without it the install dies at step 5/8 |

On save the hub generates two credentials:

- **Retrieval passphrase** (5 Hungarian words) — the ONE secret the installer carries to the box.
  Dictate or hand it to whoever runs Part C. Treat it like a password.
- **Customer API key** — internal (baked into the generated controller.yaml); never handled manually.

### A.3 Verify the Day-0 artifact manifest

Hub UI → **Configuration → Day-0 artifacts**. The manifest must vouch an **agent version** and a
**golden version** (with their sha256s — auto-resolved from Gitea). The install script refuses to
install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here
first (dropdowns; the sha is fetched authoritatively).

### A.4 Check the global controller-version floor

Hub UI → **Configuration**. Know the current `min_controller_version` floor, but understand its
limit (drill-proven, OQ-2): a fresh box lands on the golden's baked-in controller (**v0.85.1**,
which predates the floor-honoring code), so **the floor does NOT pull a fresh install up** — the
one-time update in D.1b is the required step. Raising the floor still matters for keeping the box
current afterwards. The structural fix — rebuilding + re-vouching the golden with a ≥ 0.86.0
controller — is a recorded operator follow-up.

---

## Part B — box prerequisites (confirm BEFORE install day)

Collect these from the customer / the box ahead of time. Every line is checkable in two minutes; a
"no" on any of them turns install day into debugging day.

```text
[ ] Proxmox VE 9.x installed and reachable over SSH as root
    pveversion                      # want pve-manager/9.x
[ ] Single node or cluster?
    pvesh get /nodes --output-format json | python3 -c 'import json,sys; ns=json.load(sys.stdin); print(len(ns), [n["node"] for n in ns])'
    → more than one node: the install command MUST carry --node <name>
[ ] local-lvm free space ≥ 120 GiB
    lvs /dev/pve/data -o lv_size,data_percent
[ ] Free vmid chosen from the FULL guest list (never from the hub view)
    pct list; qm list               # pick a vmid not in either list; avoid 9999 and 990000-990009
[ ] Host cores + RAM known (to size the appliance cap)
    nproc; free -m
    → shared box: cap the guest so the other guests keep breathing (Part C uses --cores/--memory)
[ ] Outbound HTTPS reachability from the box:
    curl -fsSo /dev/null -w '%{http_code}\n' https://hub.felhom.eu            # want 302 (login redirect)
    curl -fsSo /dev/null -w '%{http_code}\n' https://gitea.dooplex.hu         # want 200
    curl -fsSo /dev/null -w '%{http_code}\n' https://felhom.eu                # want 200
[ ] python3 present (PVE 9 default: yes)
    command -v python3
[ ] The operator has the retrieval passphrase from Part A at hand
```

Notes:
- The box needs **no inbound** reachability — hub, Gitea and the Cloudflare tunnel are all
  outbound-only.
- vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
  Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
  `pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only until the
  Headscale/WireGuard work lands. Go-live standard = **local backups only** (that is what the
  Part C command configures); offsite DR is retrofitted later via `--rescope-acl` /
  `--acl-storages` without reinstalling.

---

## Part C — the install run (on the box, as root)

### C.1 Fetch the installer

```bash
curl -fsSO https://felhom.eu/scripts/felhom-host-install.sh
chmod +x felhom-host-install.sh
./felhom-host-install.sh -h | head -3      # sanity: must print v1.9.1 (or newer) — the version this guide was drilled against
```

### C.2 Preview (recommended)

`--dry-run` prints every mutating command without executing anything (it still contacts the hub
read-only and asks for the passphrase):

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm" \
    --dry-run
```

### C.3 The canonical shared-box install command

```bash
./felhom-host-install.sh --customer-id <CUSTOMER-ID> --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm"
```

Placeholders and flags — what and why:

| Flag | Value | Why |
|---|---|---|
| `--customer-id` | the Part A customer ID | keys every hub call |
| `--vmid` | the free vmid from Part B | ALWAYS pass it explicitly — deterministic, and recorded in the install state for a later `--uninstall` |
| `--cores 2 --memory 4096` | the appliance cap | protects the other guests on a shared box; size to roughly half the host if the box is small (cap ≤ host resources) |
| `--force-gitea-golden` | — | a fresh box has no local golden archive; fetch it from Gitea and verify its sha256 against the hub manifest (this is the normal customer path) |
| `--acl-storages "local local-lvm"` | — | grants the agent's token write access on exactly these storages. The default set includes `felhom-pbs` (offsite), which does not exist on a customer box — granting it is harmless but misleading; scope to what is real. |
| `--node <NAME>` | only on a **cluster** | the script refuses to auto-pick a node on a multi-node cluster |

When prompted, type the **retrieval passphrase** (no echo; never on the command line). Alternative
for remote/scripted runs: put it in a 0600 file and pass `--passphrase-file <path>`.

### C.4 What the script does (so you can follow the output)

Eight steps, all idempotent/resumable (`--resume` skips completed steps after a mid-run failure):

1. **pre-flight** — root/PVE/hub checks, cluster refusal without `--node`, LVM/RAM floors,
   passphrase validated read-only against the hub, vmid guard.
2. **Proxmox API token** — `felhom-agent@pve!agent` minted (privsep), the `felhom` pool created, and
   the pool-scoped 3-role ACL applied (blast-radius containment: the token can only touch
   Felhom-pool guests + the granted storages).
3. **volume grows** — computed from free space (Docker-data/user-data sizing).
4. **host enrollment** — `POST /host-enroll` (passphrase-authed, idempotent) mints/returns the
   per-host hub credential.
5. **agent install** — binary fetched from Gitea, **sha256-verified against the hub manifest**,
   installed with the non-root service user, sudoers (visudo-validated), guarded-mkfs wrapper,
   systemd unit.
6. **agent config + service** — config written 0600, read-only selftest must pass, daemon started.
7. **golden archive** — fetched from Gitea + sha256-verified (because `--force-gitea-golden`).
8. **provision** — the agent restores the golden into vmid `<VMID>` (into the `felhom` pool),
   grows volumes, renders `bootstrap.json`, sets onboot, starts the guest. Then a verify pass.

Expected duration: minutes; dominated by the two Gitea downloads (agent ~20 MB, golden ~1-2 GB) on
the customer's uplink.

If a step fails: read the error (they are specific), fix the cause, re-run the SAME command with
`--resume` appended.

---

## Part D — post-install verification (on the box + hub)

Run through ALL of these before calling it done.

### D.1 On the box

```bash
# 1. The script's own final verdict said: "Day-0 provision SUCCESS — vmid=… host_id=…"

# 2. Agent service up, running non-root
systemctl is-active felhom-agent            # active
systemctl show felhom-agent -p User --value # felhom-agent

# 3. Agent selftest — all capabilities, including the pool read (stale-lock reaper)
felhom-agent --config /etc/felhom-agent/agent.json --selftest 2>&1 | tail -25
#    want: no FAILs; the "pve:pool-read" line ok (agent >= v0.62.0)

# 4. Guest up + onboot
pct status <VMID>                           # status: running
pct config <VMID> | grep -E '^(onboot|rootfs|mp0|mp1)'

# 5. Controller container healthy in-guest
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Status}} {{.Image}}'

# 6. Dashboard answers in-guest (the controller sits BEHIND traefik — probe with the Host header;
#    the controller's own :8080 is internal to the Docker network and not directly reachable)
pct exec <VMID> -- curl -ksSo /dev/null -w '%{http_code}\n' \
    -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/          # want 200
```

### D.1b One-time controller update (REQUIRED — do not skip)

The golden bakes controller **v0.85.1**, which predates the v0.86.0 floor-honoring code — **a fresh
install does NOT auto-update, regardless of the hub floor**. Bring it current once; from then on the
floor mechanism manages it:

```bash
# 1. ONLY if the installed agent is OLDER than v0.63.0 (check: felhom-agent --version): restart it
#    first — pre-v0.63.0 agents 401 the update's swap call until restarted (fixed in v0.63.0 by
#    token reload-on-miss; which agent you got depends on what the hub Day-0 manifest vouched)
systemctl restart felhom-agent

# 2. In the dashboard: Beállítások → "Frissítés" (update to latest), or equivalently:
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/update

# 3. Wait ~1 min, confirm the new version is running (image pull happens first, then an agent swap):
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'
```

### D.2 In the hub UI

- **Hosts**: the new host reports (status OK, fresh last-report timestamp).
- The host's guest list shows `<VMID>` running.
- **Capabilities**: 0 degraded. On a reduced-storage box (no `felhom-pbs`) the offsite-backup
  related capability is expected degraded — that is the documented go-live standard, not a fault.
- The controller version shown for the guest matches what D.1b brought it to (the golden's baked
  0.85.1 does NOT self-converge — D.1b is the required manual step until the golden is rebuilt with
  a ≥ 0.86.0 controller).

### D.3 Customer-visible

- `https://felhom.<customer-domain>` loads the dashboard through the tunnel (cloudflared up
  in-guest: `pct exec <VMID> -- docker ps --filter name=cloudflared`).

### D.4 First app deploy (smoke)

In the dashboard, deploy one small catalog app end-to-end (pick something dependency-free). The app
reaches "running" and its URL answers. This proves the full customer-visible chain: catalog →
controller → Docker → tunnel → browser.

---

## Part E — rollback / uninstall

`--uninstall` cleanly reverts an install — no hub contact, no passphrase needed:

```bash
./felhom-host-install.sh --uninstall --vmid <VMID>
```

Guardrails (all verified in the drill):

- **Ours-check**: refuses to destroy a guest without the `/etc/felhom-bootstrap` mount (i.e. not
  Felhom-provisioned) unless `--force`.
- **Typed confirmation**: you must type the vmid back before anything is destroyed.
- **Shared-box safety**: if OTHER Felhom guests remain on the host, only the named guest is
  removed; the agent/token/ACL stay (guest-only mode). `--force` overrides.
- The `felhom` pool is deleted only if empty.
- `--vmid` is optional when the install state file recorded the provisioned vmid (a fresh
  v1.9.x install records it); passing it explicitly is still the recommended, deterministic form.

What it removes (v1.9.1, residue-diff-proven — a full-filesystem diff against the pre-install
baseline showed **zero Felhom-named leftovers**): the guest, the agent (unit/sudoers/binary/
state dir/config/service user), the shared-parent unit + wrapper + `/mnt/felhom-drives`, the
guarded-mkfs wrapper, the guest-hook snippet, the lan-resolver dnsmasq snippets, the pveum
roles/ACL/token/user, the pool (if empty), the install state file.

**Expected remnants** (documented, not residue):

- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
  removing it is a direct-DB operator action (recorded follow-up).
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.

---

## Part F — troubleshooting (from the drill + live history)

| Symptom | Cause | Fix |
|---|---|---|
| step 1 dies: "this is a N-node cluster" | multi-node cluster | re-run with `--node <name>` |
| step 5 dies: "hub artifact manifest has no agent version" | Day-0 manifest unset/incomplete | Part A.3 — set it in the operator UI |
| step 5 dies: "no git token in controller.yaml" | customer created without git credentials | Part A.2 — add `git.username`/`git.token`, regenerate config |
| step 1: passphrase REJECTED (401) | typo / wrong customer | re-check with the hub UI's printed curl command |
| step 8 fails: "CT <vmid> already exists" | vmid collision with a hub-invisible guest | pick from `pct list` + `qm list` (Part B); the agent destroys nothing on collision — re-run with a free vmid and `--resume` |
| controller container missing in-guest after provision (docker ps empty) | pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach | `pct reboot <VMID>` — the unit runs on the next boot (v1.9.1 does this reboot itself) |
| self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401) | agent < v0.63.0: the daemon's token index missed a token minted by the provisioner after it started (FIXED in agent v0.63.0 — reload-on-miss) | `systemctl restart felhom-agent`, then retry (D.1b step 1; unnecessary from agent v0.63.0) |
| `local-lvm free < 120 GiB` warning | small/shared box | proceed only if you sized the grows deliberately; otherwise free space first |
| cloudflared restarting in-guest | bad/missing tunnel token | re-check A.1/A.2 token, regenerate config, `docker restart cloudflared` in-guest |

(Extended during the drill with every real hiccup encountered.)

---

## Related

- vmid picking + reserved bands: `documentation/runbooks/provisioning.md`
- The Day-0 trust chain (why the box can verify what it fetches):
  `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`
- Pool-scoped ACL model: `documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`
- Secrets handling: `documentation/runbooks/secrets.md`