felhom.eu/documentation/runbooks/day0-install.md

Runbook — Day-0 install: first-time deployment of a Felhom host

The complete, step-by-step first-time deployment guide for putting Felhom on a customer's Proxmox box — hub-side onboarding, box prerequisites, the install run, post-install verification, first app deploy, and rollback/uninstall. Every command in this guide was executed verbatim during the clean-room drill (see the header line below); placeholders are marked <LIKE-THIS>.

Drill-validated: 2026-07-03 on a clean-room nested PVE 9.2.2 VM (qemu/KVM on the build server) — install → verify → app deploy → uninstall → residue-diff → reinstall, all executed verbatim. Versions: script v1.9.1, agent v0.63.0, golden v0.98.3 (controller current on first boot — no manual update step). Evidence: documentation/audits/DRILL-day0-cleanroom-2026-07-03.md (install/uninstall path) + documentation/audits/DRILL-golden-098-2026-07-03.md (current-golden rebuild).

Scope: a shared Proxmox box (the colleague's-box model) — a box that already runs, or will run, non-Felhom guests. A dedicated box is the same procedure minus the sizing caution.

v1.10.0 (GL-2): every install now REQUIRES --mode appliance|byo — there is no default. Use appliance for a box Felhom owns/manages end-to-end (this guide's drilled path, unchanged apart from the flag); use byo for a host the operator does NOT own (the pilot's own PVE) — see §C.5. The byo profile is implemented + statically validated but not yet drill-validated (that is the supervised GL-6 run).

Who does what

Actor	Where	Does
Operator	HQ (hub UI + Cloudflare)	Part A (customer onboarding), go/no-go, on call during install
Operator (or trusted colleague with root SSH)	the customer's Proxmox box	Parts B–E
Customer	—	confirms the Part B prerequisites, receives the dashboard URL

One secret crosses to the box: the customer's retrieval passphrase (5 Hungarian words, generated by the hub in Part A). Everything else the box needs, it fetches and verifies itself.

---

Part A — hub-side customer onboarding (operator, at HQ)

A.1 Create the Cloudflare tunnel (per customer domain)

The customer's apps are published through a Cloudflare Tunnel; the in-guest controller runs cloudflared with a per-customer token. In the Cloudflare dashboard (Zero Trust → Networks → Tunnels):

1. Create a tunnel named after the customer (e.g. <customer-id>). Connector type: Cloudflared.
2. Copy the tunnel token (the long base64 string from the cloudflared service install <token> command) — this goes into the hub customer form in A.2.
3. In the customer domain's DNS zone, the controller manages per-app hostnames itself via the tunnel; make sure the domain is on Cloudflare (nameservers active).

Also have ready (optional but recommended): a Cloudflare API token with Zone edit rights for the customer's zone — the hub uses it for geo-restriction management.

A.2 Create the customer in the hub

Hub UI (https://hub.felhom.eu, operator password) → Customers → New:

Field	Value	Notes
Customer ID	e.g. sajatfelhom	letters, numbers, dots, hyphens only; immutable — it keys everything
Customer name	display name
Domain	e.g. sajatfelhom.hu	the customer's app domain (on Cloudflare)
Email	customer's email	used for customer-tier notifications (Hungarian)
CF tunnel token	from A.1	→ infrastructure.cf_tunnel_token
CF API token	from A.1 (optional)	→ infrastructure.cf_api_token (geo rules)
Git username	Gitea read account	→ git.username — required for Day-0
Git token	Gitea read token	→ git.token — required for Day-0: the install script and the controller fetch artifacts from Gitea with this credential; without it the install dies at step 5/8

On save the hub generates two credentials:

• Retrieval passphrase (5 Hungarian words) — the ONE secret the installer carries to the box. Dictate or hand it to whoever runs Part C. Treat it like a password.
• Customer API key — internal (baked into the generated controller.yaml); never handled manually.

A.3 Verify the Day-0 artifact manifest

Hub UI → Configuration → Day-0 artifacts. The manifest must vouch an agent version and a golden version (with their sha256s — auto-resolved from Gitea). The install script refuses to install anything the manifest doesn't vouch. If the vouched versions lag the fleet, bump them here first (dropdowns; the sha is fetched authoritatively). Current drilled-known-good: agent 0.63.0 + golden 0.98.3. Vouch a golden ≥ 0.98.3 — an older golden bakes a pre-floor controller and re-introduces the manual update step (Part F troubleshooting).

A.4 Check the global controller-version floor

Hub UI → Configuration. Know the current min_controller_version floor. With golden ≥ 0.98.3 vouched (the structural fix for the old OQ-2 finding — drill-proven 2026-07-03), a fresh box lands on a current, floor-honoring controller and self-manages from first boot; the floor's job is keeping it current afterwards. Recommended (1 minute, records drift-protection): raise the global floor to the current controller version whenever the golden is rebuilt.

---

Part B — box prerequisites (confirm BEFORE install day)

Collect these from the customer / the box ahead of time. Every line is checkable in two minutes; a "no" on any of them turns install day into debugging day.

[ ] Proxmox VE 9.x installed and reachable over SSH as root
    pveversion                      # want pve-manager/9.x
[ ] Single node or cluster?
    pvesh get /nodes --output-format json | python3 -c 'import json,sys; ns=json.load(sys.stdin); print(len(ns), [n["node"] for n in ns])'
    → more than one node: the install command MUST carry --node <name>
[ ] local-lvm free space ≥ 120 GiB
    lvs /dev/pve/data -o lv_size,data_percent
[ ] Free vmid chosen from the FULL guest list (never from the hub view)
    pct list; qm list               # pick a vmid not in either list; avoid 9999 and 990000-990009
[ ] Host cores + RAM known (to size the appliance cap)
    nproc; free -m
    → shared box: cap the guest so the other guests keep breathing (Part C uses --cores/--memory)
[ ] Outbound HTTPS reachability from the box:
    curl -fsSo /dev/null -w '%{http_code}\n' https://hub.felhom.eu            # want 302 (login redirect)
    curl -fsSo /dev/null -w '%{http_code}\n' https://gitea.dooplex.hu         # want 200
    curl -fsSo /dev/null -w '%{http_code}\n' https://felhom.eu                # want 200
[ ] python3 present (PVE 9 default: yes)
    command -v python3
[ ] The operator has the retrieval passphrase from Part A at hand

Notes:

• The box needs no inbound reachability — hub, Gitea and the Cloudflare tunnel are all outbound-only.
• vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows Felhom-pool guests — other guests on a shared box are invisible there. Always pick from pct list + qm list on the box (documentation/runbooks/provisioning.md).
• Offsite backup (felhom-pbs) is NOT part of a go-live today: the PBS host is LAN-only. The transport is DECIDED and DESIGNED — plain WireGuard, host-side agent-managed, box-dials-out to a public endpoint VM running the offsite PBS. Evidence: documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md; design-of-record + slice roadmap: documentation/architecture/06-offsite-connectivity.md (MVP = slices S1→S2→S3, then S4). Go-live standard = local backups only (that is what the Part C command configures); offsite DR is retrofitted later via --rescope-acl / --acl-storages without reinstalling.

---

Part C — the install run (on the box, as root)

C.1 Fetch the installer

curl -fsSO https://felhom.eu/scripts/felhom-host-install.sh
chmod +x felhom-host-install.sh
./felhom-host-install.sh -h | head -3      # sanity: must print v1.10.0 (or newer) — older scripts don't know --mode

C.2 Preview (recommended)

--dry-run prints every mutating command without executing anything (it still contacts the hub read-only and asks for the passphrase):

./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm" \
    --dry-run

C.3 The canonical shared-box install command

./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode appliance --vmid <VMID> \
    --cores 2 --memory 4096 \
    --force-gitea-golden --acl-storages "local local-lvm"

Placeholders and flags — what and why:

Flag	Value	Why
--customer-id	the Part A customer ID	keys every hub call
--mode	appliance or byo	REQUIRED, no default. appliance = a Felhom-owned/managed box (this section). byo = a host the operator does not own — use §C.5's command instead
--vmid	the free vmid from Part B	ALWAYS pass it explicitly — deterministic, and recorded in the install state for a later --uninstall
--cores 2 --memory 4096	the appliance cap	protects the other guests on a shared box; size to roughly half the host if the box is small (cap ≤ host resources)
--force-gitea-golden	—	a fresh box has no local golden archive; fetch it from Gitea and verify its sha256 against the hub manifest (this is the normal customer path)
--acl-storages "local local-lvm"	—	grants the agent's token write access on exactly these storages. The default set includes felhom-pbs (offsite), which does not exist on a customer box — granting it is harmless but misleading; scope to what is real.
--node <NAME>	only on a cluster	the script refuses to auto-pick a node on a multi-node cluster

When prompted, type the retrieval passphrase (no echo; never on the command line). Alternative for remote/scripted runs: put it in a 0600 file and pass --passphrase-file <path>.

C.4 What the script does (so you can follow the output)

Eight steps, all idempotent/resumable (--resume skips completed steps after a mid-run failure):

1. pre-flight — root/PVE/hub checks, cluster refusal without --node, LVM/RAM floors, passphrase validated read-only against the hub, vmid guard.
2. Proxmox API token — felhom-agent@pve!agent minted (privsep), the felhom pool created, and the pool-scoped 3-role ACL applied (blast-radius containment: the token can only touch Felhom-pool guests + the granted storages).
3. volume grows — computed from free space (Docker-data/user-data sizing).
4. host enrollment — POST /host-enroll (passphrase-authed, idempotent) mints/returns the per-host hub credential.
5. agent install — binary fetched from Gitea, sha256-verified against the hub manifest, installed with the non-root service user, sudoers (visudo-validated), guarded-mkfs wrapper, systemd unit.
6. agent config + service — config written 0600, read-only selftest must pass, daemon started.
7. golden archive — fetched from Gitea + sha256-verified (because --force-gitea-golden).
8. provision — the agent restores the golden into vmid <VMID> (into the felhom pool), grows volumes, renders bootstrap.json, sets onboot, starts the guest. Then a verify pass.

Expected duration: minutes; dominated by the two Gitea downloads (agent ~20 MB, golden ~1-2 GB) on the customer's uplink.

If a step fails: read the error (they are specific), fix the cause, re-run the SAME command with --resume appended (with the SAME --mode — the script refuses a mode flip on a half-done install).

C.5 BYO hosts (--mode byo) — installing on a Proxmox server the operator does NOT own

Trust model. On a BYO host the roles invert: the box owner is root and stays root; Felhom is the guest. The installer therefore (a) never touches the owner's credentials — the break-glass step (4b/8, root@pam reset + hub vault) is skipped entirely, nothing is vaulted; (b) treats the owner's workloads as the thing to protect — the CPU/RAM caps are mandatory, they are the only noisy-neighbor containment there; (c) refuses anything host-invasive that the appliance profile allows (--enable-oob, --rotate-recovery, unvalidated PVE majors, ACL grants on storages the box doesn't have); and (d) asserts the written agent config keeps lan_resolver/wg_tunnel/oob OFF — the agent must not take over the owner's DNS or open tunnels. Recovery on a BYO box = the owner's own console access, by design.

Recommended first run — preflight only (all step-1 checks incl. the byo gates, an explicit PASS/FAIL verdict, exit 0/1, writes nothing — re-runnable any number of times):

./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
    --force-gitea-golden --acl-storages "local local-lvm" \
    --preflight-only

The install command (same flags, minus --preflight-only):

./felhom-host-install.sh --customer-id <CUSTOMER-ID> --mode byo --vmid <VMID> \
    --cores <N> --memory <MiB> \
    --force-gitea-golden --acl-storages "local local-lvm"

What byo does differently (everything else matches C.4's eight steps):

• --cores AND --memory are required — size them WITH the owner (the guest must never starve the owner's guests). --acl-storages must name storages that actually exist on the box, or the preflight dies naming the missing ones.
• Preflight additionally prints what already listens on the host's :53 (informational — Felhom never configures DNS on a byo box) and the existing-guest count.
• At the end of preflight the script prints the complete host-mutation list (pveum roles/user/token/ACL/pool, agent user/binary/unit/sudoers/config/state, the watchdog + guarded wrappers, the guest + golden) and requires typing the host's short hostname to proceed — show this list to the box owner; it is the consent artifact.
• Step 4b/8 (break-glass) is skipped; the verify step asserts pool membership + the scoped ACL grants landed (both modes do this from v1.10.0).

C.5b Operator key pin — self-update authority (v1.11.0)

The agent's self-update only acts on operator-signed update ops; the verifying public keys are pinned in the agent config (authz.signers). A fresh install with no key pinned runs self-update DORMANT (the safe default — the verify step prints a WARN, not an error). To arm it at install time, either:

• fill the OPERATOR_KEY_* constants near the top of the script (one commit, after the offline key-generation ceremony), or
• pass --operator-pubkey-file <path> — one key per line, operational <authorized_keys line> or recovery <authorized_keys line>; the authorized_keys comment field is the key_id and is required. The file overrides the constants.

Rules: PUBLIC keys only (private keys never touch the box or this script); a reinstall with no keys resolved keeps an already-pinned config's signers (--preserve-from) — pinned boxes are never silently un-pinned; providing keys over a preserved config replaces them (pin rotation, logged). byo installs show the operational key_id in the disclosure block.

C.6 Post-hoc mode note for pre-v1.10.0 installs

Boxes installed by ≤ v1.9.1 have no recorded mode; their state file simply predates it. --resume on such a box accepts whichever --mode you pass — pass the mode that matches how the box is actually operated (existing Felhom-owned boxes: appliance).

---

Part D — post-install verification (on the box + hub)

Run through ALL of these before calling it done.

D.1 On the box

# 1. The script's own final verdict said: "Day-0 provision SUCCESS — vmid=… host_id=…"

# 2. Agent service up, running non-root
systemctl is-active felhom-agent            # active
systemctl show felhom-agent -p User --value # felhom-agent

# 3. Agent selftest — all capabilities, including the pool read (stale-lock reaper)
felhom-agent --config /etc/felhom-agent/agent.json --selftest 2>&1 | tail -25
#    want: no FAILs; the "pve:pool-read" line ok (agent >= v0.62.0)

# 4. Guest up + onboot
pct status <VMID>                           # status: running
pct config <VMID> | grep -E '^(onboot|rootfs|mp0|mp1)'

# 5. Controller container healthy in-guest
pct exec <VMID> -- docker ps --filter name=felhom-controller --format '{{.Status}} {{.Image}}'

# 6. Dashboard answers in-guest (the controller sits BEHIND traefik — probe with the Host header;
#    the controller's own :8080 is internal to the Docker network and not directly reachable)
pct exec <VMID> -- curl -ksSo /dev/null -w '%{http_code}\n' \
    -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/          # want 200

D.1b Controller version check (one line — the old manual update step is RETIRED)

The golden (≥ 0.98.3) bakes a current, floor-honoring controller — a fresh install is current on first boot and the hub floor manages it from here. Just verify:

# controller version current on first boot; the floor manages it from here (want: update_available false)
pct exec <VMID> -- curl -ksS -X POST -H "Host: felhom.<CUSTOMER-DOMAIN>" https://127.0.0.1/api/selfupdate/check

If this reports the baked version as BEHIND (a newer controller shipped after the golden was baked), the same endpoint's /api/selfupdate/update — or the dashboard's Beállítások → "Frissítés" button — converges it; that is the normal self-update path, not an install-day requirement. (Installed from a golden older than 0.86.0? See Part F troubleshooting.)

D.2 In the hub UI

• Hosts: the new host reports (status OK, fresh last-report timestamp).
• The host's guest list shows <VMID> running.
• Capabilities: 0 degraded. On a reduced-storage box (no felhom-pbs) the offsite-backup related capability is expected degraded — that is the documented go-live standard, not a fault.
• The controller version shown for the guest is the golden's baked (current) version — it matches what D.1b's check reported.

D.3 Customer-visible

• https://felhom.<customer-domain> loads the dashboard through the tunnel (cloudflared up in-guest: pct exec <VMID> -- docker ps --filter name=cloudflared).

D.4 First app deploy (smoke)

In the dashboard, deploy one small catalog app end-to-end (pick something dependency-free). The app reaches "running" and its URL answers. This proves the full customer-visible chain: catalog → controller → Docker → tunnel → browser.

---

Part E — rollback / uninstall

--uninstall cleanly reverts an install — no hub contact, no passphrase needed:

./felhom-host-install.sh --uninstall --vmid <VMID>

Guardrails (all verified in the drill):

• Ours-check: refuses to destroy a guest without the /etc/felhom-bootstrap mount (i.e. not Felhom-provisioned) unless --force.
• Typed confirmation: you must type the vmid back before anything is destroyed.
• Shared-box safety: if OTHER Felhom guests remain on the host, only the named guest is removed; the agent/token/ACL stay (guest-only mode). --force overrides.
• The felhom pool is deleted only if empty.
• --vmid is optional when the install state file recorded the provisioned vmid (a fresh v1.9.x install records it); passing it explicitly is still the recommended, deterministic form.

What it removes (v1.9.1, residue-diff-proven — a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers): the guest, the agent (unit/sudoers/binary/ state dir/config/service user), the shared-parent unit + wrapper + /mnt/felhom-drives, the guarded-mkfs wrapper, the guest-hook snippet, the lan-resolver dnsmasq snippets, the pveum roles/ACL/token/user, the pool (if empty), the install state file.

v1.11.0 additions: the teardown also removes the self-update artifacts (guarded wrapper, A/B slots, rollback unit, limits drop-in), unmounts enrolled drives under /mnt/felhom-drives/ (plain umount only — a busy drive gets a warning and guidance, never a forced unmount; the data is never touched and the drives are physically removable afterwards), and both modes end with an explicit KEPT-vs-WIPED statement — read it before pulling drives or closing the customer out (it lists what lives on: drive data, PBS backups, the hub record, the escrow blob, the vaulted recovery credential).

Expected remnants (documented, not residue):

• The hub host record (+ its report/guest history) — the hub currently has no host-delete; removing it is a direct-DB operator action (recorded follow-up).
• The sudo and dnsmasq packages (system packages installed as dependencies; inert).
• The golden vzdump on the archive storage — pass --remove-golden to delete it too.
• System noise: LVM archive metadata, dpkg lists, PVE's own rrd//var/lib/lxc scratch entries.

---

Part F — troubleshooting (from the drill + live history)

Symptom	Cause	Fix
dies immediately: "--mode is required"	v1.10.0+ has no default profile	add --mode appliance (Felhom-owned box) or --mode byo (owner's box, §C.5)
dies immediately: "byo mode requires explicit --cores and --memory"	byo caps are mandatory	size the caps with the box owner and pass both
dies: "install started as X; resume with --mode X"	--resume with the other mode	resume with the recorded mode, or --uninstall and start over
byo dies: "acl storage(s) not found on this box"	--acl-storages names a storage the box lacks	pass the box's real storages (check pvesm status)
step 1 dies: "this is a N-node cluster"	multi-node cluster	re-run with --node <name>
step 5 dies: "hub artifact manifest has no agent version"	Day-0 manifest unset/incomplete	Part A.3 — set it in the operator UI
step 5 dies: "no git token in controller.yaml"	customer created without git credentials	Part A.2 — add git.username/git.token, regenerate config
step 1: passphrase REJECTED (401)	typo / wrong customer	re-check with the hub UI's printed curl command
step 8 fails: "CT already exists"	vmid collision with a hub-invisible guest	pick from pct list + qm list (Part B); the agent destroys nothing on collision — re-run with a free vmid and --resume
controller container missing in-guest after provision (docker ps empty)	golden older than 0.98.3 (no baked path unit) + pre-v1.9.1 script: the controller-bootstrap unit's boot-time condition lost the race with the bootstrap-mount attach	pct reboot <VMID> — the unit runs on the next boot (v1.9.1 reboots itself; goldens ≥ 0.98.3 bake a path unit that starts the controller on the mount hot-plug, no reboot needed)
fresh box runs an OLD controller and never self-updates (hub floor ignored)	a golden older than 0.86.0 was vouched in the Day-0 manifest — its baked controller predates the floor-honoring code	one-time manual update (the retired old D.1b): dashboard Beállítások → "Frissítés", or POST /api/selfupdate/check + /api/selfupdate/update via the traefik Host-header probe; if the agent is < v0.63.0, systemctl restart felhom-agent FIRST (pre-v0.63.0 agents 401 the swap call). Structural fix: re-vouch a golden ≥ 0.98.3 (A.3)
self-update fails: "Agent swap request failed: HTTP 401" (or dashboard agent calls 401)	agent < v0.63.0: the daemon's token index missed a token minted by the provisioner after it started (FIXED in agent v0.63.0 — reload-on-miss)	systemctl restart felhom-agent, then retry (unnecessary from agent v0.63.0)
local-lvm free < 120 GiB warning	small/shared box	proceed only if you sized the grows deliberately; otherwise free space first
cloudflared restarting in-guest	bad/missing tunnel token	re-check A.1/A.2 token, regenerate config, docker restart cloudflared in-guest

(Extended during the drill with every real hiccup encountered.)

---

Related

• vmid picking + reserved bands: documentation/runbooks/provisioning.md
• The Day-0 trust chain (why the box can verify what it fetches): documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md
• Pool-scoped ACL model: documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md
• Secrets handling: documentation/runbooks/secrets.md