Files
felhom.eu/scripts
admin 2fc4a15fa3
gates / gates (push) Successful in 16s
R-389: key the operator cooldown per app for app_start_failed; gate 11 makes an unfiled observation refuse the push
The cooldown key was customerID:eventType plus the tier and run suffixes, and
none of them names an app, so every app going down inside the same hour
collapsed onto one key and only the first was mailed. Measured on demo-hp:
bookstack sent 09:27:51, privatebin suppressed 09:31:51 under
key=demo-hp:app_start_failed.

cooldownStackSuffix is the third sibling of cooldownTierSuffix and
cooldownRunSuffix, and separate for the reason the second one's docstring
already gives: the existing two keep byte-identical semantics for every type
that uses them.

It is ALLOW-LISTED to app_start_failed and takes the event type as well as the
details, unlike its siblings, and that asymmetry is the safety property. The
backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full disk
sends one digest rather than one mail per app - and crossdrive_failed is
severity error, reaches the operator leg, and carries stack_name through a
DIFFERENT struct, so a payload-shape rule would have split it silently. The
hour itself does not change.

Gate 11 refuses a push whose REPORT.md carries an observation with neither
`FILED: R-NNN` nor `NOT-A-FINDING: <reason>`. It deliberately does NOT accept a
passing mention of some other R-number: the lost item cited R-182 as an analogy,
so "cites a register row" would have passed the very item the gate exists to
catch. That discrepancy with the spec is recorded in the gate's docstring.

Registered here and in the controller and agent runners. NOT in the catalog
runner - it has no shared-gate mechanism and appends --all to every gate;
filed as R-391 rather than left as a sentence, which is this session's lesson.

PROMPT-TEMPLATE.md §15.9 corrected: "documented, NOT acted on" was the wording
that invited the gap, and it now names the markers and points at the gate.

R-390 filed for the golden-bake runbook's missing `pveam update`.
Hub tests 709 -> 716.
2026-08-23 13:53:01 +02:00
..

Felhom host scripts

Operator-side scripts for standing up a Felhom Proxmox host.

felhom-host-install.sh — Day-0 host bootstrap (operator-deploy)

Run on a freshly-PVE-installed box to fully automate Day-0: Proxmox API token → hub host enrollment (single secret) → agent install (fetch + verify + install) → agent config → golden → guest provision → verify. It composes already-proven mechanisms (the pveum role/token sequence, the hub POST /host-enroll enrollment from option C, and felhom-agent --selftest=provision). The agent renders bootstrap.json into the guest and the controller pulls its own controller.yaml in-guest — the script never fetches that.

Since v1.1.0 (BUNDLE slice) the script also installs the agent itself: it fetches the agent binary + golden from Gitea generic packages and verifies each artifact's sha256 against the hub-vouched manifest (GET /api/v1/artifacts/{id}) before installing/using it. The fetch credential is the git token already inside the customer's controller.yaml (config-retrieve) — no new credential, and the checksum trust root is the hub, not Gitea.

Grounding: documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md.

Prerequisites (manual, before running)

  1. Install Proxmox VE 9.x on the box. During the installer, use Advanced → LVM sizing so local-lvm (the pve/data thin pool) has enough room for the appliance volumes — a useful box wants ≥ ~120 GiB free on local-lvm (rootfs 32G + Docker-data ~200G + user-data ~50G after grows). The script refuses below the hard minimum.
  2. SSH into the box as root.
  3. Create the customer in the hub first (hub UI → new customer). The customer's retrieval passphrase (a 5-word Hungarian phrase) is the only secret you carry to the box.

That's it. The agent binary + golden are fetched + verified + installed by the script (provided the operator has recorded the current artifact set in the hub UI → Configs → Day-0 artifacts, and published them via felhom-agent/scripts/publish-agent.sh + configs/build-golden.sh). A local golden, if present, is still used as a fallback.

Usage

curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh
chmod +x felhom-host-install.sh

# secure no-echo passphrase prompt:
sudo ./felhom-host-install.sh --customer-id <customer>

# or from a 0600 file (no prompt):
sudo ./felhom-host-install.sh --customer-id <customer> --passphrase-file /root/.pass

# preview every mutating command without executing:
sudo ./felhom-host-install.sh --customer-id <customer> --dry-run

# resume after a fixed mid-way failure (skips completed steps):
sudo ./felhom-host-install.sh --customer-id <customer> --resume

The passphrase is read no-echo or from a 0600 file — never a CLI argument, never echoed, never written to the state file or logs. The minted Proxmox-token secret and the per-host hub api_key live only in the agent config (0600, root).

Key options

Option Default Purpose
--customer-id ID (required) customer (must already exist in the hub)
--vmid N 9201 guest VMID to provision
--golden VOLID newest vzdump-lxc-<golden-vmid> golden archive
--rootfs/--datavol/--sysdata-grow N auto-compute volume grows (GiB over the golden base 32/16/8)
--passphrase-file PATH no-echo prompt read passphrase from a 0600 file
--preserve-from PATH — merge non-Day-0 sections (PBS/local_api/privileged/authz) from an existing config
--dry-run / --resume / --force off preview / resume / clobber an existing vmid
--mode provision|dr provision dr is a documented 10D stub (not implemented)

Behaviour notes

  • Idempotent + resumable. A step-state file (/var/lib/felhom-install/state.json) records completed steps; --resume skips them. A plain re-run refuses to clobber an existing --vmid (pass --force to override).
  • Single-secret enrollment. POST /host-enroll mints on first call (201) and reuses the credential on later calls (200) — re-running never orphans a running agent's key. The global operator key is never used.
  • Token automation. Creates/normalises the 16-priv FelhomAgent role, the felhom-agent@pve user + privsep token, and both ACL grants (user and token — the ACL is applied after the token exists, because pveum user token remove purges it).
  • Agent install (v1.1.0). Fetches the binary from Gitea (/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent), verifies its sha256 against the hub manifest, then installs the non-root felhom-agent service user + binary + sudoers (0440, visudo -cf-validated) + the canonical systemd unit. Idempotent: same version already installed + service active → skips. A sha256 mismatch aborts the install (verify-before-use). The agent runs non-root (privileged.mode: "sudo" + the sudoers allowlist), never as root.
  • Golden (v1.1.0). Uses a local golden when present; otherwise fetches it from Gitea (/api/packages/admin/generic/felhom-golden/<ver>/golden.tar.zst), verifies its sha256, and imports it into the archive storage's dump dir for the restore. --force-gitea-golden forces the Gitea path even when a local golden exists.
  • DR mode (--mode dr) is a documented seam only — it restores the customer's own PBS whole-CT snapshot instead of the golden. Not implemented (10D).

Productionization hooks (not done here)

  • Serving: place this file where the felhom.eu site serves it at https://felhom.eu/scripts/felhom-host-install.sh (a static route; verify on deploy).
  • Per-customer artifact pinning: the hub manifest currently returns the global current artifact set for every customer; per-customer pinning is a future hook (GET /api/v1/artifacts/{id} already takes the customer id).
  • Unit/sudoers integrity: the binary + golden are sha256-verified against the hub; the unit + sudoers are fetched from the agent repo main (canonical text) and the sudoers is visudo -cf-validated.