Files
felhom.eu/scripts
admin e467785fd3
gates / gates (push) Successful in 5m23s
security: rotate the published secrets and de-git felhom.secret.yaml (R-925, P1)
WHY .gitignore "was not working": it was working. git never consults
.gitignore for a file it ALREADY TRACKS. The rule `*secret*` matched fine --
proved by dropping an untracked copy in and watching check-ignore name
`.gitignore:3:*secret*`. The file had been tracked since feea0606, which is
ironically the commit that de-gitted the Resend key.

WHAT THE EXPOSED VALUE ACTUALLY WAS. Not an analytics password: the GITEA
ADMIN ACCOUNT PASSWORD (is_admin true; /api/v1/admin/users answered 200), in
a repo gitea.dooplex.hu serves anonymously to the internet. That is push
access to every repo -- including the one whose website/ is git-synced live
and whose scripts/ is published by tag to every new box installer (R-110).
Re-ranked P2 -> P1 on that measurement; my first ranking had only measured
the analytics blast radius.

Every committed value was still live. Nothing had ever been rotated.

ROTATED (values never echoed; written to a 0600 file on DooPlex):
  umami-config        APP_SECRET + POSTGRES_PASSWORD. The password was
                      changed INSIDE postgres (ALTER USER) as well as in the
                      Secret -- the env var is only read at first init, so
                      patching the Secret alone would have changed nothing.
  healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD (nothing consumes them,
                      there is no healthchecks Deployment).
  gitea-creds         no longer holds the admin password at all: a SCOPED
                      token (read:package + read:repository).
  gitea admin         new random password; gitea-system/gitea-admin updated.

VERIFIED, not assumed:
  - new admin password -> 200, OLD PUBLISHED PASSWORD -> 401 (the leak is dead)
  - umami: a real beacon returns 200 (so the app authenticates to postgres and
    writes) while a bogus site id still returns 400 (so the 200 means something)
  - hub: "Registry version check: latest = 0.304.0" AND "Template fetched
    (5881 bytes)", no auth failures
  - BOTH token scopes are load-bearing, and the second was found by breaking
    it: a package-only token made the hub log "Template fetch: unexpected
    status 403", because the template fetcher reads a raw file out of the
    felhom-controller repo, not the registry.

AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the
rollout restart needed to pick up the new umami secret put umami into
CrashLoopBackOff and took stats.felhom.eu down (503) for ~4 minutes. Not the
rotation -- at memory 512Mi that pod runs for months but CANNOT RESTART:
startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137).
Raised to 1Gi IN THE MANIFEST, not just live, per .claude/rules/manifests.md
("never bare kubectl set -- the next sync reverts it and the fix silently
disappears").

THE GATE: KNOWN_BACKLOG is removed from manifest_bearer_gate.py, as its own
comment instructed. Red-proofed with a decoy: exit 1 with it, exit 0 without.
An exemption kept this visible for three months and changed nothing.

WHAT REMAINS (operator, and it is bigger than what was fixed): the same
password is still the admin password in ~12 other namespaces -- nextcloud,
paperless, bookstack (a DATABASE ROOT password), tandoor, calibre,
adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not
touch them. Also owed: a kisfenyo Gitea token sits in plaintext in the local
homelab-manifests remote URL and was printed to a session transcript during
this investigation, so it should be replaced regardless (R-580's shape).

NOT a finding: homelab-manifests is private (404 anonymously) and does not
contain the password; ArgoCD's repo credential is a separate token and was
untouched by the rotation.
2026-10-09 13:21:18 +02:00
..
…

Felhom host scripts

Operator-side scripts for standing up a Felhom Proxmox host.

felhom-host-install.sh — Day-0 host bootstrap (operator-deploy)

Run on a freshly-PVE-installed box to fully automate Day-0: Proxmox API token → hub host enrollment (single secret) → agent install (fetch + verify + install) → agent config → golden → guest provision → verify. It composes already-proven mechanisms (the pveum role/token sequence, the hub POST /host-enroll enrollment from option C, and felhom-agent --selftest=provision). The agent renders bootstrap.json into the guest and the controller pulls its own controller.yaml in-guest — the script never fetches that.

Since v1.1.0 (BUNDLE slice) the script also installs the agent itself: it fetches the agent binary + golden from Gitea generic packages and verifies each artifact's sha256 against the hub-vouched manifest (GET /api/v1/artifacts/{id}) before installing/using it. The fetch credential is the git token already inside the customer's controller.yaml (config-retrieve) — no new credential, and the checksum trust root is the hub, not Gitea.

Grounding: documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md.

Prerequisites (manual, before running)

  1. Install Proxmox VE 9.x on the box. During the installer, use Advanced → LVM sizing so local-lvm (the pve/data thin pool) has enough room for the appliance volumes — a useful box wants ≥ ~120 GiB free on local-lvm (rootfs 32G + Docker-data ~200G + user-data ~50G after grows). The script refuses below the hard minimum.
  2. SSH into the box as root.
  3. Create the customer in the hub first (hub UI → new customer). The customer's retrieval passphrase (a 5-word Hungarian phrase) is the only secret you carry to the box.

That's it. The agent binary + golden are fetched + verified + installed by the script (provided the operator has recorded the current artifact set in the hub UI → Configs → Day-0 artifacts, and published them via felhom-agent/scripts/publish-agent.sh + configs/build-golden.sh). A local golden, if present, is still used as a fallback.

Usage

curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh
chmod +x felhom-host-install.sh

# secure no-echo passphrase prompt:
sudo ./felhom-host-install.sh --customer-id <customer>

# or from a 0600 file (no prompt):
sudo ./felhom-host-install.sh --customer-id <customer> --passphrase-file /root/.pass

# preview every mutating command without executing:
sudo ./felhom-host-install.sh --customer-id <customer> --dry-run

# resume after a fixed mid-way failure (skips completed steps):
sudo ./felhom-host-install.sh --customer-id <customer> --resume

The passphrase is read no-echo or from a 0600 file — never a CLI argument, never echoed, never written to the state file or logs. The minted Proxmox-token secret and the per-host hub api_key live only in the agent config (0600, root).

Key options

Option Default Purpose
--customer-id ID (required) customer (must already exist in the hub)
--vmid N 9201 guest VMID to provision
--golden VOLID newest vzdump-lxc-<golden-vmid> golden archive
--rootfs/--datavol/--sysdata-grow N auto-compute volume grows (GiB over the golden base 32/16/8)
--passphrase-file PATH no-echo prompt read passphrase from a 0600 file
--preserve-from PATH — merge non-Day-0 sections (PBS/local_api/privileged/authz) from an existing config
--dry-run / --resume / --force off preview / resume / clobber an existing vmid
--mode provision|dr provision dr is a documented 10D stub (not implemented)

Behaviour notes

  • Idempotent + resumable. A step-state file (/var/lib/felhom-install/state.json) records completed steps; --resume skips them. A plain re-run refuses to clobber an existing --vmid (pass --force to override).
  • Single-secret enrollment. POST /host-enroll mints on first call (201) and reuses the credential on later calls (200) — re-running never orphans a running agent's key. The global operator key is never used.
  • Token automation. Creates/normalises the 16-priv FelhomAgent role, the felhom-agent@pve user + privsep token, and both ACL grants (user and token — the ACL is applied after the token exists, because pveum user token remove purges it).
  • Agent install (v1.1.0). Fetches the binary from Gitea (/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent), verifies its sha256 against the hub manifest, then installs the non-root felhom-agent service user + binary + sudoers (0440, visudo -cf-validated) + the canonical systemd unit. Idempotent: same version already installed + service active → skips. A sha256 mismatch aborts the install (verify-before-use). The agent runs non-root (privileged.mode: "sudo" + the sudoers allowlist), never as root.
  • Golden (v1.1.0). Uses a local golden when present; otherwise fetches it from Gitea (/api/packages/admin/generic/felhom-golden/<ver>/golden.tar.zst), verifies its sha256, and imports it into the archive storage's dump dir for the restore. --force-gitea-golden forces the Gitea path even when a local golden exists.
  • DR mode (--mode dr) is a documented seam only — it restores the customer's own PBS whole-CT snapshot instead of the golden. Not implemented (10D).

Productionization hooks (not done here)

  • Serving: place this file where the felhom.eu site serves it at https://felhom.eu/scripts/felhom-host-install.sh (a static route; verify on deploy).
  • Per-customer artifact pinning: the hub manifest currently returns the global current artifact set for every customer; per-customer pinning is a future hook (GET /api/v1/artifacts/{id} already takes the customer id).
  • Unit/sudoers integrity: the binary + golden are sha256-verified against the hub; the unit + sudoers are fetched from the agent repo main (canonical text) and the sudoers is visudo -cf-validated.