felhom.eu/skills/felhom-build-deploy/SKILL.md

name, description

name

description

felhom-build-deploy

Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.

Felhom build & deploy runbooks

Claude Code runs on DooPlex (192.168.0.180, Debian 13, kisfenyo) — builds are LOCAL commands. All repos live under $FELHOM_ROOT/git/. Trunk-based: commit+push to main first, always. Update the repo's CHANGELOG.md (+ REUSE.md if a shared helper changed) in the same commit.

Session invariants (set once, every session)

FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root — ALL felhom repos/build/drill/iso live HERE
                                   # repos $FELHOM_ROOT/git/felhom-*, build dirs $FELHOM_ROOT/build/felhom-*

Clean-tree gate before any build: git status --porcelain must be empty and git rev-parse HEAD must equal git rev-parse origin/main in the repo being built. An unpushed change does not exist — never build a dirty or unpushed tree. The git pull in the build step stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from elsewhere).

Host	Access	Role
DooPlex (this host, k3s)	local	build+push images/binaries ($FELHOM_ROOT/build/felhom-{controller,hub,agent}, repos under $FELHOM_ROOT/git/), sudo kubectl
Demo Proxmox host (N100)	ssh felhom-pve — via Tailscale 100.70.170.35 (location-independent; felhom-pve-lan = 192.168.0.162 LAN fallback). See documentation/operations/tailscale.md	agent deploy, pct into guests
Demo guest 9201	via ssh felhom-pve "pct exec 9201 -- bash -c '...'"	the live controller
felhotest (legacy)	ssh -p 33022 kisfenyo@router.abonet.hu	OLD /opt/docker compose mechanism — not the 9201 flow

No browser here: claude-in-chrome is not available on DooPlex. Validate at the endpoint level (invoke the exact endpoint the UI invokes) and say which method was used.

Version bumps are ldflags-only (-X main.version / -X main.Version) — build scripts inject them; no source edit.

Controller (felhom-controller → guest 9201)

9201 is golden/bootstrap-managed — NO compose file. felhom-controller-bootstrap.service docker-runs the tag written in /etc/felhom-controller-image (anonymous Gitea pull). Data volume + encryption key persist.

# 1. commit+push the repo (clean-tree gate above)
# 2. build+push image, LOCAL (build.sh does NOT git-pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push
# 3. deploy in the guest
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"
# 4. verify
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"

Check current live version first: same docker ps command, or cat /etc/felhom-controller-image.

Agent (felhom-agent → felhom-pve)

Runs as the NON-ROOT felhom-agent user: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json (systemd felhom-agent.service). Sudoers allowlist at /etc/sudoers.d/felhom-agent.

# build LOCALLY (pull first!)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent
# copy to the PVE host — ONE hop, no intermediate workstation
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/
# install with backup + restart
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"

Ship the sudoers + guarded-mkfs wrapper WITH the binary whenever configs/ changed — several Go guards exist only if the deployed sudoers/wrapper match the binary (globs must match stageTemp patterns). After restart, check the journal for a clean ReassertGuestBinds and no capability-probe degradations. (Legacy Windows workstation: the copy was two hops via the Windows box, needing cygpath -w for the local scp path — and config files scp'd through Windows risked CRLF.)

Publish to Gitea (so Day-0 self-install can fetch it): scripts/publish-agent.sh <ver> <binary> with REGISTRY_* creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is operator-password-gated (CC cannot); flag it as an operator follow-up.

Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible

Two modes, and picking the wrong one ships the wrong product.

Mode	What it is	Menu
--release	the public image. No answer.toml, no root password, no SSH key, no disk profile. Day-0 rides a .deb.	TWO interactive entries, graphical default, timeout 15s
--pairing / --bootstrap-env	operator-built appliance image: baked answer file, baked root hash, a disk profile pinned to one machine	ONE automated entry

# build the public image (clean-tree gate first — an unpushed change does not exist)
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
bash scripts/iso/build-felhom-iso.sh \
  --pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
  --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)

Before publishing, two hard gates — neither is optional and neither is a script yet:

1. documentation/runbooks/iso-release-gate.md — 13 criteria, run against the exact file you will upload, not the build inputs. It is a manual checklist; repo_gates.py does not cover it, so nothing will remind you (R-29's shape — say so if you skip it).
2. A proof install from the built image on BOTH menu entries (graphical and Terminal UI), each showing: package installed, unit enabled, unit fired on first boot, and a pairing code in /etc/felhom/appliance-pairing-code. Spike 4 reasoned the graphical path follows from shared Install.pm; the 1.26.0 run proved that reasoning insufficient in a different place — do both.

# publish — rclone in a container, env-only config, so NO credential file is ever written
source ~/.config/credentials      # ISO_S3_CLIENT_AK / _SK / ISO_S3_URL — never echo, never log
docker run --rm -v $FELHOM_ISO_OUT:/data:ro \
  -e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
  -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
  -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" \
  -e RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL" \
  -e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
  rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M
# verify by ROUND TRIP — the downloaded bytes, not the local file
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso

ListBuckets 403s — the token is object-scoped; list with lsf R2:felhom-iso, not lsd R2:.

Proof-VM traps — every one of these cost a wrong diagnosis

• Set --boot in a SEPARATE qm set, after the disk exists. qm set <id> --scsi0 … --boot order="scsi0;ide2" in one call silently yields boot: order=net0;ide2; the VM netboots, fails, falls through to the CD.
• After the install, detach the CD (qm set <id> --delete ide2; qm set <id> --boot order="scsi0") or the machine re-enters the installer on reboot — a completed install looks exactly like a stuck one. Judge completion from qm config + disk usage, never from the screen.
• Verify focus by screendump before every Enter. TUI: red-highlighted button, tab order. GTK: dashed focus ring, and Enter lands in text fields, not Next. Not checking once aborted an install.
• Proof installs register unclaimed appliances at the hub — discard them or R-131 grows: curl -u ":$HUB_PW" -X POST http://<hub-clusterIP>:8080/appliances/<id>/discard → 303. The verb is /discard, POST only (hub/internal/web/server.go:345); /delete 404s.
• Venue: demo-hp, scratch dir storage at /mnt/nvme-1tb root (a subdirectory reads disconnected forever — the agent's exactMount check). Never local-lvm. Remove the storage at teardown.

Why the shape is what it is (do not re-derive; four spikes measured it): documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md. In short — no udev property distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently wipes one, so there is no safe automated disk selection for unseen hardware; and [first-boot] is never placed on the system by an interactive install, so day-0 rides a .deb in /proxmox/packages/ instead (Install.pm:1343-1372).

Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app felhom)

The manifest is the truth. A code push + image build deploys NOTHING until manifests/hub.yaml's image: tag changes in git AND the app is synced (auto-sync is OFF). Never kubectl set image (reverted on next sync), never :latest. The live image can lag the CHANGELOG — reconcile via the manifest.

# 1. commit+push code   2. build+push image (LOCAL)
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# 3. bump manifests/hub.yaml image tag → <VER>, commit, push
# 4. hard-refresh + sync (argocd CLI is not logged in — drive the Application CR)
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"syncStrategy":{"apply":{}}}}}'
# 5. verify: Synced/Healthy + rollout + image tag + startup log
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10

Green gate before any hub commit: go build ./... && go vet ./... && go test ./... in hub/.

Website (felhom.eu/website)

Push to main = deployed (git-sync sidecar, live in ~1–2 min). Run python3 scripts/site_gates.py first, after ANY website change (BOM, emoji, nav parity, cache-bust ?v=N — bump it when css/svg change). New pages must be added to the script's PAGES list. Emergency edits: https://files.felhom.eu.

App catalog (app-catalog-felhom.eu)

Push to main = deploy: the controller's git-sync picks it up within 15 min, or trigger via the dashboard "Sablonok frissítése" button / POST /api/sync (30s debounce). Only docker-compose.yml + .felhom.yml sync; deployed app.yaml is never overwritten. Conventions: <repo>/REUSE.md.

Other k8s manifests (felhom.eu/manifests)

Same GitOps rule as the hub: edit in git, push, deliberate ArgoCD sync of app felhom. Never kubectl apply directly. Secrets: out-of-band kubectl create secret + secretKeyRef — never inline stringData (see felhom.eu/REUSE.md §3).