felhom.eu/skills/felhom-build-deploy/SKILL.md

---
name: felhom-build-deploy
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
---

# Felhom build & deploy runbooks

**Claude Code runs on DooPlex (192.168.0.180, Debian 13, `kisfenyo`) — builds are LOCAL commands.**
All repos live under `$FELHOM_ROOT/git/`. Trunk-based: commit+push to `main` first, always.
Update the repo's `CHANGELOG.md` (+ `REUSE.md` if a shared helper changed) in the same commit.

## Session invariants (set once, every session)

```bash
FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root — ALL felhom repos/build/drill/iso live HERE
                                   # repos $FELHOM_ROOT/git/felhom-*, build dirs $FELHOM_ROOT/build/felhom-*
```

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

| Host | Access | Role |
|---|---|---|
| **DooPlex (this host, k3s)** | local | build+push images/binaries (`$FELHOM_ROOT/build/felhom-{controller,hub,agent}`, repos under `$FELHOM_ROOT/git/`), `sudo kubectl` |
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent; `felhom-pve-lan` = `192.168.0.162` LAN fallback). See `documentation/operations/tailscale.md` | agent deploy, `pct` into guests |
| Demo guest 9201 | via `ssh felhom-pve "pct exec 9201 -- bash -c '...'"` | the live controller |
| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism — not the 9201 flow |

**No browser here:** `claude-in-chrome` is not available on DooPlex. Validate at the endpoint level
(invoke the exact endpoint the UI invokes) and say which method was used.

Version bumps are ldflags-only (`-X main.version` / `-X main.Version`) — build scripts inject them; no source edit.

## Controller (felhom-controller → guest 9201)

9201 is golden/bootstrap-managed — **NO compose file**. `felhom-controller-bootstrap.service` docker-runs
the tag written in `/etc/felhom-controller-image` (anonymous Gitea pull). Data volume + encryption key persist.

```bash
# 1. commit+push the repo (clean-tree gate above)
# 2. build+push image, LOCAL (build.sh does NOT git-pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push
# 3. deploy in the guest
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"
# 4. verify
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
```

Check current live version first: same `docker ps` command, or `cat /etc/felhom-controller-image`.

## Agent (felhom-agent → felhom-pve)

Runs as the NON-ROOT `felhom-agent` user: `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`
(systemd `felhom-agent.service`). Sudoers allowlist at `/etc/sudoers.d/felhom-agent`.

```bash
# build LOCALLY (pull first!)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent
# copy to the PVE host — ONE hop, no intermediate workstation
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/
# install with backup + restart
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"
```

**Ship the sudoers + guarded-mkfs wrapper WITH the binary whenever `configs/` changed** — several Go
guards exist only if the deployed sudoers/wrapper match the binary (globs must match `stageTemp`
patterns). After restart, check the journal for a clean `ReassertGuestBinds` and no capability-probe
degradations. (Legacy Windows workstation: the copy was two hops via the Windows box, needing
`cygpath -w` for the local scp path — and config files scp'd through Windows risked CRLF.)

Publish to Gitea (so Day-0 self-install can fetch it): `scripts/publish-agent.sh <ver> <binary>` with
`REGISTRY_*` creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is
operator-password-gated (CC cannot); flag it as an operator follow-up.

## Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible

**Two modes, and picking the wrong one ships the wrong product.**

| Mode | What it is | Menu |
|---|---|---|
| `--release` | the **public** image. No `answer.toml`, no root password, no SSH key, no disk profile. Day-0 rides a `.deb`. | TWO interactive entries, graphical default, timeout 15s |
| `--pairing` / `--bootstrap-env` | operator-built **appliance** image: baked answer file, baked root hash, a disk profile pinned to one machine | ONE automated entry |

```bash
# build the public image (clean-tree gate first — an unpushed change does not exist)
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
bash scripts/iso/build-felhom-iso.sh \
  --pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
  --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)
```

**Before publishing, two hard gates — neither is optional and neither is a script yet:**

1. **`documentation/runbooks/iso-release-gate.md`** — 13 criteria, run against **the exact file you
   will upload**, not the build inputs. It is a manual checklist; `repo_gates.py` does **not** cover
   it, so nothing will remind you (R-29's shape — say so if you skip it).
2. **A proof install from the built image on BOTH menu entries** (graphical *and* Terminal UI), each
   showing: package installed, unit `enabled`, unit fired on first boot, and a pairing code in
   `/etc/felhom/appliance-pairing-code`. Spike 4 *reasoned* the graphical path follows from shared
   `Install.pm`; the 1.26.0 run proved that reasoning insufficient in a different place — do both.

```bash
# publish — rclone in a container, env-only config, so NO credential file is ever written
source ~/.config/credentials      # ISO_S3_CLIENT_AK / _SK / ISO_S3_URL — never echo, never log
docker run --rm -v $FELHOM_ISO_OUT:/data:ro \
  -e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
  -e RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
  -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" \
  -e RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL" \
  -e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
  rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M
# verify by ROUND TRIP — the downloaded bytes, not the local file
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso
```

`ListBuckets` 403s — the token is object-scoped; list with `lsf R2:felhom-iso`, not `lsd R2:`.

### Proof-VM traps — every one of these cost a wrong diagnosis

- **Set `--boot` in a SEPARATE `qm set`, after the disk exists.** `qm set <id> --scsi0 … --boot order="scsi0;ide2"`
  in one call silently yields `boot: order=net0;ide2`; the VM netboots, fails, falls through to the CD.
- **After the install, detach the CD** (`qm set <id> --delete ide2; qm set <id> --boot order="scsi0"`)
  or the machine re-enters the installer on reboot — **a completed install looks exactly like a stuck one.**
  Judge completion from `qm config` + disk usage, never from the screen.
- **Verify focus by screendump before every `Enter`.** TUI: red-highlighted button, tab order. GTK:
  dashed focus ring, and `Enter` lands in text *fields*, not `Next`. Not checking once aborted an install.
- **Proof installs register unclaimed appliances at the hub — discard them** or R-131 grows:
  `curl -u ":$HUB_PW" -X POST http://<hub-clusterIP>:8080/appliances/<id>/discard` → 303.
  The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s.
- Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads
  `disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.

**Why the shape is what it is** (do not re-derive; four spikes measured it):
`documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md`. In short — no udev property
distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently
wipes one, so **there is no safe automated disk selection for unseen hardware**; and `[first-boot]`
is never placed on the system by an interactive install, so day-0 rides a `.deb` in
`/proxmox/packages/` instead (`Install.pm:1343-1372`).

## Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app `felhom`)

**The manifest is the truth.** A code push + image build deploys NOTHING until `manifests/hub.yaml`'s
`image:` tag changes in git AND the app is synced (auto-sync is OFF). Never `kubectl set image`
(reverted on next sync), never `:latest`. The live image can lag the CHANGELOG — reconcile via the manifest.

```bash
# 1. commit+push code   2. build+push image (LOCAL)
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# 3. bump manifests/hub.yaml image tag → <VER>, commit, push
# 4. hard-refresh + sync (argocd CLI is not logged in — drive the Application CR)
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"syncStrategy":{"apply":{}}}}}'
# 5. verify: Synced/Healthy + rollout + image tag + startup log
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10
```

Green gate before any hub commit: `go build ./... && go vet ./... && go test ./...` in `hub/`.

## Website (felhom.eu/website)

Push to `main` = deployed (git-sync sidecar, live in ~1–2 min). **Run `python3 scripts/site_gates.py`
first, after ANY website change** (BOM, emoji, nav parity, cache-bust `?v=N` — bump it when css/svg
change). New pages must be added to the script's `PAGES` list. Emergency edits: https://files.felhom.eu.

## App catalog (app-catalog-felhom.eu)

Push to `main` = deploy: the controller's git-sync picks it up within 15 min, or trigger via the
dashboard "Sablonok frissítése" button / `POST /api/sync` (30s debounce). Only `docker-compose.yml` +
`.felhom.yml` sync; deployed `app.yaml` is never overwritten. Conventions: `<repo>/REUSE.md`.

## Other k8s manifests (felhom.eu/manifests)

Same GitOps rule as the hub: edit in git, push, deliberate ArgoCD sync of app `felhom`. Never
`kubectl apply` directly. Secrets: out-of-band `kubectl create secret` + `secretKeyRef` — never inline
`stringData` (see felhom.eu/REUSE.md §3).