felhom.eu/skills/felhom-build-deploy/SKILL.md

---
name: felhom-build-deploy
description: Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.
---

# Felhom build & deploy runbooks

**Claude Code runs on DooPlex (192.168.0.180, Debian 13, `kisfenyo`) — builds are LOCAL commands.**
All repos live under `$FELHOM_ROOT/git/`. Trunk-based: commit+push to `main` first, always.
Update the repo's `CHANGELOG.md` (+ `REUSE.md` if a shared helper changed) in the same commit.

## Session invariants (set once, every session)

```bash
FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root — ALL felhom repos/build/drill/iso live HERE
                                   # repos $FELHOM_ROOT/git/felhom-*, build dirs $FELHOM_ROOT/build/felhom-*
```

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

| Host | Access | Role |
|---|---|---|
| **DooPlex (this host, k3s)** | local | build+push images/binaries (`$FELHOM_ROOT/build/felhom-{controller,hub,agent}`, repos under `$FELHOM_ROOT/git/`), `sudo kubectl` |
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent; `felhom-pve-lan` = `192.168.0.162` LAN fallback). See `documentation/operations/tailscale.md` | agent deploy, `pct` into guests |
| Demo guest 9201 | via `ssh felhom-pve "pct exec 9201 -- bash -c '...'"` | the live controller |
| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism — not the 9201 flow |

**No browser here:** `claude-in-chrome` is not available on DooPlex. Validate at the endpoint level
(invoke the exact endpoint the UI invokes) and say which method was used.

Version bumps are ldflags-only (`-X main.version` / `-X main.Version`) — build scripts inject them; no source edit.

## Controller (felhom-controller → guest 9201)

9201 is golden/bootstrap-managed — **NO compose file**. `felhom-controller-bootstrap.service` docker-runs
the tag written in `/etc/felhom-controller-image` (anonymous Gitea pull). Data volume + encryption key persist.

**Never deploy these controller tags** <!-- spoiled-controller-tags --> `0.315.0` `0.316.3` — withdrawn builds (R-1081,
R-1085): their image is not the code their number says, and neither is ever re-pushed. The hub refuses them as a floor
and holds one that arrives any other way; step 3 below refuses them by hand. The hub's list
(`hub/internal/store/spoiled_tags.go`) and these two lines must agree — `TestSpoiledControllerTags_MatchTheSkill`.

```bash
# 1. commit+push the repo (clean-tree gate above)
# 2. build+push image, LOCAL (build.sh does NOT git-pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push
# 3. deploy in the guest — refused for a withdrawn build (the list is the marked line above; spoiled-controller-tags)
VER=<VER>; case " 0.315.0 0.316.3 " in *" ${VER#v} "*) echo "REFUSED: $VER is a withdrawn build (R-1085)"; false;; esac && \
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"
# 4. verify
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
```

Check current live version first: same `docker ps` command, or `cat /etc/felhom-controller-image`.

### Releasing a controller (`09` §3 decision 250, operator 2026-10-15 — a release needs NO golden)

1. **Test images get an `-rc.N` tag** (`./build.sh 0.313.0-rc.1 --push`). The household's update button takes the HIGHEST
   tag in the registry, and an `-rc` tag is not a version to it (`util.ParseVersion` wants X.Y.Z) — so it is never offered.
   **Since controller 0.314.0 a box on `X.Y.Z-rc.N` is BELOW `X.Y.Z` for the floor** (`util.ParseVersionPre`, R-1053), so
   the release's floor moves it by itself (proven: Tester 1, rc.1 → 0.314.0). A box on an rc built BEFORE 0.314.0 runs the
   old check and is never moved: put it on a new rc (or the release tag) by hand, through `/etc/felhom-controller-image`.
2. **Push the release tag only when it is ready for EVERY box** — from that second every household's button offers it.
3. **The three test boxes through the normal update path:** a per-customer floor for Tester 1, demo-hp and demo-felhom at
   the release, MinAgent declared — `POST /customers/<id>/floor` with the same token and fields as step 5 (ids: `tester-1`,
   `demo-hp`, `demo-felhom`); read back each box's `docker ps` and its `SetFloor`/self-update log lines. A box can take up to
   its next 15-minute report (R-1061). After the global floor, clear them: the same POST with both fields empty.
4. **The box walk green on all three** — `python3 scripts/https_only_check.py` included (felhom.eu).
5. **Raise the global floor**, MinAgent declared (from the CHANGELOG header), with **CC's hub token** — never the
   operator password (`09` §3 decision 251; hub ≥ 0.152.0, `05` §17). The token is `HUB_TOKEN` in the CC secret
   store — use it through `sec run` (cc-secrets `ARCHITECTURE.md`), and hand it to curl through a header file
   (`printf` is a shell builtin, so the value is never on a command line):
   ```bash
   sec run HUB_TOKEN -- sh -c 'H=$(mktemp -p "$XDG_RUNTIME_DIR"); printf "Authorization: Bearer %s\n" "$HUB_TOKEN" > "$H"
     curl -s -o /dev/null -D - -H @"$H" --data-urlencode min_controller_version=<VER> --data-urlencode min_agent=<MA> \
       http://<hub-clusterIP>:8080/configuration/global-floor | grep -iE "^HTTP|^location"; rm -f "$H"'
   ```
   **Success is `303` with `Location: /configuration?flash=floor_set`** — a 303 to any other flash is a REFUSAL
   (`floor_needs_min_agent`, `floor_invalid`, …). `401` = token expired/revoked; `403` names the missing scope. Then read
   back from the hub DB **with its WAL** (`hub_settings.min_controller_version` and `…_declared_min_agent`). The vouch
   (`/configuration/artifacts`) is NOT CC's — the token cannot do it (403), and must not be widened to.
6. Not at night unless the night brief says so. **Agent releases still stop for the operator's vouch**; never set a floor
   whose MinAgent is above the vouched agent. The golden is baked weekly (Sunday, `nightly-rotation.md`); CC never vouches.

## Agent (felhom-agent → felhom-pve)

Runs as the NON-ROOT `felhom-agent` user: `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`
(systemd `felhom-agent.service`). Sudoers allowlist at `/etc/sudoers.d/felhom-agent`.

> **Release and delivery today (corrected 2026-10-13, R-998 session — the copy-and-restart block below is the
> pre-R-840 path):** release with `scripts/release-agent.sh <ver>` (push the code first, the CHANGELOG entry after;
> package creds = the gitea entry of `~/.docker/config.json`, read into env, never printed). A box takes it ONLY by a
> signed job per box: `felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1 -key <operational key>
> -agent-version <ver> -sha256 <sha> -upload http://<hub ClusterIP>:8080 -hub-key "$REPORT_API_KEY"`, and when
> `configs/` changed also `-op agent_config_update -agent-version <ver> -bundle-sha256 <bundle sha>` (root files ride
> only the signed bundle). Picked up in 4–15 min; observables: `signedjobs: signed op COMPLETED`, `selfupdate: update
> committed`, `osupdate: config bundle INSTALLED` + the capability probe. Measured on three boxes 2026-10-13.
> **After the operator's vouch (or a floor that declares a new MinAgent):** set `pinned.day0_agent` (or
> `pinned.hub_min_agent`) in `felhom-agent/scripts/retention-policy.json` in the same session — the agent CI asserts the
> newest 5 versions plus those two pins, because it cannot read the hub (D50 = A, `09` §3 decision 262, R-1290).

```bash
# build LOCALLY (pull first!)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent
# copy to the PVE host — ONE hop, no intermediate workstation
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/
# install with backup + restart
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"
```

**Ship the sudoers + guarded-mkfs wrapper WITH the binary whenever `configs/` changed** — several Go
guards exist only if the deployed sudoers/wrapper match the binary (globs must match `stageTemp`
patterns). After restart, check the journal for a clean `ReassertGuestBinds` and no capability-probe
degradations. (Legacy Windows workstation: the copy was two hops via the Windows box, needing
`cygpath -w` for the local scp path — and config files scp'd through Windows risked CRLF.)

Publish to Gitea (so Day-0 self-install can fetch it): `scripts/publish-agent.sh <ver> <binary>` with
`REGISTRY_*` creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is
operator-password-gated (CC cannot); flag it as an operator follow-up.

## Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible

**Two modes, and picking the wrong one ships the wrong product.**

| Mode | What it is | Menu |
|---|---|---|
| `--release` | the **public** image. No `answer.toml`, no root password, no SSH key, no disk profile. Day-0 rides a `.deb`. | TWO interactive entries, graphical default, timeout 15s |
| `--pairing` / `--bootstrap-env` | operator-built **appliance** image: baked answer file, baked root hash, a disk profile pinned to one machine | ONE automated entry |

```bash
# build the public image (clean-tree gate first — an unpushed change does not exist)
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
bash scripts/iso/build-felhom-iso.sh \
  --pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
  --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)
```

**Before publishing, two hard gates — neither is optional and neither is a script yet:**

1. **`documentation/runbooks/iso-release-gate.md`** — 13 criteria, run against **the exact file you
   will upload**, not the build inputs. It is a manual checklist; `repo_gates.py` does **not** cover
   it, so nothing will remind you (R-29's shape — say so if you skip it).
2. **A proof install from the built image on BOTH menu entries** (graphical *and* Terminal UI), each
   showing: package installed, unit `enabled`, unit fired on first boot, and a pairing code in
   `/etc/felhom/appliance-pairing-code`. Spike 4 *reasoned* the graphical path follows from shared
   `Install.pm`; the 1.26.0 run proved that reasoning insufficient in a different place — do both.

```bash
# publish — rclone in a container, env-only config, so NO credential file is ever written.
# The keys come from the CC secret store through `sec run` (never `source` a credentials file — R-1122), and reach
# docker as `-e NAME` with no value, so they are never on docker's command line either. Proven 2026-10-17 with a
# read-only `lsf R2:felhom-iso` through this exact shape.
# R-587: a *.rootpw.txt in the publish source STOPS the upload — the --include pattern is not a control
if ls "$FELHOM_ISO_OUT"/*.rootpw.txt 2>/dev/null; then echo "STOP: shred -u the file(s) above, then publish (R-587)"; else
sec run ISO_S3_CLIENT_AK,ISO_S3_CLIENT_SK,ISO_S3_URL -- sh -c 'export RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
    RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL"
  docker run --rm -v '"$FELHOM_ISO_OUT"':/data:ro \
    -e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
    -e RCLONE_CONFIG_R2_ACCESS_KEY_ID -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY -e RCLONE_CONFIG_R2_ENDPOINT \
    -e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
    rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M'
fi
# verify by ROUND TRIP — the downloaded bytes, not the local file
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso
```

`ListBuckets` 403s — the token is object-scoped; list with `lsf R2:felhom-iso`, not `lsd R2:`.

### Proof-VM traps — every one of these cost a wrong diagnosis

- **Set `--boot` in a SEPARATE `qm set`, after the disk exists.** `qm set <id> --scsi0 … --boot order="scsi0;ide2"`
  in one call silently yields `boot: order=net0;ide2`; the VM netboots, fails, falls through to the CD.
- **After the install, detach the CD** (`qm set <id> --delete ide2; qm set <id> --boot order="scsi0"`)
  or the machine re-enters the installer on reboot — **a completed install looks exactly like a stuck one.**
  Judge completion from `qm config` + disk usage, never from the screen.
- **Verify focus by screendump before every `Enter`.** TUI: red-highlighted button, tab order. GTK:
  dashed focus ring, and `Enter` lands in text *fields*, not `Next`. Not checking once aborted an install.
- **Proof installs register unclaimed appliances at the hub — discard them** or R-131 grows. **The discard is session
  only** (hub ≥ 0.152.0, `05` §17.3 — no token can do it, and CC holds no hub password any more): ask the operator to
  press Discard on the hub's Hosts page, naming the appliance id. (A Basic-auth POST — the operator's own script —
  still needs `-H "X-Felhom-Operator: cli"`, R-135.)

> **Never pair `-w '%{redirect_url}'` with `-u` or `--netrc` (R-580).** curl rebuilds the request URL
> for that variable **with the credentials in it**, so the hub password is printed even though it
> never appeared in the command. `%{url_effective}` and `%{referer}` do the same. Print
> `%{http_code}` alone; read the destination from `-D-` headers if you need it.
  The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s.
- Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads
  `disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.

**Why the shape is what it is** (do not re-derive; four spikes measured it):
`documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md`. In short — no udev property
distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently
wipes one, so **there is no safe automated disk selection for unseen hardware**; and `[first-boot]`
is never placed on the system by an interactive install, so day-0 rides a `.deb` in
`/proxmox/packages/` instead (`Install.pm:1343-1372`).

## Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app `felhom`)

**The manifest is the truth.** A code push + image build deploys NOTHING until `manifests/hub.yaml`'s
`image:` tag changes in git AND the app is synced (auto-sync is OFF). Never `kubectl set image`
(reverted on next sync), never `:latest`. The live image can lag the CHANGELOG — reconcile via the manifest.

```bash
# 1. commit+push code   2. build+push image (LOCAL)
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# 3. bump manifests/hub.yaml image tag → <VER>, commit, push
# 4. hard-refresh, then READ what is OutOfSync before syncing. If anything besides Deployment/hub is OutOfSync
#    (since 2026-10-09: three Secrets de-gitted by R-925 + Deployment/umami), sync ONLY the hub — a whole-app sync
#    pushes git's view over rotated live Secrets:
#    sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"revision":"<sha>","resources":[{"group":"apps","kind":"Deployment","name":"hub","namespace":"felhom-system"}]}}}'
#    Image push 401 "unauthorized" from DooPlex = its saved Docker login predates the R-925 rotation (see R-925).
# (argocd CLI is not logged in — drive the Application CR)
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"syncStrategy":{"apply":{}}}}}'
# 5. verify: Synced/Healthy + rollout + image tag + startup log
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10
```

Green gate before any hub commit: `go build ./... && go vet ./... && go test ./...` in `hub/`.

## Website (felhom.eu/website)

Since 2026-10-30 (D35) the public site is Cloudflare Pages: push to `main`, then `sec run CF_PAGES_TOKEN --
scripts/publish-site-cloudflare.sh` (`--dry-run` first; DooPlex's git-sync copy was retired 2026-11-07, R-1228 — the installer tag is in `INSTALLER-TAG`). **Run `python3 scripts/site_gates.py`
first, after ANY website change** (BOM, emoji, nav parity, cache-bust `?v=N` — bump it when css/svg
change). New pages must be added to the script's `PAGES` list. Emergency edits: https://files.felhom.eu.

## App catalog (app-catalog-felhom.eu)

Push to `main` = deploy: the controller's git-sync picks it up within 15 min, or trigger via the
dashboard "Sablonok frissítése" button / `POST /api/sync` (30s debounce). Only `docker-compose.yml` +
`.felhom.yml` sync; deployed `app.yaml` is never overwritten. Conventions: `<repo>/REUSE.md`.

## Other k8s manifests (felhom.eu/manifests)

Same GitOps rule as the hub: edit in git, push, deliberate ArgoCD sync of app `felhom`. Never
`kubectl apply` directly. Secrets: out-of-band `kubectl create secret` + `secretKeyRef` — never inline
`stringData` (see felhom.eu/REUSE.md §3).