felhom.eu/skills/felhom-build-deploy/SKILL.md

name, description

name

description

felhom-build-deploy

Build, deploy, publish, or verify ANY Felhom artifact — felhom-controller image (guest 9201 bootstrap deploy), felhom-agent binary (felhom-pve), felhom-hub (GitOps/ArgoCD), the felhom.eu website (git-sync), the app catalog, or the PUBLIC installer ISO (iso.felhom.eu). Use whenever the task says build, deploy, ship, release, publish, bump version, restart the controller/agent/hub, or verify what version is live. Contains the exact verified commands and the gotchas that silently break deploys.

Felhom build & deploy runbooks

Claude Code runs on DooPlex (192.168.0.180, Debian 13, kisfenyo) — builds are LOCAL commands. All repos live under $FELHOM_ROOT/git/. Trunk-based: commit+push to main first, always. Update the repo's CHANGELOG.md (+ REUSE.md if a shared helper changed) in the same commit.

Session invariants (set once, every session)

FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root — ALL felhom repos/build/drill/iso live HERE
                                   # repos $FELHOM_ROOT/git/felhom-*, build dirs $FELHOM_ROOT/build/felhom-*

Clean-tree gate before any build: git status --porcelain must be empty and git rev-parse HEAD must equal git rev-parse origin/main in the repo being built. An unpushed change does not exist — never build a dirty or unpushed tree. The git pull in the build step stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from elsewhere).

Host	Access	Role
DooPlex (this host, k3s)	local	build+push images/binaries ($FELHOM_ROOT/build/felhom-{controller,hub,agent}, repos under $FELHOM_ROOT/git/), sudo kubectl
Demo Proxmox host (N100)	ssh felhom-pve — via Tailscale 100.70.170.35 (location-independent; felhom-pve-lan = 192.168.0.162 LAN fallback). See documentation/operations/tailscale.md	agent deploy, pct into guests
Demo guest 9201	via ssh felhom-pve "pct exec 9201 -- bash -c '...'"	the live controller
felhotest (legacy)	ssh -p 33022 kisfenyo@router.abonet.hu	OLD /opt/docker compose mechanism — not the 9201 flow

No browser here: claude-in-chrome is not available on DooPlex. Validate at the endpoint level (invoke the exact endpoint the UI invokes) and say which method was used.

Version bumps are ldflags-only (-X main.version / -X main.Version) — build scripts inject them; no source edit.

Controller (felhom-controller → guest 9201)

9201 is golden/bootstrap-managed — NO compose file. felhom-controller-bootstrap.service docker-runs the tag written in /etc/felhom-controller-image (anonymous Gitea pull). Data volume + encryption key persist.

Never deploy these controller tags 0.315.0 0.316.3 — withdrawn builds (R-1081, R-1085): their image is not the code their number says, and neither is ever re-pushed. The hub refuses them as a floor and holds one that arrives any other way; step 3 below refuses them by hand. The hub's list (hub/internal/store/spoiled_tags.go) and these two lines must agree — TestSpoiledControllerTags_MatchTheSkill.

# 1. commit+push the repo (clean-tree gate above)
# 2. build+push image, LOCAL (build.sh does NOT git-pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push
# 3. deploy in the guest — refused for a withdrawn build (the list is the marked line above; spoiled-controller-tags)
VER=<VER>; case " 0.315.0 0.316.3 " in *" ${VER#v} "*) echo "REFUSED: $VER is a withdrawn build (R-1085)"; false;; esac && \
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"
# 4. verify
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"

Check current live version first: same docker ps command, or cat /etc/felhom-controller-image.

Releasing a controller (09 §3 decision 250, operator 2026-10-15 — a release needs NO golden)

1. Test images get an -rc.N tag (./build.sh 0.313.0-rc.1 --push). The household's update button takes the HIGHEST tag in the registry, and an -rc tag is not a version to it (util.ParseVersion wants X.Y.Z) — so it is never offered. Since controller 0.314.0 a box on X.Y.Z-rc.N is BELOW X.Y.Z for the floor (util.ParseVersionPre, R-1053), so the release's floor moves it by itself (proven: Tester 1, rc.1 → 0.314.0). A box on an rc built BEFORE 0.314.0 runs the old check and is never moved: put it on a new rc (or the release tag) by hand, through /etc/felhom-controller-image.
2. Push the release tag only when it is ready for EVERY box — from that second every household's button offers it.
3. The three test boxes through the normal update path: a per-customer floor for Tester 1, demo-hp and demo-felhom at the release, MinAgent declared — POST /customers/<id>/floor with the same token and fields as step 5 (ids: tester-1, demo-hp, demo-felhom); read back each box's docker ps and its SetFloor/self-update log lines. A box can take up to its next 15-minute report (R-1061). After the global floor, clear them: the same POST with both fields empty.
4. The box walk green on all three — python3 scripts/https_only_check.py included (felhom.eu).
5. Raise the global floor, MinAgent declared (from the CHANGELOG header), with CC's hub token — never the operator password (09 §3 decision 251; hub ≥ 0.152.0, 05 §17). The token is HUB_TOKEN in the CC secret store — use it through sec run (cc-secrets ARCHITECTURE.md), and hand it to curl through a header file (printf is a shell builtin, so the value is never on a command line):

   sec run HUB_TOKEN -- sh -c 'H=$(mktemp -p "$XDG_RUNTIME_DIR"); printf "Authorization: Bearer %s\n" "$HUB_TOKEN" > "$H"
     curl -s -o /dev/null -D - -H @"$H" --data-urlencode min_controller_version=<VER> --data-urlencode min_agent=<MA> \
       http://<hub-clusterIP>:8080/configuration/global-floor | grep -iE "^HTTP|^location"; rm -f "$H"'
   

   Success is 303 with Location: /configuration?flash=floor_set — a 303 to any other flash is a REFUSAL (floor_needs_min_agent, floor_invalid, …). 401 = token expired/revoked; 403 names the missing scope. Then read back from the hub DB with its WAL (hub_settings.min_controller_version and …_declared_min_agent). The vouch (/configuration/artifacts) is NOT CC's — the token cannot do it (403), and must not be widened to.
6. Not at night unless the night brief says so. Agent releases still stop for the operator's vouch; never set a floor whose MinAgent is above the vouched agent. The golden is baked weekly (Sunday, nightly-rotation.md); CC never vouches.

Agent (felhom-agent → felhom-pve)

Runs as the NON-ROOT felhom-agent user: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json (systemd felhom-agent.service). Sudoers allowlist at /etc/sudoers.d/felhom-agent.

Release and delivery today (corrected 2026-10-13, R-998 session — the copy-and-restart block below is the pre-R-840 path): release with scripts/release-agent.sh <ver> (push the code first, the CHANGELOG entry after; package creds = the gitea entry of ~/.docker/config.json, read into env, never printed). A box takes it ONLY by a signed job per box: felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1 -key <operational key> -agent-version <ver> -sha256 <sha> -upload http://<hub ClusterIP>:8080 -hub-key "$REPORT_API_KEY", and when configs/ changed also -op agent_config_update -agent-version <ver> -bundle-sha256 <bundle sha> (root files ride only the signed bundle). Picked up in 4–15 min; observables: signedjobs: signed op COMPLETED, selfupdate: update committed, osupdate: config bundle INSTALLED + the capability probe. Measured on three boxes 2026-10-13.

# build LOCALLY (pull first!)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent
# copy to the PVE host — ONE hop, no intermediate workstation
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/
# install with backup + restart
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"

Ship the sudoers + guarded-mkfs wrapper WITH the binary whenever configs/ changed — several Go guards exist only if the deployed sudoers/wrapper match the binary (globs must match stageTemp patterns). After restart, check the journal for a clean ReassertGuestBinds and no capability-probe degradations. (Legacy Windows workstation: the copy was two hops via the Windows box, needing cygpath -w for the local scp path — and config files scp'd through Windows risked CRLF.)

Publish to Gitea (so Day-0 self-install can fetch it): scripts/publish-agent.sh <ver> <binary> with REGISTRY_* creds. The hub's Day-0 artifact manifest must then vouch the new version — that UI is operator-password-gated (CC cannot); flag it as an operator follow-up.

Installer ISO (felhom.eu/scripts/iso → iso.felhom.eu) — PUBLIC, irreversible

Two modes, and picking the wrong one ships the wrong product.

Mode	What it is	Menu
--release	the public image. No answer.toml, no root password, no SSH key, no disk profile. Day-0 rides a .deb.	TWO interactive entries, graphical default, timeout 15s
--pairing / --bootstrap-env	operator-built appliance image: baked answer file, baked root hash, a disk profile pinned to one machine	ONE automated entry

# build the public image (clean-tree gate first — an unpushed change does not exist)
export FELHOM_ISO_OUT=$FELHOM_ROOT/felhom-iso/out
bash scripts/iso/build-felhom-iso.sh \
  --pve-iso $FELHOM_ROOT/drill/proxmox-ve_9.2-1.iso \
  --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c --release
# -> $FELHOM_ISO_OUT/felhom-installer-<VER>-pve<PVE>.iso + .sha256 + .manifest.txt (NO .rootpw.txt)

Before publishing, two hard gates — neither is optional and neither is a script yet:

1. documentation/runbooks/iso-release-gate.md — 13 criteria, run against the exact file you will upload, not the build inputs. It is a manual checklist; repo_gates.py does not cover it, so nothing will remind you (R-29's shape — say so if you skip it).
2. A proof install from the built image on BOTH menu entries (graphical and Terminal UI), each showing: package installed, unit enabled, unit fired on first boot, and a pairing code in /etc/felhom/appliance-pairing-code. Spike 4 reasoned the graphical path follows from shared Install.pm; the 1.26.0 run proved that reasoning insufficient in a different place — do both.

# publish — rclone in a container, env-only config, so NO credential file is ever written.
# The keys come from the CC secret store through `sec run` (never `source` a credentials file — R-1122), and reach
# docker as `-e NAME` with no value, so they are never on docker's command line either. Proven 2026-10-17 with a
# read-only `lsf R2:felhom-iso` through this exact shape.
# R-587: a *.rootpw.txt in the publish source STOPS the upload — the --include pattern is not a control
if ls "$FELHOM_ISO_OUT"/*.rootpw.txt 2>/dev/null; then echo "STOP: shred -u the file(s) above, then publish (R-587)"; else
sec run ISO_S3_CLIENT_AK,ISO_S3_CLIENT_SK,ISO_S3_URL -- sh -c 'export RCLONE_CONFIG_R2_ACCESS_KEY_ID="$ISO_S3_CLIENT_AK" \
    RCLONE_CONFIG_R2_SECRET_ACCESS_KEY="$ISO_S3_CLIENT_SK" RCLONE_CONFIG_R2_ENDPOINT="$ISO_S3_URL"
  docker run --rm -v '"$FELHOM_ISO_OUT"':/data:ro \
    -e RCLONE_CONFIG_R2_TYPE=s3 -e RCLONE_CONFIG_R2_PROVIDER=Cloudflare \
    -e RCLONE_CONFIG_R2_ACCESS_KEY_ID -e RCLONE_CONFIG_R2_SECRET_ACCESS_KEY -e RCLONE_CONFIG_R2_ENDPOINT \
    -e RCLONE_CONFIG_R2_REGION=auto -e RCLONE_CONFIG_R2_NO_CHECK_BUCKET=true \
    rclone/rclone:latest copy /data R2:felhom-iso --include "felhom-installer-<VER>*" --s3-chunk-size 64M'
fi
# verify by ROUND TRIP — the downloaded bytes, not the local file
curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.iso && sha256sum /tmp/rt.iso

ListBuckets 403s — the token is object-scoped; list with lsf R2:felhom-iso, not lsd R2:.

Proof-VM traps — every one of these cost a wrong diagnosis

• Set --boot in a SEPARATE qm set, after the disk exists. qm set <id> --scsi0 … --boot order="scsi0;ide2" in one call silently yields boot: order=net0;ide2; the VM netboots, fails, falls through to the CD.
• After the install, detach the CD (qm set <id> --delete ide2; qm set <id> --boot order="scsi0") or the machine re-enters the installer on reboot — a completed install looks exactly like a stuck one. Judge completion from qm config + disk usage, never from the screen.
• Verify focus by screendump before every Enter. TUI: red-highlighted button, tab order. GTK: dashed focus ring, and Enter lands in text fields, not Next. Not checking once aborted an install.
• Proof installs register unclaimed appliances at the hub — discard them or R-131 grows. The discard is session only (hub ≥ 0.152.0, 05 §17.3 — no token can do it, and CC holds no hub password any more): ask the operator to press Discard on the hub's Hosts page, naming the appliance id. (A Basic-auth POST — the operator's own script — still needs -H "X-Felhom-Operator: cli", R-135.)

Never pair -w '%{redirect_url}' with -u or --netrc (R-580). curl rebuilds the request URL for that variable with the credentials in it, so the hub password is printed even though it never appeared in the command. %{url_effective} and %{referer} do the same. Print %{http_code} alone; read the destination from -D- headers if you need it. The verb is /discard, POST only (hub/internal/web/server.go:345); /delete 404s.

• Venue: demo-hp, scratch dir storage at /mnt/nvme-1tb root (a subdirectory reads disconnected forever — the agent's exactMount check). Never local-lvm. Remove the storage at teardown.

Why the shape is what it is (do not re-derive; four spikes measured it): documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md. In short — no udev property distinguishes an internal disk from a customer's backup drive and a two-disk filter match silently wipes one, so there is no safe automated disk selection for unseen hardware; and [first-boot] is never placed on the system by an interactive install, so day-0 rides a .deb in /proxmox/packages/ instead (Install.pm:1343-1372).

Hub (felhom.eu/hub → k3s, GitOps via ArgoCD app felhom)

The manifest is the truth. A code push + image build deploys NOTHING until manifests/hub.yaml's image: tag changes in git AND the app is synced (auto-sync is OFF). Never kubectl set image (reverted on next sync), never :latest. The live image can lag the CHANGELOG — reconcile via the manifest.

# 1. commit+push code   2. build+push image (LOCAL)
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# 3. bump manifests/hub.yaml image tag → <VER>, commit, push
# 4. hard-refresh, then READ what is OutOfSync before syncing. If anything besides Deployment/hub is OutOfSync
#    (since 2026-10-09: three Secrets de-gitted by R-925 + Deployment/umami), sync ONLY the hub — a whole-app sync
#    pushes git's view over rotated live Secrets:
#    sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"revision":"<sha>","resources":[{"group":"apps","kind":"Deployment","name":"hub","namespace":"felhom-system"}]}}}'
#    Image push 401 "unauthorized" from DooPlex = its saved Docker login predates the R-925 rotation (see R-925).
# (argocd CLI is not logged in — drive the Application CR)
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"cc"},"sync":{"syncStrategy":{"apply":{}}}}}'
# 5. verify: Synced/Healthy + rollout + image tag + startup log
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10

Green gate before any hub commit: go build ./... && go vet ./... && go test ./... in hub/.

Website (felhom.eu/website)

Push to main = deployed (git-sync sidecar, live in ~1–2 min). Run python3 scripts/site_gates.py first, after ANY website change (BOM, emoji, nav parity, cache-bust ?v=N — bump it when css/svg change). New pages must be added to the script's PAGES list. Emergency edits: https://files.felhom.eu.

App catalog (app-catalog-felhom.eu)

Push to main = deploy: the controller's git-sync picks it up within 15 min, or trigger via the dashboard "Sablonok frissítése" button / POST /api/sync (30s debounce). Only docker-compose.yml + .felhom.yml sync; deployed app.yaml is never overwritten. Conventions: <repo>/REUSE.md.

Other k8s manifests (felhom.eu/manifests)

Same GitOps rule as the hub: edit in git, push, deliberate ArgoCD sync of app felhom. Never kubectl apply directly. Secrets: out-of-band kubectl create secret + secretKeyRef — never inline stringData (see felhom.eu/REUSE.md §3).