# RUNBOOK — manual build / deploy / publish (agent · controller · golden · hub) **Audience:** the operator, on DooPlex (192.168.0.180, Debian 13) — the same environment CC uses. Builds are local commands; felhom-pve is one `ssh` hop. **Written:** 2026-07-11, from the verified `felhom.eu/skills/felhom-build-deploy/SKILL.md` command set + the publish-train procedure (`documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md`) + `documentation/runbooks/publish-train-rules.md`. If this doc and the skill ever disagree, the skill wins. --- ## 0. The mental model (why the hub dropdown "lags") There is **no CI** — nothing builds on push, ever. Every artifact moves through two separate, deliberate verbs: | Verb | Means | Who sees it | |---|---|---| | **DEPLOY** | build + install on the DEMO (felhom-pve / guest 9201 / k3s) | only the demo box | | **PUBLISH** | upload the artifact to Gitea packages + vouch it in the hub Day-0 manifest | Day-0 installs, the manifest dropdowns, and the remote-rollout machinery (signed agent ops, controller floor) | A version can be live-on-demo for days without being published (agent 0.82–0.84 right now). The hub Configuration dropdowns list **published** artifacts only — that screen showing 0.81.0/0.113.0 is correct, not stale. Remote rollout to customer boxes is a third, separate step and has its own runbook pattern: **agent** = operator-signed `agent_update` op (per box); **controller** = the global floor (DB row — acts immediately; save it LAST). Rules: `publish-train-rules.md`. ## 1. Session setup (every session, first) ```bash FELHOM_ROOT=/mnt/5_hdd/felhom.eu # working root: ALL felhom repos/build/drill/iso live HERE ``` **Clean-tree gate before any build:** `git status --porcelain` must be empty and `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed change does not exist. | Host | Access | Role | |---|---|---| | DooPlex (this host) | local | builds (`$FELHOM_ROOT/git/felhom-agent`, `$FELHOM_ROOT/build/felhom-controller`, `$FELHOM_ROOT/build/felhom-hub`), kubectl | | Demo PVE host | `ssh felhom-pve` (root@192.168.0.162) | agent install, `pct exec 9201` | | Hub UI | hub.felhom.eu → Configuration | manifest vouch, MinAgent, floor (operator password) | Housekeeping note: `$FELHOM_ROOT/build/felhom-agent` on 180 is a stale pre-June-23 leftover — agent builds live in `$FELHOM_ROOT/git/felhom-agent` now. Safe to remove the old dir. ## 2. Agent (felhom-agent binary → felhom-pve, then optionally publish) Always commit+push to `main` first (an unpushed change does not exist). ```bash # BUILD on 180 (the explicit git pull is load-bearing) cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=' -o /tmp/felhom-agent- ./cmd/felhom-agent # PUSH to the PVE host — ONE hop, the binary is already local scp /tmp/felhom-agent- felhom-pve:/tmp/ # DEPLOY with backup + restart + verify ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent- /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager" ``` Gotchas (earned): if `configs/` changed in the repo, ship the **sudoers + guarded wrapper WITH the binary** (several Go guards only exist when the deployed configs match); beware CRLF when scp-ing configs through Windows; after restart the journal must show a clean `ReassertGuestBinds` and (since 0.84) the network-mount reassert, with no capability degradations. **PUBLISH (makes it real for the fleet):** ```bash # from the agent repo, with REGISTRY_* creds set; use the LIVE-DEPLOYED bytes, sha-verified across hops scripts/publish-agent.sh ``` Pre-gate: the package GET for `` must be **404** before (published artifacts are immutable — never republish over an existing version). The script prints the **sha256 — record it**: the same sha goes into the hub manifest AND any signed `agent_update` op. One sha, three places, byte-identical. Then hub → Configuration → Day-0 artifacts: pick the new Agent version (sha auto-read from Gitea), set **Min agent** if the paired controller release declares `MinAgent:` in its CHANGELOG, **Save artifact manifest**. (This save does NOT move the floor — that's a separate card since hub v0.45.) ## 3. Controller image (felhom-controller → guest 9201, remote via floor) 9201 is golden/bootstrap-managed — no compose file. The bootstrap service runs whatever tag is in `/etc/felhom-controller-image` (anonymous pull). ```bash # BUILD+PUSH the image (build.sh does NOT pull — the explicit pull is load-bearing) cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh --push # DEPLOY on the demo guest ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller: && echo gitea.dooplex.hu/admin/felhom-controller: > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'" # VERIFY ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'" ``` **Remote rollout** = the hub floor (Configuration → Managed updates). Order rules (train rules doc): the manifest vouches the target FIRST; any MinAgent must be satisfied fleet-wide (the hub now HOLDS boxes below it automatically, and flags them); **save the floor LAST** — the DB row acts immediately on every box below it, on their next report. ## 4. Golden image (fresh Day-0 installs) The golden is a pre-baked controller-era guest image built in the **drill VM** on 180 (`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`). ### 4.0 The canonical drill-VM launch — captured from a real bake, not reconstructed Until 2026-07-31 this section told the reader to "use the RECORDED qemu launch line" from `RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — while that line is itself labelled **"PASS (reconstructed — DEVIATION)"** and its own §Deviations says *"the canonical `qemu-system` one-liner was **never saved**"*. The document forbade improvising and pointed at an improvisation. The lines below were **captured verbatim from the 0.188.0 bake on 2026-07-31** and are now the canonical invocation. Re-capture them (do not retype from memory) if the bake host or disk layout ever changes. ```bash # 1. revert the disk to virgin (non-destructive to the snapshot; succeeding also proves no qemu holds the qcow2) qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2 # 2. COLD-boot it (the snapshot is disk-only, 0 B VM_SIZE — never -loadvm) qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \ -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \ -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \ -device virtio-net-pci,netdev=n0 -display none -daemonize \ -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid ``` `if=virtio` is load-bearing (the guest expects `/dev/vda`). SSH answers on `:2222` in ~40 s (`ssh -i /mnt/5_hdd/felhom.eu/drill/id_drill -p 2222 root@localhost`); `pveversion` read `pve-manager/9.2.2` on 2026-07-31. **Liveness check:** `ps -eo comm | grep qemu-system-x86` — `pgrep -f qemu-system-x86_64` self-matches your own command line and reports a false "still running". ### 4.1 Bake + publish 1. Revert + boot per §4.0. 2. The debian template is **absent on `virgin`** and **the exact point release rots** — list the current one (`pveam available --section system | grep debian-13`) and `pveam download local `. It was `debian-13-standard_13.6-1_amd64.tar.zst` on 2026-07-31. 3. `scp` in `build-golden.sh` (agent repo `configs/`) + the Gitea token (`~/.gitea-token` on 180, 0600), then run it as a transient unit so it survives a session close, reading the token from the file **inside** the VM so it never reaches a command line: ```bash GT=$(cat /root/.gitea-token); systemd-run --unit=golden-bake --collect \ --setenv=GITEA_USER=admin --setenv=GITEA_TOKEN=$GT \ --setenv=REGISTRY_USER=admin --setenv=REGISTRY_TOKEN=$GT \ bash -c "/root/build-golden.sh 9100 local:vztmpl/