felhom.eu/documentation/runbooks/RUNBOOK-manual-build.md

# RUNBOOK — manual build / deploy / publish (agent · controller · golden · hub)

**Audience:** the operator, on DooPlex (192.168.0.180, Debian 13) — the same environment CC uses.
Builds are local commands; felhom-pve is one `ssh` hop.
**Written:** 2026-07-11, from the verified `felhom.eu/skills/felhom-build-deploy/SKILL.md` command set
+ the publish-train procedure (`documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md`) +
`documentation/runbooks/publish-train-rules.md`. If this doc and the skill ever disagree, the skill wins.

---

## 0. The mental model (why the hub dropdown "lags")

There is **no CI** — nothing builds on push, ever. Every artifact moves through two separate,
deliberate verbs:

| Verb | Means | Who sees it |
|---|---|---|
| **DEPLOY** | build + install on the DEMO (felhom-pve / guest 9201 / k3s) | only the demo box |
| **PUBLISH** | upload the artifact to Gitea packages + vouch it in the hub Day-0 manifest | Day-0 installs, the manifest dropdowns, and the remote-rollout machinery (signed agent ops, controller floor) |

A version can be live-on-demo for days without being published (agent 0.82–0.84 right now). The hub
Configuration dropdowns list **published** artifacts only — that screen showing 0.81.0/0.113.0 is
correct, not stale.

Remote rollout to customer boxes is a third, separate step and has its own runbook pattern:
**agent** = operator-signed `agent_update` op (per box); **controller** = the global floor (DB row —
acts immediately; save it LAST). Rules: `publish-train-rules.md`.

## 1. Session setup (every session, first)

```bash
FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root: ALL felhom repos/build/drill/iso live HERE
```

**Clean-tree gate before any build:** `git status --porcelain` must be empty and `git rev-parse HEAD`
must equal `git rev-parse origin/main` in the repo being built. An unpushed change does not exist.

| Host | Access | Role |
|---|---|---|
| DooPlex (this host) | local | builds (`$FELHOM_ROOT/git/felhom-agent`, `$FELHOM_ROOT/build/felhom-controller`, `$FELHOM_ROOT/build/felhom-hub`), kubectl |
| Demo PVE host | `ssh felhom-pve` (root@192.168.0.162) | agent install, `pct exec 9201` |
| Hub UI | hub.felhom.eu → Configuration | manifest vouch, MinAgent, floor (operator password) |

Housekeeping note: `$FELHOM_ROOT/build/felhom-agent` on 180 is a stale pre-June-23 leftover — agent
builds live in `$FELHOM_ROOT/git/felhom-agent` now. Safe to remove the old dir.

## 2. Agent (felhom-agent binary → felhom-pve, then optionally publish)

Always commit+push to `main` first (an unpushed change does not exist).

```bash
# BUILD on 180 (the explicit git pull is load-bearing)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent

# PUSH to the PVE host — ONE hop, the binary is already local
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/

# DEPLOY with backup + restart + verify
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"
```

Gotchas (earned): if `configs/` changed in the repo, ship the **sudoers + guarded wrapper WITH the
binary** (several Go guards only exist when the deployed configs match); beware CRLF when scp-ing
configs through Windows; after restart the journal must show a clean `ReassertGuestBinds` and (since
0.84) the network-mount reassert, with no capability degradations.

**PUBLISH (makes it real for the fleet):**
```bash
# from the agent repo, with REGISTRY_* creds set; use the LIVE-DEPLOYED bytes, sha-verified across hops
scripts/publish-agent.sh <VER> <path-to-binary>
```
Pre-gate: the package GET for `<VER>` must be **404** before (published artifacts are immutable —
never republish over an existing version). The script prints the **sha256 — record it**: the same sha
goes into the hub manifest AND any signed `agent_update` op. One sha, three places, byte-identical.

Then hub → Configuration → Day-0 artifacts: pick the new Agent version (sha auto-read from Gitea),
set **Min agent** if the paired controller release declares `MinAgent:` in its CHANGELOG, **Save
artifact manifest**. (This save does NOT move the floor — that's a separate card since hub v0.45.)

## 3. Controller image (felhom-controller → guest 9201, remote via floor)

9201 is golden/bootstrap-managed — no compose file. The bootstrap service runs whatever tag is in
`/etc/felhom-controller-image` (anonymous pull).

```bash
# BUILD+PUSH the image (build.sh does NOT pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push

# DEPLOY on the demo guest
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"

# VERIFY
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
```

**Remote rollout** = the hub floor (Configuration → Managed updates). Order rules (train rules doc):
the manifest vouches the target FIRST; any MinAgent must be satisfied fleet-wide (the hub now HOLDS
boxes below it automatically, and flags them); **save the floor LAST** — the DB row acts immediately
on every box below it, on their next report.

**Raise the floor to a release with no golden (hub v0.112.0+, R-472).** Between bakes this is the
normal route — do not hand-deploy.

1. Build and push the controller image (above). Do not install it on any box.
2. Read the MinAgent from the release's header: `grep -m1 -A3 '^## v<VER>' CHANGELOG.md` shows
   `**MinAgent: X.Y.Z**`. `controller_gates.py --fast` refuses a release whose newest header lacks it.
3. Hub → Configuration → Managed updates: type `<VER>` in the floor field **and** that value in
   `min_agent`. Save. Without `min_agent` the form refuses with *"This floor is above the vouched
   golden — declare its MinAgent"* and stores nothing.
4. Verify: `sudo kubectl -n felhom-system logs deploy/hub | grep 'managed floor SERVED'` shows
   `from "declared"`, and each box logs `SetFloor: floor "…" → "<VER>"` within about a minute.

### 3.1 Park the controller (stop the agent restarting it) — agent ≥ 0.131.0

The agent restarts a stopped `felhom-controller` within about a minute (R-523). To keep it stopped on
purpose, on the **Proxmox host**: `touch /var/lib/felhom-agent/guests/<vmid>/controller-parked`. The
agent logs `controller is not running and the guest is PARKED — leaving it`. To unpark:
`rm /var/lib/felhom-agent/guests/<vmid>/controller-parked`. A controller swap is never fought either.

## 4. Golden image (fresh Day-0 installs)

The golden is a pre-baked controller-era guest image built in the **drill VM** on 180
(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`).

### 4.0 The canonical drill-VM launch — captured from a real bake, not reconstructed

Until 2026-07-31 this section told the reader to "use the RECORDED qemu launch line" from
`RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — while that line is itself labelled
**"PASS (reconstructed — DEVIATION)"** and its own §Deviations says *"the canonical `qemu-system`
one-liner was **never saved**"*. The document forbade improvising and pointed at an improvisation. The
lines below were **captured verbatim from the 0.188.0 bake on 2026-07-31** and are now the canonical
invocation. Re-capture them (do not retype from memory) if the bake host or disk layout ever changes.

```bash
# 1. revert the disk to virgin (non-destructive to the snapshot; succeeding also proves no qemu holds the qcow2)
qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2

# 2. COLD-boot it (the snapshot is disk-only, 0 B VM_SIZE — never -loadvm)
qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
  -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \
  -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \
  -device virtio-net-pci,netdev=n0 -display none -daemonize \
  -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid
```

`if=virtio` is load-bearing (the guest expects `/dev/vda`). SSH answers on `:2222` in ~40 s
(`ssh -i /mnt/5_hdd/felhom.eu/drill/id_drill -p 2222 root@localhost`); `pveversion` read
`pve-manager/9.2.2` on 2026-07-31. **Liveness check:** `ps -eo comm | grep qemu-system-x86` —
`pgrep -f qemu-system-x86_64` self-matches your own command line and reports a false "still running".

### 4.1 Bake + publish

1. Revert + boot per §4.0.
2. Run **`pveam update` first** — the `virgin` snapshot's template INDEX is stale too, and a stale index fails as a bogus
   `400 no such template` (0.289.1 bake, 2026-10-03). The debian template is **absent on `virgin`** and **the exact point release rots** — list the
   current one (`pveam available --section system | grep 'debian-13-standard_.*_amd64'`) and `pveam download local <that>`.
   **Filter on `_amd64`:** the index also lists an `_arm64` build of the same point release, and a version sort
   picks it (2026-09-28: the 0.276.0 bake's first attempt did, and was stopped before `pct create` finished; the
   2026-09-26 bench LXC did too and would not start).
   It was `debian-13-standard_13.6-1_amd64.tar.zst` on 2026-07-31.
3. `scp` in `build-golden.sh` (agent repo `configs/`) + the Gitea token (`~/.gitea-token` on 180,
   0600), then run it as a transient unit so it survives a session close, reading the token from the
   file **inside** the VM so it never reaches a command line:

   Put the invocation in a **runner script inside the VM** that reads the token itself, and launch
   that. The older `--setenv=GITEA_TOKEN=$GT` form put the value on a command line and into the
   transient unit's properties, where `systemctl show` prints it:

   ```bash
   cat > /root/bake-run.sh <<'EOS'
   #!/bin/bash
   GT=$(cat /root/.gitea-token)
   export GITEA_USER=admin GITEA_TOKEN="$GT" REGISTRY_USER=admin REGISTRY_TOKEN="$GT"
   exec /root/build-golden.sh 9100 local:vztmpl/<template> local-lvm local vmbr0 \
     gitea.dooplex.hu/admin/felhom-controller:<VER>
   EOS
   chmod 0700 /root/bake-run.sh
   systemd-run --unit=golden-bake --collect bash -c "/root/bake-run.sh > /root/bake.log 2>&1"
   ```

   Confirm the value went nowhere: `systemctl show golden-bake -p Environment -p ExecStart |
   grep -c -F "$(cat /root/.gitea-token)"` must be `0`. Copy the token in **file → file** (`scp`), so
   it never crosses a shell on either side.

   `CONTROLLER_IMAGE` is a **required** argument (a hand-bumped default rotted twice) and
   `GOLDEN_VERSION` is derived from it — the golden's version IS the controller it bakes.
   **The script is the publisher**: it uploads to Gitea and prints `GOLDEN_VERSION` + `GOLDEN_SHA256`.
   Pass markers, **each captured from a real log rather than paraphrased** — two of the three named
   here until 2026-08-06 could not match anything the script prints (see the note below):
   `docker OK (overlay2`, `including mount point` for rootfs **and mp0** — there is no mp1 — with no
   `excluding`/`FATAL`, `upload OK (HTTP 201)`. The 404 pre-gate applies to the package URL, which is
   `…/generic/felhom-golden/<VER>/golden.tar.zst` — the filename is `golden.tar.zst`, **not**
   `felhom-golden-<VER>.tar.zst`.

<!--
2026-08-06, R-233: this line listed `overlay2 OK` and demanded `including mount point` for `mp1`.
The script prints neither. The storage-driver line it means is `  docker OK (overlay2; data-root
/var/lib/docker)`, and mp1 stopped existing in build-golden.sh v3.0.0 (R-165 collapsed the two data
volumes into one). A reader following this literally greps for a string that can never appear and
reads 0 — the "an instrument that can silently drop results is not a measurement" class, aimed at the
bake's own acceptance check. The real guard was never weak: the script's own
`[ "$drv" = "overlay2" ] || { echo FATAL; exit 1; }` is fail-closed. The DOCUMENT was the broken part.
-->

4. Teardown: `pct destroy 9100 --purge`, `shred -u` the token/script/log **after** copying the log out
   for evidence, `poweroff`, wait for qemu to exit, `qemu-img snapshot -a virgin`. Token-leak grep on
   the saved log = `grep -c -F "$(cat ~/.gitea-token)"` (the literal value — a broad `[a-f0-9]{40}`
   pattern false-hits image shas). **A `0` is only evidence once the grep is shown to work**: append
   the token to a throwaway copy of the log, grep that (must be `1`), `shred -u` the copy, and only
   then believe the `0`. Grep the copy that gets **committed**, not just the one in the VM.
5. Hub → Configuration → Day-0 artifacts: pick the new Golden, Save. **The R-120 gate lives on this
   save** (`hub/internal/web/configs.go:1165`) and REFUSES a golden older than the newest controller
   the fleet reports. It does **not** run on a controller image deploy — it is not a general drift net.

   **Vouching is a THREE-field change, not one.** `golden_version` alone ships a controller onto an
   older agent than it declares it needs. Read the golden's controller `CHANGELOG.md` header — it
   carries `MinAgent <X>` — and move all three together:
   `golden_version` → the new golden, `agent_version` → ≥ that `MinAgent`, `min_agent` → that
   `MinAgent`. The `min_agent` field is what the hub HOLDS a box's floor against; blank means an
   uncoupled release with no gating. Setting `min_agent` **above** the vouched agent is the R-216
   shape and hub v0.97.0 now HOLDS it rather than serving past it.

   **Vouching is reversible**: re-select the previous values and Save. The old golden's package is
   never deleted by a bake (the publish step's pre-delete targets only its own version), so rolling
   back is a form submission, not a rebuild.

A golden is only needed when a publish train wants fresh installs current — demo deploys never need it.
**Since 2026-09-13 the cadence is §4.2: weekly, and before any drill or fresh install.**
The full 0.188.0 run, with the observables: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.

### 4.1a Registry clean-up — a person's act (`09` §3 decision 62)

When the Gitea volume fills, `~/git/misc-scripts/gitea-image-prune.sh` (repo `admin/misc-scripts`) prunes old versions.
**Dry-run first, always** (`./gitea-image-prune.sh --all prune`, then `--type generic --all prune`): it keeps the newest
20 of each package plus every version in use — the floor, the vouched golden, the vouched agent and `min_agent` (read
from the hub with `HUB_PW`), the images the vouched golden baked (its `bake.log` here), the hub `manifests/hub.yaml`
runs — and prints each with its reason. It refuses when that list cannot be read. **`--apply` is a person's act**,
after reading the dry-run; nothing schedules it. The August 2026 `--keep 7` run (HM-024) predates the rule.

### 4.2 Cadence — goldens are WEEKLY and before any install, not per release (operator ruling 2026-09-13)

**What was measured before the ruling:** 25 goldens in 26 days in August, almost one per release,
because `golden_currency_gate.py` trips on every release by design and the only honest ways past it
were a bake or a declared `--no-verify` (thirteen of those by 2026-09-01 — R-404/R-417). **The
ruling:** bake on a cadence. The ruling assumed every release would still raise the FLOOR (§4.1 step
5) and reach both demo boxes in ~20 s, with only the golden moving to a cadence. **⚠ CORRECTED THE SAME DAY (R-472): the hub held any floor above the vouched golden (publish-train rule 1), so releases between bakes reached the demo boxes only by hand-deploy.** **RESOLVED in hub v0.112.0:** a floor above the golden is served when it carries a
declared MinAgent (§3 "Raise the floor to a release with no golden"). A release between bakes rides
the floor again; only an undeclared floor is still held.

**The cadence, and it is a step in a routine, not a memory:**

1. **Weekly.** Bake + vouch + raise the floor per §4.1, one golden carrying whatever is newest.
2. **Before ANY drill or fresh install**, whether or not the week is up. A drill on a fresh install
   with a stale golden measures the wrong controller (that is how R-120 surfaced).
3. **The first external install retires the arrangement.** The waiver's own register row says so.

**The gate reads a dated waiver** — `documentation/tests/golden-waiver.yml`, four lines:

```yaml
# documentation/tests/golden-waiver.yml — read by scripts/golden_currency_gate.py
issued: 2026-09-13
expires: 2026-09-27          # HARD LIMIT: at most 14 days after issued, or the gate refuses the waiver
reason: pre-customer development; goldens on a weekly cadence (operator ruling 2026-09-13)
register_row: R-468          # must exist as a `**R-468**` row in OPEN-ITEMS.md
```

While it is valid, a golden BEHIND the record makes the gate print a loud **ADVISORY** and exit 0;
when it expires the gate is **red again** until someone bakes or renews. **It never covers a golden
that is UNRECORDED** (no `## vX.Y.Z` heading — R-385): that is not a cadence choice. A waiver longer
than 14 days, with a missing or unparseable date, an empty reason, or a row that does not exist, is
**INCONCLUSIVE (exit 2)** — refused, and refused out loud. The cap lives in the gate
(`WAIVER_MAX_DAYS`), not here, because a cap in prose is what failed the first time (R-242).

**To issue or renew:** edit the two dates (issue = today, expiry ≤ 14 days later), keep the row,
commit it **on its own** with a message saying why the bake is deferred, and push. Renewal is a diff
someone can see — that is the point. **Do not issue a waiver over a bake that could be done today**;
that is the habit the gate's docstring warns about. **To retire it:** delete the file in the commit
that records the first external install.

**What the waiver does NOT do:** it does not touch the vouch half of R-242 (still open — nothing
gates the vouch), it does not change what the gate checks, and it does not stop the floor.

## 5. Hub (felhom.eu/hub → k3s, GitOps)

**The manifest is the truth** — a built image deploys NOTHING until `manifests/hub.yaml`'s `image:`
tag changes in git and the ArgoCD app is deliberately synced (auto-sync is OFF; never
`kubectl set image`, never `:latest`).

```bash
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# edit manifests/hub.yaml image tag → <VER>; commit; push
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"op"},"sync":{"syncStrategy":{"apply":{}}}}}'
# verify: Synced/Healthy + rollout + live image tag + logs
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10
```

## 6. "What's live right now?" one-liners

```bash
ssh felhom-pve "/usr/local/bin/felhom-agent --version"                                    # agent on demo
ssh felhom-pve "pct exec 9201 -- cat /etc/felhom-controller-image"                        # controller on demo
sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'  # hub
# published = the hub Configuration dropdowns (they read Gitea packages live)
# fleet = hub Dashboard per-host rows (agent_version + controller version per box)
```

## 7. State snapshot as of 2026-07-11 (so the screens make sense)

| Artifact | Live on demo | Published / vouched | Peti |
|---|---|---|---|
| agent | 0.84.0 | **0.81.0** | 0.81.0 |
| controller | 0.117.0 | golden **0.113.0**, floor 0.113.0 (DB = env, aligned) | 0.113.0 |
| hub | 0.46.0 | n/a (central) | n/a |

The gap between columns 1 and 2 is the pending publish train (agent 0.84 + golden/floor 0.117 +
MinAgent 0.81 + Peti's journal one-liner + temp-creds deletion) — its runbook follows the
0.81/0.113 pattern with these numbers.

## 8. Iron rules (recap)

Never `:latest`; never republish over an existing package version (404 pre-gate); never
`kubectl set image` / bare `kubectl apply` against GitOps surfaces; manifest before floor, floor
LAST; one sha in three places, byte-identical; secrets never in transcripts or commits; an unpushed
change does not exist.