# RUNBOOK — manual build / deploy / publish (agent · controller · golden · hub)

**Audience:** the operator, on DooPlex (192.168.0.180, Debian 13) — the same environment CC uses.
Builds are local commands; felhom-pve is one `ssh` hop.
**Written:** 2026-07-11, from the verified `felhom.eu/skills/felhom-build-deploy/SKILL.md` command set
+ the publish-train procedure (`documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md`) +
`documentation/runbooks/publish-train-rules.md`. If this doc and the skill ever disagree, the skill wins.

---

## 0. The mental model (why the hub dropdown "lags")

There is **no CI** — nothing builds on push, ever. Every artifact moves through two separate,
deliberate verbs:

| Verb | Means | Who sees it |
|---|---|---|
| **DEPLOY** | build + install on the DEMO (felhom-pve / guest 9201 / k3s) | only the demo box |
| **PUBLISH** | upload the artifact to Gitea packages + vouch it in the hub Day-0 manifest | Day-0 installs, the manifest dropdowns, and the remote-rollout machinery (signed agent ops, controller floor) |

A version can be live-on-demo for days without being published (agent 0.82–0.84 right now). The hub
Configuration dropdowns list **published** artifacts only — that screen showing 0.81.0/0.113.0 is
correct, not stale.

Remote rollout to customer boxes is a third, separate step and has its own runbook pattern:
**agent** = operator-signed `agent_update` op (per box); **controller** = the global floor (DB row —
acts immediately; save it LAST). Rules: `publish-train-rules.md`.

## 1. Session setup (every session, first)

```bash
FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root: ALL felhom repos/build/drill/iso live HERE
```

**Clean-tree gate before any build:** `git status --porcelain` must be empty and `git rev-parse HEAD`
must equal `git rev-parse origin/main` in the repo being built. An unpushed change does not exist.

| Host | Access | Role |
|---|---|---|
| DooPlex (this host) | local | builds (`$FELHOM_ROOT/git/felhom-agent`, `$FELHOM_ROOT/build/felhom-controller`, `$FELHOM_ROOT/build/felhom-hub`), kubectl |
| Demo PVE host | `ssh felhom-pve` (root@192.168.0.162) | agent install, `pct exec 9201` |
| Hub UI | hub.felhom.eu → Configuration | manifest vouch, MinAgent, floor (operator password) |

Housekeeping note: `$FELHOM_ROOT/build/felhom-agent` on 180 is a stale pre-June-23 leftover — agent
builds live in `$FELHOM_ROOT/git/felhom-agent` now. Safe to remove the old dir.

## 2. Agent (felhom-agent binary → felhom-pve, then optionally publish)

Always commit+push to `main` first (an unpushed change does not exist).

```bash
# BUILD on 180 (the explicit git pull is load-bearing)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent

# PUSH to the PVE host — ONE hop, the binary is already local
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/

# DEPLOY with backup + restart + verify
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"
```

Gotchas (earned): if `configs/` changed in the repo, ship the **sudoers + guarded wrapper WITH the
binary** (several Go guards only exist when the deployed configs match); beware CRLF when scp-ing
configs through Windows; after restart the journal must show a clean `ReassertGuestBinds` and (since
0.84) the network-mount reassert, with no capability degradations.

**PUBLISH (makes it real for the fleet):**
```bash
# from the agent repo, with REGISTRY_* creds set; use the LIVE-DEPLOYED bytes, sha-verified across hops
scripts/publish-agent.sh <VER> <path-to-binary>
```
Pre-gate: the package GET for `<VER>` must be **404** before (published artifacts are immutable —
never republish over an existing version). The script prints the **sha256 — record it**: the same sha
goes into the hub manifest AND any signed `agent_update` op. One sha, three places, byte-identical.

Then hub → Configuration → Day-0 artifacts: pick the new Agent version (sha auto-read from Gitea),
set **Min agent** if the paired controller release declares `MinAgent:` in its CHANGELOG, **Save
artifact manifest**. (This save does NOT move the floor — that's a separate card since hub v0.45.)

## 3. Controller image (felhom-controller → guest 9201, remote via floor)

9201 is golden/bootstrap-managed — no compose file. The bootstrap service runs whatever tag is in
`/etc/felhom-controller-image` (anonymous pull).

```bash
# BUILD+PUSH the image (build.sh does NOT pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push

# DEPLOY on the demo guest
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"

# VERIFY
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
```

**Remote rollout** = the hub floor (Configuration → Managed updates). Order rules (train rules doc):
the manifest vouches the target FIRST; any MinAgent must be satisfied fleet-wide (the hub now HOLDS
boxes below it automatically, and flags them); **save the floor LAST** — the DB row acts immediately
on every box below it, on their next report.

**Raise the floor to a release with no golden (hub v0.112.0+, R-472).** **This IS the release route** (`09` §3 decision
250, 2026-10-15: a controller release needs no golden; decision 247 of the same morning was reversed). Only after the
release ran on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk is green on all three.

1. Build and push the controller image (above). Do not install it on any box.
2. Read the MinAgent from the release's header: `grep -m1 -A3 '^## v<VER>' CHANGELOG.md` shows
   `**MinAgent: X.Y.Z**`. `controller_gates.py --fast` refuses a release whose newest header lacks it.
3. Hub → Configuration → Managed updates: type `<VER>` in the floor field **and** that value in
   `min_agent`. Save. Without `min_agent` the form refuses with *"This floor is above the vouched
   golden — declare its MinAgent"* and stores nothing.
4. Verify: `sudo kubectl -n felhom-system logs deploy/hub | grep 'managed floor SERVED'` shows
   `from "declared"`, and each box logs `SetFloor: floor "…" → "<VER>"` within about a minute.

### 3.1 Park the controller (stop the agent restarting it) — agent ≥ 0.131.0

The agent restarts a stopped `felhom-controller` within about a minute (R-523). To keep it stopped on
purpose, on the **Proxmox host**: `touch /var/lib/felhom-agent/guests/<vmid>/controller-parked`. The
agent logs `controller is not running and the guest is PARKED — leaving it`. To unpark:
`rm /var/lib/felhom-agent/guests/<vmid>/controller-parked`. A controller swap is never fought either.

## 4. Golden image (fresh Day-0 installs)

The golden is a pre-baked controller-era guest image built in the **drill VM** on 180
(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`).

### 4.0 The canonical drill-VM launch — captured from a real bake, not reconstructed

Until 2026-07-31 this section told the reader to "use the RECORDED qemu launch line" from
`RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — while that line is itself labelled
**"PASS (reconstructed — DEVIATION)"** and its own §Deviations says *"the canonical `qemu-system`
one-liner was **never saved**"*. The document forbade improvising and pointed at an improvisation. The
lines below were **captured verbatim from the 0.188.0 bake on 2026-07-31** and are now the canonical
invocation. Re-capture them (do not retype from memory) if the bake host or disk layout ever changes.

```bash
# 1. revert the disk to virgin (non-destructive to the snapshot; succeeding also proves no qemu holds the qcow2)
qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2

# 2. COLD-boot it (the snapshot is disk-only, 0 B VM_SIZE — never -loadvm)
qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
  -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \
  -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \
  -device virtio-net-pci,netdev=n0 -display none -daemonize \
  -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid
```

`if=virtio` is load-bearing (the guest expects `/dev/vda`). SSH answers on `:2222` in ~40 s
(`ssh -i /mnt/5_hdd/felhom.eu/drill/id_drill -p 2222 root@localhost`); `pveversion` read
`pve-manager/9.2.2` on 2026-07-31. **Liveness check:** `ps -eo comm | grep qemu-system-x86` —
`pgrep -f qemu-system-x86_64` self-matches your own command line and reports a false "still running".

### 4.1 Bake + publish

1. Revert + boot per §4.0.
2. Run **`pveam update` first** — the `virgin` snapshot's template INDEX is stale too, and a stale index fails as a bogus
   `400 no such template` (0.289.1 bake, 2026-10-03). The debian template is **absent on `virgin`** and **the exact point release rots** — list the
   current one (`pveam available --section system | grep 'debian-13-standard_.*_amd64'`) and `pveam download local <that>`.
   **Filter on `_amd64`:** the index also lists an `_arm64` build of the same point release, and a version sort
   picks it (2026-09-28: the 0.276.0 bake's first attempt did, and was stopped before `pct create` finished; the
   2026-09-26 bench LXC did too and would not start).
   It was `debian-13-standard_13.6-1_amd64.tar.zst` on 2026-07-31.
3. `scp` in `build-golden.sh` (agent repo `configs/`) + the Gitea token (`~/.gitea-token` on 180,
   0600), then run it as a transient unit so it survives a session close, reading the token from the
   file **inside** the VM so it never reaches a command line:

   Put the invocation in a **runner script inside the VM** that reads the token itself, and launch
   that. The older `--setenv=GITEA_TOKEN=$GT` form put the value on a command line and into the
   transient unit's properties, where `systemctl show` prints it:

   ```bash
   cat > /root/bake-run.sh <<'EOS'
   #!/bin/bash
   GT=$(cat /root/.gitea-token)
   export GOLDEN_DOCKER_PKGS="<the approved Docker set: all six pkg=version pairs>"
   export GITEA_USER=admin GITEA_TOKEN="$GT" REGISTRY_USER=admin REGISTRY_TOKEN="$GT"
   exec /root/build-golden.sh 9100 local:vztmpl/<template> local-lvm local vmbr0 \
     gitea.dooplex.hu/admin/felhom-controller:<VER>
   EOS
   chmod 0700 /root/bake-run.sh
   systemd-run --unit=golden-bake --collect bash -c "/root/bake-run.sh > /root/bake.log 2>&1"
   ```

   **`GOLDEN_DOCKER_PKGS` is not optional.** Without it the script installs the NEWEST stable Docker, not the approved
   one, and says so only as a `WARNING` line in `bake.log` (2026-10-06, the 0.298.0 bake: missed, re-baked). Copy the
   six pairs from the previous bake's `Docker engine set PINNED to the approved release:` line, or from the hub's newest
   approved Docker release (System page). Pass marker: that `PINNED` line, and no `GOLDEN_DOCKER_PKGS not set`.

   Confirm the value went nowhere: `systemctl show golden-bake -p Environment -p ExecStart |
   grep -c -F "$(cat /root/.gitea-token)"` must be `0`. Copy the token in **file → file** (`scp`), so
   it never crosses a shell on either side.

   `CONTROLLER_IMAGE` is a **required** argument (a hand-bumped default rotted twice) and
   `GOLDEN_VERSION` is derived from it — the golden's version IS the controller it bakes.
   **The script is the publisher**: it uploads to Gitea and prints `GOLDEN_VERSION` + `GOLDEN_SHA256`.
   Pass markers, **each captured from a real log rather than paraphrased** — two of the three named
   here until 2026-08-06 could not match anything the script prints (see the note below):
   `docker OK (overlay2`, `including mount point` for rootfs **and mp0** — there is no mp1 — with no
   `excluding`/`FATAL`, `upload OK (HTTP 201)`. The 404 pre-gate applies to the package URL, which is
   `…/generic/felhom-golden/<VER>/golden.tar.zst` — the filename is `golden.tar.zst`, **not**
   `felhom-golden-<VER>.tar.zst`.

<!--
2026-08-06, R-233: this line listed `overlay2 OK` and demanded `including mount point` for `mp1`.
The script prints neither. The storage-driver line it means is `  docker OK (overlay2; data-root
/var/lib/docker)`, and mp1 stopped existing in build-golden.sh v3.0.0 (R-165 collapsed the two data
volumes into one). A reader following this literally greps for a string that can never appear and
reads 0 — the "an instrument that can silently drop results is not a measurement" class, aimed at the
bake's own acceptance check. The real guard was never weak: the script's own
`[ "$drv" = "overlay2" ] || { echo FATAL; exit 1; }` is fail-closed. The DOCUMENT was the broken part.
-->

4. Teardown: `pct destroy 9100 --purge`, `shred -u` the token/script/log **after** copying the log out
   for evidence, `poweroff`, wait for qemu to exit, `qemu-img snapshot -a virgin`. Token-leak grep on
   the saved log = `grep -c -F "$(cat ~/.gitea-token)"` (the literal value — a broad `[a-f0-9]{40}`
   pattern false-hits image shas). **A `0` is only evidence once the grep is shown to work**: append
   the token to a throwaway copy of the log, grep that (must be `1`), `shred -u` the copy, and only
   then believe the `0`. Grep the copy that gets **committed**, not just the one in the VM.
5. Hub → Configuration → Day-0 artifacts: pick the new Golden, Save. **The R-120 gate lives on this
   save** (`hub/internal/web/configs.go:1165`) and REFUSES a golden older than the newest controller
   the fleet reports. It does **not** run on a controller image deploy — it is not a general drift net.

   **Vouching is a THREE-field change, not one.** `golden_version` alone ships a controller onto an
   older agent than it declares it needs. Read the golden's controller `CHANGELOG.md` header — it
   carries `MinAgent <X>` — and move all three together:
   `golden_version` → the new golden, `agent_version` → ≥ that `MinAgent`, `min_agent` → that
   `MinAgent`. The `min_agent` field is what the hub HOLDS a box's floor against; blank means an
   uncoupled release with no gating. Setting `min_agent` **above** the vouched agent is the R-216
   shape and hub v0.97.0 now HOLDS it rather than serving past it.

   **Vouching is reversible**: re-select the previous values and Save. The old golden's package is
   never deleted by a bake (the publish step's pre-delete targets only its own version), so rolling
   back is a form submission, not a rebuild.

   **A re-bake under the SAME version: re-vouch at once (R-857).** The publish pre-deletes and re-uploads
   that version's package, so the hub keeps vouching the FIRST bake's sha, which the registry no longer
   holds — a fresh install in that window fails its sha check. Re-select the golden and Save right after
   the publish. Put the evidence in `golden-<VER>-<DATE>-rebake/`; `golden_currency_gate.py` reads the
   newest bake of a version (a later date, then the suffixed directory of the same date).

A golden is only needed when a publish train wants fresh installs current — demo deploys never need it.
**Since 2026-09-13 the cadence is §4.2: weekly, and before any drill or fresh install.**
The full 0.188.0 run, with the observables: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.

### 4.1a Registry clean-up — a person's act (`09` §3 decision 62)

When the Gitea volume fills, `~/git/misc-scripts/gitea-image-prune.sh` (repo `admin/misc-scripts`) prunes old versions.
**Dry-run first, always** (`./gitea-image-prune.sh --all prune`, then `--type generic --all prune`): it keeps the newest
20 of each package plus every version in use — the floor, the vouched golden, the vouched agent and `min_agent` (read
from the hub with the scoped token `HUB_TOKEN`, `floor:read`), the images the vouched golden baked (its `bake.log` here), the hub `manifests/hub.yaml`
runs — and prints each with its reason. It refuses when that list cannot be read. **`--apply` is a person's act**,
after reading the dry-run; nothing schedules it. The August 2026 `--keep 7` run (HM-024) predates the rule.

### 4.2 Cadence — goldens are WEEKLY and before any install, not per release (operator ruling 2026-09-13)

> **2026-10-15 (`09` §3 decision 250):** the weekly cadence below STANDS (the same morning's „bake first", decision 247,
> was reversed). The weekly bake is a Sunday-night step of `nightly-rotation.md`; the push gate judges the golden by its
> AGE (7 days) while the floor is above it, not by the newest CHANGELOG heading.

**What was measured before the ruling:** 25 goldens in 26 days in August, almost one per release,
because `golden_currency_gate.py` trips on every release by design and the only honest ways past it
were a bake or a declared `--no-verify` (thirteen of those by 2026-09-01 — R-404/R-417). **The
ruling:** bake on a cadence. The ruling assumed every release would still raise the FLOOR (§4.1 step
5) and reach both demo boxes in ~20 s, with only the golden moving to a cadence. **⚠ CORRECTED THE SAME DAY (R-472): the hub held any floor above the vouched golden (publish-train rule 1), so releases between bakes reached the demo boxes only by hand-deploy.** **RESOLVED in hub v0.112.0:** a floor above the golden is served when it carries a
declared MinAgent (§3 "Raise the floor to a release with no golden"). A release between bakes rides
the floor again; only an undeclared floor is still held.

**The cadence, and it is a step in a routine, not a memory:**

1. **Weekly.** Bake + vouch + raise the floor per §4.1, one golden carrying whatever is newest.
2. **Before ANY drill or fresh install**, whether or not the week is up. A drill on a fresh install
   with a stale golden measures the wrong controller (that is how R-120 surfaced).
3. **The first external install retires the arrangement.** The waiver's own register row says so.

**The gate reads a dated waiver** — `documentation/tests/golden-waiver.yml`, four lines:

```yaml
# documentation/tests/golden-waiver.yml — read by scripts/golden_currency_gate.py
issued: 2026-09-13
expires: 2026-09-27          # HARD LIMIT: at most 14 days after issued, or the gate refuses the waiver
reason: pre-customer development; goldens on a weekly cadence (operator ruling 2026-09-13)
register_row: R-468          # must exist as a `**R-468**` row in OPEN-ITEMS.md
```

While it is valid, a golden BEHIND the record makes the gate print a loud **ADVISORY** and exit 0;
when it expires the gate is **red again** until someone bakes or renews. **It never covers a golden
that is UNRECORDED** (no `## vX.Y.Z` heading — R-385): that is not a cadence choice. A waiver longer
than 14 days, with a missing or unparseable date, an empty reason, or a row that does not exist, is
**INCONCLUSIVE (exit 2)** — refused, and refused out loud. The cap lives in the gate
(`WAIVER_MAX_DAYS`), not here, because a cap in prose is what failed the first time (R-242).

**To issue or renew:** edit the two dates (issue = today, expiry ≤ 14 days later), keep the row,
commit it **on its own** with a message saying why the bake is deferred, and push. Renewal is a diff
someone can see — that is the point. **Do not issue a waiver over a bake that could be done today**;
that is the habit the gate's docstring warns about. **To retire it:** delete the file in the commit
that records the first external install.

**What the waiver does NOT do:** it does not touch the vouch half of R-242 (still open — nothing
gates the vouch), it does not change what the gate checks, and it does not stop the floor.

## 5. Hub (felhom.eu/hub → k3s, GitOps)

**The manifest is the truth** — a built image deploys NOTHING until `manifests/hub.yaml`'s `image:`
tag changes in git and the ArgoCD app is deliberately synced (auto-sync is OFF; never
`kubectl set image`, never `:latest`).

```bash
cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# edit manifests/hub.yaml image tag → <VER>; commit; push
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"op"},"sync":{"syncStrategy":{"apply":{}}}}}'
# verify: Synced/Healthy + rollout + live image tag + logs
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10
```

## 6. "What's live right now?" one-liners

```bash
ssh felhom-pve "/usr/local/bin/felhom-agent --version"                                    # agent on demo
ssh felhom-pve "pct exec 9201 -- cat /etc/felhom-controller-image"                        # controller on demo
sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'  # hub
# published = the hub Configuration dropdowns (they read Gitea packages live)
# fleet = hub Dashboard per-host rows (agent_version + controller version per box)
```

## 7. State snapshot as of 2026-07-11 (so the screens make sense)

| Artifact | Live on demo | Published / vouched | Peti |
|---|---|---|---|
| agent | 0.84.0 | **0.81.0** | 0.81.0 |
| controller | 0.117.0 | golden **0.113.0**, floor 0.113.0 (DB = env, aligned) | 0.113.0 |
| hub | 0.46.0 | n/a (central) | n/a |

The gap between columns 1 and 2 is the pending publish train (agent 0.84 + golden/floor 0.117 +
MinAgent 0.81 + Peti's journal one-liner + temp-creds deletion) — its runbook follows the
0.81/0.113 pattern with these numbers.

## 8. Iron rules (recap)

Never `:latest`; never republish over an existing package version (404 pre-gate); never
`kubectl set image` / bare `kubectl apply` against GitOps surfaces; manifest before floor, floor
LAST; one sha in three places, byte-identical; secrets never in transcripts or commits; an unpushed
change does not exist.