Files

T

gates / gates (push) Successful in 6m33s

Details

CC uses its hub token: live C3 proof (floor no-op 303 floor_set, vouch 403, session-only 403); R-1051 closed; skill floor step + prune script on the token; STATUS: change the hub password

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb

2026-10-15 13:10:00 +02:00

22 KiB

Raw Blame History

RUNBOOK — manual build / deploy / publish (agent · controller · golden · hub)

Audience: the operator, on DooPlex (192.168.0.180, Debian 13) — the same environment CC uses. Builds are local commands; felhom-pve is one ssh hop. Written: 2026-07-11, from the verified felhom.eu/skills/felhom-build-deploy/SKILL.md command set

the publish-train procedure (documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md) + documentation/runbooks/publish-train-rules.md. If this doc and the skill ever disagree, the skill wins.

There is no CI — nothing builds on push, ever. Every artifact moves through two separate, deliberate verbs:

Verb	Means	Who sees it
DEPLOY	build + install on the DEMO (felhom-pve / guest 9201 / k3s)	only the demo box
PUBLISH	upload the artifact to Gitea packages + vouch it in the hub Day-0 manifest	Day-0 installs, the manifest dropdowns, and the remote-rollout machinery (signed agent ops, controller floor)

A version can be live-on-demo for days without being published (agent 0.82–0.84 right now). The hub Configuration dropdowns list published artifacts only — that screen showing 0.81.0/0.113.0 is correct, not stale.

Remote rollout to customer boxes is a third, separate step and has its own runbook pattern: agent = operator-signed agent_update op (per box); controller = the global floor (DB row — acts immediately; save it LAST). Rules: publish-train-rules.md.

1. Session setup (every session, first)

FELHOM_ROOT=/mnt/5_hdd/felhom.eu   # working root: ALL felhom repos/build/drill/iso live HERE

Clean-tree gate before any build: git status --porcelain must be empty and git rev-parse HEAD must equal git rev-parse origin/main in the repo being built. An unpushed change does not exist.

Host	Access	Role
DooPlex (this host)	local	builds (`$FELHOM_ROOT/git/felhom-agent`, `$FELHOM_ROOT/build/felhom-controller`, `$FELHOM_ROOT/build/felhom-hub`), kubectl
Demo PVE host	`ssh felhom-pve` (root@192.168.0.162)	agent install, `pct exec 9201`
Hub UI	hub.felhom.eu → Configuration	manifest vouch, MinAgent, floor (operator password)

Housekeeping note: $FELHOM_ROOT/build/felhom-agent on 180 is a stale pre-June-23 leftover — agent builds live in $FELHOM_ROOT/git/felhom-agent now. Safe to remove the old dir.

2. Agent (felhom-agent binary → felhom-pve, then optionally publish)

Always commit+push to main first (an unpushed change does not exist).

# BUILD on 180 (the explicit git pull is load-bearing)
cd $FELHOM_ROOT/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent

# PUSH to the PVE host — ONE hop, the binary is already local
scp /tmp/felhom-agent-<VER> felhom-pve:/tmp/

# DEPLOY with backup + restart + verify
ssh felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"

Gotchas (earned): if configs/ changed in the repo, ship the sudoers + guarded wrapper WITH the binary (several Go guards only exist when the deployed configs match); beware CRLF when scp-ing configs through Windows; after restart the journal must show a clean ReassertGuestBinds and (since 0.84) the network-mount reassert, with no capability degradations.

PUBLISH (makes it real for the fleet):

# from the agent repo, with REGISTRY_* creds set; use the LIVE-DEPLOYED bytes, sha-verified across hops
scripts/publish-agent.sh <VER> <path-to-binary>

Pre-gate: the package GET for <VER> must be 404 before (published artifacts are immutable — never republish over an existing version). The script prints the sha256 — record it: the same sha goes into the hub manifest AND any signed agent_update op. One sha, three places, byte-identical.

Then hub → Configuration → Day-0 artifacts: pick the new Agent version (sha auto-read from Gitea), set Min agent if the paired controller release declares MinAgent: in its CHANGELOG, Save artifact manifest. (This save does NOT move the floor — that's a separate card since hub v0.45.)

3. Controller image (felhom-controller → guest 9201, remote via floor)

9201 is golden/bootstrap-managed — no compose file. The bootstrap service runs whatever tag is in /etc/felhom-controller-image (anonymous pull).

# BUILD+PUSH the image (build.sh does NOT pull — the explicit pull is load-bearing)
cd $FELHOM_ROOT/build/felhom-controller && git -C $FELHOM_ROOT/git/felhom-controller pull && ./build.sh <VER> --push

# DEPLOY on the demo guest
ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"

# VERIFY
ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"

Remote rollout = the hub floor (Configuration → Managed updates). Order rules (train rules doc): the manifest vouches the target FIRST; any MinAgent must be satisfied fleet-wide (the hub now HOLDS boxes below it automatically, and flags them); save the floor LAST — the DB row acts immediately on every box below it, on their next report.

Raise the floor to a release with no golden (hub v0.112.0+, R-472). This IS the release route (09 §3 decision 250, 2026-10-15: a controller release needs no golden; decision 247 of the same morning was reversed). Only after the release ran on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk is green on all three.

Build and push the controller image (above). Do not install it on any box.
Read the MinAgent from the release's header: grep -m1 -A3 '^## v<VER>' CHANGELOG.md shows **MinAgent: X.Y.Z**. controller_gates.py --fast refuses a release whose newest header lacks it.
Hub → Configuration → Managed updates: type <VER> in the floor field and that value in min_agent. Save. Without min_agent the form refuses with "This floor is above the vouched golden — declare its MinAgent" and stores nothing.
Verify: sudo kubectl -n felhom-system logs deploy/hub | grep 'managed floor SERVED' shows from "declared", and each box logs SetFloor: floor "…" → "<VER>" within about a minute.

3.1 Park the controller (stop the agent restarting it) — agent ≥ 0.131.0

The agent restarts a stopped felhom-controller within about a minute (R-523). To keep it stopped on purpose, on the Proxmox host: touch /var/lib/felhom-agent/guests/<vmid>/controller-parked. The agent logs controller is not running and the guest is PARKED — leaving it. To unpark: rm /var/lib/felhom-agent/guests/<vmid>/controller-parked. A controller swap is never fought either.

4. Golden image (fresh Day-0 installs)

The golden is a pre-baked controller-era guest image built in the drill VM on 180 (/mnt/5_hdd/felhom.eu/drill/drill.qcow2, internal snapshot virgin).

4.0 The canonical drill-VM launch — captured from a real bake, not reconstructed

Until 2026-07-31 this section told the reader to "use the RECORDED qemu launch line" from RUNBOOK-publish-0.79-0.110-2026-07-10.md Phase C — while that line is itself labelled "PASS (reconstructed — DEVIATION)" and its own §Deviations says "the canonical qemu-system one-liner was never saved". The document forbade improvising and pointed at an improvisation. The lines below were captured verbatim from the 0.188.0 bake on 2026-07-31 and are now the canonical invocation. Re-capture them (do not retype from memory) if the bake host or disk layout ever changes.

# 1. revert the disk to virgin (non-destructive to the snapshot; succeeding also proves no qemu holds the qcow2)
qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2

# 2. COLD-boot it (the snapshot is disk-only, 0 B VM_SIZE — never -loadvm)
qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
  -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \
  -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \
  -device virtio-net-pci,netdev=n0 -display none -daemonize \
  -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid

if=virtio is load-bearing (the guest expects /dev/vda). SSH answers on :2222 in ~40 s (ssh -i /mnt/5_hdd/felhom.eu/drill/id_drill -p 2222 root@localhost); pveversion read pve-manager/9.2.2 on 2026-07-31. Liveness check: ps -eo comm | grep qemu-system-x86 — pgrep -f qemu-system-x86_64 self-matches your own command line and reports a false "still running".

4.1 Bake + publish

Revert + boot per §4.0.
Run pveam update first — the virgin snapshot's template INDEX is stale too, and a stale index fails as a bogus 400 no such template (0.289.1 bake, 2026-10-03). The debian template is absent on virgin and the exact point release rots — list the current one (pveam available --section system | grep 'debian-13-standard_.*_amd64') and pveam download local <that>. Filter on _amd64: the index also lists an _arm64 build of the same point release, and a version sort picks it (2026-09-28: the 0.276.0 bake's first attempt did, and was stopped before pct create finished; the 2026-09-26 bench LXC did too and would not start). It was debian-13-standard_13.6-1_amd64.tar.zst on 2026-07-31.
scp in build-golden.sh (agent repo configs/) + the Gitea token (~/.gitea-token on 180, 0600), then run it as a transient unit so it survives a session close, reading the token from the file inside the VM so it never reaches a command line:

Put the invocation in a runner script inside the VM that reads the token itself, and launch that. The older --setenv=GITEA_TOKEN=$GT form put the value on a command line and into the transient unit's properties, where systemctl show prints it:
```
cat > /root/bake-run.sh <<'EOS'
#!/bin/bash
GT=$(cat /root/.gitea-token)
export GOLDEN_DOCKER_PKGS="<the approved Docker set: all six pkg=version pairs>"
export GITEA_USER=admin GITEA_TOKEN="$GT" REGISTRY_USER=admin REGISTRY_TOKEN="$GT"
exec /root/build-golden.sh 9100 local:vztmpl/<template> local-lvm local vmbr0 \
  gitea.dooplex.hu/admin/felhom-controller:<VER>
EOS
chmod 0700 /root/bake-run.sh
systemd-run --unit=golden-bake --collect bash -c "/root/bake-run.sh > /root/bake.log 2>&1"
```
GOLDEN_DOCKER_PKGS is not optional. Without it the script installs the NEWEST stable Docker, not the approved one, and says so only as a WARNING line in bake.log (2026-10-06, the 0.298.0 bake: missed, re-baked). Copy the six pairs from the previous bake's Docker engine set PINNED to the approved release: line, or from the hub's newest approved Docker release (System page). Pass marker: that PINNED line, and no GOLDEN_DOCKER_PKGS not set.

Confirm the value went nowhere: systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "$(cat /root/.gitea-token)" must be 0. Copy the token in file → file (scp), so it never crosses a shell on either side.

CONTROLLER_IMAGE is a required argument (a hand-bumped default rotted twice) and GOLDEN_VERSION is derived from it — the golden's version IS the controller it bakes. The script is the publisher: it uploads to Gitea and prints GOLDEN_VERSION + GOLDEN_SHA256. Pass markers, each captured from a real log rather than paraphrased — two of the three named here until 2026-08-06 could not match anything the script prints (see the note below): docker OK (overlay2, including mount point for rootfs and mp0 — there is no mp1 — with no excluding/FATAL, upload OK (HTTP 201). The 404 pre-gate applies to the package URL, which is …/generic/felhom-golden/<VER>/golden.tar.zst — the filename is golden.tar.zst, not felhom-golden-<VER>.tar.zst.

Teardown: pct destroy 9100 --purge, shred -u the token/script/log after copying the log out for evidence, poweroff, wait for qemu to exit, qemu-img snapshot -a virgin. Token-leak grep on the saved log = grep -c -F "$(cat ~/.gitea-token)" (the literal value — a broad [a-f0-9]{40} pattern false-hits image shas). A 0 is only evidence once the grep is shown to work: append the token to a throwaway copy of the log, grep that (must be 1), shred -u the copy, and only then believe the 0. Grep the copy that gets committed, not just the one in the VM.
Hub → Configuration → Day-0 artifacts: pick the new Golden, Save. The R-120 gate lives on this save (hub/internal/web/configs.go:1165) and REFUSES a golden older than the newest controller the fleet reports. It does not run on a controller image deploy — it is not a general drift net.

Vouching is a THREE-field change, not one. golden_version alone ships a controller onto an older agent than it declares it needs. Read the golden's controller CHANGELOG.md header — it carries MinAgent <X> — and move all three together: golden_version → the new golden, agent_version → ≥ that MinAgent, min_agent → that MinAgent. The min_agent field is what the hub HOLDS a box's floor against; blank means an uncoupled release with no gating. Setting min_agent above the vouched agent is the R-216 shape and hub v0.97.0 now HOLDS it rather than serving past it.

Vouching is reversible: re-select the previous values and Save. The old golden's package is never deleted by a bake (the publish step's pre-delete targets only its own version), so rolling back is a form submission, not a rebuild.

A re-bake under the SAME version: re-vouch at once (R-857). The publish pre-deletes and re-uploads that version's package, so the hub keeps vouching the FIRST bake's sha, which the registry no longer holds — a fresh install in that window fails its sha check. Re-select the golden and Save right after the publish. Put the evidence in golden-<VER>-<DATE>-rebake/; golden_currency_gate.py reads the newest bake of a version (a later date, then the suffixed directory of the same date).

A golden is only needed when a publish train wants fresh installs current — demo deploys never need it. Since 2026-09-13 the cadence is §4.2: weekly, and before any drill or fresh install. The full 0.188.0 run, with the observables: documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md.

4.1a Registry clean-up — a person's act (`09` §3 decision 62)

When the Gitea volume fills, ~/git/misc-scripts/gitea-image-prune.sh (repo admin/misc-scripts) prunes old versions. Dry-run first, always (./gitea-image-prune.sh --all prune, then --type generic --all prune): it keeps the newest 20 of each package plus every version in use — the floor, the vouched golden, the vouched agent and min_agent (read from the hub with the scoped token HUB_TOKEN, floor:read), the images the vouched golden baked (its bake.log here), the hub manifests/hub.yaml runs — and prints each with its reason. It refuses when that list cannot be read. --apply is a person's act, after reading the dry-run; nothing schedules it. The August 2026 --keep 7 run (HM-024) predates the rule.

4.2 Cadence — goldens are WEEKLY and before any install, not per release (operator ruling 2026-09-13)

2026-10-15 (09 §3 decision 250): the weekly cadence below STANDS (the same morning's „bake first", decision 247, was reversed). The weekly bake is a Sunday-night step of nightly-rotation.md; the push gate judges the golden by its AGE (7 days) while the floor is above it, not by the newest CHANGELOG heading.

What was measured before the ruling: 25 goldens in 26 days in August, almost one per release, because golden_currency_gate.py trips on every release by design and the only honest ways past it were a bake or a declared --no-verify (thirteen of those by 2026-09-01 — R-404/R-417). The ruling: bake on a cadence. The ruling assumed every release would still raise the FLOOR (§4.1 step 5) and reach both demo boxes in ~20 s, with only the golden moving to a cadence. ⚠ CORRECTED THE SAME DAY (R-472): the hub held any floor above the vouched golden (publish-train rule 1), so releases between bakes reached the demo boxes only by hand-deploy. RESOLVED in hub v0.112.0: a floor above the golden is served when it carries a declared MinAgent (§3 "Raise the floor to a release with no golden"). A release between bakes rides the floor again; only an undeclared floor is still held.

The cadence, and it is a step in a routine, not a memory:

Weekly. Bake + vouch + raise the floor per §4.1, one golden carrying whatever is newest.
Before ANY drill or fresh install, whether or not the week is up. A drill on a fresh install with a stale golden measures the wrong controller (that is how R-120 surfaced).
The first external install retires the arrangement. The waiver's own register row says so.

The gate reads a dated waiver — documentation/tests/golden-waiver.yml, four lines:

# documentation/tests/golden-waiver.yml — read by scripts/golden_currency_gate.py
issued: 2026-09-13
expires: 2026-09-27          # HARD LIMIT: at most 14 days after issued, or the gate refuses the waiver
reason: pre-customer development; goldens on a weekly cadence (operator ruling 2026-09-13)
register_row: R-468          # must exist as a `**R-468**` row in OPEN-ITEMS.md

While it is valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. It never covers a golden that is UNRECORDED (no ## vX.Y.Z heading — R-385): that is not a cadence choice. A waiver longer than 14 days, with a missing or unparseable date, an empty reason, or a row that does not exist, is INCONCLUSIVE (exit 2) — refused, and refused out loud. The cap lives in the gate (WAIVER_MAX_DAYS), not here, because a cap in prose is what failed the first time (R-242).

To issue or renew: edit the two dates (issue = today, expiry ≤ 14 days later), keep the row, commit it on its own with a message saying why the bake is deferred, and push. Renewal is a diff someone can see — that is the point. Do not issue a waiver over a bake that could be done today; that is the habit the gate's docstring warns about. To retire it: delete the file in the commit that records the first external install.

What the waiver does NOT do: it does not touch the vouch half of R-242 (still open — nothing gates the vouch), it does not change what the gate checks, and it does not stop the floor.

5. Hub (felhom.eu/hub → k3s, GitOps)

The manifest is the truth — a built image deploys NOTHING until manifests/hub.yaml's image: tag changes in git and the ArgoCD app is deliberately synced (auto-sync is OFF; never kubectl set image, never :latest).

cd $FELHOM_ROOT/build/felhom-hub && ./build.sh <VER> --push
# edit manifests/hub.yaml image tag → <VER>; commit; push
sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{"\n"}'
sudo kubectl -n argocd patch application felhom --type merge -p '{"operation":{"initiatedBy":{"username":"op"},"sync":{"syncStrategy":{"apply":{}}}}}'
# verify: Synced/Healthy + rollout + live image tag + logs
sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10

6. "What's live right now?" one-liners

ssh felhom-pve "/usr/local/bin/felhom-agent --version"                                    # agent on demo
ssh felhom-pve "pct exec 9201 -- cat /etc/felhom-controller-image"                        # controller on demo
sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'  # hub
# published = the hub Configuration dropdowns (they read Gitea packages live)
# fleet = hub Dashboard per-host rows (agent_version + controller version per box)

7. State snapshot as of 2026-07-11 (so the screens make sense)

Artifact	Live on demo	Published / vouched	Peti
agent	0.84.0	0.81.0	0.81.0
controller	0.117.0	golden 0.113.0, floor 0.113.0 (DB = env, aligned)	0.113.0
hub	0.46.0	n/a (central)	n/a

The gap between columns 1 and 2 is the pending publish train (agent 0.84 + golden/floor 0.117 + MinAgent 0.81 + Peti's journal one-liner + temp-creds deletion) — its runbook follows the 0.81/0.113 pattern with these numbers.

8. Iron rules (recap)

Never :latest; never republish over an existing package version (404 pre-gate); never kubectl set image / bare kubectl apply against GitOps surfaces; manifest before floor, floor LAST; one sha in three places, byte-identical; secrets never in transcripts or commits; an unpushed change does not exist.

22 KiB Raw Blame History Unescape Escape