build runbook

documentation/runbooks/RUNBOOK-manual-build.md

@@ -0,0 +1,157 @@
+# RUNBOOK — manual build / deploy / publish (agent · controller · golden · hub)
+
+**Audience:** the operator, from the Windows workstation (Git Bash), same environment CC uses.
+**Written:** 2026-07-11, from the verified `felhom.eu/skills/felhom-build-deploy/SKILL.md` command set
++ the publish-train procedure (`documentation/pilot/RUNBOOK-publish-0.79-0.110-2026-07-10.md`) +
+`documentation/runbooks/publish-train-rules.md`. If this doc and the skill ever disagree, the skill wins.
+
+---
+
+## 0. The mental model (why the hub dropdown "lags")
+
+There is **no CI** — nothing builds on push, ever. Every artifact moves through two separate,
+deliberate verbs:
+
+| Verb | Means | Who sees it |
+|---|---|---|
+| **DEPLOY** | build + install on the DEMO (felhom-pve / guest 9201 / k3s) | only the demo box |
+| **PUBLISH** | upload the artifact to Gitea packages + vouch it in the hub Day-0 manifest | Day-0 installs, the manifest dropdowns, and the remote-rollout machinery (signed agent ops, controller floor) |
+
+A version can be live-on-demo for days without being published (agent 0.82–0.84 right now). The hub
+Configuration dropdowns list **published** artifacts only — that screen showing 0.81.0/0.113.0 is
+correct, not stale.
+
+Remote rollout to customer boxes is a third, separate step and has its own runbook pattern:
+**agent** = operator-signed `agent_update` op (per box); **controller** = the global floor (DB row —
+acts immediately; save it LAST). Rules: `publish-train-rules.md`.
+
+## 1. Session setup (every session, first)
+
+```bash
+SSH=/c/Windows/System32/OpenSSH/ssh.exe   # Git Bash's own ssh fails silently against the Windows agent
+export MSYS_NO_PATHCONV=1                 # before any felhom-pve command with absolute paths (pct etc.)
+```
+
+| Host | Access | Role |
+|---|---|---|
+| Build server | `$SSH kisfenyo@192.168.0.180` | builds (`~/git/felhom-agent`, `~/build/felhom-controller`, `~/build/felhom-hub`), kubectl |
+| Demo PVE host | `$SSH felhom-pve` (root@192.168.0.162) | agent install, `pct exec 9201` |
+| Hub UI | hub.felhom.eu → Configuration | manifest vouch, MinAgent, floor (operator password) |
+
+Housekeeping note: `~/build/felhom-agent` on 180 is a stale pre-June-23 leftover — agent builds live
+in `~/git/felhom-agent` now. Safe to remove the old dir.
+
+## 2. Agent (felhom-agent binary → felhom-pve, then optionally publish)
+
+Always commit+push to `main` first (an unpushed change does not exist).
+
+```bash
+# BUILD on 180 (the explicit git pull is load-bearing)
+$SSH kisfenyo@192.168.0.180 "cd ~/git/felhom-agent && git pull && go build -ldflags '-X main.version=<VER>' -o /tmp/felhom-agent-<VER> ./cmd/felhom-agent"
+
+# FETCH locally, PUSH to the PVE host (Windows scp needs cygpath for the local side)
+scp kisfenyo@192.168.0.180:/tmp/felhom-agent-<VER> "$(cygpath -w /tmp/felhom-agent-<VER>)"
+scp "$(cygpath -w /tmp/felhom-agent-<VER>)" felhom-pve:/tmp/
+
+# DEPLOY with backup + restart + verify
+$SSH felhom-pve "cp /usr/local/bin/felhom-agent /usr/local/bin/felhom-agent.bak-\$(/usr/local/bin/felhom-agent --version | awk '{print \$2}') && install -m0755 /tmp/felhom-agent-<VER> /usr/local/bin/felhom-agent && systemctl restart felhom-agent && sleep 3 && /usr/local/bin/felhom-agent --version && journalctl -u felhom-agent -n 20 --no-pager"
+```
+
+Gotchas (earned): if `configs/` changed in the repo, ship the **sudoers + guarded wrapper WITH the
+binary** (several Go guards only exist when the deployed configs match); beware CRLF when scp-ing
+configs through Windows; after restart the journal must show a clean `ReassertGuestBinds` and (since
+0.84) the network-mount reassert, with no capability degradations.
+
+**PUBLISH (makes it real for the fleet):**
+```bash
+# from the agent repo, with REGISTRY_* creds set; use the LIVE-DEPLOYED bytes, sha-verified across hops
+scripts/publish-agent.sh <VER> <path-to-binary>
+```
+Pre-gate: the package GET for `<VER>` must be **404** before (published artifacts are immutable —
+never republish over an existing version). The script prints the **sha256 — record it**: the same sha
+goes into the hub manifest AND any signed `agent_update` op. One sha, three places, byte-identical.
+
+Then hub → Configuration → Day-0 artifacts: pick the new Agent version (sha auto-read from Gitea),
+set **Min agent** if the paired controller release declares `MinAgent:` in its CHANGELOG, **Save
+artifact manifest**. (This save does NOT move the floor — that's a separate card since hub v0.45.)
+
+## 3. Controller image (felhom-controller → guest 9201, remote via floor)
+
+9201 is golden/bootstrap-managed — no compose file. The bootstrap service runs whatever tag is in
+`/etc/felhom-controller-image` (anonymous pull).
+
+```bash
+# BUILD+PUSH the image (build.sh does NOT pull — the explicit pull is load-bearing)
+$SSH kisfenyo@192.168.0.180 "cd ~/build/felhom-controller && git -C ~/git/felhom-controller pull && ./build.sh <VER> --push"
+
+# DEPLOY on the demo guest
+$SSH felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"
+
+# VERIFY
+$SSH felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"
+```
+
+**Remote rollout** = the hub floor (Configuration → Managed updates). Order rules (train rules doc):
+the manifest vouches the target FIRST; any MinAgent must be satisfied fleet-wide (the hub now HOLDS
+boxes below it automatically, and flags them); **save the floor LAST** — the DB row acts immediately
+on every box below it, on their next report.
+
+## 4. Golden image (fresh Day-0 installs)
+
+The golden is a pre-baked controller-era guest image built in the **drill VM** on 180 — procedure per
+`RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C (use its RECORDED qemu launch line; do not
+reconstruct from memory):
+
+1. Revert the drill VM to its `virgin` internal snapshot; boot it with the recorded qemu one-liner.
+2. Run `build-golden.sh <controller VER>` (agent repo `configs/`) — bakes
+   `felhom-controller:<VER>` as a transient unit, uploads the golden to Gitea packages (404 pre-gate
+   applies), prints the **sha256 — record it**.
+3. Teardown: purge the build guest, shred the token/script/log, revert to `virgin`, and grep the
+   transcript for token leaks (must be 0 hits).
+4. Hub → Configuration → Day-0 artifacts: pick the new Golden, Save.
+
+A golden is only needed when a publish train wants fresh installs current — demo deploys never need it.
+
+## 5. Hub (felhom.eu/hub → k3s, GitOps)
+
+**The manifest is the truth** — a built image deploys NOTHING until `manifests/hub.yaml`'s `image:`
+tag changes in git and the ArgoCD app is deliberately synced (auto-sync is OFF; never
+`kubectl set image`, never `:latest`).
+
+```bash
+$SSH kisfenyo@192.168.0.180 "cd ~/build/felhom-hub && ./build.sh <VER> --push"
+# edit manifests/hub.yaml image tag → <VER>; commit; push
+$SSH kisfenyo@192.168.0.180 "sudo kubectl -n argocd annotate application felhom argocd.argoproj.io/refresh=hard --overwrite; sleep 8; sudo kubectl -n argocd get application felhom -o jsonpath='{.status.sync.status} {.status.sync.revision}{\"\n\"}'"
+$SSH kisfenyo@192.168.0.180 "sudo kubectl -n argocd patch application felhom --type merge -p '{\"operation\":{\"initiatedBy\":{\"username\":\"op\"},\"sync\":{\"syncStrategy\":{\"apply\":{}}}}}'"
+# verify: Synced/Healthy + rollout + live image tag + logs
+$SSH kisfenyo@192.168.0.180 "sudo kubectl -n argocd get application felhom -o jsonpath='sync={.status.sync.status} health={.status.health.status}{\"\n\"}'; sudo kubectl -n felhom-system rollout status deploy/hub --timeout=90s; sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'; echo; sudo kubectl -n felhom-system logs -l app=hub --tail 10"
+```
+
+## 6. "What's live right now?" one-liners
+
+```bash
+$SSH felhom-pve "/usr/local/bin/felhom-agent --version"                                    # agent on demo
+$SSH felhom-pve "pct exec 9201 -- cat /etc/felhom-controller-image"                        # controller on demo
+$SSH kisfenyo@192.168.0.180 "sudo kubectl -n felhom-system get deploy hub -o jsonpath='{.spec.template.spec.containers[0].image}'"  # hub
+# published = the hub Configuration dropdowns (they read Gitea packages live)
+# fleet = hub Dashboard per-host rows (agent_version + controller version per box)
+```
+
+## 7. State snapshot as of 2026-07-11 (so the screens make sense)
+
+| Artifact | Live on demo | Published / vouched | Peti |
+|---|---|---|---|
+| agent | 0.84.0 | **0.81.0** | 0.81.0 |
+| controller | 0.117.0 | golden **0.113.0**, floor 0.113.0 (DB = env, aligned) | 0.113.0 |
+| hub | 0.46.0 | n/a (central) | n/a |
+
+The gap between columns 1 and 2 is the pending publish train (agent 0.84 + golden/floor 0.117 +
+MinAgent 0.81 + Peti's journal one-liner + temp-creds deletion) — its runbook follows the
+0.81/0.113 pattern with these numbers.
+
+## 8. Iron rules (recap)
+
+Never `:latest`; never republish over an existing package version (404 pre-gate); never
+`kubectl set image` / bare `kubectl apply` against GitOps surfaces; manifest before floor, floor
+LAST; one sha in three places, byte-identical; secrets never in transcripts or commits; an unpushed
+change does not exist.