RUNBOOK results — publish train agent 0.81.0 + golden 0.113.0 + floor, incl. Peti remote rollout (2026-07-11)

Class: publish/signing runbook (GL-1 pattern) — the NAS verify-before-commit train. Status: Phases 0/A/B/C/D/F COMPLETE — Peti (peti-felhom-86d37d, sajatfelhom.hu) is LIVE on agent 0.81.0 + controller 0.113.0, healthy over consecutive reports; the floor-driven self-update auto-confirm proof is captured (the pending v0.112 item). Phase E (journal group) + G1 (temp-creds deletion) + G2 (NAS page look) = operator follow-ups below. No credential or signature-key material in this doc.

The four operator values (saved in the hub Day-0 manifest — DB-verified byte-identical)

AGENT_VERSION=0.81.0
AGENT_SHA256=914bb3b597195365a5d1c7ac48a360755be763e15df8d2efaf663dd3b530df0b
GOLDEN_VERSION=0.113.0
GOLDEN_SHA256=66de5e645caea429eb00d8c3ef18e026f52fd6a3ab681f39c7260a91dc82054c
FLOOR=0.113.0   (hub_settings row via UI + env fallback via GitOps 16390d0)

Gate outcomes (Phase 0)

Gate	Result	Evidence
0a Peti reporting	PASS w/ deviation	fresh reports; agent 0.79.0; controller 0.112.0 (the runbook assumed 0.110.0 — the v0.112 bump had already reached him); host cloudflared inactive + `wg-handshake-read` degraded = the known no-tunnel/no-OOB state of his box
0b temp Git Sync creds	OBSOLETE	moot on 0.112 (anonymous registry pulls); the G1 deletion item stands
0c zero NAS shares	PASS	his report's storage array holds only `/` (the demo, in contrast, lists its registered `/mnt/felhom-drives/felhom-usb` — proving registered paths do surface there); the `NAS-storage` lvmthin in his host report is a PVE storage, not a felhom share
0d demo	PASS	9201 controller 0.113.0 healthy; felhom-pve agent 0.81.0
0e published-state pre-gate	PASS	agent 0.81.0 GET 404 + golden 0.113.0 GET 404 before publish
0f live bytes	PASS	felhom-pve binary v0.81.0 sha `914bb3b5…30df0b`; `<StateDir>/selfupdate/` absent = no pending marker
0g controller image	PASS	`docker manifest inspect …:0.113.0` OK on 180
0h identifiers	PASS	host_id `peti-felhom-86d37d` read fresh from the hub `hosts` table; pinned key-id `felhom-op-1` (host-install ceremony constant) — operator signed with the matching `felhom-op-operational` key

Phase A — agent 0.81.0 published

Live felhom-pve bytes (the deployed artifact of felhom-agent main@1e60e88); sha identical across felhom-pve → CC → 180; publish-agent.sh 0.81.0: pre-GET 404 → PUT 201 → round-trip sha match → anonymous off-LAN GET 200 (13 511 166 bytes, sha == AGENT_SHA256). Hop copies removed; token-leak grep of transcripts: 0.

Phase B — golden 0.113.0 baked + published

Recorded drill-VM procedure (RUNBOOK-publish-0.79-0.110 Phase C, incl. the recorded qemu one-liner — booted first try): qemu-img snapshot -a virgin → cold boot → PVE 9.2.2 up on :2222 in ~10 s → gitea 200 in-VM → debian-13 template pveam download (checksum OK; absent on virgin, as last time) → build-golden.sh v2.0.0 as transient unit golden-bake baking felhom-controller:0.113.0: docker OK (overlay2), all 3 mounts included (rootfs+mp0+mp1, no exclusions/FATAL), archive 583 MB → pre-delete 404 → upload 201. Verified: anonymous GET 200, 612 257 363 bytes, sha == GOLDEN_SHA256 + off-LAN HEAD 200. Teardown: guest 9100 purged, token/script/log shredded in-VM, clean poweroff, qcow2 reverted to virgin (snapshot intact), ~/drill/bake-0.113.0.log retained, saved-log token-leak grep 0.

Phase C — hub Day-0 manifest (operator)

Saved by Viktor; DB-verified (hub_settings): artifact_agent_version=0.81.0 + artifact_agent_sha256=914bb3b5…30df0b, artifact_golden_version=0.113.0 + artifact_golden_sha256=66de5e64…82054c — byte-identical to the published artifacts. One sha per artifact, three places (publish / manifest / signed op), all matching.

Phase D — signed agent_update → Peti on 0.81.0

Signed by Viktor (felhom-opsign -op agent_update -host peti-felhom-86d37d -key-id felhom-op-1 … -agent-version 0.81.0 -sha256 914bb3b5… -ttl 30m), uploaded to the hub jobs queue at 10:39:23Z (job 56c3d29e6e925c1d). Consumed on the next heartbeat window: host report 10:51:52Z shows agent 0.81.0, queue empty, no rollback activation, no pending marker; capability sweep 59 probes, 4 non-ok all pre-existing/expected (wg-handshake-read = no WG tunnel on his box; pbsdr-create/reconcile/grant = the felhom-pbs-apply wrapper lands in the pending supervised PBS-DR phase; all critical:false). 0.79.0 fallback stays published for a reverse agent_update if ever needed (sha 35a1625476…d27561).

Phase F — floor → 0.113.0 + THE AUTO-CONFIRM PROOF

Finding (deviation, root-caused): the hub UI screen Viktor used for Phase C also carries the GLOBAL floor as a DB-backed hub_settings.min_controller_version row, which takes precedence over the DEFAULT_MIN_CONTROLLER_VERSION env (store.go GetGlobalMinControllerVersion) and acts immediately — no GitOps needed. He saved 0.113.0 there with the manifest. (This also retro-explains how Peti reached 0.112.0 earlier with the env floor still at 0.87.0.)

The floor-driven self-update evidence chain (the pending v0.112 live auto-confirm proof) — CAPTURED:

~10:3x Z  hub_settings.min_controller_version = 0.113.0 (operator UI save)
10:41:38Z Peti report: controller 0.112.0, health ok      (floor seen on this cycle)
          → anonymous registry pull (NO Git Sync creds needed — the v0.112 anonymous
            token dance, now proven on PETI's box, not just the demo)
          → in-guest pre-pull → POST /controller/swap via his agent → bootstrap restart
10:42:02Z Peti report: controller 0.113.0, health ok      (24 s report-to-report flip)
10:56:57Z Peti report: controller 0.113.0, health ok      (stable next cycle)

CC then aligned the GitOps env fallback: manifests/hub.yaml DEFAULT_MIN_CONTROLLER_VERSION 0.87.0 → 0.113.0 (commit 16390d0), ArgoCD hard-refresh + deliberate sync → Synced/Healthy, rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (already 0.113.0).

Deviations from the letter of the runbook

Peti entered on controller 0.112.0, not the assumed 0.110.0 (the v0.112 per-train pending item had already been executed) — gate 0b thereby obsolete (anonymous pulls).
The forbidden skew (controller 0.113 + agent 0.79) transiently existed ~10:42→10:51Z: the UI floor save acted immediately, BEFORE the agent op was consumed — the runbook assumed the floor was GitOps-only and CC-sequenced. Benign exactly because gate 0c held (zero NAS shares, no adds in the window). Lesson for the next train: the manifest-save screen carries the floor — ask the operator to save the floor value LAST, after the agent lands, or leave the floor field untouched at manifest time.
The auto-confirm proof ran off the DB floor, not the env floor — same mechanism (EffectiveMinControllerVersion), stronger precedence; the env bump became the consistency fallback rather than the trigger.
pgrep/pkill -f qemu-system self-match footgun during teardown (the pattern matched the ssh command line itself) — use pgrep -f "[q]emu-system"; the VM had in fact powered off cleanly.

Security notes

No key material or tokens in any transcript; publish creds stayed in env-substitution on 180; drill-VM token shredded; both saved-log leak greps 0.
⚠ ROTATION NOW DUE: the hub operator bearer key is COMMITTED in git (manifests/hub.yaml — one of the three known committed secrets on the rotate+de-git backlog) and was additionally exposed in a session screenshot during Phase D. Rotate it + move to a secretKeyRef (the Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task. DISPOSITION 2026-07-13 (hub v0.53.0): de-git DONE (secretKeyRef: report-api/REPORT_API_KEY
- scripts/manifest_bearer_gate.py); the value rotation is the supervised procedure in documentation/runbooks/secrets.md §"Operator/global bearer key".

END STATE

Item	State
Agent 0.81.0 published + anon-fetchable + sha-verified	✅ `914bb3b5…30df0b`
Golden 0.113.0 published + anon-fetchable + sha-verified	✅ `66de5e64…82054c` (612 257 363 B)
Hub manifest vouches both	✅ DB-verified
Global floor 0.113.0	✅ hub_settings (UI) + env fallback (GitOps `16390d0`, Synced/Healthy)
Peti: agent 0.81.0 + controller 0.113.0, healthy	✅ consecutive ok reports; auto-confirm proof captured
Demo: unchanged (0.113.0 / 0.81.0)	✅ no churn
Drill VM	✅ reverted to virgin; bake log retained
Phase E — journal group on Peti's host	⏳ operator choice: OOB unlikely (WG down) → message Peti the one-liner (`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`) or defer (NAS errors degrade to generic text until then)
G1 — delete Peti's temp Git Sync creds	⏳ operator/guided-Peti: Settings → Git Sync → delete → verify "Registry: nyilvános (hitelesítés nélkül)" + `/api/selfupdate/check` ok (pulls are proven anonymous on his box as of this train)
G2 — NAS page render on Peti's box	⏳ optional; his first real NAS add is the first external run of the verify pipeline
Hub bearer key rotation	⏳ operator — see Security notes

9.4 KiB Raw Blame History