# RUNBOOK results — publish train agent 0.81.0 + golden 0.113.0 + floor, incl. Peti remote rollout (2026-07-11)

**Class:** publish/signing runbook (GL-1 pattern) — the NAS verify-before-commit train.
**Status:** Phases 0/A/B/C/D/F **COMPLETE** — Peti (`peti-felhom-86d37d`, sajatfelhom.hu) is LIVE on
**agent 0.81.0 + controller 0.113.0**, healthy over consecutive reports; the **floor-driven
self-update auto-confirm proof is captured** (the pending v0.112 item). Phase E (journal group) +
G1 (temp-creds deletion) + G2 (NAS page look) = operator follow-ups below.
No credential or signature-key material in this doc.

## The four operator values (saved in the hub Day-0 manifest — DB-verified byte-identical)

```
AGENT_VERSION=0.81.0
AGENT_SHA256=914bb3b597195365a5d1c7ac48a360755be763e15df8d2efaf663dd3b530df0b
GOLDEN_VERSION=0.113.0
GOLDEN_SHA256=66de5e645caea429eb00d8c3ef18e026f52fd6a3ab681f39c7260a91dc82054c
FLOOR=0.113.0   (hub_settings row via UI + env fallback via GitOps 16390d0)
```

## Gate outcomes (Phase 0)

| Gate | Result | Evidence |
|---|---|---|
| 0a Peti reporting | **PASS w/ deviation** | fresh reports; agent 0.79.0; controller **0.112.0** (the runbook assumed 0.110.0 — the v0.112 bump had already reached him); host cloudflared inactive + `wg-handshake-read` degraded = the known no-tunnel/no-OOB state of his box |
| 0b temp Git Sync creds | **OBSOLETE** | moot on 0.112 (anonymous registry pulls); the G1 deletion item stands |
| 0c zero NAS shares | **PASS** | his report's storage array holds only `/` (the demo, in contrast, lists its registered `/mnt/felhom-drives/felhom-usb` — proving registered paths do surface there); the `NAS-storage` lvmthin in his host report is a PVE storage, not a felhom share |
| 0d demo | **PASS** | 9201 controller 0.113.0 healthy; felhom-pve agent 0.81.0 |
| 0e published-state pre-gate | **PASS** | agent 0.81.0 GET 404 + golden 0.113.0 GET 404 before publish |
| 0f live bytes | **PASS** | felhom-pve binary v0.81.0 sha `914bb3b5…30df0b`; `<StateDir>/selfupdate/` absent = no pending marker |
| 0g controller image | **PASS** | `docker manifest inspect …:0.113.0` OK on 180 |
| 0h identifiers | **PASS** | host_id `peti-felhom-86d37d` read fresh from the hub `hosts` table; pinned key-id `felhom-op-1` (host-install ceremony constant) — operator signed with the matching `felhom-op-operational` key |

## Phase A — agent 0.81.0 published

Live felhom-pve bytes (the deployed artifact of felhom-agent `main@1e60e88`); sha **identical across
felhom-pve → CC → 180**; `publish-agent.sh 0.81.0`: pre-GET 404 → PUT **201** → round-trip sha match
→ **anonymous off-LAN GET 200** (13 511 166 bytes, sha == AGENT_SHA256). Hop copies removed;
token-leak grep of transcripts: 0.

## Phase B — golden 0.113.0 baked + published

Recorded drill-VM procedure (RUNBOOK-publish-0.79-0.110 Phase C, incl. the recorded qemu one-liner —
booted first try): `qemu-img snapshot -a virgin` → cold boot → PVE 9.2.2 up on :2222 in ~10 s →
gitea 200 in-VM → debian-13 template `pveam download` (checksum OK; absent on virgin, as last time)
→ `build-golden.sh` v2.0.0 as transient unit `golden-bake` baking `felhom-controller:0.113.0`:
`docker OK (overlay2)`, **all 3 mounts included** (rootfs+mp0+mp1, no exclusions/FATAL), archive
583 MB → pre-delete 404 → upload **201**. Verified: **anonymous GET 200, 612 257 363 bytes, sha ==
GOLDEN_SHA256** + off-LAN HEAD 200. Teardown: guest 9100 purged, token/script/log shredded in-VM,
clean poweroff, qcow2 **reverted to virgin** (snapshot intact), `~/drill/bake-0.113.0.log` retained,
saved-log token-leak grep **0**.

## Phase C — hub Day-0 manifest (operator)

Saved by Viktor; **DB-verified** (`hub_settings`): `artifact_agent_version=0.81.0` +
`artifact_agent_sha256=914bb3b5…30df0b`, `artifact_golden_version=0.113.0` +
`artifact_golden_sha256=66de5e64…82054c` — byte-identical to the published artifacts. One sha per
artifact, three places (publish / manifest / signed op), all matching.

## Phase D — signed agent_update → Peti on 0.81.0

Signed by Viktor (`felhom-opsign -op agent_update -host peti-felhom-86d37d -key-id felhom-op-1 …
-agent-version 0.81.0 -sha256 914bb3b5… -ttl 30m`), uploaded to the hub jobs queue at **10:39:23Z**
(job `56c3d29e6e925c1d`). Consumed on the next heartbeat window: host report **10:51:52Z shows agent
0.81.0**, queue empty, **no rollback activation, no pending marker**; capability sweep 59 probes,
4 non-ok all pre-existing/expected (`wg-handshake-read` = no WG tunnel on his box;
`pbsdr-create/reconcile/grant` = the felhom-pbs-apply wrapper lands in the pending supervised PBS-DR
phase; all `critical:false`). 0.79.0 fallback stays published for a reverse `agent_update` if ever
needed (sha `35a1625476…d27561`).

## Phase F — floor → 0.113.0 + THE AUTO-CONFIRM PROOF

**Finding (deviation, root-caused):** the hub UI screen Viktor used for Phase C also carries the
GLOBAL floor as a **DB-backed `hub_settings.min_controller_version` row, which takes precedence over
the `DEFAULT_MIN_CONTROLLER_VERSION` env** (`store.go GetGlobalMinControllerVersion`) and acts
immediately — no GitOps needed. He saved 0.113.0 there with the manifest. (This also retro-explains
how Peti reached 0.112.0 earlier with the env floor still at 0.87.0.)

**The floor-driven self-update evidence chain (the pending v0.112 live auto-confirm proof) — CAPTURED:**

```
~10:3x Z  hub_settings.min_controller_version = 0.113.0 (operator UI save)
10:41:38Z Peti report: controller 0.112.0, health ok      (floor seen on this cycle)
          → anonymous registry pull (NO Git Sync creds needed — the v0.112 anonymous
            token dance, now proven on PETI's box, not just the demo)
          → in-guest pre-pull → POST /controller/swap via his agent → bootstrap restart
10:42:02Z Peti report: controller 0.113.0, health ok      (24 s report-to-report flip)
10:56:57Z Peti report: controller 0.113.0, health ok      (stable next cycle)
```

CC then aligned the GitOps env fallback: `manifests/hub.yaml` `DEFAULT_MIN_CONTROLLER_VERSION`
0.87.0 → **0.113.0** (commit `16390d0`), ArgoCD hard-refresh + deliberate sync → Synced/Healthy,
rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (already 0.113.0).

## Deviations from the letter of the runbook

1. **Peti entered on controller 0.112.0**, not the assumed 0.110.0 (the v0.112 per-train pending
   item had already been executed) — gate 0b thereby obsolete (anonymous pulls).
2. **The forbidden skew (controller 0.113 + agent 0.79) transiently existed** ~10:42→10:51Z: the UI
   floor save acted immediately, BEFORE the agent op was consumed — the runbook assumed the floor
   was GitOps-only and CC-sequenced. Benign exactly because gate 0c held (zero NAS shares, no adds
   in the window). **Lesson for the next train:** the manifest-save screen carries the floor — ask
   the operator to save the floor value LAST, after the agent lands, or leave the floor field
   untouched at manifest time.
3. **The auto-confirm proof ran off the DB floor, not the env floor** — same mechanism
   (`EffectiveMinControllerVersion`), stronger precedence; the env bump became the consistency
   fallback rather than the trigger.
4. `pgrep/pkill -f qemu-system` self-match footgun during teardown (the pattern matched the ssh
   command line itself) — use `pgrep -f "[q]emu-system"`; the VM had in fact powered off cleanly.

## Security notes

- No key material or tokens in any transcript; publish creds stayed in env-substitution on 180;
  drill-VM token shredded; both saved-log leak greps 0.
- **⚠ ROTATION NOW DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml` —
  one of the three known committed secrets on the rotate+de-git backlog) and was additionally
  exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
  Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
  **DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY`
  + `scripts/manifest_bearer_gate.py`); the value rotation is the supervised procedure in
  documentation/runbooks/secrets.md §"Operator/global bearer key".

## END STATE

| Item | State |
|---|---|
| Agent 0.81.0 published + anon-fetchable + sha-verified | ✅ `914bb3b5…30df0b` |
| Golden 0.113.0 published + anon-fetchable + sha-verified | ✅ `66de5e64…82054c` (612 257 363 B) |
| Hub manifest vouches both | ✅ DB-verified |
| Global floor 0.113.0 | ✅ hub_settings (UI) + env fallback (GitOps `16390d0`, Synced/Healthy) |
| **Peti: agent 0.81.0 + controller 0.113.0, healthy** | ✅ consecutive ok reports; auto-confirm proof captured |
| Demo: unchanged (0.113.0 / 0.81.0) | ✅ no churn |
| Drill VM | ✅ reverted to virgin; bake log retained |
| Phase E — journal group on Peti's host | ⏳ operator choice: OOB unlikely (WG down) → message Peti the one-liner (`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`) or defer (NAS errors degrade to generic text until then) |
| G1 — delete Peti's temp Git Sync creds | ⏳ operator/guided-Peti: Settings → Git Sync → delete → verify "Registry: nyilvános (hitelesítés nélkül)" + `/api/selfupdate/check` ok (pulls are proven anonymous on his box as of this train) |
| G2 — NAS page render on Peti's box | ⏳ optional; his first real NAS add is the first external run of the verify pipeline |
| Hub bearer key rotation | ⏳ operator — see Security notes |
