diff --git a/CONTEXT.md b/CONTEXT.md index 1784461..32c2498 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0.** + Record: `documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md`. Agent 0.81.0 published + (sha `914bb3b5…30df0b`, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha + `66de5e64…82054c`, drill VM, virgin-reverted); manifest + signed `agent_update` + (key-id `felhom-op-1`) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); **floor auto-confirm + proof CAPTURED** (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING: + the hub UI manifest screen ALSO sets the GLOBAL floor as a DB `hub_settings.min_controller_version` + row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113 + before the GitOps env bump `16390d0`); a transient forbidden skew (ctrl 0.113 + agent 0.79, + ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST. + OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), **hub bearer key + rotation now DUE** (committed in manifests/hub.yaml + screenshot-exposed this session). + - **2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo; host-install v1.13.0).** The spike's production task landed same-day: add = agent verify (retry=0 unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) → diff --git a/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md b/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md new file mode 100644 index 0000000..d477596 --- /dev/null +++ b/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md @@ -0,0 +1,133 @@ +# RUNBOOK results — publish train agent 0.81.0 + golden 0.113.0 + floor, incl. Peti remote rollout (2026-07-11) + +**Class:** publish/signing runbook (GL-1 pattern) — the NAS verify-before-commit train. +**Status:** Phases 0/A/B/C/D/F **COMPLETE** — Peti (`peti-felhom-86d37d`, sajatfelhom.hu) is LIVE on +**agent 0.81.0 + controller 0.113.0**, healthy over consecutive reports; the **floor-driven +self-update auto-confirm proof is captured** (the pending v0.112 item). Phase E (journal group) + +G1 (temp-creds deletion) + G2 (NAS page look) = operator follow-ups below. +No credential or signature-key material in this doc. + +## The four operator values (saved in the hub Day-0 manifest — DB-verified byte-identical) + +``` +AGENT_VERSION=0.81.0 +AGENT_SHA256=914bb3b597195365a5d1c7ac48a360755be763e15df8d2efaf663dd3b530df0b +GOLDEN_VERSION=0.113.0 +GOLDEN_SHA256=66de5e645caea429eb00d8c3ef18e026f52fd6a3ab681f39c7260a91dc82054c +FLOOR=0.113.0 (hub_settings row via UI + env fallback via GitOps 16390d0) +``` + +## Gate outcomes (Phase 0) + +| Gate | Result | Evidence | +|---|---|---| +| 0a Peti reporting | **PASS w/ deviation** | fresh reports; agent 0.79.0; controller **0.112.0** (the runbook assumed 0.110.0 — the v0.112 bump had already reached him); host cloudflared inactive + `wg-handshake-read` degraded = the known no-tunnel/no-OOB state of his box | +| 0b temp Git Sync creds | **OBSOLETE** | moot on 0.112 (anonymous registry pulls); the G1 deletion item stands | +| 0c zero NAS shares | **PASS** | his report's storage array holds only `/` (the demo, in contrast, lists its registered `/mnt/felhom-drives/felhom-usb` — proving registered paths do surface there); the `NAS-storage` lvmthin in his host report is a PVE storage, not a felhom share | +| 0d demo | **PASS** | 9201 controller 0.113.0 healthy; felhom-pve agent 0.81.0 | +| 0e published-state pre-gate | **PASS** | agent 0.81.0 GET 404 + golden 0.113.0 GET 404 before publish | +| 0f live bytes | **PASS** | felhom-pve binary v0.81.0 sha `914bb3b5…30df0b`; `/selfupdate/` absent = no pending marker | +| 0g controller image | **PASS** | `docker manifest inspect …:0.113.0` OK on 180 | +| 0h identifiers | **PASS** | host_id `peti-felhom-86d37d` read fresh from the hub `hosts` table; pinned key-id `felhom-op-1` (host-install ceremony constant) — operator signed with the matching `felhom-op-operational` key | + +## Phase A — agent 0.81.0 published + +Live felhom-pve bytes (the deployed artifact of felhom-agent `main@1e60e88`); sha **identical across +felhom-pve → CC → 180**; `publish-agent.sh 0.81.0`: pre-GET 404 → PUT **201** → round-trip sha match +→ **anonymous off-LAN GET 200** (13 511 166 bytes, sha == AGENT_SHA256). Hop copies removed; +token-leak grep of transcripts: 0. + +## Phase B — golden 0.113.0 baked + published + +Recorded drill-VM procedure (RUNBOOK-publish-0.79-0.110 Phase C, incl. the recorded qemu one-liner — +booted first try): `qemu-img snapshot -a virgin` → cold boot → PVE 9.2.2 up on :2222 in ~10 s → +gitea 200 in-VM → debian-13 template `pveam download` (checksum OK; absent on virgin, as last time) +→ `build-golden.sh` v2.0.0 as transient unit `golden-bake` baking `felhom-controller:0.113.0`: +`docker OK (overlay2)`, **all 3 mounts included** (rootfs+mp0+mp1, no exclusions/FATAL), archive +583 MB → pre-delete 404 → upload **201**. Verified: **anonymous GET 200, 612 257 363 bytes, sha == +GOLDEN_SHA256** + off-LAN HEAD 200. Teardown: guest 9100 purged, token/script/log shredded in-VM, +clean poweroff, qcow2 **reverted to virgin** (snapshot intact), `~/drill/bake-0.113.0.log` retained, +saved-log token-leak grep **0**. + +## Phase C — hub Day-0 manifest (operator) + +Saved by Viktor; **DB-verified** (`hub_settings`): `artifact_agent_version=0.81.0` + +`artifact_agent_sha256=914bb3b5…30df0b`, `artifact_golden_version=0.113.0` + +`artifact_golden_sha256=66de5e64…82054c` — byte-identical to the published artifacts. One sha per +artifact, three places (publish / manifest / signed op), all matching. + +## Phase D — signed agent_update → Peti on 0.81.0 + +Signed by Viktor (`felhom-opsign -op agent_update -host peti-felhom-86d37d -key-id felhom-op-1 … +-agent-version 0.81.0 -sha256 914bb3b5… -ttl 30m`), uploaded to the hub jobs queue at **10:39:23Z** +(job `56c3d29e6e925c1d`). Consumed on the next heartbeat window: host report **10:51:52Z shows agent +0.81.0**, queue empty, **no rollback activation, no pending marker**; capability sweep 59 probes, +4 non-ok all pre-existing/expected (`wg-handshake-read` = no WG tunnel on his box; +`pbsdr-create/reconcile/grant` = the felhom-pbs-apply wrapper lands in the pending supervised PBS-DR +phase; all `critical:false`). 0.79.0 fallback stays published for a reverse `agent_update` if ever +needed (sha `35a1625476…d27561`). + +## Phase F — floor → 0.113.0 + THE AUTO-CONFIRM PROOF + +**Finding (deviation, root-caused):** the hub UI screen Viktor used for Phase C also carries the +GLOBAL floor as a **DB-backed `hub_settings.min_controller_version` row, which takes precedence over +the `DEFAULT_MIN_CONTROLLER_VERSION` env** (`store.go GetGlobalMinControllerVersion`) and acts +immediately — no GitOps needed. He saved 0.113.0 there with the manifest. (This also retro-explains +how Peti reached 0.112.0 earlier with the env floor still at 0.87.0.) + +**The floor-driven self-update evidence chain (the pending v0.112 live auto-confirm proof) — CAPTURED:** + +``` +~10:3x Z hub_settings.min_controller_version = 0.113.0 (operator UI save) +10:41:38Z Peti report: controller 0.112.0, health ok (floor seen on this cycle) + → anonymous registry pull (NO Git Sync creds needed — the v0.112 anonymous + token dance, now proven on PETI's box, not just the demo) + → in-guest pre-pull → POST /controller/swap via his agent → bootstrap restart +10:42:02Z Peti report: controller 0.113.0, health ok (24 s report-to-report flip) +10:56:57Z Peti report: controller 0.113.0, health ok (stable next cycle) +``` + +CC then aligned the GitOps env fallback: `manifests/hub.yaml` `DEFAULT_MIN_CONTROLLER_VERSION` +0.87.0 → **0.113.0** (commit `16390d0`), ArgoCD hard-refresh + deliberate sync → Synced/Healthy, +rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (already 0.113.0). + +## Deviations from the letter of the runbook + +1. **Peti entered on controller 0.112.0**, not the assumed 0.110.0 (the v0.112 per-train pending + item had already been executed) — gate 0b thereby obsolete (anonymous pulls). +2. **The forbidden skew (controller 0.113 + agent 0.79) transiently existed** ~10:42→10:51Z: the UI + floor save acted immediately, BEFORE the agent op was consumed — the runbook assumed the floor + was GitOps-only and CC-sequenced. Benign exactly because gate 0c held (zero NAS shares, no adds + in the window). **Lesson for the next train:** the manifest-save screen carries the floor — ask + the operator to save the floor value LAST, after the agent lands, or leave the floor field + untouched at manifest time. +3. **The auto-confirm proof ran off the DB floor, not the env floor** — same mechanism + (`EffectiveMinControllerVersion`), stronger precedence; the env bump became the consistency + fallback rather than the trigger. +4. `pgrep/pkill -f qemu-system` self-match footgun during teardown (the pattern matched the ssh + command line itself) — use `pgrep -f "[q]emu-system"`; the VM had in fact powered off cleanly. + +## Security notes + +- No key material or tokens in any transcript; publish creds stayed in env-substitution on 180; + drill-VM token shredded; both saved-log leak greps 0. +- **⚠ ROTATION NOW DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml` — + one of the three known committed secrets on the rotate+de-git backlog) and was additionally + exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the + Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task. + +## END STATE + +| Item | State | +|---|---| +| Agent 0.81.0 published + anon-fetchable + sha-verified | ✅ `914bb3b5…30df0b` | +| Golden 0.113.0 published + anon-fetchable + sha-verified | ✅ `66de5e64…82054c` (612 257 363 B) | +| Hub manifest vouches both | ✅ DB-verified | +| Global floor 0.113.0 | ✅ hub_settings (UI) + env fallback (GitOps `16390d0`, Synced/Healthy) | +| **Peti: agent 0.81.0 + controller 0.113.0, healthy** | ✅ consecutive ok reports; auto-confirm proof captured | +| Demo: unchanged (0.113.0 / 0.81.0) | ✅ no churn | +| Drill VM | ✅ reverted to virgin; bake log retained | +| Phase E — journal group on Peti's host | ⏳ operator choice: OOB unlikely (WG down) → message Peti the one-liner (`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`) or defer (NAS errors degrade to generic text until then) | +| G1 — delete Peti's temp Git Sync creds | ⏳ operator/guided-Peti: Settings → Git Sync → delete → verify "Registry: nyilvános (hitelesítés nélkül)" + `/api/selfupdate/check` ok (pulls are proven anonymous on his box as of this train) | +| G2 — NAS page render on Peti's box | ⏳ optional; his first real NAS add is the first external run of the verify pipeline | +| Hub bearer key rotation | ⏳ operator — see Security notes |