pilot: publish-train 0.81/0.113 results — Peti live, floor auto-confirm proof captured
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,133 @@
|
||||
# RUNBOOK results — publish train agent 0.81.0 + golden 0.113.0 + floor, incl. Peti remote rollout (2026-07-11)
|
||||
|
||||
**Class:** publish/signing runbook (GL-1 pattern) — the NAS verify-before-commit train.
|
||||
**Status:** Phases 0/A/B/C/D/F **COMPLETE** — Peti (`peti-felhom-86d37d`, sajatfelhom.hu) is LIVE on
|
||||
**agent 0.81.0 + controller 0.113.0**, healthy over consecutive reports; the **floor-driven
|
||||
self-update auto-confirm proof is captured** (the pending v0.112 item). Phase E (journal group) +
|
||||
G1 (temp-creds deletion) + G2 (NAS page look) = operator follow-ups below.
|
||||
No credential or signature-key material in this doc.
|
||||
|
||||
## The four operator values (saved in the hub Day-0 manifest — DB-verified byte-identical)
|
||||
|
||||
```
|
||||
AGENT_VERSION=0.81.0
|
||||
AGENT_SHA256=914bb3b597195365a5d1c7ac48a360755be763e15df8d2efaf663dd3b530df0b
|
||||
GOLDEN_VERSION=0.113.0
|
||||
GOLDEN_SHA256=66de5e645caea429eb00d8c3ef18e026f52fd6a3ab681f39c7260a91dc82054c
|
||||
FLOOR=0.113.0 (hub_settings row via UI + env fallback via GitOps 16390d0)
|
||||
```
|
||||
|
||||
## Gate outcomes (Phase 0)
|
||||
|
||||
| Gate | Result | Evidence |
|
||||
|---|---|---|
|
||||
| 0a Peti reporting | **PASS w/ deviation** | fresh reports; agent 0.79.0; controller **0.112.0** (the runbook assumed 0.110.0 — the v0.112 bump had already reached him); host cloudflared inactive + `wg-handshake-read` degraded = the known no-tunnel/no-OOB state of his box |
|
||||
| 0b temp Git Sync creds | **OBSOLETE** | moot on 0.112 (anonymous registry pulls); the G1 deletion item stands |
|
||||
| 0c zero NAS shares | **PASS** | his report's storage array holds only `/` (the demo, in contrast, lists its registered `/mnt/felhom-drives/felhom-usb` — proving registered paths do surface there); the `NAS-storage` lvmthin in his host report is a PVE storage, not a felhom share |
|
||||
| 0d demo | **PASS** | 9201 controller 0.113.0 healthy; felhom-pve agent 0.81.0 |
|
||||
| 0e published-state pre-gate | **PASS** | agent 0.81.0 GET 404 + golden 0.113.0 GET 404 before publish |
|
||||
| 0f live bytes | **PASS** | felhom-pve binary v0.81.0 sha `914bb3b5…30df0b`; `<StateDir>/selfupdate/` absent = no pending marker |
|
||||
| 0g controller image | **PASS** | `docker manifest inspect …:0.113.0` OK on 180 |
|
||||
| 0h identifiers | **PASS** | host_id `peti-felhom-86d37d` read fresh from the hub `hosts` table; pinned key-id `felhom-op-1` (host-install ceremony constant) — operator signed with the matching `felhom-op-operational` key |
|
||||
|
||||
## Phase A — agent 0.81.0 published
|
||||
|
||||
Live felhom-pve bytes (the deployed artifact of felhom-agent `main@1e60e88`); sha **identical across
|
||||
felhom-pve → CC → 180**; `publish-agent.sh 0.81.0`: pre-GET 404 → PUT **201** → round-trip sha match
|
||||
→ **anonymous off-LAN GET 200** (13 511 166 bytes, sha == AGENT_SHA256). Hop copies removed;
|
||||
token-leak grep of transcripts: 0.
|
||||
|
||||
## Phase B — golden 0.113.0 baked + published
|
||||
|
||||
Recorded drill-VM procedure (RUNBOOK-publish-0.79-0.110 Phase C, incl. the recorded qemu one-liner —
|
||||
booted first try): `qemu-img snapshot -a virgin` → cold boot → PVE 9.2.2 up on :2222 in ~10 s →
|
||||
gitea 200 in-VM → debian-13 template `pveam download` (checksum OK; absent on virgin, as last time)
|
||||
→ `build-golden.sh` v2.0.0 as transient unit `golden-bake` baking `felhom-controller:0.113.0`:
|
||||
`docker OK (overlay2)`, **all 3 mounts included** (rootfs+mp0+mp1, no exclusions/FATAL), archive
|
||||
583 MB → pre-delete 404 → upload **201**. Verified: **anonymous GET 200, 612 257 363 bytes, sha ==
|
||||
GOLDEN_SHA256** + off-LAN HEAD 200. Teardown: guest 9100 purged, token/script/log shredded in-VM,
|
||||
clean poweroff, qcow2 **reverted to virgin** (snapshot intact), `~/drill/bake-0.113.0.log` retained,
|
||||
saved-log token-leak grep **0**.
|
||||
|
||||
## Phase C — hub Day-0 manifest (operator)
|
||||
|
||||
Saved by Viktor; **DB-verified** (`hub_settings`): `artifact_agent_version=0.81.0` +
|
||||
`artifact_agent_sha256=914bb3b5…30df0b`, `artifact_golden_version=0.113.0` +
|
||||
`artifact_golden_sha256=66de5e64…82054c` — byte-identical to the published artifacts. One sha per
|
||||
artifact, three places (publish / manifest / signed op), all matching.
|
||||
|
||||
## Phase D — signed agent_update → Peti on 0.81.0
|
||||
|
||||
Signed by Viktor (`felhom-opsign -op agent_update -host peti-felhom-86d37d -key-id felhom-op-1 …
|
||||
-agent-version 0.81.0 -sha256 914bb3b5… -ttl 30m`), uploaded to the hub jobs queue at **10:39:23Z**
|
||||
(job `56c3d29e6e925c1d`). Consumed on the next heartbeat window: host report **10:51:52Z shows agent
|
||||
0.81.0**, queue empty, **no rollback activation, no pending marker**; capability sweep 59 probes,
|
||||
4 non-ok all pre-existing/expected (`wg-handshake-read` = no WG tunnel on his box;
|
||||
`pbsdr-create/reconcile/grant` = the felhom-pbs-apply wrapper lands in the pending supervised PBS-DR
|
||||
phase; all `critical:false`). 0.79.0 fallback stays published for a reverse `agent_update` if ever
|
||||
needed (sha `35a1625476…d27561`).
|
||||
|
||||
## Phase F — floor → 0.113.0 + THE AUTO-CONFIRM PROOF
|
||||
|
||||
**Finding (deviation, root-caused):** the hub UI screen Viktor used for Phase C also carries the
|
||||
GLOBAL floor as a **DB-backed `hub_settings.min_controller_version` row, which takes precedence over
|
||||
the `DEFAULT_MIN_CONTROLLER_VERSION` env** (`store.go GetGlobalMinControllerVersion`) and acts
|
||||
immediately — no GitOps needed. He saved 0.113.0 there with the manifest. (This also retro-explains
|
||||
how Peti reached 0.112.0 earlier with the env floor still at 0.87.0.)
|
||||
|
||||
**The floor-driven self-update evidence chain (the pending v0.112 live auto-confirm proof) — CAPTURED:**
|
||||
|
||||
```
|
||||
~10:3x Z hub_settings.min_controller_version = 0.113.0 (operator UI save)
|
||||
10:41:38Z Peti report: controller 0.112.0, health ok (floor seen on this cycle)
|
||||
→ anonymous registry pull (NO Git Sync creds needed — the v0.112 anonymous
|
||||
token dance, now proven on PETI's box, not just the demo)
|
||||
→ in-guest pre-pull → POST /controller/swap via his agent → bootstrap restart
|
||||
10:42:02Z Peti report: controller 0.113.0, health ok (24 s report-to-report flip)
|
||||
10:56:57Z Peti report: controller 0.113.0, health ok (stable next cycle)
|
||||
```
|
||||
|
||||
CC then aligned the GitOps env fallback: `manifests/hub.yaml` `DEFAULT_MIN_CONTROLLER_VERSION`
|
||||
0.87.0 → **0.113.0** (commit `16390d0`), ArgoCD hard-refresh + deliberate sync → Synced/Healthy,
|
||||
rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (already 0.113.0).
|
||||
|
||||
## Deviations from the letter of the runbook
|
||||
|
||||
1. **Peti entered on controller 0.112.0**, not the assumed 0.110.0 (the v0.112 per-train pending
|
||||
item had already been executed) — gate 0b thereby obsolete (anonymous pulls).
|
||||
2. **The forbidden skew (controller 0.113 + agent 0.79) transiently existed** ~10:42→10:51Z: the UI
|
||||
floor save acted immediately, BEFORE the agent op was consumed — the runbook assumed the floor
|
||||
was GitOps-only and CC-sequenced. Benign exactly because gate 0c held (zero NAS shares, no adds
|
||||
in the window). **Lesson for the next train:** the manifest-save screen carries the floor — ask
|
||||
the operator to save the floor value LAST, after the agent lands, or leave the floor field
|
||||
untouched at manifest time.
|
||||
3. **The auto-confirm proof ran off the DB floor, not the env floor** — same mechanism
|
||||
(`EffectiveMinControllerVersion`), stronger precedence; the env bump became the consistency
|
||||
fallback rather than the trigger.
|
||||
4. `pgrep/pkill -f qemu-system` self-match footgun during teardown (the pattern matched the ssh
|
||||
command line itself) — use `pgrep -f "[q]emu-system"`; the VM had in fact powered off cleanly.
|
||||
|
||||
## Security notes
|
||||
|
||||
- No key material or tokens in any transcript; publish creds stayed in env-substitution on 180;
|
||||
drill-VM token shredded; both saved-log leak greps 0.
|
||||
- **⚠ ROTATION NOW DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml` —
|
||||
one of the three known committed secrets on the rotate+de-git backlog) and was additionally
|
||||
exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
|
||||
Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
|
||||
|
||||
## END STATE
|
||||
|
||||
| Item | State |
|
||||
|---|---|
|
||||
| Agent 0.81.0 published + anon-fetchable + sha-verified | ✅ `914bb3b5…30df0b` |
|
||||
| Golden 0.113.0 published + anon-fetchable + sha-verified | ✅ `66de5e64…82054c` (612 257 363 B) |
|
||||
| Hub manifest vouches both | ✅ DB-verified |
|
||||
| Global floor 0.113.0 | ✅ hub_settings (UI) + env fallback (GitOps `16390d0`, Synced/Healthy) |
|
||||
| **Peti: agent 0.81.0 + controller 0.113.0, healthy** | ✅ consecutive ok reports; auto-confirm proof captured |
|
||||
| Demo: unchanged (0.113.0 / 0.81.0) | ✅ no churn |
|
||||
| Drill VM | ✅ reverted to virgin; bake log retained |
|
||||
| Phase E — journal group on Peti's host | ⏳ operator choice: OOB unlikely (WG down) → message Peti the one-liner (`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`) or defer (NAS errors degrade to generic text until then) |
|
||||
| G1 — delete Peti's temp Git Sync creds | ⏳ operator/guided-Peti: Settings → Git Sync → delete → verify "Registry: nyilvános (hitelesítés nélkül)" + `/api/selfupdate/check` ok (pulls are proven anonymous on his box as of this train) |
|
||||
| G2 — NAS page render on Peti's box | ⏳ optional; his first real NAS add is the first external run of the verify pipeline |
|
||||
| Hub bearer key rotation | ⏳ operator — see Security notes |
|
||||
Reference in New Issue
Block a user