118 lines
7.5 KiB
Markdown
118 lines
7.5 KiB
Markdown
# RUNBOOK results — publish train agent 0.85.0 + golden 0.120.0 + floor (fleet half; Peti parked) (2026-07-12)
|
||
|
||
**Class:** publish/signing runbook (GL-1 pattern) — the CAMPAIGN-3 wave fleet train.
|
||
**Status:** Fleet half **COMPLETE** — Phases 0/A/B/C done, Phase F floor moved + verified on **both**
|
||
sources (env GitOps + DB hub_settings, aligned at 0.120.0). Phase D/E/G (Peti `peti-felhom-86d37d`,
|
||
sajatfelhom.hu) **DEFERRED — no ETA**: his server is offline (dead fan) with no known return date, so
|
||
no op was signed for it. The MinAgent-0.81 gate protects him automatically whenever he does return;
|
||
the fleet is not waiting on him. No credential or signature-key material in this doc.
|
||
|
||
## The operator values (saved in the hub Day-0 manifest — UI-confirmed byte-identical)
|
||
|
||
```
|
||
AGENT_VERSION=0.85.0
|
||
AGENT_SHA256=31babb2c4f8fa5a1f961428519a0dfbb3a19b6e536fba01c2664afca339da93d
|
||
GOLDEN_VERSION=0.120.0
|
||
GOLDEN_SHA256=f7d7d02c76b49891719df9cf68624c9ae193e438e2a3a63b948a2a9d12887596
|
||
MIN_AGENT=0.81.0
|
||
FLOOR (env, GitOps 0eb9dc2)=0.120.0 ← live in the pod
|
||
FLOOR (DB hub_settings)=0.120.0 ← operator-saved 11:19:43Z (hub-logged); effective floor = 0.120.0
|
||
```
|
||
|
||
## Gate outcomes (Phase 0)
|
||
|
||
| Gate | Result | Evidence |
|
||
|---|---|---|
|
||
| 0a Peti online | **OFFLINE (expected)** | dead fan; fleet publish proceeds, Peti rollout parks — never sign an op for an offline host |
|
||
| 0c immutability pre-gate | **PASS** | agent 0.85.0 GET **404** + golden 0.120.0 GET **404**; controls (agent 0.79.0 / golden 0.113.0) = 200 → proves anonymous GET works, so the 404s are truly-unpublished not auth failures |
|
||
| 0d live bytes | **PASS** | felhom-pve binary v0.85.0, sha `31babb2c…39da93d`, 13 629 759 B (deployed artifact of `main@bcb8dad2`) |
|
||
| 0e golden source | **PASS** | `docker manifest inspect …/felhom-controller:0.120.0` OK on 180 |
|
||
| 0f hub healthy | **PASS** | ArgoCD app `felhom` Synced/Healthy; hub image 0.48.0 |
|
||
| 0b/0g Peti creds+key | **PARKED** | Peti-only; deferred with D/E/G until his box returns |
|
||
|
||
## Phase A — agent 0.85.0 published
|
||
|
||
Live felhom-pve bytes; sha **identical across felhom-pve → CC → 180** (`31babb2c…39da93d`);
|
||
`publish-agent.sh 0.85.0`: pre-GET **404** → PUT **201** → round-trip sha match → **anonymous off-LAN
|
||
GET 200** (13 629 759 B, sha == AGENT_SHA256). Token from `~/.gitea-token` env-substituted on 180;
|
||
token-leak grep of the transcript: 0.
|
||
|
||
## Phase B — golden 0.120.0 baked + published
|
||
|
||
Recorded drill-VM procedure (RUNBOOK-publish-0.79-0.110 Phase C): `qemu-img snapshot -a virgin` →
|
||
cold boot → PVE 9.2.2 up on :2222 in ~9 s → debian-13 template `pveam download` (checksum OK; absent
|
||
on virgin) → `build-golden.sh` v2.0.0 as transient unit `golden-bake` baking
|
||
`felhom-controller:0.120.0`: `docker OK (overlay2)`, both extra volumes separate mounts, **all 3
|
||
mounts included** (rootfs+mp0+mp1, no exclusions/FATAL), archive 583 MB → pre-delete **404** → upload
|
||
**201**. Verified: integrity round-trip **anonymous GET | sha256sum == GOLDEN_SHA256** (612 309 180 B)
|
||
+ off-LAN ranged GET **206**. Teardown: guest 9100 purged, token/script/log **shredded in-VM**, clean
|
||
poweroff, qcow2 **reverted to virgin** (snapshot intact), `~/drill/bake-0.120.0.log` retained,
|
||
saved-log token-leak grep **0**.
|
||
|
||
## Phase C — hub Day-0 manifest (operator)
|
||
|
||
Saved by Viktor; UI banner "Artifact manifest saved" + the Day-0 card shows agent **0.85.0** /
|
||
`31babb2c…39da93d`, golden **0.120.0** / `f7d7d02c…887596`, Min agent **0.81.0**. The sha the hub
|
||
auto-reads from Gitea matched the published artifacts byte-for-byte — independent cross-validation of
|
||
the right bytes. Save did **not** move the floor (separate card since v0.45), as intended.
|
||
|
||
## Phase D/E/G — Peti (PARKED, box offline)
|
||
|
||
Not executed. Peti's box (`peti-felhom-86d37d`) is offline (dead fan). Per the runbook, no signed op
|
||
is issued for an offline host. When he returns, the sequence is: sign `agent_update` 0.79→0.85
|
||
(sha `31babb2c…39da93d`, ttl 30m) → confirm agent 0.85 reports → THEN his controller updates. Until
|
||
his agent reports ≥0.81 the conditional-floor gate **holds his controller at 0.113** — the intended
|
||
safety, not a fault. His 0.79.0 fallback stays published for a reverse op if needed
|
||
(sha `35a162547646737ebef265f5ad69d042e897914f6ca799c007872d1c3cd27561`). Journal-group one-liner (E)
|
||
and temp-Git-Sync-creds deletion (G1) travel with his return; both LOW priority as he has no NAS
|
||
share yet (F12 fix is latent-protective for him).
|
||
|
||
## Phase F — floor
|
||
|
||
**Env fallback (GitOps, CC):** `manifests/hub.yaml` `DEFAULT_MIN_CONTROLLER_VERSION` 0.113.0 →
|
||
**0.120.0** (commit `0eb9dc2`), ArgoCD hard-refresh + deliberate sync → **Synced/Healthy**, hub
|
||
rolled out, **live pod `printenv` == 0.120.0**. Demo: no churn (already 0.120.0).
|
||
|
||
**DB global floor (operator, LAST):** saved by Viktor to **0.120.0** at **11:19:43Z** — hub log
|
||
`Global controller-version floor set to "0.120.0"` (store write-path). DB wins over env, so the
|
||
fleet's **effective floor is now 0.120.0**, aligned with the env fallback. Blast radius was zero
|
||
online boxes (demo already 0.120; Peti offline). When Peti returns and his agent reaches ≥0.81, this
|
||
floor lands his controller at 0.120 automatically.
|
||
|
||
## Deviations from the letter of the runbook
|
||
|
||
1. **Peti offline at train time, no return ETA** → the entire Peti arc (D/E/G) deferred indefinitely;
|
||
only the fleet artifacts (A–C) + floor (F) ran. The fleet does not wait on him — his box is gated
|
||
(agent 0.79 < MinAgent 0.81) and the floor lands him whenever he returns + updates his agent.
|
||
2. **Floor moved cleanly on both sources this train** (env first via GitOps, then the operator's DB
|
||
save last per the 9-minute-skew rule) — no transient skew, unlike the 0.81/0.113 train where the
|
||
UI floor save acted before the agent op. Aligned at 0.120.0.
|
||
|
||
## Security notes
|
||
|
||
- No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)`
|
||
env-substitution on 180; drill-VM token shredded; saved-log leak grep 0.
|
||
- ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
|
||
carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern,
|
||
documentation/runbooks/secrets.md) as an operator hygiene task.~~
|
||
**DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY`
|
||
(literal removed from the manifest; `scripts/manifest_bearer_gate.py` blocks reintroduction).
|
||
The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md
|
||
§"Operator/global bearer key" — the git-history copy is dead only after it runs.
|
||
|
||
## END STATE
|
||
|
||
| Item | State |
|
||
|---|---|
|
||
| Agent 0.85.0 published + anon-fetchable + sha-verified | ✅ `31babb2c…39da93d` (13 629 759 B) |
|
||
| Golden 0.120.0 published + anon-fetchable + sha-verified | ✅ `f7d7d02c…887596` (612 309 180 B) |
|
||
| Hub manifest vouches both (MinAgent 0.81.0) | ✅ UI-confirmed |
|
||
| Env floor 0.120.0 (GitOps `0eb9dc2`) | ✅ Synced/Healthy, live pod verified |
|
||
| DB global floor 0.120.0 | ✅ operator-saved 11:19:43Z (hub-logged); effective floor 0.120.0, aligned with env |
|
||
| Demo: unchanged (0.120.0 / 0.85.0) | ✅ no churn |
|
||
| Drill VM | ✅ reverted to virgin; `~/drill/bake-0.120.0.log` retained |
|
||
| Peti agent 0.85 + controller 0.120 | ⏸ **DEFERRED — box offline (dead fan), no ETA**; fleet not waiting; gate holds him at 0.113 until his agent lands |
|
||
| Phase E journal group on Peti's host | ⏸ with his return (LOW — no NAS share yet) |
|
||
| G1 delete Peti temp Git Sync creds | ⏸ with his return |
|
||
| Hub bearer key rotation | ⏳ operator — see Security notes |
|