122 lines
7.9 KiB
Markdown
122 lines
7.9 KiB
Markdown
# RUNBOOK — agent-only publish train 0.93.0: hyphen-free escrow wordlist reaches the fleet (2026-07-22)
|
||
|
||
**Class:** Runbook-style operational run (NO code changed anywhere — v0.93.0 source was already on
|
||
`main` at `a452dc33`/`eba040d0`). **Executor:** CC, with one password-gated **STOP → Viktor** point
|
||
(hub Day-0 manifest save). **Status: ALL PHASES COMPLETE.** Both boxes on agent 0.93.0, manifest
|
||
vouched, controller floor + golden byte-identical before/after.
|
||
|
||
**Why:** (1) `runbooks/RUNBOOK-onboarding-draft-v4.md` §A item 5 — agent ≥ 0.93.0 must be BUILT +
|
||
DEPLOYED box-side before the first real tester's escrow ceremony (source-only does not count);
|
||
(2) Friday's Peti reinstall pulls the Day-0 manifest — vouched at 0.93.0, his box lands on the fixed
|
||
wordlist with zero extra work.
|
||
|
||
---
|
||
|
||
## Results at a glance
|
||
|
||
| Phase | What | Outcome |
|
||
|---|---|---|
|
||
| 0 | baselines + pre-gates + green gate | ✅ all confirmed at live source |
|
||
| 1 | build + publish felhom-agent 0.93.0 | ✅ sha-chain identical build → package → anon GET |
|
||
| 2 | **STOP → Viktor:** Day-0 manifest vouch | ✅ saved + CC-verified in hub DB **before any deploy** (one deviation, ruled — below) |
|
||
| 3 | deploy felhom-pve → demo-hp | ✅ both clean-restart 5/5, hub confirms 0.93.0 |
|
||
| 4 | docs (this file, REPORTs, §A5 discharge) | ✅ |
|
||
|
||
```
|
||
AGENT_VERSION=0.93.0
|
||
AGENT_SHA256=a68b2ff73200622ea9e23aed71a0a8ff885808626cab9e9d554a5a5df5ced28f (13 817 690 bytes)
|
||
Package: https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.93.0/felhom-agent
|
||
```
|
||
|
||
## Phase 0 — live baseline confirmation ✅
|
||
|
||
All re-confirmed at source 2026-07-22 (~06:45–06:52 UTC), not trusted from the spec:
|
||
|
||
- **felhom-agent `main`** = `eba040d0` = origin/main, clean tree; CHANGELOG head **v0.93.0**
|
||
(fix commit `a452dc33`). `git diff --stat 935904f..HEAD`: escrow code + tests + docs only —
|
||
**no `configs/` change** → binary-only deploy, no sudoers/wrapper ship.
|
||
- **Both boxes 0.92.1**, from the hub `hosts` table (read-only streamed DB copy) AND direct
|
||
`felhom-agent --version` on each: `demo-felhom-8363b5` (last report 06:47:10Z),
|
||
`demo-hp-bb76ea` (06:35:43Z). Both services `active`.
|
||
- **Day-0 manifest before** (hub DB `hub_settings`): agent **0.92.1** / `7424bc1c3c533eff…`,
|
||
golden 0.153.0 / `15fdd191…`, MinAgent 0.92.1, floor `min_controller_version` **0.156.0**,
|
||
wrapper sha `104db0a4…`. Hub live at **0.70.0** (deploy image tag).
|
||
- **Package pre-gate:** `felhom-agent/0.93.0` anon GET = **404** (immutability respected).
|
||
- **Green gate on the exact HEAD:** `go build ./... && go vet ./... && go test ./...` — green.
|
||
**The ex-flaky escrow package additionally run `-count=10` fresh: 10/10 pass** (135.2 s). That
|
||
flake WAS this defect (~23 % per 50-draw run pre-fix), so a clean 10× run is itself evidence.
|
||
- **Deploy channel** (per `RUNBOOK-publish-0.90-0.143-2026-07-18.md` Phase 1 + the
|
||
felhom-build-deploy skill agent legs): build local (DooPlex) → `publish-agent.sh` →
|
||
scp one hop → `.bak-<old>` → `install -m0755` → restart → journal + hub verification.
|
||
- **demo-hp access:** key auth denied (no key baked — TASK-H posture unchanged) → hub-vaulted G1
|
||
break-glass (`host_recovery` row `demo-hp-bb76ea`), extracted file→file (never stdout), used via
|
||
`SSH_ASKPASS`, **shredded after** together with all hub DB copies.
|
||
|
||
## Phase 1 — build + publish ✅
|
||
|
||
- Built on DooPlex from `eba040d0`: `CGO_ENABLED=0 go build -ldflags '-X main.version=0.93.0'`
|
||
(matching `publish-agent.sh`'s own build line). `--version` → `felhom-agent 0.93.0`.
|
||
- `scripts/publish-agent.sh 0.93.0 <binary>`: pre-delete **404**, PUT **201**, authed round-trip
|
||
GET sha match. Creds sourced from the build host's docker-config gitea auth, never echoed.
|
||
- **Sha-chain, three independent points, identical:** built binary → published package →
|
||
independent **anonymous** GET (HTTP 200, 13 817 690 bytes) — all
|
||
`a68b2ff73200622ea9e23aed71a0a8ff885808626cab9e9d554a5a5df5ced28f`.
|
||
|
||
## Phase 2 — Day-0 manifest vouch (rule 1: before any box moved) ✅ + one ruled deviation
|
||
|
||
Viktor saved the manifest (~06:53 UTC); CC verified in the live hub DB **before Phase 3 started**:
|
||
`artifact_agent_version=0.93.0`, `artifact_agent_sha256=a68b2ff7…` — exact published values.
|
||
Golden (0.153.0 / `15fdd191…`), floor (0.156.0) and wrapper sha (`104db0a4…`) byte-identical.
|
||
|
||
**Deviation, surfaced + ruled:** `artifact_min_agent` moved **0.92.1 → 0.93.0** at the save (spec
|
||
said MinAgent do-not-touch; rule 3 has MinAgent declared by the controller release, and 0.156.0's
|
||
declaration was 0.92.1). Flagged to Viktor before deploying; **ruled: leave at 0.93.0.** Practical
|
||
effect was a transient held-floor window (minutes) for boxes still reporting 0.92.1 — harmless,
|
||
both already ran controller ≥ 0.156.0 — and it closed when both boxes reported 0.93.0
|
||
(06:56:52Z / 06:57:52Z). Recorded, not papered over.
|
||
|
||
## Phase 3 — deploy, felhom-pve first, then demo-hp ✅
|
||
|
||
Per box: scp → **sha256 verified on-box = `a68b2ff7…`** → `cp` to `.bak-0.92.1` →
|
||
`install -m0755` → `systemctl restart`. Journal timestamps below are **host-local CEST = UTC+2**.
|
||
|
||
| | felhom-pve (`demo-felhom-8363b5`) | demo-hp (`demo-hp-bb76ea`, break-glass) |
|
||
|---|---|---|
|
||
| restart (UTC) | 06:56:49 | 06:57:48 |
|
||
| `--version` / active / NRestarts | 0.93.0 / active / 0 | 0.93.0 / active / 0 |
|
||
| ① no restart loop, clean startup | ✅ `daemon starting version=0.93.0` | ✅ same |
|
||
| ② capability rows | ✅ **68/68 ok, degraded=0, inactive=0** | ✅ **68/68 ok, degraded=0, inactive=0** |
|
||
| ③ ReassertGuestBinds | ✅ bound ONLY the enrolled `uuid:47a3361a…` (`/mnt/hdd_1` → 9201); nothing non-enrolled | ✅ bound ONLY the enrolled `uuid:91d2dc2d…` (`/mnt/nvme-1tb` → 9201); nothing non-enrolled |
|
||
| ④ guestnet sweep | ✅ watchdog started; several 1-min sweeps: **0 heal/error lines, 0 ERROR/WARN total** | ✅ same |
|
||
| ⑤ hub `agent_version` | ✅ **0.93.0** (report landed at startup, 06:56:52Z) | ✅ **0.93.0** (06:57:52Z) |
|
||
|
||
Also observed healthy on felhom-pve: desired-state synced (gen 5), PBS verify cycle complete
|
||
(felhom-offsite, 1 snapshot); on demo-hp: desired gen 2, PBS verify cycle (0 snapshots — no PBS-DR
|
||
descriptor yet). Escape hatch (restore `.bak-0.92.1`) **not needed**; the backups remain on both
|
||
boxes.
|
||
|
||
## Post-train state
|
||
|
||
- Fleet: **both boxes agent 0.93.0**, reporting, all checks green.
|
||
- Day-0 manifest: agent **0.93.0 / a68b2ff7…**, MinAgent **0.93.0** (ruled), golden 0.153.0,
|
||
floor 0.156.0, wrapper `104db0a4…` — floor + golden byte-identical before/after this train.
|
||
- `RUNBOOK-onboarding-draft-v4.md` **§A item 5 gate = DISCHARGED** for the current fleet and for
|
||
every future manifest-driven install (Friday's Peti reinstall included) — annotated in place.
|
||
- `peti-felhom` deliberately NOT deployed to (box down; gets 0.93.0 via Friday's reinstall + the
|
||
vouched manifest — by design).
|
||
|
||
## Observations (recorded, NOT acted on)
|
||
|
||
1. **The spec's "ROADMAP wordlist row" does not exist** — no wordlist/hyphen row anywhere in
|
||
`backlog/ROADMAP.md`. The gate lives in `runbooks/RUNBOOK-onboarding-draft-v4.md` §A item 5;
|
||
the discharge annotation went there (the coupling's real home) instead.
|
||
2. **demo-hp's 1TB NVMe is now ENROLLED** (`/mnt/nvme-1tb`, `uuid:91d2dc2d…`, reasserted into
|
||
9201) — TASK-H (2026-07-21) recorded it unenrolled/NTFS. It evidently joined via Tárhely as
|
||
designed. Fleet notes (`operations/nodes.md`) may want updating.
|
||
3. **demo-hp remains key-less** — this train deployed over G1 break-glass (worked cleanly, secret
|
||
file→file + shredded). Standing open item: the tailnet key expiry (2027-01-17) toggle.
|
||
4. `felhom-agent/CONTEXT.md`'s top bullet ("agent DOWN at remote site, deployed 0.90.0") was stale
|
||
history — superseded note added this train.
|
||
5. Journal `--since` on the boxes is interpreted in **host-local CEST**; first query this run was
|
||
2 h off before correction (the standing host=CEST vs UTC gotcha).
|