diff --git a/REPORT.md b/REPORT.md index fe4b9c8..09b44f9 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,56 +1,49 @@ -# REPORT — Runbook refresh: RUNBOOK-onboarding-draft v3 → v4 (docs-only, XS) +# REPORT — Agent-only publish train 0.93.0 executed (operational run, docs-only in this repo) -**Date:** 2026-07-22 · **Repo:** `felhom.eu` · **Scope:** documentation only — no code, no version -bump, no deploy. The onboarding runbook was refreshed against what shipped 2026-07-18..21 and -against the second full onboarding on virgin hardware (HP t740, 2026-07-21). +**Date:** 2026-07-22 · **Repo:** `felhom.eu` (runbook record + §A5 discharge; no code, no version +bump, no hub deploy) · **Full record:** `documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md` -## What changed +## What happened -`documentation/runbooks/RUNBOOK-onboarding-draft-v3.md` → **`RUNBOOK-onboarding-draft-v4.md`** -(git mv, one commit), per the project-Claude refresh brief of 2026-07-21: +felhom-agent **v0.93.0** (hyphen-free escrow wordlist, source-only since 2026-07-21 at `a452dc33`) +was **built, published, manifest-vouched and deployed to both fleet boxes** by CC, with one +password-gated STOP → Viktor (the Day-0 manifest save, verified by CC in the hub DB **before** any +box moved — publish-train rule 1). -1. **R-39 interim blocks DELETED** (B4 descriptor caveat + E1 manual `pvesm status` check) — - replaced with one sentence each: the DR tier self-detects a dead credential (loud `auth_failed`) - and self-heals via damped re-issue; the hub gauge is trustworthy. Cited: `backlog/ROADMAP.md` - R-39 (CLOSED, proven live 2026-07-21, hub 0.68.1 + agent 0.91.2, 13 s chain - `applied → auth_failed → applied`). -2. **R-36 workaround block in C4 DELETED** (re-onboarding manual-Save) — replaced with the shipped - behaviour: enabled-but-unprovisioned raises an amber banner (hub v0.67.0). R-31 (Save-once/504) - and the R-32 orphan-card note KEPT. Consistency follow-through: A1 and the B3 note now say the - self-bind link is **auto-minted at create/RESET** (also R-36) — the manual mint is the re-mint - fallback. -3. **C1 freemail gate** — freemail.hu PROVEN 2026-07-21 (operator test-send received; R-4 COMPLETE - per `backlog/ROADMAP.md`); the check-spam habit line kept. -4. **A1 golden note** — golden 0.153.0 carries all four infra images; floor-lift proven on TWO - shapes (rehearsal 0.143→0.145 in 5 s; virgin HP t740 0.153.0→0.156.0 during day-0, `CONTEXT.md`). - "Rebuild before first tester" softened to a pointer at the standing rule in F. -5. **A3 loader note** — the single F1 line replaced with the loader table. **Deviation from the - brief, on the record's side:** the brief said "N100/AMI = shim"; the record - (`operations/nodes.md` + `tests/VALIDATION-n100-baremetal-2026-07-16.md` F1) says the N100 - installed with **mkimage, SB off** — the AMI AN3PLUS firmware cannot relocate the signed GRUB, - which is exactly why shim was not usable there. Table written per the record, as the brief's - "(SB state as recorded)" directs. HP t740 = shim with Secure Boot ENABLED (2026-07-21). -6. **B2/B3 second datapoint** — the HP onboarding walked register → pairing banner → unclaimed - list → self-bind (code + passphrase) → day-0 → dashboard on never-seen hardware (cited - `CONTEXT.md` + `operations/nodes.md`). New B2 gotcha: on multi-NIC boards verify the CABLED - port got the lease during install (the t740 4-port trap; R-59/R-60 pending). -7. **New A5 checklist item** — agent train ≥ v0.93.0 (hyphen-free recovery wordlist, - `felhom-agent/CHANGELOG.md`) must be BUILT + DEPLOYED to the customer's box before the FIRST - real tester's escrow ceremony; ceremonies run box-side, source-only does not count. -8. **D offboarding pointer** — R-25b ruling noted: full teardown = DELETE cascade (TASK-I), - RESET = identity-preserving re-onboarding; pointer to §G. -9. **KEPT unchanged:** DRAFT status, the C7 graduation-gate paragraph (still the last unwalked - step), all reference wall-clocks, all F rules, all of §G. +- **Artifact:** sha256 `a68b2ff73200622ea9e23aed71a0a8ff885808626cab9e9d554a5a5df5ced28f` + (13 817 690 bytes), sha-chain identical: built binary → Gitea package (PUT 201) → independent + anonymous GET. +- **Manifest:** agent 0.92.1→**0.93.0** + new sha. Golden (0.153.0), floor (**0.156.0**) and + wrapper sha **byte-identical before/after**. **Deviation, ruled:** MinAgent moved 0.92.1→0.93.0 + at the operator save (spec said do-not-touch); flagged before deploy, Viktor ruled leave it — + the held-floor window was transient (minutes) and harmless. +- **Deploys** (felhom-pve first, then demo-hp over G1 break-glass — secret file→file, shredded): + both boxes clean-restart **5/5** — active/NRestarts=0, capabilities **68/68 degraded=0**, + ReassertGuestBinds touched only enrolled drives, guestnet sweeps quiet (0 ERROR/WARN), hub + `hosts.agent_version` = **0.93.0** for both (06:56:52Z / 06:57:52Z). `.bak-0.92.1` retained on + both; escape hatch not needed. +- **Green gate** on the exact HEAD: build/vet/test green; the ex-flaky escrow package **10/10** + fresh runs (`-count=10`) — that flake was this defect, so the clean run is itself evidence. -Also: `backlog/ROADMAP.md` R-25b's live pointer updated `…draft-v3.md` → `…draft-v4.md` so the -citation resolves (historical references in the rehearsal validation doc and past REPORTs left -as-is — they describe the file as it was named then). +## Changes in this repo -## Not done / unchanged +1. **`documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`** — the executed record + (timestamps, hashes, per-box evidence, the MinAgent deviation + ruling, observations). +2. **`documentation/runbooks/RUNBOOK-onboarding-draft-v4.md`** §A item 5 — the "agent ≥ 0.93.0 + box-side before the first real escrow ceremony" gate marked **DISCHARGED 2026-07-22**, citing + the runbook record. (The spec pointed at a ROADMAP "wordlist row" — **none exists**; the gate's + real home is §A5, so the annotation went there. Recorded as a deviation, not improvised into a + new ROADMAP row.) -- C7 not flipped; the capability-map row untouched; no other runbook touched. -- No CHANGELOG entry: this repo's changelogs are per-area (`hub/`, `scripts/`, `website/`) and none - of those areas changed; prior docs-only commits (e.g. TASK-H nodes.md) followed the same pattern. +## Fleet after this train -*(Previous REPORT — TASK-I, hub v0.69.0/v0.70.0 DELETE cascade — preserved in git history at -`9b3381b` and summarized in `backlog/ROADMAP.md` R-25b.)* +Both boxes agent **0.93.0**; Day-0 manifest agent 0.93.0 / MinAgent 0.93.0 / golden 0.153.0 / +floor 0.156.0. `peti-felhom` untouched by design — Friday's reinstall pulls the vouched manifest +and lands on 0.93.0. + +## Observations (not acted on) + +- demo-hp's 1TB NVMe is now **enrolled** (`/mnt/nvme-1tb`, reasserted into 9201) — TASK-H's + "unenrolled/NTFS" note is outdated; `operations/nodes.md` may want a refresh. +- demo-hp still key-less (break-glass worked cleanly); tailnet key-expiry toggle still open. +- Host journals are CEST; all runbook evidence stamps note the offset. diff --git a/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md b/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md new file mode 100644 index 0000000..145770e --- /dev/null +++ b/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md @@ -0,0 +1,121 @@ +# RUNBOOK — agent-only publish train 0.93.0: hyphen-free escrow wordlist reaches the fleet (2026-07-22) + +**Class:** Runbook-style operational run (NO code changed anywhere — v0.93.0 source was already on +`main` at `a452dc33`/`eba040d0`). **Executor:** CC, with one password-gated **STOP → Viktor** point +(hub Day-0 manifest save). **Status: ALL PHASES COMPLETE.** Both boxes on agent 0.93.0, manifest +vouched, controller floor + golden byte-identical before/after. + +**Why:** (1) `runbooks/RUNBOOK-onboarding-draft-v4.md` §A item 5 — agent ≥ 0.93.0 must be BUILT + +DEPLOYED box-side before the first real tester's escrow ceremony (source-only does not count); +(2) Friday's Peti reinstall pulls the Day-0 manifest — vouched at 0.93.0, his box lands on the fixed +wordlist with zero extra work. + +--- + +## Results at a glance + +| Phase | What | Outcome | +|---|---|---| +| 0 | baselines + pre-gates + green gate | ✅ all confirmed at live source | +| 1 | build + publish felhom-agent 0.93.0 | ✅ sha-chain identical build → package → anon GET | +| 2 | **STOP → Viktor:** Day-0 manifest vouch | ✅ saved + CC-verified in hub DB **before any deploy** (one deviation, ruled — below) | +| 3 | deploy felhom-pve → demo-hp | ✅ both clean-restart 5/5, hub confirms 0.93.0 | +| 4 | docs (this file, REPORTs, §A5 discharge) | ✅ | + +``` +AGENT_VERSION=0.93.0 +AGENT_SHA256=a68b2ff73200622ea9e23aed71a0a8ff885808626cab9e9d554a5a5df5ced28f (13 817 690 bytes) +Package: https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.93.0/felhom-agent +``` + +## Phase 0 — live baseline confirmation ✅ + +All re-confirmed at source 2026-07-22 (~06:45–06:52 UTC), not trusted from the spec: + +- **felhom-agent `main`** = `eba040d0` = origin/main, clean tree; CHANGELOG head **v0.93.0** + (fix commit `a452dc33`). `git diff --stat 935904f..HEAD`: escrow code + tests + docs only — + **no `configs/` change** → binary-only deploy, no sudoers/wrapper ship. +- **Both boxes 0.92.1**, from the hub `hosts` table (read-only streamed DB copy) AND direct + `felhom-agent --version` on each: `demo-felhom-8363b5` (last report 06:47:10Z), + `demo-hp-bb76ea` (06:35:43Z). Both services `active`. +- **Day-0 manifest before** (hub DB `hub_settings`): agent **0.92.1** / `7424bc1c3c533eff…`, + golden 0.153.0 / `15fdd191…`, MinAgent 0.92.1, floor `min_controller_version` **0.156.0**, + wrapper sha `104db0a4…`. Hub live at **0.70.0** (deploy image tag). +- **Package pre-gate:** `felhom-agent/0.93.0` anon GET = **404** (immutability respected). +- **Green gate on the exact HEAD:** `go build ./... && go vet ./... && go test ./...` — green. + **The ex-flaky escrow package additionally run `-count=10` fresh: 10/10 pass** (135.2 s). That + flake WAS this defect (~23 % per 50-draw run pre-fix), so a clean 10× run is itself evidence. +- **Deploy channel** (per `RUNBOOK-publish-0.90-0.143-2026-07-18.md` Phase 1 + the + felhom-build-deploy skill agent legs): build local (DooPlex) → `publish-agent.sh` → + scp one hop → `.bak-` → `install -m0755` → restart → journal + hub verification. +- **demo-hp access:** key auth denied (no key baked — TASK-H posture unchanged) → hub-vaulted G1 + break-glass (`host_recovery` row `demo-hp-bb76ea`), extracted file→file (never stdout), used via + `SSH_ASKPASS`, **shredded after** together with all hub DB copies. + +## Phase 1 — build + publish ✅ + +- Built on DooPlex from `eba040d0`: `CGO_ENABLED=0 go build -ldflags '-X main.version=0.93.0'` + (matching `publish-agent.sh`'s own build line). `--version` → `felhom-agent 0.93.0`. +- `scripts/publish-agent.sh 0.93.0 `: pre-delete **404**, PUT **201**, authed round-trip + GET sha match. Creds sourced from the build host's docker-config gitea auth, never echoed. +- **Sha-chain, three independent points, identical:** built binary → published package → + independent **anonymous** GET (HTTP 200, 13 817 690 bytes) — all + `a68b2ff73200622ea9e23aed71a0a8ff885808626cab9e9d554a5a5df5ced28f`. + +## Phase 2 — Day-0 manifest vouch (rule 1: before any box moved) ✅ + one ruled deviation + +Viktor saved the manifest (~06:53 UTC); CC verified in the live hub DB **before Phase 3 started**: +`artifact_agent_version=0.93.0`, `artifact_agent_sha256=a68b2ff7…` — exact published values. +Golden (0.153.0 / `15fdd191…`), floor (0.156.0) and wrapper sha (`104db0a4…`) byte-identical. + +**Deviation, surfaced + ruled:** `artifact_min_agent` moved **0.92.1 → 0.93.0** at the save (spec +said MinAgent do-not-touch; rule 3 has MinAgent declared by the controller release, and 0.156.0's +declaration was 0.92.1). Flagged to Viktor before deploying; **ruled: leave at 0.93.0.** Practical +effect was a transient held-floor window (minutes) for boxes still reporting 0.92.1 — harmless, +both already ran controller ≥ 0.156.0 — and it closed when both boxes reported 0.93.0 +(06:56:52Z / 06:57:52Z). Recorded, not papered over. + +## Phase 3 — deploy, felhom-pve first, then demo-hp ✅ + +Per box: scp → **sha256 verified on-box = `a68b2ff7…`** → `cp` to `.bak-0.92.1` → +`install -m0755` → `systemctl restart`. Journal timestamps below are **host-local CEST = UTC+2**. + +| | felhom-pve (`demo-felhom-8363b5`) | demo-hp (`demo-hp-bb76ea`, break-glass) | +|---|---|---| +| restart (UTC) | 06:56:49 | 06:57:48 | +| `--version` / active / NRestarts | 0.93.0 / active / 0 | 0.93.0 / active / 0 | +| ① no restart loop, clean startup | ✅ `daemon starting version=0.93.0` | ✅ same | +| ② capability rows | ✅ **68/68 ok, degraded=0, inactive=0** | ✅ **68/68 ok, degraded=0, inactive=0** | +| ③ ReassertGuestBinds | ✅ bound ONLY the enrolled `uuid:47a3361a…` (`/mnt/hdd_1` → 9201); nothing non-enrolled | ✅ bound ONLY the enrolled `uuid:91d2dc2d…` (`/mnt/nvme-1tb` → 9201); nothing non-enrolled | +| ④ guestnet sweep | ✅ watchdog started; several 1-min sweeps: **0 heal/error lines, 0 ERROR/WARN total** | ✅ same | +| ⑤ hub `agent_version` | ✅ **0.93.0** (report landed at startup, 06:56:52Z) | ✅ **0.93.0** (06:57:52Z) | + +Also observed healthy on felhom-pve: desired-state synced (gen 5), PBS verify cycle complete +(felhom-offsite, 1 snapshot); on demo-hp: desired gen 2, PBS verify cycle (0 snapshots — no PBS-DR +descriptor yet). Escape hatch (restore `.bak-0.92.1`) **not needed**; the backups remain on both +boxes. + +## Post-train state + +- Fleet: **both boxes agent 0.93.0**, reporting, all checks green. +- Day-0 manifest: agent **0.93.0 / a68b2ff7…**, MinAgent **0.93.0** (ruled), golden 0.153.0, + floor 0.156.0, wrapper `104db0a4…` — floor + golden byte-identical before/after this train. +- `RUNBOOK-onboarding-draft-v4.md` **§A item 5 gate = DISCHARGED** for the current fleet and for + every future manifest-driven install (Friday's Peti reinstall included) — annotated in place. +- `peti-felhom` deliberately NOT deployed to (box down; gets 0.93.0 via Friday's reinstall + the + vouched manifest — by design). + +## Observations (recorded, NOT acted on) + +1. **The spec's "ROADMAP wordlist row" does not exist** — no wordlist/hyphen row anywhere in + `backlog/ROADMAP.md`. The gate lives in `runbooks/RUNBOOK-onboarding-draft-v4.md` §A item 5; + the discharge annotation went there (the coupling's real home) instead. +2. **demo-hp's 1TB NVMe is now ENROLLED** (`/mnt/nvme-1tb`, `uuid:91d2dc2d…`, reasserted into + 9201) — TASK-H (2026-07-21) recorded it unenrolled/NTFS. It evidently joined via Tárhely as + designed. Fleet notes (`operations/nodes.md`) may want updating. +3. **demo-hp remains key-less** — this train deployed over G1 break-glass (worked cleanly, secret + file→file + shredded). Standing open item: the tailnet key expiry (2027-01-17) toggle. +4. `felhom-agent/CONTEXT.md`'s top bullet ("agent DOWN at remote site, deployed 0.90.0") was stale + history — superseded note added this train. +5. Journal `--since` on the boxes is interpreted in **host-local CEST**; first query this run was + 2 h off before correction (the standing host=CEST vs UTC gotcha). diff --git a/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md b/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md index 0d75e5a..70e69bc 100644 --- a/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md +++ b/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md @@ -54,6 +54,10 @@ (`felhom-agent/CHANGELOG.md` v0.93.0, 2026-07-21) — ceremonies run **box-side**, so the fix protects a ceremony only once the train is **BUILT + DEPLOYED to that box**; source-only does not count. Already-issued codes stay valid. + **DISCHARGED 2026-07-22** — agent 0.93.0 published (sha `a68b2ff7…`), Day-0-manifest-vouched, + and deployed to both fleet boxes (`demo-felhom-8363b5` + `demo-hp-bb76ea`), clean-restart + verified; any future manifest-driven install also lands ≥ 0.93.0. Record: + `pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`. ## B. Install + bind (~15 min wall-clock)