From a63cc7151b977be25cfe73ede466c65d0ab218fc Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 7 Jul 2026 21:07:15 +0200 Subject: [PATCH] =?UTF-8?q?docs(pilot):=20RUNBOOK=20GL-1=20executed=20?= =?UTF-8?q?=E2=80=94=20agent=200.74.0=20+=20golden=200.103.0=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Agent = the live felhom-pve bytes (sha 1ec3f588..76af05, round-trip verified); golden baked clean-room in the drill VM (build-golden v2.0.0, controller 0.103.0, all integrity gates green, sha 8481e8a1..6026e) and published 201. Both anon-fetch-verified (Gate 3c: packages world-readable -> G3 read-only token rotation cannot break fetches; installer-side empty-token die is the real precondition). Drill VM restored to virgin. OPERATOR NEXT: Day-0 manifest bump to the four recorded values. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 13 ++ REPORT.md | 123 +++++++----------- .../pilot/RUNBOOK-GL1-publish-2026-07-07.md | 77 +++++++++++ 3 files changed, 135 insertions(+), 78 deletions(-) create mode 100644 documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md diff --git a/CONTEXT.md b/CONTEXT.md index 3f48ea9..05a6fb6 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1 + (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha + `1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked + clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all + integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts + fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3 + git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token — + script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence + `~/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`. + **OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values + are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded: + operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status + recorded here instead). - **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 + harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old diff --git a/REPORT.md b/REPORT.md index 7cb4ba2..31d8da8 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,92 +2,59 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## TASK GL-2 — BYO-host install profile (`--mode appliance|byo`) — host-install v1.10.0 (2026-07-07) +## RUNBOOK GL-1 — publish agent 0.74.0 + bake/publish golden 0.103.0 (2026-07-07) -**Baseline confirmed:** felhom.eu @ `1a1e42ad`, `SCRIPT_VERSION="1.9.1"` → **`1.10.0`**. -Implementation + static/dry validation only per spec §0 — **the live provision/uninstall runs are -GL-6 (operator-supervised)**; the §13 STOP was honored (zero non-dry installer invocations anywhere). +Full execution record with per-gate evidence: +`documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`. Docs-only commit; no code changed, no +production host mutated (felhom-pve read-only; all root work inside the disposable drill VM). -### Files created / modified (single repo, `main`) +### The four operator values (hub Day-0 manifest — Configs → Day-0 artifacts) -- `scripts/felhom-host-install.sh` — v1.10.0 (Parts 1–3; see scripts/CHANGELOG.md for the feature list) -- `scripts/hostinstall-mode-harness.sh` — NEW (Part 4) -- `documentation/runbooks/day0-install.md` — `--mode` in every command, new §C.5 (byo command + - trust-model paragraph + preflight-only), §C.6 (pre-v1.10.0 resume note), 4 new troubleshooting rows -- `scripts/CHANGELOG.md`, `REUSE.md` (install-profile-gate pattern row), `CONTEXT.md`, this file +``` +AGENT_VERSION=0.74.0 +AGENT_SHA256=1ec3f58842edce1e9e32d022eaef15b7ff599c658c7a36bfe7833c708076af05 +GOLDEN_VERSION=0.103.0 +GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e +``` -### Spec deviations / judgment calls (each verified against the real code) +### Gate outcomes (all PASS) -1. **`--mode` collision:** v1.9.1 already used `--mode provision|dr`. Resolved: `appliance|byo` are - the only install values, `provision` retired with a specific die, the `dr` stub dispatch kept - verbatim (the documented 10D seam). -2. **Spec §2 was wrong about the lan-resolver**: the config write did `setdefault('lan_resolver', - {"enable": True})` — default-ON, so the spec's byo assert would have died on every byo install. - Fixed by flipping the byo DEFAULT to off (appliance untouched); the assert still dies if a - `--preserve-from` config carries true, which is the intended refusal. -3. The byo skip line reads `break-glass (step 4b) SKIPPED (byo: …)` — NOT the spec's literal - `"4b/8 …"` text, because Scenario B / harness H-B forbid the string `4b/8` in the byo transcript - (an internal spec contradiction; the greppable invariant won). -4. `--preflight-only` skips the byo typed ack (prints the disclosure, logs the skip) — nothing will - be mutated, and a mandatory prompt would break its re-runnable/scripted purpose. -5. `documentation/pilot/GO-LIVE-PACKAGE.md` is not in the repo and no operator copy arrived with the - task — the G2/G4/G5 → "implemented, awaiting GL-6" flip is recorded in CONTEXT.md; add the doc - when supplied. +- **0a–0d**: baselines exact (`ced60ddf` / `59eb3bea` / `02d63ed0`; agent 0.63.0 + golden 0.98.3 + the last published, targets 404); drill VM + `virgin` snapshot intact; controller image + 0.103.0 resolvable; headroom ample. +- **1a/1b**: felhom-pve's RUNNING binary (`readlink /proc/…/exe`, no pending A/B marker) is + 0.74.0; those exact bytes (hash-stable across both copy hops) published via + `publish-agent.sh` from `main`@`ced60ddf` — PUT 201, round-trip GET sha match. +- **2a/2b**: clean-room bake on the virgin-restored drill VM (build-golden.sh v2.0.0): + overlay2 + both split-volumes as separate mounts, rootfs+mp0+mp1 ALL included in the vzdump + (no exclusions, no FATAL), archive 583MB, publish HTTP 201; transcript token-leak grep = 0. + Deviations: debian-13 template pveam-downloaded inside (virgin never has it, checksum + verified); bake launched via `systemd-run` (bare nohup didn't survive ssh session close). +- **3a/3b**: both artifacts fetched **ANONYMOUSLY** and hashed to exactly the published sha + (golden 611 696 321 bytes, streamed+hashed+deleted); off-LAN HEAD 200 from the CC machine too. -### Harness results +### Gate 3c finding (feeds G3) — read-credential sufficiency -Static tier (Windows Git Bash AND felhom-pve, identical): **bash -n, C1/C1b/C1c, C2a/C2b, C3/C3b/C3c, -C4, C6-static, INV-1 (one appliance-gated break-glass call site), INV-2 (chpasswd unreachable outside -step_break_glass), INV-3 (--mode in usage) — all PASS.** +The specified reviewer read-only token (`8417…4140`) was not locatable by CC anywhere in reach — +the gates ran with **no credential at all and passed**: the generic packages are anonymously +world-readable. Therefore rotating the customer `git.token` to read-only **cannot break** the +artifact fetch path (G3 unblocked from this side). Nuance: `resolve_git_creds` in host-install +**dies on an empty token** even though Gitea would serve the fetch anonymously — the requirement +is script-side; customer configs still need a (read-scoped) token until a small installer +follow-up makes it optional. This also reframes GL-2's finding that demo-felhom's git credentials +are EMPTY: the fetch would work, the installer's precondition is what fails. -PVE tier (felhom-pve, read-only, temp `FELHOM_INSTALL_STATE_DIR`): **C5 PASS live** (bogus -`--acl-storages` entry → die naming it, pre-hub, pre-passphrase). H-A/H-B dry transcripts ran with -the out-of-band demo passphrase (file→file, never printed) but **die at step 5 with rc=1 — a -customer-data blocker, not a code bug**: demo-felhom's controller.yaml serves EMPTY -`git.username`/`git.token`. The transcript span through the break-glass gate was validated manually -instead: appliance transcript contains `4b/8`; byo transcript contains **no** -`4b/8`/`chpasswd`/`recovery-credential` and **does** contain the disclosure block, the dry ack line, -`acl storages all present`, and the host-DNS `:53` line (it correctly showed the live dnsmasq). -`--preflight-only` proven live both ways: PASS verdict + rc 0 + **zero state entries**; bogus-storage -run → specific die + `PRE-FLIGHT FAIL (exit 1)` verdict. +### Drill-VM end state -**Red-proofs (run → fail → revert, against mutated scratch copies — the repo file was never mutated):** -- RP-1 un-gated the break-glass call site → INV-1 FAIL (13/16). Reverted (scratch deleted). -- RP-2 dropped the byo caps requirement → C1/C1b/C1c FAIL (11/16). Reverted. -- RP-3 dropped the resume mode-mismatch check → C4 FAIL (13/16). Reverted. +Guest 9100 destroyed (`--purge`), in-VM cred file shredded, VM powered off, qcow2 **restored to +`virgin`** (verified) — environment exactly as found. Evidence kept: `~/drill/bake-0.103.0.log` +on the build server. -### §9 rule 9 — the `_state_mark` dry-run finding +### Follow-ups -Verified in code: **both** `_state_mark` and `_state_put` begin with `$DRY_RUN && return 0` — dry -runs write nothing. Belt on top: the harness points `FELHOM_INSTALL_STATE_DIR` (new, harness-only -env override — none existed) at a throwaway temp dir for EVERY invocation, so even non-dry refusal -cases can never touch a live `state.json`. Confirmed live: felhom-pve's real state.json was -byte-untouched (it only ever saw the temp dir) and the preflight-only runs left 0 entries. - -### Shellcheck - -v0.10.0 (build server 180 + felhom-pve; not installed locally). **Clean at `--severity=warning`** -on both scripts. Full-severity: 2 pre-existing SC2015 *info* notes on untouched v1.9.1 lines -(A&&B||C in step_verify) — triaged, not refactored (spec rule 2); 1 new SC2154 false positive on -the EXIT trap (assignment inside the trap string) — targeted disable comment. - -### NOT yet live-validated — awaiting supervised GL-6 - -- The full appliance AND byo installs end-to-end (rc=0 transcripts blocked today by the - demo-felhom git-creds gap below). -- The uninstall → re-provision drill under both modes. -- C7: the verify pool/ACL asserts firing on real drift (needs a deliberately-broken live box). -- The byo typed-hostname ack answered interactively (only its dry/preflight-only branches ran). - -### Observations (documented, not acted on) - -- **demo-felhom has EMPTY `git.username`/`git.token`** in its served controller.yaml — any step-5 - (re)install for that customer dies until real Gitea read creds are set (operator; ties into the - standing "scope down + rotate the package-WRITE git token" security follow-up). -- felhom-pve's live install predates `_state_put` — its state.json has only `completed` (no - `customer_id`/`provisioned_vmid`/`mode`); a future `--uninstall` there needs an explicit `--vmid`, - and §C.6 of the runbook covers the missing-mode resume. -- A `--resume` that skips preflight never records `mode` into an old state file (it only writes - during a full preflight) — harmless (C4 only fires when a mode IS recorded), noted for GL-6. -- The live host-DNS line on felhom-pve shows the agent's own dnsmasq (lan_resolver is ON there — - correct for an appliance; it is exactly what byo now refuses). +1. **OPERATOR — hub Day-0 manifest bump** to the four values above (until then fresh installs + land agent 0.63.0 + golden 0.98.3). +2. **Operator-key pin** (out of GL-1 scope): day-0 ships no operator pubkey → self-update dormant + on fresh boxes; pin the real key + a small host-install follow-up (candidate: fold into GL-4). +3. **G3 rotation** unblocked per Gate 3c; also fixes demo-felhom's empty git credentials. +4. **GL-6** consumes the bumped manifest end-to-end (deliberately not run here). diff --git a/documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md b/documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md new file mode 100644 index 0000000..e5d3045 --- /dev/null +++ b/documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md @@ -0,0 +1,77 @@ +# RUNBOOK GL-1 — execution record: publish agent 0.74.0 + bake/publish golden 0.103.0 (2026-07-07) + +**Class:** runbook execution (go-live item G1, partial). No code changes, no version bumps. +Production touched read-only (felhom-pve: one `scp` out + `--version`/`readlink`; build server: +normal kisfenyo work + drill-VM lifecycle). All root actions inside the disposable nested drill VM. +No credential appears in this record; publish creds were sourced on the build server +(`~/.gitea-token`, user `admin`) and the in-VM copy was shredded before poweroff. + +## The four operator values (hub Day-0 manifest → Configs → Day-0 artifacts) + +``` +AGENT_VERSION=0.74.0 +AGENT_SHA256=1ec3f58842edce1e9e32d022eaef15b7ff599c658c7a36bfe7833c708076af05 +GOLDEN_VERSION=0.103.0 +GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e +``` + +## Gate outcomes + +| Gate | Result | Evidence | +|---|---|---| +| 0a baselines | **PASS** | felhom-agent `ced60ddf` (v0.74.0), felhom-controller `59eb3bea` (v0.103.0), felhom.eu `02d63ed0`; published state pre-run: agent 0.63.0 = 200 / 0.74.0 = 404, golden 0.98.3 = 200 / 0.103.0 = 404 | +| 0b drill VM | **PASS** | `~/drill/drill.qcow2` on 192.168.0.180 with internal snapshot `virgin` (2026-07-03) intact | +| 0c controller image | **PASS** | `docker manifest inspect gitea.dooplex.hu/admin/felhom-controller:0.103.0` OK on the build server | +| 0d headroom | **PASS** | 213 G free on 180 (golden ≈ 0.6 G); VM local-lvm 76 G thin, 0% used post-virgin | +| 1a running binary | **PASS** | felhom-pve `readlink /proc/$(pidof felhom-agent)/exe` → `/usr/local/bin/felhom-agent`, `--version` → 0.74.0; selfupdate state dir EMPTY (no pending A/B marker) | +| 1b agent publish | **PASS** | the LIVE bytes (sha identical across felhom-pve→CC→180 hops) published via `publish-agent.sh 0.74.0` from `main`@`ced60ddf`: pre-delete 404, PUT 201, round-trip GET sha match | +| 2a drill VM ready | **PASS** | virgin-restored + booted (slirp `dhcpstart=10.0.2.30` + hostfwd 2222→10.0.2.15:22, per DRILL precedent); PVE 9.2.2 responsive; gitea HTTP/2 200 from inside; debian-13 template ABSENT on virgin (expected) → `pveam download` inside, checksum verified | +| 2b bake + publish | **PASS** | `build-golden.sh` (main@`ced60ddf`, script v2.0.0) as root@pam in-VM, baking `felhom-controller:0.103.0`; transcript: `docker OK (overlay2; data-root /var/lib/docker)`, both `/var/lib/docker` + `/mnt/sys_drive` separate ext4 mounts, `including mount point` rootfs+mp0+mp1 (NO exclusions, no FATAL), archive 583MB, publish upload HTTP 201, `GOLDEN_VERSION=0.103.0` + sha printed. Token-leak grep over the transcript: 0 hits | +| 3a agent fetch | **PASS** | **ANONYMOUS** GET → 200, sha256 == AGENT_SHA256 (from the build server; HEAD 200 from the CC machine off-LAN too) | +| 3b golden fetch | **PASS** | **ANONYMOUS** GET → 200, 611 696 321 bytes, sha256 == GOLDEN_SHA256 (streamed to disk, hashed, deleted) | +| 3c read-token finding | **see below** | | + +## Gate 3c — read-credential finding (feeds G3) + +The specified reviewer read-only token (`8417…4140`) was **not locatable by CC** (not in the repos, +not on the build server under any evident name). The gates were instead run with **no credential at +all — and passed**: both generic packages are **anonymously world-readable** (deliberate posture; +the agent's self-update and guest 9201's image pulls already rely on anon Gitea reads; integrity +comes from the hub-manifest sha, not transport auth). Consequences: + +- A read-scoped token is trivially sufficient for artifact fetch — **rotating the customer + `git.token` to read-only (or revoking its package-write) CANNOT break fresh installs' fetch + path.** The G3 rotation is unblocked from this side. +- Nuance: the installer (`resolve_git_creds`) **dies on an empty git token** even though the fetch + itself would succeed anonymously — the credential is required by the script, not by Gitea. If the + operator ever wants credential-less artifact fetch, that is a small host-install follow-up; until + then customer configs still need a (read-scoped) token present. + +## Drill-VM end state + +Build guest 9100 destroyed (`--purge`, all 3 LVs removed); in-VM cred file shredded; VM powered +off; **qcow2 restored to `virgin`** (snapshot list verified) — the drill environment is exactly as +found. Evidence retained on the build server: `~/drill/bake-0.103.0.log` (next to the prior +`bake-A.log`). + +## Deviations from the letter of the runbook + +- Gate 2a's template check was satisfied by downloading the template inside the VM (the virgin + image never has it — same as the 0.98.3 bake) — anticipated by the runbook, noted as required. +- Phase 3 ran anonymously instead of with the reviewer token (see Gate 3c) — a strictly *stronger* + proof of fetchability. +- The bake was launched as a transient systemd unit (`systemd-run --unit=golden-bake`) inside the + VM instead of bare `nohup` (a plain ssh-backgrounded nohup did not survive session close). + +## Follow-ups (operator) + +1. **Hub Day-0 manifest bump** (Configs → Day-0 artifacts): enter the four values above. Until + then fresh installs still fetch agent 0.63.0 + golden 0.98.3. +2. **Operator-key pin** (out of scope here): day-0 installs no operator pubkey, so agent + self-update stays dormant on fresh boxes; (a) pin the real key on felhom-pve, (b) small + host-install follow-up ships the pinned pubkey at day-0 — candidate to fold into GL-4. +3. **G3 git.token rotation**: unblocked per Gate 3c — issue read-scoped tokens (or rely on anon + + keep a placeholder until the installer follow-up); also fixes demo-felhom's currently-EMPTY + git credentials (found in GL-2; blocks any step-5 (re)install for that customer). +4. GL-6 supervised drill consumes the bumped manifest end-to-end (Scenario-D-style proof for + 0.74.0/0.103.0 — deliberately NOT run here).