docs(pilot): RUNBOOK GL-1 executed — agent 0.74.0 + golden 0.103.0 published
Agent = the live felhom-pve bytes (sha 1ec3f588..76af05, round-trip verified); golden baked clean-room in the drill VM (build-golden v2.0.0, controller 0.103.0, all integrity gates green, sha 8481e8a1..6026e) and published 201. Both anon-fetch-verified (Gate 3c: packages world-readable -> G3 read-only token rotation cannot break fetches; installer-side empty-token die is the real precondition). Drill VM restored to virgin. OPERATOR NEXT: Day-0 manifest bump to the four recorded values. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+13
@@ -3,6 +3,19 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
|
||||
(partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
|
||||
`1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
|
||||
clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
|
||||
integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
|
||||
fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
|
||||
git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
|
||||
script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
|
||||
`~/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
|
||||
**OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
|
||||
are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
|
||||
operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
|
||||
recorded here instead).
|
||||
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
|
||||
harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
|
||||
provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
|
||||
|
||||
@@ -2,92 +2,59 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## TASK GL-2 — BYO-host install profile (`--mode appliance|byo`) — host-install v1.10.0 (2026-07-07)
|
||||
## RUNBOOK GL-1 — publish agent 0.74.0 + bake/publish golden 0.103.0 (2026-07-07)
|
||||
|
||||
**Baseline confirmed:** felhom.eu @ `1a1e42ad`, `SCRIPT_VERSION="1.9.1"` → **`1.10.0`**.
|
||||
Implementation + static/dry validation only per spec §0 — **the live provision/uninstall runs are
|
||||
GL-6 (operator-supervised)**; the §13 STOP was honored (zero non-dry installer invocations anywhere).
|
||||
Full execution record with per-gate evidence:
|
||||
`documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`. Docs-only commit; no code changed, no
|
||||
production host mutated (felhom-pve read-only; all root work inside the disposable drill VM).
|
||||
|
||||
### Files created / modified (single repo, `main`)
|
||||
### The four operator values (hub Day-0 manifest — Configs → Day-0 artifacts)
|
||||
|
||||
- `scripts/felhom-host-install.sh` — v1.10.0 (Parts 1–3; see scripts/CHANGELOG.md for the feature list)
|
||||
- `scripts/hostinstall-mode-harness.sh` — NEW (Part 4)
|
||||
- `documentation/runbooks/day0-install.md` — `--mode` in every command, new §C.5 (byo command +
|
||||
trust-model paragraph + preflight-only), §C.6 (pre-v1.10.0 resume note), 4 new troubleshooting rows
|
||||
- `scripts/CHANGELOG.md`, `REUSE.md` (install-profile-gate pattern row), `CONTEXT.md`, this file
|
||||
```
|
||||
AGENT_VERSION=0.74.0
|
||||
AGENT_SHA256=1ec3f58842edce1e9e32d022eaef15b7ff599c658c7a36bfe7833c708076af05
|
||||
GOLDEN_VERSION=0.103.0
|
||||
GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e
|
||||
```
|
||||
|
||||
### Spec deviations / judgment calls (each verified against the real code)
|
||||
### Gate outcomes (all PASS)
|
||||
|
||||
1. **`--mode` collision:** v1.9.1 already used `--mode provision|dr`. Resolved: `appliance|byo` are
|
||||
the only install values, `provision` retired with a specific die, the `dr` stub dispatch kept
|
||||
verbatim (the documented 10D seam).
|
||||
2. **Spec §2 was wrong about the lan-resolver**: the config write did `setdefault('lan_resolver',
|
||||
{"enable": True})` — default-ON, so the spec's byo assert would have died on every byo install.
|
||||
Fixed by flipping the byo DEFAULT to off (appliance untouched); the assert still dies if a
|
||||
`--preserve-from` config carries true, which is the intended refusal.
|
||||
3. The byo skip line reads `break-glass (step 4b) SKIPPED (byo: …)` — NOT the spec's literal
|
||||
`"4b/8 …"` text, because Scenario B / harness H-B forbid the string `4b/8` in the byo transcript
|
||||
(an internal spec contradiction; the greppable invariant won).
|
||||
4. `--preflight-only` skips the byo typed ack (prints the disclosure, logs the skip) — nothing will
|
||||
be mutated, and a mandatory prompt would break its re-runnable/scripted purpose.
|
||||
5. `documentation/pilot/GO-LIVE-PACKAGE.md` is not in the repo and no operator copy arrived with the
|
||||
task — the G2/G4/G5 → "implemented, awaiting GL-6" flip is recorded in CONTEXT.md; add the doc
|
||||
when supplied.
|
||||
- **0a–0d**: baselines exact (`ced60ddf` / `59eb3bea` / `02d63ed0`; agent 0.63.0 + golden 0.98.3
|
||||
the last published, targets 404); drill VM + `virgin` snapshot intact; controller image
|
||||
0.103.0 resolvable; headroom ample.
|
||||
- **1a/1b**: felhom-pve's RUNNING binary (`readlink /proc/…/exe`, no pending A/B marker) is
|
||||
0.74.0; those exact bytes (hash-stable across both copy hops) published via
|
||||
`publish-agent.sh` from `main`@`ced60ddf` — PUT 201, round-trip GET sha match.
|
||||
- **2a/2b**: clean-room bake on the virgin-restored drill VM (build-golden.sh v2.0.0):
|
||||
overlay2 + both split-volumes as separate mounts, rootfs+mp0+mp1 ALL included in the vzdump
|
||||
(no exclusions, no FATAL), archive 583MB, publish HTTP 201; transcript token-leak grep = 0.
|
||||
Deviations: debian-13 template pveam-downloaded inside (virgin never has it, checksum
|
||||
verified); bake launched via `systemd-run` (bare nohup didn't survive ssh session close).
|
||||
- **3a/3b**: both artifacts fetched **ANONYMOUSLY** and hashed to exactly the published sha
|
||||
(golden 611 696 321 bytes, streamed+hashed+deleted); off-LAN HEAD 200 from the CC machine too.
|
||||
|
||||
### Harness results
|
||||
### Gate 3c finding (feeds G3) — read-credential sufficiency
|
||||
|
||||
Static tier (Windows Git Bash AND felhom-pve, identical): **bash -n, C1/C1b/C1c, C2a/C2b, C3/C3b/C3c,
|
||||
C4, C6-static, INV-1 (one appliance-gated break-glass call site), INV-2 (chpasswd unreachable outside
|
||||
step_break_glass), INV-3 (--mode in usage) — all PASS.**
|
||||
The specified reviewer read-only token (`8417…4140`) was not locatable by CC anywhere in reach —
|
||||
the gates ran with **no credential at all and passed**: the generic packages are anonymously
|
||||
world-readable. Therefore rotating the customer `git.token` to read-only **cannot break** the
|
||||
artifact fetch path (G3 unblocked from this side). Nuance: `resolve_git_creds` in host-install
|
||||
**dies on an empty token** even though Gitea would serve the fetch anonymously — the requirement
|
||||
is script-side; customer configs still need a (read-scoped) token until a small installer
|
||||
follow-up makes it optional. This also reframes GL-2's finding that demo-felhom's git credentials
|
||||
are EMPTY: the fetch would work, the installer's precondition is what fails.
|
||||
|
||||
PVE tier (felhom-pve, read-only, temp `FELHOM_INSTALL_STATE_DIR`): **C5 PASS live** (bogus
|
||||
`--acl-storages` entry → die naming it, pre-hub, pre-passphrase). H-A/H-B dry transcripts ran with
|
||||
the out-of-band demo passphrase (file→file, never printed) but **die at step 5 with rc=1 — a
|
||||
customer-data blocker, not a code bug**: demo-felhom's controller.yaml serves EMPTY
|
||||
`git.username`/`git.token`. The transcript span through the break-glass gate was validated manually
|
||||
instead: appliance transcript contains `4b/8`; byo transcript contains **no**
|
||||
`4b/8`/`chpasswd`/`recovery-credential` and **does** contain the disclosure block, the dry ack line,
|
||||
`acl storages all present`, and the host-DNS `:53` line (it correctly showed the live dnsmasq).
|
||||
`--preflight-only` proven live both ways: PASS verdict + rc 0 + **zero state entries**; bogus-storage
|
||||
run → specific die + `PRE-FLIGHT FAIL (exit 1)` verdict.
|
||||
### Drill-VM end state
|
||||
|
||||
**Red-proofs (run → fail → revert, against mutated scratch copies — the repo file was never mutated):**
|
||||
- RP-1 un-gated the break-glass call site → INV-1 FAIL (13/16). Reverted (scratch deleted).
|
||||
- RP-2 dropped the byo caps requirement → C1/C1b/C1c FAIL (11/16). Reverted.
|
||||
- RP-3 dropped the resume mode-mismatch check → C4 FAIL (13/16). Reverted.
|
||||
Guest 9100 destroyed (`--purge`), in-VM cred file shredded, VM powered off, qcow2 **restored to
|
||||
`virgin`** (verified) — environment exactly as found. Evidence kept: `~/drill/bake-0.103.0.log`
|
||||
on the build server.
|
||||
|
||||
### §9 rule 9 — the `_state_mark` dry-run finding
|
||||
### Follow-ups
|
||||
|
||||
Verified in code: **both** `_state_mark` and `_state_put` begin with `$DRY_RUN && return 0` — dry
|
||||
runs write nothing. Belt on top: the harness points `FELHOM_INSTALL_STATE_DIR` (new, harness-only
|
||||
env override — none existed) at a throwaway temp dir for EVERY invocation, so even non-dry refusal
|
||||
cases can never touch a live `state.json`. Confirmed live: felhom-pve's real state.json was
|
||||
byte-untouched (it only ever saw the temp dir) and the preflight-only runs left 0 entries.
|
||||
|
||||
### Shellcheck
|
||||
|
||||
v0.10.0 (build server 180 + felhom-pve; not installed locally). **Clean at `--severity=warning`**
|
||||
on both scripts. Full-severity: 2 pre-existing SC2015 *info* notes on untouched v1.9.1 lines
|
||||
(A&&B||C in step_verify) — triaged, not refactored (spec rule 2); 1 new SC2154 false positive on
|
||||
the EXIT trap (assignment inside the trap string) — targeted disable comment.
|
||||
|
||||
### NOT yet live-validated — awaiting supervised GL-6
|
||||
|
||||
- The full appliance AND byo installs end-to-end (rc=0 transcripts blocked today by the
|
||||
demo-felhom git-creds gap below).
|
||||
- The uninstall → re-provision drill under both modes.
|
||||
- C7: the verify pool/ACL asserts firing on real drift (needs a deliberately-broken live box).
|
||||
- The byo typed-hostname ack answered interactively (only its dry/preflight-only branches ran).
|
||||
|
||||
### Observations (documented, not acted on)
|
||||
|
||||
- **demo-felhom has EMPTY `git.username`/`git.token`** in its served controller.yaml — any step-5
|
||||
(re)install for that customer dies until real Gitea read creds are set (operator; ties into the
|
||||
standing "scope down + rotate the package-WRITE git token" security follow-up).
|
||||
- felhom-pve's live install predates `_state_put` — its state.json has only `completed` (no
|
||||
`customer_id`/`provisioned_vmid`/`mode`); a future `--uninstall` there needs an explicit `--vmid`,
|
||||
and §C.6 of the runbook covers the missing-mode resume.
|
||||
- A `--resume` that skips preflight never records `mode` into an old state file (it only writes
|
||||
during a full preflight) — harmless (C4 only fires when a mode IS recorded), noted for GL-6.
|
||||
- The live host-DNS line on felhom-pve shows the agent's own dnsmasq (lan_resolver is ON there —
|
||||
correct for an appliance; it is exactly what byo now refuses).
|
||||
1. **OPERATOR — hub Day-0 manifest bump** to the four values above (until then fresh installs
|
||||
land agent 0.63.0 + golden 0.98.3).
|
||||
2. **Operator-key pin** (out of GL-1 scope): day-0 ships no operator pubkey → self-update dormant
|
||||
on fresh boxes; pin the real key + a small host-install follow-up (candidate: fold into GL-4).
|
||||
3. **G3 rotation** unblocked per Gate 3c; also fixes demo-felhom's empty git credentials.
|
||||
4. **GL-6** consumes the bumped manifest end-to-end (deliberately not run here).
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# RUNBOOK GL-1 — execution record: publish agent 0.74.0 + bake/publish golden 0.103.0 (2026-07-07)
|
||||
|
||||
**Class:** runbook execution (go-live item G1, partial). No code changes, no version bumps.
|
||||
Production touched read-only (felhom-pve: one `scp` out + `--version`/`readlink`; build server:
|
||||
normal kisfenyo work + drill-VM lifecycle). All root actions inside the disposable nested drill VM.
|
||||
No credential appears in this record; publish creds were sourced on the build server
|
||||
(`~/.gitea-token`, user `admin`) and the in-VM copy was shredded before poweroff.
|
||||
|
||||
## The four operator values (hub Day-0 manifest → Configs → Day-0 artifacts)
|
||||
|
||||
```
|
||||
AGENT_VERSION=0.74.0
|
||||
AGENT_SHA256=1ec3f58842edce1e9e32d022eaef15b7ff599c658c7a36bfe7833c708076af05
|
||||
GOLDEN_VERSION=0.103.0
|
||||
GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e
|
||||
```
|
||||
|
||||
## Gate outcomes
|
||||
|
||||
| Gate | Result | Evidence |
|
||||
|---|---|---|
|
||||
| 0a baselines | **PASS** | felhom-agent `ced60ddf` (v0.74.0), felhom-controller `59eb3bea` (v0.103.0), felhom.eu `02d63ed0`; published state pre-run: agent 0.63.0 = 200 / 0.74.0 = 404, golden 0.98.3 = 200 / 0.103.0 = 404 |
|
||||
| 0b drill VM | **PASS** | `~/drill/drill.qcow2` on 192.168.0.180 with internal snapshot `virgin` (2026-07-03) intact |
|
||||
| 0c controller image | **PASS** | `docker manifest inspect gitea.dooplex.hu/admin/felhom-controller:0.103.0` OK on the build server |
|
||||
| 0d headroom | **PASS** | 213 G free on 180 (golden ≈ 0.6 G); VM local-lvm 76 G thin, 0% used post-virgin |
|
||||
| 1a running binary | **PASS** | felhom-pve `readlink /proc/$(pidof felhom-agent)/exe` → `/usr/local/bin/felhom-agent`, `--version` → 0.74.0; selfupdate state dir EMPTY (no pending A/B marker) |
|
||||
| 1b agent publish | **PASS** | the LIVE bytes (sha identical across felhom-pve→CC→180 hops) published via `publish-agent.sh 0.74.0` from `main`@`ced60ddf`: pre-delete 404, PUT 201, round-trip GET sha match |
|
||||
| 2a drill VM ready | **PASS** | virgin-restored + booted (slirp `dhcpstart=10.0.2.30` + hostfwd 2222→10.0.2.15:22, per DRILL precedent); PVE 9.2.2 responsive; gitea HTTP/2 200 from inside; debian-13 template ABSENT on virgin (expected) → `pveam download` inside, checksum verified |
|
||||
| 2b bake + publish | **PASS** | `build-golden.sh` (main@`ced60ddf`, script v2.0.0) as root@pam in-VM, baking `felhom-controller:0.103.0`; transcript: `docker OK (overlay2; data-root /var/lib/docker)`, both `/var/lib/docker` + `/mnt/sys_drive` separate ext4 mounts, `including mount point` rootfs+mp0+mp1 (NO exclusions, no FATAL), archive 583MB, publish upload HTTP 201, `GOLDEN_VERSION=0.103.0` + sha printed. Token-leak grep over the transcript: 0 hits |
|
||||
| 3a agent fetch | **PASS** | **ANONYMOUS** GET → 200, sha256 == AGENT_SHA256 (from the build server; HEAD 200 from the CC machine off-LAN too) |
|
||||
| 3b golden fetch | **PASS** | **ANONYMOUS** GET → 200, 611 696 321 bytes, sha256 == GOLDEN_SHA256 (streamed to disk, hashed, deleted) |
|
||||
| 3c read-token finding | **see below** | |
|
||||
|
||||
## Gate 3c — read-credential finding (feeds G3)
|
||||
|
||||
The specified reviewer read-only token (`8417…4140`) was **not locatable by CC** (not in the repos,
|
||||
not on the build server under any evident name). The gates were instead run with **no credential at
|
||||
all — and passed**: both generic packages are **anonymously world-readable** (deliberate posture;
|
||||
the agent's self-update and guest 9201's image pulls already rely on anon Gitea reads; integrity
|
||||
comes from the hub-manifest sha, not transport auth). Consequences:
|
||||
|
||||
- A read-scoped token is trivially sufficient for artifact fetch — **rotating the customer
|
||||
`git.token` to read-only (or revoking its package-write) CANNOT break fresh installs' fetch
|
||||
path.** The G3 rotation is unblocked from this side.
|
||||
- Nuance: the installer (`resolve_git_creds`) **dies on an empty git token** even though the fetch
|
||||
itself would succeed anonymously — the credential is required by the script, not by Gitea. If the
|
||||
operator ever wants credential-less artifact fetch, that is a small host-install follow-up; until
|
||||
then customer configs still need a (read-scoped) token present.
|
||||
|
||||
## Drill-VM end state
|
||||
|
||||
Build guest 9100 destroyed (`--purge`, all 3 LVs removed); in-VM cred file shredded; VM powered
|
||||
off; **qcow2 restored to `virgin`** (snapshot list verified) — the drill environment is exactly as
|
||||
found. Evidence retained on the build server: `~/drill/bake-0.103.0.log` (next to the prior
|
||||
`bake-A.log`).
|
||||
|
||||
## Deviations from the letter of the runbook
|
||||
|
||||
- Gate 2a's template check was satisfied by downloading the template inside the VM (the virgin
|
||||
image never has it — same as the 0.98.3 bake) — anticipated by the runbook, noted as required.
|
||||
- Phase 3 ran anonymously instead of with the reviewer token (see Gate 3c) — a strictly *stronger*
|
||||
proof of fetchability.
|
||||
- The bake was launched as a transient systemd unit (`systemd-run --unit=golden-bake`) inside the
|
||||
VM instead of bare `nohup` (a plain ssh-backgrounded nohup did not survive session close).
|
||||
|
||||
## Follow-ups (operator)
|
||||
|
||||
1. **Hub Day-0 manifest bump** (Configs → Day-0 artifacts): enter the four values above. Until
|
||||
then fresh installs still fetch agent 0.63.0 + golden 0.98.3.
|
||||
2. **Operator-key pin** (out of scope here): day-0 installs no operator pubkey, so agent
|
||||
self-update stays dormant on fresh boxes; (a) pin the real key on felhom-pve, (b) small
|
||||
host-install follow-up ships the pinned pubkey at day-0 — candidate to fold into GL-4.
|
||||
3. **G3 git.token rotation**: unblocked per Gate 3c — issue read-scoped tokens (or rely on anon +
|
||||
keep a placeholder until the installer follow-up); also fixes demo-felhom's currently-EMPTY
|
||||
git credentials (found in GL-2; blocks any step-5 (re)install for that customer).
|
||||
4. GL-6 supervised drill consumes the bumped manifest end-to-end (Scenario-D-style proof for
|
||||
0.74.0/0.103.0 — deliberately NOT run here).
|
||||
Reference in New Issue
Block a user