admin
b4edc087fa
§7.2 answer: YES. A real day-0 from the existing v1.25.0 ISO reached a claimable,
app-serving box in ~10 minutes unattended, and an app's data came back from the
drive with the guest's app.yaml gone — proven readable by the application over
its own TCP path, with a discriminator (PRE-BACKUP row = 1, POST-BACKUP row = 0).
Part 0: NO ISO rebuild needed, verified against the ISO on disk rather than from
source. It bakes only felhom-bootstrap.sh, its unit and the secret-free pairing
env (full-base64 match, 1 hit each) and 0 hits for any installer, controller or
golden marker. The installer is fetched at run time; the live URL is byte-identical
to repo HEAD (v1.22.0, six days newer than the ISO) and the fresh box ran it.
Part 1: baked 0.188.0 rather than the brief's 0.187.0 — 0.187.0 lacks D5, which
is the very claim Part 2 step 6 tests. Published (404 pre-gate with a 200 control;
anonymous download, 649310288 bytes, sha match), vouched, and consumed by a real
box. R-120's gate exercised BOTH ways: 0.185.1 refused with no write, 0.188.0
allowed — evaluated, not silently skipped.
Part 3: RUNBOOK-manual-build.md cited a "RECORDED" qemu line that is itself
labelled reconstructed and whose source says it was never saved. The real
invocation is now captured from this bake as §4.0, with the bake/publish/teardown
steps; the old entry is marked SUPERSEDED.
Teardown all three layers, hub disposition stated: VM destroyed, scratch storage
removed with space returned exactly, customer sess-g DELETED via full cascade.
sess-f deliberately left (R-131) with its command recorded.
Filed, none fixed: R-128 (false ISO_VERSION invariant comment), R-129 (demo-hp's
"no baked SSH key" is stale — key auth works), R-130 (HARD_MIN_LVM_GIB warns and
proceeds), R-131 (fourth orphaned scratch customer), R-132 (curl's %{redirect_url}
printed the hub operator password into a transcript — HUB_PW needs rotating).
93 lines
7.8 KiB
Markdown
93 lines
7.8 KiB
Markdown
# RUNBOOK — publish train agent 0.79.0 + golden 0.110.0 + floor (2026-07-10) — A/B/C DONE, D/E operator-gated
|
|
|
|
**Class:** publish/signing runbook (GL-1 pattern). **Status:** Phases A, B, **C COMPLETE** (agent + golden
|
|
both published + verified; drill VM restored to virgin); **Phase D = the operator handoff** (one UI save +
|
|
the floor bump); Phase E qualified (demo already on 0.110.0). No credential/signature-key material here.
|
|
|
|
## The four operator values (hub Day-0 manifest → Configs → Day-0 artifacts) — READY
|
|
|
|
```
|
|
AGENT_VERSION=0.79.0
|
|
AGENT_SHA256=35a162547646737ebef265f5ad69d042e897914f6ca799c007872d1c3cd27561
|
|
GOLDEN_VERSION=0.110.0
|
|
GOLDEN_SHA256=1593597d98af61d94c4923cfd526e444e038aa21bd63ddc4e89067f5f010f3d4
|
|
FLOOR=0.110.0
|
|
```
|
|
|
|
## Gate outcomes
|
|
|
|
| Gate | Result | Evidence |
|
|
|---|---|---|
|
|
| A — C2 code shipped | **PASS (prior task)** | controller v0.110.0 on main + live on 9201 (`Up (healthy)`); lock self-heal + C1 unit-tested + red-proofed |
|
|
| B-0a published-state pre | **PASS** | agent 0.79.0 GET = 404, golden 0.110.0 GET = 404 (the GL-1 pre-gate) |
|
|
| B-1a live bytes | **PASS** | felhom-pve running binary = `/usr/local/bin/felhom-agent` v0.79.0, sha `35a1625476…`; selfupdate state dir empty (no pending A/B marker); sha IDENTICAL across felhom-pve→CC→180 hops |
|
|
| B-1b agent publish | **PASS** | `publish-agent.sh 0.79.0 <live-bytes>` on 180 from main@`a6e8bcb`: pre-delete 404, PUT **201**, round-trip GET sha match |
|
|
| B-3a anon fetch | **PASS** | **anonymous** GET → 200, sha256 == AGENT_SHA256 (off-LAN CC machine); hop copies removed (180 + local); token-leak grep of transcript: 0 hits |
|
|
| C-0b drill VM | **PASS** | `~/drill/drill.qcow2` on 180, internal snapshot `virgin` (2026-07-03) intact |
|
|
| C-0c controller image | **PASS** | `docker manifest inspect …/felhom-controller:0.110.0` OK on 180 |
|
|
| C-0d headroom | **PASS** | 196 G free on 180 (golden ≈ 0.6 G); virgin snapshot 0 B |
|
|
| C-2a boot drill VM | **PASS (reconstructed — DEVIATION)** | operator authorized reconstruction; `qemu-img snapshot -a virgin` then `qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 -drive …drill.qcow2,if=virtio -netdev user,…dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 -display none -daemonize`; PVE 9.2.2 up + SSH on :2222 in ~10 s (virtio bus correct); gitea 200 from inside; debian-13 template absent on virgin → `pveam download` (checksum OK) |
|
|
| C-2b bake + publish | **PASS** | `build-golden.sh` v2.0.0 as root@pam in-VM (transient `systemd-run --unit=golden-bake`), baking `felhom-controller:0.110.0`: `docker OK (overlay2; data-root /var/lib/docker)`, all 3 mounts included (rootfs+mp0+mp1, NO exclusions/FATAL), archive 583 MB, upload **HTTP 201**, `GOLDEN_VERSION=0.110.0` + sha printed; in-VM token-leak grep 0 |
|
|
| C-3b golden fetch | **PASS** | **anonymous** GET → 200, **611 696 157 bytes**, sha256 == GOLDEN_SHA256; off-LAN HEAD 200 |
|
|
| C-teardown | **PASS** | build guest 9100 `--purge`d; `/root/.gitea-token`+script+log shredded; VM powered off; qcow2 reverted to `virgin` (snapshot intact, no qemu); saved-log token-leak grep (literal token) = **0**; `~/drill/bake-0.110.0.log` retained next to priors |
|
|
| **D — manifest + floor** | **OPERATOR** | 🛑 one hub UI save (artifact manifest) + the floor bump — see handoff below |
|
|
| E — demo self-update proof | **QUALIFIED** | the demo is ALREADY on 0.110.0 (manually deployed in the C2 task), so the floor-driven 0.109.1→0.110.0 self-update can't be demonstrated on it; the Day-0 DRY proof (anon-fetch BOTH artifacts + sha-verify) is DONE + green above (what Peti's installer does) |
|
|
|
|
## Phase D — operator handoff (order matters: manifest BEFORE floor)
|
|
|
|
1. **🛑 Viktor — artifact manifest (DB-backed operator UI, session-gated — CC cannot):** hub → Configs →
|
|
Day-0 artifacts → Agent **0.79.0** / sha `35a1625476…d27561`, Golden **0.110.0** / sha `1593597d98…f010f3d4`
|
|
→ Save. (The sha the UI auto-reads must equal these.)
|
|
2. **Global floor 0.87.0 → 0.110.0 — AFTER the manifest save.** This is the `DEFAULT_MIN_CONTROLLER_VERSION`
|
|
env in `manifests/hub.yaml` (GitOps, NOT session-gated) — CC can bump it via commit+ArgoCD sync once
|
|
Viktor confirms the manifest is saved (respecting the order rule: never raise the floor before the
|
|
manifest vouches the target). Or Viktor sets it. Effect: any box below 0.110.0 auto-updates next report
|
|
(the demo is already there → no-op; Peti's future box lands current).
|
|
|
|
## Signing note (correcting the runbook's Phase-B assumption)
|
|
|
|
`publish-agent.sh` does **NOT** sign anything — it PUTs the sha-addressed binary and prints the sha; that
|
|
matches the GL-1 precedent (its Phase B was 404→201→sha, no signing). The **operator signature is a RUNTIME
|
|
concern**, not a publish-time one: the agent self-update integrity root is the SHA256 (`selfupdate/executor.go`
|
|
— "the signed sha is the ONLY integrity root"), and the operator-signed `agent_update` job is what the hub
|
|
pushes when an operator TRIGGERS a managed agent update (GL-6 proved that round-trip against the pinned key).
|
|
So Phase B correctly required **no operator signature** — integrity for day-0 fetch comes from the
|
|
hub-manifest sha (Phase D) + anonymous fetchability (proven above). Nothing for Viktor to sign in this train
|
|
unless/until a managed agent_update is triggered.
|
|
|
|
## READY-FOR-PETI checklist
|
|
|
|
| Item | State |
|
|
|---|---|
|
|
| Agent 0.79.0 published + anon-fetchable + sha-verified | ✅ (sha `35a1625476…d27561`) |
|
|
| Golden 0.110.0 published + anon-fetchable + sha-verified | ✅ (sha `1593597d98…f010f3d4`, 611 696 157 B) |
|
|
| Both artifacts fetch ANONYMOUSLY (what Peti's installer does) | ✅ (off-LAN HEAD 200 both) |
|
|
| Agent self-update signature path armed vs the pinned key | ✅ (pinned in GL-4 v1.11.1; signed round-trip proven GL-6) |
|
|
| Hub artifact manifest vouches 0.79.0 / 0.110.0 | ⏳ Phase D.1 (🛑 Viktor UI save) |
|
|
| Global floor at 0.110.0 | ⏳ Phase D.2 (GitOps, after D.1) |
|
|
| Demo on 0.110.0 | ✅ (already deployed; escrowed offsite intact) |
|
|
| Peti hub-side provisioning armed | ✅ (sub-account 269130, descriptor+fingerprint served, one-time secret unconsumed) |
|
|
|
|
## Deviations from the letter of the runbook
|
|
|
|
- **Phase B needed NO operator signature** (the runbook assumed one). `publish-agent.sh` doesn't sign;
|
|
the integrity root is the SHA256 (`selfupdate/executor.go`) + the hub-manifest sha (Phase D). The
|
|
operator-signed `agent_update` is a RUNTIME managed-update concern (GL-6 proved it), not publish-time —
|
|
same as the GL-1 precedent (its Phase B was 404→201→sha, no signing).
|
|
- **Phase C drill launch RECONSTRUCTED** (operator-authorized): the canonical `qemu-system` one-liner was
|
|
never saved; reconstructed from the audits' parameters (recorded above); virtio bus + slirp/hostfwd
|
|
booted cleanly first try; virgin snapshot reverted afterwards, exactly as found.
|
|
> **SUPERSEDED 2026-07-31 — do not use the line above as an authority.** It is a reconstruction, and
|
|
> `RUNBOOK-manual-build.md` used to cite it while telling the reader not to improvise. The canonical
|
|
> invocation is now captured from a real bake and lives in **`runbooks/RUNBOOK-manual-build.md` §4.0**.
|
|
> This entry stays only as the historical record of the deviation.
|
|
- **Phase E qualified:** the demo was already on 0.110.0 (C2 task), so the floor-driven self-update proof
|
|
isn't demonstrable on it; the anon-fetch Day-0 dry proof stands in (a stronger fetchability proof).
|
|
|
|
## Install-window items for Viktor (NOT this train — the Peti install session)
|
|
- Two install-form items for the Peti window (NOT this train): cores 8 vs 12/16 (confirm with Peti);
|
|
`--acl-storages` for his storage layout.
|
|
- Send Peti the retrieval passphrase out-of-band; recommend `--preflight-only` first.
|
|
- Peti hub-side provisioning is already armed (sub-account 269130, descriptor+fingerprint served, one-time
|
|
secret unconsumed) — waiting on his box to enroll + report (DIAG 2026-07-10).
|