slice 7 Phase 1: unified bring-up reconcile job (provision + guest-loss DR) (v0.8.0)

The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.

- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
  verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
  host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
  dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
  (SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
  reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.

Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-09 21:27:49 +02:00
parent 9f6753de0f
commit 57405c1a99
10 changed files with 1184 additions and 73 deletions
+55 -57
View File
@@ -1,70 +1,68 @@
# REPORT — Restore-test must not false-fail on benign start warnings (v0.7.0) (2026-06-09)
# REPORT — Slice 7 Phase 1: unified bring-up reconcile job (v0.8.0) (2026-06-09)
> Overwrite-latest report (most recent significant work only). Cumulative history lives in
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Restore-test must not false-fail on benign
> start warnings`. **Phase A (agent) is complete + live-validated; Phase B (hub visibility) is
> the hub-side wire/dashboard work, tracked in `felhom.eu`.**
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Slice 7 Phase 1: unified bring-up reconcile job`.
> **Agent-only — no hub/wire change** (the new guest auto-appears in the host-report via `ListLXC`).
## Problem
## Outcome
The live hub-enrollment runbook surfaced it: the self-restore-test reported `pass:false` on
**every** modern-distro guest. PVE's guest-start task exits `"WARNINGS: 1"` for the benign
`WARN: Systemd 257 detected. You may need to enable nesting.` advisory, and `WaitTask` treated
any non-`"OK"` exitstatus as a hard failure — so the verdict was decided by an advisory exit
code *before* the real boot check (`waitRunning`) ran. The guest boots fine. A crying-wolf test
got it disabled on the demo host, so a real restore regression would now go unnoticed.
The shared **front half** of provision and guest-loss DR shipped as a journaled reconcile job
(`internal/reconcile/bringup.go`), mirroring the slice-6 restore-test's crash-safety but KEEPING
the guest on success and applying a **scenario-specific identity policy**. Built from the slice-7
spike findings (commit `3342993`): F1 (MAC reset unconditional on provision), F3 (host keys via a
baked golden first-boot unit, not an agent guest-internal op), F4 (transient config-lock retry).
## Decision (encoded as an invariant)
## What landed
**Verdict = liveness, not exitstatus.** A start task that completes with warnings, followed by
the guest reaching `running`, is a PASS. Warnings are always fetched and surfaced but never
decide pass/fail. **Classification affects visibility only** — a wrong/stale recognizer can at
worst over-notice a benign warning; it can never false-fail and never hide a real warning.
- **`RunBringUp(BringUpSpec) BringUpResult`** — restore → identity reset → size → attach mounts →
start LINK-UP, each mutation preceded by journaling the owning entry. **Verdict is liveness**
(`waitRunning`), never the start exitstatus (reuses the v0.7.0 WARNINGS surface). **Success keeps
the guest** (the key difference from the restore-test).
- **Identity policy (doc 03 §9):** *provision* resets MAC unconditionally (`PUT net0`, hwaddr
omitted → PVE regenerates; F1) + hostname, host-side via the token; machine-id + SSH host keys
regenerate guest-side on first boot (systemd + the baked unit) — the agent never touches guest
internals. *dr_guest_loss* preserves continuity (keep hostname; keep MAC unless `KeepMAC=false`);
never resets restic/tunnel/hub identity.
- **Compensating rollback:** any mid-flight failure destroys the just-created guest
(`ClassGuestDestroy` benign via `Provenance{SameTxnCreated:true}`, gated). New journal flag
`Rollback` + `Recover.recoverBringUp` reap a half-built guest from a mid-job crash (idempotent,
via `ListLXC`) — distinct from the scratch path's audit label, same destroy machinery.
- **F4:** identity+sizing+mounts coalesced into ONE `PUT config`; rootfs grow kept separate;
`setConfigWithLockRetry` retries ONLY the transient PVE config-lock 500 (`pveConfigLock`), never
a real error.
- **`--selftest=bring-up`** (`-mode provision|dr -archive -vmid -hostname [-keep]`) — runs the real
job after a `Recover`, then tears the guest down unless `-keep`.
- **`configs/build-golden.sh`** — the validated golden recipe incl. the F3 first-boot
`felhom-regen-hostkeys.service` (Condition-gated: fires on provision, no-ops on DR). The spike's
golden archive (no unit) is superseded.
## What landed (Phase A — agent; v0.7.0, single bump for the agent's A+B work)
## Tests (assert the effect)
- **`proxmox.WaitOptions.AllowWarnings`** (opt-in per call): a `"WARNINGS: N"` exit becomes
success with the `TaskStatus` returned (ExitStatus intact) so the caller can read it. Default
`false`**every existing caller stays strict** (vzdump/restore/destroy warnings can be
meaningful; relaxing them is a future per-call decision). Any non-WARNINGS non-OK exit is
still a `*TaskError`.
- **Restore-test start step** (`reconcile/restoretest.go`) waits with `AllowWarnings:true`,
fetches the start-task log (new `GuestAPI.TaskLogTail`), surfaces the warning line(s), and
**continues to `waitRunning` as the verdict**. Restore + scratch-teardown WaitTasks stay strict.
- **`RestoreTestResult.StartWarnings` / `.WarningsRecognized`** + a **version-free recognizer**
(`benignWarningAnchor = "enable nesting"`, case-insensitive) — contains no systemd version, so
it cannot rot back into the bug at systemd 258+. `extractWarningLines` keeps `WARN…` log lines.
- **Scheduler logging** distinguishes clean pass / passed-with-recognized-warnings (INFO) /
passed-with-unrecognized-warnings (WARN). Nothing silent.
- **Agent-side wire fields** (`hub.RestoreTest.warnings` / `.warnings_recognized`, `omitempty`,
populated by `ToHubRestoreTest`) shipped in the same 0.7.0 binary so the agent is built once.
They're additive — the deployed v0.7.4 hub ignores them; hub **v0.7.5** (Phase B) consumes them.
`go test ./...` green; `-race` green on the build server. Provision happy path (fresh MAC = net0
without hwaddr, hostname, coalesced sizing+mount, rootfs-grow separate, started, **guest NOT
destroyed**); compensating rollback injected at each step (restore / config / start-task /
waitRunning → asserts the guest **was** destroyed); DR continuity (MAC kept, hostname not reset) +
DR `KeepMAC=false` resets MAC; liveness verdict (warnings+running pass / not-running fail); F4
(lock-500 → retry → proceed; non-lock 500 → fail without retry); owning entry journaled **before**
restore; reserved/existing VMID refused; `Recover` rolls back / clean.
## Tests (assert the effect, not the call)
## Live validation (demo-felhom)
`go test ./...` green locally (Go 1.26); `-race` on the build server.
- `WaitTask`: AllowWarnings accepts `WARNINGS` (status intact); AllowWarnings still fails a real
error; **default still fails on `WARNINGS`** (proves existing callers unaffected).
- Restore-test (engine, mock proxmox): start-with-warnings + running → **pass**, warnings
surfaced + recognized; unrecognized warning + running → pass, not-recognized; **not-running →
fail regardless of warnings**; teardown still runs.
- **Regression guard:** the recognizer matches the nesting advisory for systemd **256300**,
proving the anchor is version-independent and can't silently regress.
Built the real golden via `configs/build-golden.sh` (with the host-key unit) and exercised the
job live:
- **provision** (`--selftest=bring-up -mode provision`): restore → fresh MAC → hostname set →
start link-up → Docker runs; **SSH host keys regenerated by the baked unit** (`ssh.service`
active; the agent issued no `ssh-keygen`); machine-id unique; hostname propagated; fresh MAC +
clean DHCP lease, no collision → torn down.
- **dr** (`-mode dr`): continuity-identity branch — hostname + host keys preserved (unit no-op),
MAC kept.
- **Recover**: a deliberate mid-restore crash left a half-built guest → restart → `Recover` reaped
the orphan (idempotent).
## Live re-validation (Phase A — A.6)
*(Concrete volids/MACs/leases captured in the run; see CHANGELOG + the slice-7 findings doc.)*
Re-enabled `backup.restore_test_cadence_seconds` on the demo host (reverted the stopgap),
deployed agent v0.7.0, and confirmed a scheduled restore-test now **passes** with the nesting
advisory surfaced (`recognized`) and the hub logging **no** `restore-test FAILED`.
*(Filled in at deploy time; see CHANGELOG for the live evidence.)*
## Deferred (stated, not built)
## Phase B — hub visibility (tracked in `felhom.eu`)
Wire fields already emitted by this agent (v0.7.0). Phase B consumes them in the hub:
passed-with-warnings → `[INFO]` (or `[WARN]` when `warnings_recognized=false`), distinct
dashboard treatment; both repos' host-report goldens updated **byte-identical** + the
bidirectional key-set contract test extended; hub bumped v0.7.4 → v0.7.5 and deployed via GitOps.
## No secrets
No secrets touched or committed. Live config/token references are by location only.
Provisioning BACK HALF (controller deploy, bootstrap, per-guest token mint) → slice 8; host-loss
DR + escrow consumption → slice 10; the SOURCE of a `BringUpSpec` (hub desired-state) → slice 10
(`GuestMount` defined minimally, no hub coupling). No secrets committed.