slice 7 Phase 1: unified bring-up reconcile job (provision + guest-loss DR) (v0.8.0)
The shared front half of provision and guest-loss DR as a journaled reconcile job (internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping the guest on success and applying a scenario-specific identity policy. Agent-only; no hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4. - RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up; verdict is liveness (waitRunning), success KEEPS the guest. - identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname, host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit). dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false). - compensating rollback: mid-flight failure destroys the just-created guest (SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp reap a half-built guest from a crash. - F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only. - --selftest=bring-up (mode/archive/vmid/hostname/keep). - configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit. - doc-03 §9 + identity-reset settled/implemented. Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption and the BringUpSpec source (hub desired-state) -> slice 10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,70 +1,68 @@
|
||||
# REPORT — Restore-test must not false-fail on benign start warnings (v0.7.0) (2026-06-09)
|
||||
# REPORT — Slice 7 Phase 1: unified bring-up reconcile job (v0.8.0) (2026-06-09)
|
||||
|
||||
> Overwrite-latest report (most recent significant work only). Cumulative history lives in
|
||||
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Restore-test must not false-fail on benign
|
||||
> start warnings`. **Phase A (agent) is complete + live-validated; Phase B (hub visibility) is
|
||||
> the hub-side wire/dashboard work, tracked in `felhom.eu`.**
|
||||
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Slice 7 Phase 1: unified bring-up reconcile job`.
|
||||
> **Agent-only — no hub/wire change** (the new guest auto-appears in the host-report via `ListLXC`).
|
||||
|
||||
## Problem
|
||||
## Outcome
|
||||
|
||||
The live hub-enrollment runbook surfaced it: the self-restore-test reported `pass:false` on
|
||||
**every** modern-distro guest. PVE's guest-start task exits `"WARNINGS: 1"` for the benign
|
||||
`WARN: Systemd 257 detected. You may need to enable nesting.` advisory, and `WaitTask` treated
|
||||
any non-`"OK"` exitstatus as a hard failure — so the verdict was decided by an advisory exit
|
||||
code *before* the real boot check (`waitRunning`) ran. The guest boots fine. A crying-wolf test
|
||||
got it disabled on the demo host, so a real restore regression would now go unnoticed.
|
||||
The shared **front half** of provision and guest-loss DR shipped as a journaled reconcile job
|
||||
(`internal/reconcile/bringup.go`), mirroring the slice-6 restore-test's crash-safety but KEEPING
|
||||
the guest on success and applying a **scenario-specific identity policy**. Built from the slice-7
|
||||
spike findings (commit `3342993`): F1 (MAC reset unconditional on provision), F3 (host keys via a
|
||||
baked golden first-boot unit, not an agent guest-internal op), F4 (transient config-lock retry).
|
||||
|
||||
## Decision (encoded as an invariant)
|
||||
## What landed
|
||||
|
||||
**Verdict = liveness, not exitstatus.** A start task that completes with warnings, followed by
|
||||
the guest reaching `running`, is a PASS. Warnings are always fetched and surfaced but never
|
||||
decide pass/fail. **Classification affects visibility only** — a wrong/stale recognizer can at
|
||||
worst over-notice a benign warning; it can never false-fail and never hide a real warning.
|
||||
- **`RunBringUp(BringUpSpec) BringUpResult`** — restore → identity reset → size → attach mounts →
|
||||
start LINK-UP, each mutation preceded by journaling the owning entry. **Verdict is liveness**
|
||||
(`waitRunning`), never the start exitstatus (reuses the v0.7.0 WARNINGS surface). **Success keeps
|
||||
the guest** (the key difference from the restore-test).
|
||||
- **Identity policy (doc 03 §9):** *provision* resets MAC unconditionally (`PUT net0`, hwaddr
|
||||
omitted → PVE regenerates; F1) + hostname, host-side via the token; machine-id + SSH host keys
|
||||
regenerate guest-side on first boot (systemd + the baked unit) — the agent never touches guest
|
||||
internals. *dr_guest_loss* preserves continuity (keep hostname; keep MAC unless `KeepMAC=false`);
|
||||
never resets restic/tunnel/hub identity.
|
||||
- **Compensating rollback:** any mid-flight failure destroys the just-created guest
|
||||
(`ClassGuestDestroy` benign via `Provenance{SameTxnCreated:true}`, gated). New journal flag
|
||||
`Rollback` + `Recover.recoverBringUp` reap a half-built guest from a mid-job crash (idempotent,
|
||||
via `ListLXC`) — distinct from the scratch path's audit label, same destroy machinery.
|
||||
- **F4:** identity+sizing+mounts coalesced into ONE `PUT config`; rootfs grow kept separate;
|
||||
`setConfigWithLockRetry` retries ONLY the transient PVE config-lock 500 (`pveConfigLock`), never
|
||||
a real error.
|
||||
- **`--selftest=bring-up`** (`-mode provision|dr -archive -vmid -hostname [-keep]`) — runs the real
|
||||
job after a `Recover`, then tears the guest down unless `-keep`.
|
||||
- **`configs/build-golden.sh`** — the validated golden recipe incl. the F3 first-boot
|
||||
`felhom-regen-hostkeys.service` (Condition-gated: fires on provision, no-ops on DR). The spike's
|
||||
golden archive (no unit) is superseded.
|
||||
|
||||
## What landed (Phase A — agent; v0.7.0, single bump for the agent's A+B work)
|
||||
## Tests (assert the effect)
|
||||
|
||||
- **`proxmox.WaitOptions.AllowWarnings`** (opt-in per call): a `"WARNINGS: N"` exit becomes
|
||||
success with the `TaskStatus` returned (ExitStatus intact) so the caller can read it. Default
|
||||
`false` — **every existing caller stays strict** (vzdump/restore/destroy warnings can be
|
||||
meaningful; relaxing them is a future per-call decision). Any non-WARNINGS non-OK exit is
|
||||
still a `*TaskError`.
|
||||
- **Restore-test start step** (`reconcile/restoretest.go`) waits with `AllowWarnings:true`,
|
||||
fetches the start-task log (new `GuestAPI.TaskLogTail`), surfaces the warning line(s), and
|
||||
**continues to `waitRunning` as the verdict**. Restore + scratch-teardown WaitTasks stay strict.
|
||||
- **`RestoreTestResult.StartWarnings` / `.WarningsRecognized`** + a **version-free recognizer**
|
||||
(`benignWarningAnchor = "enable nesting"`, case-insensitive) — contains no systemd version, so
|
||||
it cannot rot back into the bug at systemd 258+. `extractWarningLines` keeps `WARN…` log lines.
|
||||
- **Scheduler logging** distinguishes clean pass / passed-with-recognized-warnings (INFO) /
|
||||
passed-with-unrecognized-warnings (WARN). Nothing silent.
|
||||
- **Agent-side wire fields** (`hub.RestoreTest.warnings` / `.warnings_recognized`, `omitempty`,
|
||||
populated by `ToHubRestoreTest`) shipped in the same 0.7.0 binary so the agent is built once.
|
||||
They're additive — the deployed v0.7.4 hub ignores them; hub **v0.7.5** (Phase B) consumes them.
|
||||
`go test ./...` green; `-race` green on the build server. Provision happy path (fresh MAC = net0
|
||||
without hwaddr, hostname, coalesced sizing+mount, rootfs-grow separate, started, **guest NOT
|
||||
destroyed**); compensating rollback injected at each step (restore / config / start-task /
|
||||
waitRunning → asserts the guest **was** destroyed); DR continuity (MAC kept, hostname not reset) +
|
||||
DR `KeepMAC=false` resets MAC; liveness verdict (warnings+running pass / not-running fail); F4
|
||||
(lock-500 → retry → proceed; non-lock 500 → fail without retry); owning entry journaled **before**
|
||||
restore; reserved/existing VMID refused; `Recover` rolls back / clean.
|
||||
|
||||
## Tests (assert the effect, not the call)
|
||||
## Live validation (demo-felhom)
|
||||
|
||||
`go test ./...` green locally (Go 1.26); `-race` on the build server.
|
||||
- `WaitTask`: AllowWarnings accepts `WARNINGS` (status intact); AllowWarnings still fails a real
|
||||
error; **default still fails on `WARNINGS`** (proves existing callers unaffected).
|
||||
- Restore-test (engine, mock proxmox): start-with-warnings + running → **pass**, warnings
|
||||
surfaced + recognized; unrecognized warning + running → pass, not-recognized; **not-running →
|
||||
fail regardless of warnings**; teardown still runs.
|
||||
- **Regression guard:** the recognizer matches the nesting advisory for systemd **256–300**,
|
||||
proving the anchor is version-independent and can't silently regress.
|
||||
Built the real golden via `configs/build-golden.sh` (with the host-key unit) and exercised the
|
||||
job live:
|
||||
- **provision** (`--selftest=bring-up -mode provision`): restore → fresh MAC → hostname set →
|
||||
start link-up → Docker runs; **SSH host keys regenerated by the baked unit** (`ssh.service`
|
||||
active; the agent issued no `ssh-keygen`); machine-id unique; hostname propagated; fresh MAC +
|
||||
clean DHCP lease, no collision → torn down.
|
||||
- **dr** (`-mode dr`): continuity-identity branch — hostname + host keys preserved (unit no-op),
|
||||
MAC kept.
|
||||
- **Recover**: a deliberate mid-restore crash left a half-built guest → restart → `Recover` reaped
|
||||
the orphan (idempotent).
|
||||
|
||||
## Live re-validation (Phase A — A.6)
|
||||
*(Concrete volids/MACs/leases captured in the run; see CHANGELOG + the slice-7 findings doc.)*
|
||||
|
||||
Re-enabled `backup.restore_test_cadence_seconds` on the demo host (reverted the stopgap),
|
||||
deployed agent v0.7.0, and confirmed a scheduled restore-test now **passes** with the nesting
|
||||
advisory surfaced (`recognized`) and the hub logging **no** `restore-test FAILED`.
|
||||
*(Filled in at deploy time; see CHANGELOG for the live evidence.)*
|
||||
## Deferred (stated, not built)
|
||||
|
||||
## Phase B — hub visibility (tracked in `felhom.eu`)
|
||||
|
||||
Wire fields already emitted by this agent (v0.7.0). Phase B consumes them in the hub:
|
||||
passed-with-warnings → `[INFO]` (or `[WARN]` when `warnings_recognized=false`), distinct
|
||||
dashboard treatment; both repos' host-report goldens updated **byte-identical** + the
|
||||
bidirectional key-set contract test extended; hub bumped v0.7.4 → v0.7.5 and deployed via GitOps.
|
||||
|
||||
## No secrets
|
||||
|
||||
No secrets touched or committed. Live config/token references are by location only.
|
||||
Provisioning BACK HALF (controller deploy, bootstrap, per-guest token mint) → slice 8; host-loss
|
||||
DR + escrow consumption → slice 10; the SOURCE of a `BringUpSpec` (hub desired-state) → slice 10
|
||||
(`GuestMount` defined minimally, no hub coupling). No secrets committed.
|
||||
|
||||
Reference in New Issue
Block a user