admin
6e86483185
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook: PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict was decided by an advisory exit code before the real boot check ran. Every modern-distro restore-test reported pass:false. - proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict) - restore-test start step accepts warnings, surfaces them, verdict stays waitRunning - RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail - hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5) - scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings - tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free regression guard (systemd 256-300) Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
71 lines
4.4 KiB
Markdown
71 lines
4.4 KiB
Markdown
# REPORT — Restore-test must not false-fail on benign start warnings (v0.7.0) (2026-06-09)
|
||
|
||
> Overwrite-latest report (most recent significant work only). Cumulative history lives in
|
||
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Restore-test must not false-fail on benign
|
||
> start warnings`. **Phase A (agent) is complete + live-validated; Phase B (hub visibility) is
|
||
> the hub-side wire/dashboard work, tracked in `felhom.eu`.**
|
||
|
||
## Problem
|
||
|
||
The live hub-enrollment runbook surfaced it: the self-restore-test reported `pass:false` on
|
||
**every** modern-distro guest. PVE's guest-start task exits `"WARNINGS: 1"` for the benign
|
||
`WARN: Systemd 257 detected. You may need to enable nesting.` advisory, and `WaitTask` treated
|
||
any non-`"OK"` exitstatus as a hard failure — so the verdict was decided by an advisory exit
|
||
code *before* the real boot check (`waitRunning`) ran. The guest boots fine. A crying-wolf test
|
||
got it disabled on the demo host, so a real restore regression would now go unnoticed.
|
||
|
||
## Decision (encoded as an invariant)
|
||
|
||
**Verdict = liveness, not exitstatus.** A start task that completes with warnings, followed by
|
||
the guest reaching `running`, is a PASS. Warnings are always fetched and surfaced but never
|
||
decide pass/fail. **Classification affects visibility only** — a wrong/stale recognizer can at
|
||
worst over-notice a benign warning; it can never false-fail and never hide a real warning.
|
||
|
||
## What landed (Phase A — agent; v0.7.0, single bump for the agent's A+B work)
|
||
|
||
- **`proxmox.WaitOptions.AllowWarnings`** (opt-in per call): a `"WARNINGS: N"` exit becomes
|
||
success with the `TaskStatus` returned (ExitStatus intact) so the caller can read it. Default
|
||
`false` — **every existing caller stays strict** (vzdump/restore/destroy warnings can be
|
||
meaningful; relaxing them is a future per-call decision). Any non-WARNINGS non-OK exit is
|
||
still a `*TaskError`.
|
||
- **Restore-test start step** (`reconcile/restoretest.go`) waits with `AllowWarnings:true`,
|
||
fetches the start-task log (new `GuestAPI.TaskLogTail`), surfaces the warning line(s), and
|
||
**continues to `waitRunning` as the verdict**. Restore + scratch-teardown WaitTasks stay strict.
|
||
- **`RestoreTestResult.StartWarnings` / `.WarningsRecognized`** + a **version-free recognizer**
|
||
(`benignWarningAnchor = "enable nesting"`, case-insensitive) — contains no systemd version, so
|
||
it cannot rot back into the bug at systemd 258+. `extractWarningLines` keeps `WARN…` log lines.
|
||
- **Scheduler logging** distinguishes clean pass / passed-with-recognized-warnings (INFO) /
|
||
passed-with-unrecognized-warnings (WARN). Nothing silent.
|
||
- **Agent-side wire fields** (`hub.RestoreTest.warnings` / `.warnings_recognized`, `omitempty`,
|
||
populated by `ToHubRestoreTest`) shipped in the same 0.7.0 binary so the agent is built once.
|
||
They're additive — the deployed v0.7.4 hub ignores them; hub **v0.7.5** (Phase B) consumes them.
|
||
|
||
## Tests (assert the effect, not the call)
|
||
|
||
`go test ./...` green locally (Go 1.26); `-race` on the build server.
|
||
- `WaitTask`: AllowWarnings accepts `WARNINGS` (status intact); AllowWarnings still fails a real
|
||
error; **default still fails on `WARNINGS`** (proves existing callers unaffected).
|
||
- Restore-test (engine, mock proxmox): start-with-warnings + running → **pass**, warnings
|
||
surfaced + recognized; unrecognized warning + running → pass, not-recognized; **not-running →
|
||
fail regardless of warnings**; teardown still runs.
|
||
- **Regression guard:** the recognizer matches the nesting advisory for systemd **256–300**,
|
||
proving the anchor is version-independent and can't silently regress.
|
||
|
||
## Live re-validation (Phase A — A.6)
|
||
|
||
Re-enabled `backup.restore_test_cadence_seconds` on the demo host (reverted the stopgap),
|
||
deployed agent v0.7.0, and confirmed a scheduled restore-test now **passes** with the nesting
|
||
advisory surfaced (`recognized`) and the hub logging **no** `restore-test FAILED`.
|
||
*(Filled in at deploy time; see CHANGELOG for the live evidence.)*
|
||
|
||
## Phase B — hub visibility (tracked in `felhom.eu`)
|
||
|
||
Wire fields already emitted by this agent (v0.7.0). Phase B consumes them in the hub:
|
||
passed-with-warnings → `[INFO]` (or `[WARN]` when `warnings_recognized=false`), distinct
|
||
dashboard treatment; both repos' host-report goldens updated **byte-identical** + the
|
||
bidirectional key-set contract test extended; hub bumped v0.7.4 → v0.7.5 and deployed via GitOps.
|
||
|
||
## No secrets
|
||
|
||
No secrets touched or committed. Live config/token references are by location only.
|