felhom-agent/REPORT.md

REPORT — Restore-test must not false-fail on benign start warnings (v0.7.0) (2026-06-09)

Overwrite-latest report (most recent significant work only). Cumulative history lives in CHANGELOG.md. Implements TASK — Restore-test must not false-fail on benign start warnings. Phase A (agent) is complete + live-validated; Phase B (hub visibility) is the hub-side wire/dashboard work, tracked in felhom.eu.

Problem

The live hub-enrollment runbook surfaced it: the self-restore-test reported pass:false on every modern-distro guest. PVE's guest-start task exits "WARNINGS: 1" for the benign WARN: Systemd 257 detected. You may need to enable nesting. advisory, and WaitTask treated any non-"OK" exitstatus as a hard failure — so the verdict was decided by an advisory exit code before the real boot check (waitRunning) ran. The guest boots fine. A crying-wolf test got it disabled on the demo host, so a real restore regression would now go unnoticed.

Decision (encoded as an invariant)

Verdict = liveness, not exitstatus. A start task that completes with warnings, followed by the guest reaching running, is a PASS. Warnings are always fetched and surfaced but never decide pass/fail. Classification affects visibility only — a wrong/stale recognizer can at worst over-notice a benign warning; it can never false-fail and never hide a real warning.

What landed (Phase A — agent; v0.7.0, single bump for the agent's A+B work)

• proxmox.WaitOptions.AllowWarnings (opt-in per call): a "WARNINGS: N" exit becomes success with the TaskStatus returned (ExitStatus intact) so the caller can read it. Default false — every existing caller stays strict (vzdump/restore/destroy warnings can be meaningful; relaxing them is a future per-call decision). Any non-WARNINGS non-OK exit is still a *TaskError.
• Restore-test start step (reconcile/restoretest.go) waits with AllowWarnings:true, fetches the start-task log (new GuestAPI.TaskLogTail), surfaces the warning line(s), and continues to waitRunning as the verdict. Restore + scratch-teardown WaitTasks stay strict.
• RestoreTestResult.StartWarnings / .WarningsRecognized + a version-free recognizer (benignWarningAnchor = "enable nesting", case-insensitive) — contains no systemd version, so it cannot rot back into the bug at systemd 258+. extractWarningLines keeps WARN… log lines.
• Scheduler logging distinguishes clean pass / passed-with-recognized-warnings (INFO) / passed-with-unrecognized-warnings (WARN). Nothing silent.
• Agent-side wire fields (hub.RestoreTest.warnings / .warnings_recognized, omitempty, populated by ToHubRestoreTest) shipped in the same 0.7.0 binary so the agent is built once. They're additive — the deployed v0.7.4 hub ignores them; hub v0.7.5 (Phase B) consumes them.

Tests (assert the effect, not the call)

go test ./... green locally (Go 1.26); -race on the build server.

• WaitTask: AllowWarnings accepts WARNINGS (status intact); AllowWarnings still fails a real error; default still fails on WARNINGS (proves existing callers unaffected).
• Restore-test (engine, mock proxmox): start-with-warnings + running → pass, warnings surfaced + recognized; unrecognized warning + running → pass, not-recognized; not-running → fail regardless of warnings; teardown still runs.
• Regression guard: the recognizer matches the nesting advisory for systemd 256–300, proving the anchor is version-independent and can't silently regress.

Live re-validation (Phase A — A.6)

Re-enabled backup.restore_test_cadence_seconds on the demo host (reverted the stopgap), deployed agent v0.7.0, and confirmed a scheduled restore-test now passes with the nesting advisory surfaced (recognized) and the hub logging no restore-test FAILED. (Filled in at deploy time; see CHANGELOG for the live evidence.)

Phase B — hub visibility (tracked in felhom.eu)

Wire fields already emitted by this agent (v0.7.0). Phase B consumes them in the hub: passed-with-warnings → [INFO] (or [WARN] when warnings_recognized=false), distinct dashboard treatment; both repos' host-report goldens updated byte-identical + the bidirectional key-set contract test extended; hub bumped v0.7.4 → v0.7.5 and deployed via GitOps.

No secrets

No secrets touched or committed. Live config/token references are by location only.