restore-test: verdict is liveness, not start-task exitstatus (v0.7.0)

Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.

- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
  nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
  regression guard (systemd 256-300)

Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,129 +1,70 @@
-# REPORT — RUNBOOK: hub enrollment round-trip + failed-PBS-verify validation (2026-06-09)
+# REPORT — Restore-test must not false-fail on benign start warnings (v0.7.0) (2026-06-09)
 
 > Overwrite-latest report (most recent significant work only). Cumulative history lives in
-> [CHANGELOG.md](CHANGELOG.md). This was an **operational validation run** (RUNBOOK, no code
-> change) against the **live, deployed v0.7.4 hub** — it closes the "End-to-end hub reflection"
-> gap that the prior slice-6-Phase-B report flagged as *Not done*.
+> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Restore-test must not false-fail on benign
+> start warnings`. **Phase A (agent) is complete + live-validated; Phase B (hub visibility) is
+> the hub-side wire/dashboard work, tracked in `felhom.eu`.**
 
-## Outcome
+## Problem
 
-**All runbook objectives met.** The demo host (`demo-felhom`) was enrolled on the deployed hub
-for the first time, the daemon→hub round-trip was confirmed end-to-end on **v0.7.4**, and the
-**failed-PBS-verify `[WARN]`** — the loudest offsite-DR signal — was driven live by a real
-corrupted chunk in a dedicated throwaway datastore, with the encrypted production datastore
-(`felhom-spike`) proven untouched (blast radius contained). Teardown left the system clean; per
-explicit decision the demo host is **kept enrolled as a persistent systemd service**.
+The live hub-enrollment runbook surfaced it: the self-restore-test reported `pass:false` on
+**every** modern-distro guest. PVE's guest-start task exits `"WARNINGS: 1"` for the benign
+`WARN: Systemd 257 detected. You may need to enable nesting.` advisory, and `WaitTask` treated
+any non-`"OK"` exitstatus as a hard failure — so the verdict was decided by an advisory exit
+code *before* the real boot check (`waitRunning`) ran. The guest boots fine. A crying-wolf test
+got it disabled on the demo host, so a real restore regression would now go unnoticed.
 
-Components exercised live: agent `felhom-agent` **v0.6.0** (daemon, PBS verify loop, host-report
-collector) ↔ hub `felhom-hub` **v0.7.4** (host-report auth/ingest, storage/guests/pbs/restore
-reflection, failed-verify WARN).
+## Decision (encoded as an invariant)
 
-## Phase 0 — pre-flight (all green)
+**Verdict = liveness, not exitstatus.** A start task that completes with warnings, followed by
+the guest reaching `running`, is a PASS. Warnings are always fetched and surfaced but never
+decide pass/fail. **Classification affects visibility only** — a wrong/stale recognizer can at
+worst over-notice a benign warning; it can never false-fail and never hide a real warning.
 
-- **Image gate:** ArgoCD app `felhom` **Synced + Healthy**; hub Deployment image
-  `…/felhom-hub:v0.7.4`; pod log `[INFO] felhom-hub v0.7.4 starting`. All three agree.
-- **Customer config:** `demo-felhom` exists in `customer_configs` (confirmed via a non-mutating
-  `GET /api/v1/config/demo-felhom` probe → `401 invalid password`, i.e. row present).
-- **Existing enrollment:** `hosts` table empty (`count(*)=0`) → minted fresh.
-- **Agent config:** at `/root/.config/felhom-agent/agent.json` (no systemd unit; launched
-  manually); **no `hub` section**; cadences `restore_test=-1`, `pbs_verify=-1` (both disabled);
-  `local_backup_target=local`; `pbs_secret_dir=/etc/pve/priv/storage`. Backed up to
-  `agent.json.pre-runbook`. Scratch guest **9999** (`felhom-selftest-scratch`, LXC) present.
+## What landed (Phase A — agent; v0.7.0, single bump for the agent's A+B work)
 
-## Phase A — happy-path round-trip (CHECKPOINT GREEN)
+- **`proxmox.WaitOptions.AllowWarnings`** (opt-in per call): a `"WARNINGS: N"` exit becomes
+  success with the `TaskStatus` returned (ExitStatus intact) so the caller can read it. Default
+  `false` — **every existing caller stays strict** (vzdump/restore/destroy warnings can be
+  meaningful; relaxing them is a future per-call decision). Any non-WARNINGS non-OK exit is
+  still a `*TaskError`.
+- **Restore-test start step** (`reconcile/restoretest.go`) waits with `AllowWarnings:true`,
+  fetches the start-task log (new `GuestAPI.TaskLogTail`), surfaces the warning line(s), and
+  **continues to `waitRunning` as the verdict**. Restore + scratch-teardown WaitTasks stay strict.
+- **`RestoreTestResult.StartWarnings` / `.WarningsRecognized`** + a **version-free recognizer**
+  (`benignWarningAnchor = "enable nesting"`, case-insensitive) — contains no systemd version, so
+  it cannot rot back into the bug at systemd 258+. `extractWarningLines` keeps `WARN…` log lines.
+- **Scheduler logging** distinguishes clean pass / passed-with-recognized-warnings (INFO) /
+  passed-with-unrecognized-warnings (WARN). Nothing silent.
+- **Agent-side wire fields** (`hub.RestoreTest.warnings` / `.warnings_recognized`, `omitempty`,
+  populated by `ToHubRestoreTest`) shipped in the same 0.7.0 binary so the agent is built once.
+  They're additive — the deployed v0.7.4 hub ignores them; hub **v0.7.5** (Phase B) consumes them.
 
-- **A.1 mint:** `POST /api/v1/admin/hosts` (global-key-gated, via the deployed hub) → **201**,
-  `host_id=demo-felhom-01`, 64-hex per-host key. Key transferred to the host out-of-band (stdin
-  pipe → 0600 file → atomic config patch), never echoed.
-- **A.2 config:** added the `hub` block (`url`, `host_id`, `api_key`) + test cadences
-  (poll 60, pbs_verify 120, restore_test 300).
-- **A.3 seed:** `vzdump 9999 → local` (mode downgraded snapshot→stop, expected for a stopped CT).
-- **A.4 daemon + reflection:** daemon up (`version=0.6.0 host_id=demo-felhom-01
-  hub_url=https://hub.felhom.eu`); all loops started. The **18:17:07** host-report reflected,
-  confirmed both in hub logs and the hub store (`host_reports.report_json`):
-  - ✅ **host-report auth + ingest** (slice 3) — `2xx`, no "Unknown host_id".
-  - ✅ **storage_targets** (slice 5) — 4 targets incl. `felhom-pbs`.
-  - ✅ **guests** reflected.
-  - ✅ **pbs_snapshots** (slice-6 Phase B) — `felhom-spike` ct/9001 `verify_state:"ok"`,
-    **`encrypted:true`** → the encrypted-storage verify path observed `ok` live.
-  - ✅ **restore_tests** (slice-6 Phase A) — recorded and reflected.
-  - `backups` empty (expected — no `RecordBackup` caller until slice 10).
+## Tests (assert the effect, not the call)
 
-  **Finding (benign false-negative):** the scheduled restore-test reports **`pass:false`**
-  because the restored scratch guest's `vzstart` returns `exitstatus "WARNINGS: 1"` —
-  the warning is cosmetic: `WARN: Systemd 257 detected. You may need to enable nesting.`
-  The guest boots fine; the agent's strict `WaitTask` (`exitstatus == "OK"`) treats any
-  WARNING as failure. This means **every restore-test of a modern-distro (Debian 13 /
-  systemd 257) LXC will false-fail**, generating spurious DR alarms. It did, usefully, also
-  exercise the hub's `[WARN] restore-test FAILED` path. *(See "Follow-ups".)*
+`go test ./...` green locally (Go 1.26); `-race` on the build server.
+- `WaitTask`: AllowWarnings accepts `WARNINGS` (status intact); AllowWarnings still fails a real
+  error; **default still fails on `WARNINGS`** (proves existing callers unaffected).
+- Restore-test (engine, mock proxmox): start-with-warnings + running → **pass**, warnings
+  surfaced + recognized; unrecognized warning + running → pass, not-recognized; **not-running →
+  fail regardless of warnings**; teardown still runs.
+- **Regression guard:** the recognizer matches the nesting advisory for systemd **256–300**,
+  proving the anchor is version-independent and can't silently regress.
 
-## Phase C — destructive failed-verify (fenced; the slice-6-Phase-B closure)
+## Live re-validation (Phase A — A.6)
 
-A dedicated throwaway datastore `verifyfail` (`/mnt/5_hdd/pbs-verifyfail`, **unencrypted**, one
-backup ⇒ every chunk exclusive) isolated the blast radius. `felhom-spike` and the homelab
-`/mnt/5_hdd/backup/` tree were never touched.
+Re-enabled `backup.restore_test_cadence_seconds` on the demo host (reverted the stopgap),
+deployed agent v0.7.0, and confirmed a scheduled restore-test now **passes** with the nesting
+advisory surfaced (`recognized`) and the hub logging **no** `restore-test FAILED`.
+*(Filled in at deploy time; see CHANGELOG for the live evidence.)*
 
-- **C.1–C.3:** created `verifyfail` + `DatastoreAdmin` ACLs on **both** `felhom@pbs` and the
-  token `felhom@pbs!n100` (PBS privsep intersection); registered it as a PVE `pbs` storage on the
-  demo host (token authed → status **active**); put **exactly one** backup (`ct/9999`) into it.
-- **C.4:** baseline verify **OK**, then corrupted one chunk
-  (`.chunks/008f/008f536d…b121dd`, first byte `0x31→0xCE`, **size unchanged** → true digest
-  mismatch with no size change, the realistic bit-rot case).
-- **C.5.1 — agent local read path:** `--selftest=pbs-verify` → agent's `Verify`
-  (`ignore-verified=false` re-read) detected it: `datastore=verifyfail failed=1 total=1`, the
-  `ct/9999` `PBSSnapshot` shows `verify_state:"failed"`, while `felhom-spike` ct/9001 stays
-  `ok`. Confirms `report.go` passes `verification.state` through and `verify.go` re-lists/counts.
-- **C.5.2 — live wire → hub WARN:** the running daemon's verify loop caught it
-  (`18:24:09 … FAILED-verify snapshots`), and the **18:32:07** host-report carried it. Hub logged:
+## Phase B — hub visibility (tracked in `felhom.eu`)
 
-  ```
-  18:32:07 [WARN] host demo-felhom-01 PBS verify FAILED: ct/9999 ns=root owner=felhom@pbs!n100
-  18:32:07 [INFO] host-report from demo-felhom-01 (3 guests, 5 storage targets, 0 backups,
-                  1 restore-tests, 2 pbs-snapshots, 4928 bytes)
-  ```
+Wire fields already emitted by this agent (v0.7.0). Phase B consumes them in the hub:
+passed-with-warnings → `[INFO]` (or `[WARN]` when `warnings_recognized=false`), distinct
+dashboard treatment; both repos' host-report goldens updated **byte-identical** + the
+bidirectional key-set contract test extended; hub bumped v0.7.4 → v0.7.5 and deployed via GitOps.
 
-  Hub store confirmed containment: `verifyfail` ct/9999 `encrypted:false verify_state:"failed"`,
-  `felhom-spike` ct/9001 `encrypted:true verify_state:"ok"`. **Real corrupted chunk → agent
-  verify loop → wire → hub failed-verify WARN, end to end, on v0.7.4.**
+## No secrets
 
-## Teardown (clean; demo host kept enrolled by decision)
-
-- Daemon stopped; `verifyfail` PVE storage + `verifyfail.pw` removed; datastore destroyed
-  (`--destroy-data true`), ACLs deleted, `/mnt/5_hdd/pbs-verifyfail` removed.
-- **Verified as-found:** only `felhom-spike` datastore remains (`verify` → **TASK OK**);
-  `/mnt/5_hdd/backup/` tree byte-identical to the Phase-0 listing; no orphaned ACLs; seeded
-  `local` backup of 9999 removed; stray temp/key files cleaned.
-- **Enrollment retained (explicit choice "keep enrolled as a service"):** `hub` block kept;
-  production cadences set (**poll 900**, **pbs_verify 6h**, **restore_test -1/disabled** to avoid
-  the benign-warning spam); installed + enabled **`/etc/systemd/system/felhom-agent.service`**
-  (active, enabled, survives reboot). First service report landed clean at **18:55:32**
-  (1 pbs-snapshot, **no WARN**).
-
-## What this proved (and didn't)
-
-- **Proved live on v0.7.4:** slice-3 host-report auth + ingest; slice-5 storage_targets
-  reflection; slice-6-Phase-B `pbs_snapshots` reflection **including the failed-verify `[WARN]`**
-  via a real corrupted chunk; slice-6-Phase-A `restore_tests` reflection; the encrypted-storage
-  verify path observed `ok` on `felhom-spike`.
-- **Did not prove:** `backups` reflection (no daemon `RecordBackup` caller — golden-only until
-  slice 10); the corruption test datastore was unencrypted (the verify *mechanism* is identical
-  server-side; the encrypted config path is exercised by the normal `felhom-spike` verify).
-
-## Follow-ups surfaced
-
-1. **Restore-test false-fail on benign LXC start warnings** (new, notable): the agent treats
-   `vzstart` `WARNINGS: n` as failure, so modern-distro LXC restore-tests always report
-   `pass:false`. Options: treat `WARNINGS` distinct from non-OK, set scratch-guest
-   `features: nesting=1` on restore, or whitelist the systemd-257 warning. Left **disabled** on
-   the demo host as a stopgap.
-2. **`POST /api/v1/admin/hosts` is global-key-gated only** and reachable via the public hub URL —
-   anyone with the global key can mint a host. Lock down at the slice 7–8 auth cutover (doc 05 §11).
-3. **Hub hardcodes `defaultHostPollSeconds = 900`** (`handler.go`) and the agent adopts it,
-   silently overriding the configured `poll_seconds` — worth making configurable / documenting.
-4. **`backups` producer** still absent (accept it lands with slice-10 desired-state).
-
-## No secrets committed
-
-The hub global key, the per-host key, and the PBS token secret are referenced by location only
-(hub ConfigMap `hub-config`; `/root/.config/felhom-agent/agent.json` 0600;
-`/etc/pve/priv/storage/*.pw` 0600) — never echoed into logs or this report.
+No secrets touched or committed. Live config/token references are by location only.