REPORT: v0.136.0 and v0.137.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

.claude/rules/backup.md

@@ -0,0 +1,46 @@
+---
+paths: ["internal/backup/**", "internal/pbs/**", "internal/pbsdr/**", "internal/dr/**"]
+---
+
+# Backup, PBS and DR
+
+`internal/backup/` is the vzdump runner, restore-test scheduler and report store. `internal/pbs/` is
+the fingerprint-pinned PBS-API client plus the verify maintenance loop. `internal/pbsdr/` and
+`internal/dr/` carry the DR tier and recipe halves.
+
+## The three PBS laws
+
+1. **Set-only.** `pvesm remove` **DELETES the encryption key**. Re-apply configuration; never remove
+   and re-add a PBS storage to change it.
+2. **Secret on stdin.** A token secret is passed on stdin, never as an argv the process table shows.
+3. **Verify the pin BEFORE consuming the secret.** A fingerprint check after the secret has been sent
+   protects nothing.
+
+## Verify is server-side, and its default skips the work
+
+The agent drives verification **remotely** via the PBS API; `proxmox-backup-client` has **no** verify
+subcommand. `POST .../verify` defaults to **`ignore-verified=true`, which SKIPS already-verified
+snapshots** — send `ignore-verified=false` to actually re-read and detect corruption. A verify that
+skipped everything reports success.
+
+## Presence is not success
+
+A timestamp recording an **attempt** is not evidence of a **result**. Where a status field travels
+beside a timestamp, the verdict must consult **both** — or the timestamp must record only successes.
+Ask of any timestamp: *what exactly must have happened for this to be set?* If the answer is "we
+tried", it cannot answer "did it work".
+
+**Corollary:** when a verdict changes which field it counts from, the alarm text changes with it.
+Leaving a message reading `last run 8h ago` while alarming on a six-day-old **success** turns a true
+alarm into one the operator dismisses.
+
+## Prune is server-side now
+
+`DatastoreBackup` carries **no** `Datastore.Prune`. Boxes set `keep_last: 0` and the off-site endpoint
+runs the prune jobs. **Box tokens stay write-only — never widen that grant** (R-89).
+
+<!--
+The ignore-verified default is the sharpest instance of the "absent log line" class in this repo: a
+verify that silently skipped every snapshot completes fast, exits clean, and reports the same shape
+as one that read every byte.
+-->

.claude/rules/health-checks.md

@@ -0,0 +1,26 @@
+---
+paths: ["internal/capability/**", "internal/storage/**", "internal/localapi/**", "internal/hub/**", "internal/guesthook/**"]
+---
+
+# A health check issues no block I/O
+
+No `statfs`, no `getdents`, no read, write or `fsync` — **not even behind a timeout**.
+
+A probe that touches a wedged device enters uninterruptible sleep, survives `SIGKILL`, and cannot be
+recovered until the device returns or the host reboots — so `systemctl restart` hangs too. A timeout
+protects the caller's control flow and nothing else: the blocked thread remains.
+
+**Liveness is decided from `/proc` and the kernel's own state**, never by reading or writing the
+filesystem.
+
+<!--
+Measured, R-117 spike §6.3 (felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md):
+a probe stayed in D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT
+needs journal access); and statfs/getdents returned HEALTHY on a namespace that EIOs every byte —
+fast, and wrong.
+
+This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
+felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
+path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
+file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
+-->

.claude/rules/localapi.md

@@ -0,0 +1,44 @@
+---
+paths: ["internal/localapi/**", "internal/authz/**", "internal/guesthook/**"]
+---
+
+# Local API, authz and guest hooks — the per-guest blast radius
+
+`internal/localapi/` is the narrow per-guest local API: token store, disks/format, guest binds,
+controller swap, stale-lock recovery, pinned self-signed leaf. `internal/authz/` is the operator
+signed-op verifier (SSHSIG) plus the durable nonce store. `internal/guesthook/` installs the
+pre-start self-heal hookscript.
+
+> **Overlap note:** `health-checks.md` also matches `internal/localapi/**` and
+> `internal/guesthook/**`. That is deliberate — both rules apply there and both load. Neither
+> supersedes the other.
+
+## Scoping is the whole security property
+
+This API is reachable **from inside a customer guest**. Every route must be scoped to the guest that
+called it — a route that can name another guest's id has escaped its blast radius. Fail **safe to
+protected**: an unrecognised or unresolvable caller gets less access, never more.
+
+## Replay protection must survive a restart
+
+**`authz.MemoryNonceStore` on a real host is a defect** — replay protection dies on restart. Use
+`authz.FileNonceStore`. The memory store exists for tests.
+
+## The token is a hash on disk, plaintext only at mint
+
+The store keeps **hashes**. The plaintext token exists in exactly one place, `bootstrap.json` on the
+PVE host — so a "read the token" step means reading that file, and a lost token is re-minted, never
+recovered.
+
+## Binds can brick guest boot
+
+| Do not | Because | Use |
+|---|---|---|
+| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can **brick guest boot** (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
+| `isHostMountpoint` to reconcile bind state | a boolean cannot converge stacked double-binds (the `/mnt` doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
+
+<!--
+Why fail-safe-to-protected rather than fail-closed: this API also carries the recovery paths. A hard
+refusal on an unresolvable caller would make a half-broken guest unrecoverable through the very
+interface built to recover it. Less access, never none.
+-->

.claude/rules/proxmox.md

@@ -0,0 +1,44 @@
+---
+paths: ["internal/proxmox/**", "internal/reconcile/**", "internal/signedjobs/**"]
+---
+
+# Proxmox — the API contract, and how destructive work is gated
+
+`internal/proxmox/` is the API-first `Client` plus the fenced root-CLI `Privileged`.
+`internal/reconcile/` is the reconcile engine, reversibility gate, op journal and crash recovery.
+`internal/signedjobs/` holds the operator-signed destructive executors (wipe, decommission).
+
+## A 200 on the POST is not success
+
+**Every mutating op is async**: it returns a **UPID**, and `WaitTask` must assert
+`exitstatus == "OK"`. Authorization can fail at *task execution* long after the HTTP call returned
+200. Treating the POST's status as the result is how a failed destroy reads as a successful one.
+
+## The privsep token gotcha
+
+A `--privsep 1` token's rights are the **intersection** of the backing user's permissions **and** the
+token's own ACLs. The role must be granted on **both** or every call 403s. The same intersection rule
+bites on PBS (`token ∩ user`).
+
+## TLS
+
+**SHA-256 leaf-cert pinning** against the self-signed host cert. **No insecure default**, ever. The
+pin is the raw leaf-DER sha — the SAN is never checked, so a cert rotation changes the pin and the
+agent must be re-pinned.
+
+## The destructive path — never the direct call
+
+| Do not | Because | Use |
+|---|---|---|
+| `Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
+| add a method to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers `Cmnd_Alias` + `validate.go`-style checks |
+| treat `ListLXC` output as "guests we own" | audit A1 — pre-v0.62.0 the stale-lock reaper did exactly this, contained only by the pool-scoped token | intersect with `Client.Pool` membership (`staleLockController.Guests()`); **fail safe on read failure** |
+
+Full trap table: `REUSE.md` §3. Every guest joins the `felhom` pool — `VM.Audit` comes from the
+`/pool` grant, not from a per-guest ACL.
+
+<!--
+The fence is not stylistic. It is what makes this component auditable: two types, one of which can
+only speak HTTP and one of which can only shell out, with a test asserting neither crosses. A single
+convenience method on Privileged that also makes an HTTP call would end that property silently.
+-->

.claude/rules/storage.md

@@ -0,0 +1,49 @@
+---
+paths: ["internal/storage/**", "internal/escrow/**"]
+---
+
+# Storage and escrow — format safety and zero-knowledge recovery
+
+`internal/storage/` is the storage observer, durable IDs, role/claim classifiers, `SudoHostOps` and
+the watchdog. `internal/escrow/` is the PBS-key escrow with its zero-knowledge recovery code.
+
+> **Overlap note:** `health-checks.md` also matches `internal/storage/**`. Deliberate — both rules
+> apply there and both load.
+
+## Never format the device you inspected
+
+**AGENT-001 is a TOCTOU:** acting on the caller's `req.Device` (or any remembered `/dev` path) after
+inspection lets `/dev` re-enumeration retarget the node to a **different physical disk**. Format the
+**re-resolved** device — `Server.reresolveWipe` / `reresolveBlank`.
+
+**Never exec raw `mkfs.*`** (including `Binaries.MkfsExt4`/`MkfsXfs`): sudoers no longer allowlists
+raw mkfs, and going direct bypasses the claim filter and the wrapper's re-checks. Use
+`SudoHostOps.Format`, which routes through `felhom-mkfs-guarded`.
+
+## The two durable-ID schemes refuse each other
+
+They are not interchangeable, and each returns a `binding_mismatch` for the other's scheme:
+
+| Purpose | Scheme | Resolver |
+|---|---|---|
+| wipe confirmation | `byid:` / `byuuid:` | `ResolveDurableDevice`, `DiskInfo.WipeDurableID` |
+| enrolled-storage remount | `uuid:` | `ResolveStorageDevice` |
+
+Using `DiskInfo.DurableID` (a `uuid:`) as a wipe-confirmation id is F20-BUG2.
+
+## Drive data is never taken by force
+
+Plain `umount` only — **never `-l`, never `-f`**, and never any format operation under
+`/mnt/felhom-drives`.
+
+## Escrow is zero-knowledge, and a fetch failure is not a wrong code
+
+The server holds no client key; a no-key restore fails with `missing key`. **A fetch failure must
+never be reported as a wrong recovery code** — that told a customer their correct code was bad, in
+hundredths of a second, when checking a code actually takes about one. Distinguish "we could not
+reach the store" from "the code did not match", always.
+
+<!--
+The escrow recovery-code "flake" was a REAL defect, not a flake. "Known flake, re-run" needs evidence
+before it is said out loud — that phrase cost this project a real finding once.
+-->

.gitea/workflows/gates.yml

@@ -0,0 +1,116 @@
+# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
+# pushed or what they typed.
+#
+#   *** THIS REPORTS. IT CANNOT REFUSE. ***
+#
+# felhom repos push straight to `main` with no pull request, so there is no merge for a status
+# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
+# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
+# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
+#
+# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
+# the runner is a host-mode container with python3 and git and nothing else (see
+# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
+# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
+#
+# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
+# layer up. That is the last step, and it runs ONLY on failure.
+name: gates
+on: [push]
+
+jobs:
+  gates:
+    runs-on: felhom-gates
+    steps:
+      - name: Fetch the pushed commit and the sibling clone it needs
+        # This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
+        # door and is deliberately never copied here — so CI has to reproduce the workspace's
+        # sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
+        # needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
+        run: |
+          # Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
+          # which can move under us if two pushes race.
+          mkdir -p ws/felhom-agent
+          cd ws/felhom-agent
+          git init -q .
+          git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
+          git fetch -q --depth 1 origin "$GITHUB_SHA"
+          git checkout -q FETCH_HEAD
+          echo "checked out $(git rev-parse HEAD)"
+          cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
+          echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
+
+      - name: Run the gate entry point
+        # The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
+        # exit code IS the result: no `|| true`, no pipe that could swallow it.
+        #
+        # THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
+        # container runtime" and exists for `.githooks/pre-push`, where a push must not fail
+        # because Gitea blinked or because someone is on a train. CI is the opposite machine: it
+        # has the network, it is not in anyone's way, and it is the half that emails. The
+        # published-versions gate — the R-115 mechanism, which asks Gitea whether a released
+        # version can actually be downloaded — is network-bound and therefore runs ONLY here.
+        # Leaving `--fast` in place would have registered that gate and never run it, which is the
+        # built-but-never-wired failure this project has shipped four times.
+        env:
+          # In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
+          GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
+        run: cd ws/felhom-agent && python3 scripts/agent_gates.py
+
+      - name: Alarm on failure
+        # THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
+        # notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
+        # is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
+        # existing transactional path (Resend, the same one the hub uses), and prints the provider's
+        # accepted id so "a message left the machine" is an observable, not an assumption.
+        #
+        # Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
+        # else on purpose, and the first version of this step died on `curl: command not found`.
+        # Reaching for a bigger image to send one HTTP request would have been the wrong trade.
+        if: failure()
+        env:
+          RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
+        run: |
+          python3 - <<'PY'
+          import json, os, sys, urllib.request, urllib.error
+
+          key = os.environ.get("RESEND_API_KEY", "")
+          if not key:
+              sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
+                       "silent alarm is worse than none. Set the user-level Actions secret.")
+
+          repo = os.environ.get("GITHUB_REPOSITORY", "?")
+          sha  = os.environ.get("GITHUB_SHA", "?")
+          run  = os.environ.get("GITHUB_RUN_NUMBER", "?")
+          srv  = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
+
+          body = json.dumps({
+              "from": "Felhom CI <monitoring@felhom.eu>",
+              "to": ["admin@felhom.eu"],
+              "subject": "[felhom CI] gates FAILED in %s" % repo,
+              "text": (
+                  "The gate entry point exited non-zero.\n\n"
+                  "Repository : %s\n"
+                  "Commit     : %s\n"
+                  "Run        : %s/%s/actions/runs/%s\n\n"
+                  "The failing gate names itself in the run log.\n\n"
+                  "If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
+                  "disagree - that is a finding about the gates themselves, not about CI, and it\n"
+                  "outranks whatever the push was for.\n"
+              ) % (repo, sha, srv, repo, run),
+          }).encode()
+
+          req = urllib.request.Request(
+              "https://api.resend.com/emails", data=body, method="POST",
+              headers={"Authorization": "Bearer %s" % key,
+                       "Content-Type": "application/json",
+                       # Cloudflare fronts api.resend.com and BLOCKS the default
+                       # "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
+                       # exactly like an auth failure and is not one. Measured 2026-08-02.
+                       "User-Agent": "felhom-ci/1.0"})
+          try:
+              with urllib.request.urlopen(req, timeout=30) as r:
+                  print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
+          except urllib.error.HTTPError as e:
+              sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
+          PY

.githooks/pre-push

@@ -0,0 +1,82 @@
+#!/bin/sh
+# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
+#
+# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
+# container runtime, so a push stays a push and never pulls images or starts containers. The slow
+# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
+# the bypass becomes the habit.
+#
+# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
+# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
+#
+# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
+#   * per-clone   — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
+#                     git config core.hooksPath .githooks
+#                   Any manual entry-point run WARNS when the clone is unarmed.
+#   * skippable   — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
+#                   hatch that cannot be reached is one that gets removed the first time it is
+#                   inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
+# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
+#
+# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
+# is the repo root whether `git push` is issued from the root or from any subdirectory. The
+# explicit rev-parse below does not depend on that.
+set -u
+
+root=$(git rev-parse --show-toplevel 2>/dev/null) || {
+    echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
+    exit 1
+}
+cd "$root" || exit 1
+
+# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
+# Refuse a push from a clone outside the felhom workspace.
+#
+# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
+# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
+# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
+# fixed by writing it down again — it has to be asserted where it can bite.
+#
+# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
+# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
+#
+# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
+# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
+# failed — this hook must not brick a legitimate clone on a different host.
+#
+# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
+FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
+if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
+    ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
+    root_real=$(pwd -P) || root_real=""
+    if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
+        case "$root_real/" in
+            "$ws_real"/*) : ;;  # inside the workspace — proceed
+            *)
+                echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
+                echo "          clone:    $root_real" >&2
+                echo "          expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
+                echo "          Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
+                echo "          and state that you did in the session report." >&2
+                exit 1
+                ;;
+        esac
+    fi
+fi
+
+if ! command -v python3 >/dev/null 2>&1; then
+    echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
+    echo "          pass by default. Install python3, or push with --no-verify and say so." >&2
+    exit 1
+fi
+
+echo "pre-push [felhom-agent]: running scripts/agent_gates.py --fast ..."
+python3 "scripts/agent_gates.py" --fast
+rc=$?
+if [ "$rc" -ne 0 ]; then
+    echo "pre-push [felhom-agent]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
+    echo "          'git push --no-verify' and state that you did in the session report." >&2
+else
+    echo "pre-push [felhom-agent]: gates OK - push proceeding."
+fi
+exit $rc

.gitignore

@@ -0,0 +1,11 @@
+# build output
+/felhom-agent
+/felhom-agent.exe
+/dist/
+
+# local config that may carry a real token secret
+/agent.json
+*.local.json
+
+# go
+/vendor/

AGENT-001-FIX-NOTES.md

@@ -0,0 +1,52 @@
+# AGENT-001 fix — inline customer-confirmed wipe: anti-retarget re-resolution
+
+**Branch:** `fix/agent-001-wipe-durable-reresolve` (off `main` @ d17b5ab) — **PENDING REVIEW, NOT deployed.**
+**Status:** branch-only. No agent build, no deploy, no golden rebake, no re-provision. That is reserved for a supervised session (this fix touches the destructive storage-wipe path).
+
+## The bug
+
+`internal/localapi/disks.go` `handleDiskFormat`, the data-bearing **customer-confirmed** branch
+(`dec.Allowed`), inspected and gate-bound the device by its durable id but then called
+`s.disks.Format(ctx, req.Device, …)` on the **caller-supplied mutable `/dev` path**. The durable id
+only bound the *confirmation*; it was never re-resolved to the device just before `mkfs`. If a USB
+re-enumeration reassigned `/dev/sdbN` to a **different physical disk** between inspection and `mkfs`
+(the classify→mkfs TOCTOU), the wipe hit the wrong drive — data loss on an unintended disk.
+
+The signed-jobs path (`internal/signedjobs/wipe.go` `WipeExecutor.Execute`) already does this
+correctly: resolve durable→device, re-derive the device's durable id and require an exact match,
+re-inspect, then format the *re-resolved* device. The inline path did not.
+
+## What changed
+
+- **New `internal/localapi/wipe_reresolve.go`:**
+  - `antiRetargetResolve(durableID, resolve, derive, inspect)` — pure, injected-deps helper mirroring
+    `WipeExecutor`: refuse empty durable id; resolve→current device; re-derive and require exact
+    match (a `/dev` node now pointing at a different disk derives a different id → refuse); re-inspect
+    and require still-`DataBearing()`. Returns the **re-resolved device** to format.
+  - `(*Server).reresolveDurableForWipe` wires it with the real `storage.ResolveDurableDevice` /
+    `storage.DeviceDurableID` / `s.disks.InspectDevice`.
+- **`internal/localapi/disks.go`:** the `dec.Allowed` branch now calls `s.reresolveWipe(ctx, deviceDurable)`
+  and formats the **returned** device (never `req.Device`). On any refusal it returns `409 Conflict`
+  with the reason and does **not** call `mkfs`. The response/log now reference the re-resolved device.
+- **`internal/localapi/server.go`:** new injectable field `reresolveWipe`, defaulted in `NewServer` to
+  `s.reresolveDurableForWipe`. Production behaviour unchanged (same real functions); the seam exists so
+  the inline path is unit-testable without real `/dev/disk/by-*`.
+- **Tests:** `wipe_reresolve_test.go` covers `antiRetargetResolve` directly — happy path, empty durable,
+  no-longer-resolves, **durable-id mismatch (the core retarget case)**, no-longer-data-bearing,
+  re-inspect error. The existing integration test `TestFormat_DataBearing_UserDataConfirmed_Formats`
+  stubs `reresolveWipe` (the test helper sets it). Full `go build/vet/test ./...` green at the branch HEAD.
+
+## Why it can't break legitimate wipes
+
+`storage.DeviceDurableID` produces `byid:`/`byuuid:` ids and `storage.ResolveDurableDevice` accepts
+exactly those — they round-trip. A normal user-data drive resolves back to itself, re-derives the same
+id, and inspects data-bearing → format proceeds. Only three new refusals occur, all correct:
+durable id gone, durable-id mismatch (retarget), or no-longer-data-bearing.
+
+## Review checklist for the supervised merge+deploy+golden-rebake session
+
+1. Confirm `antiRetargetResolve` refuses on mismatch/gone/blank/empty (run `go test ./internal/localapi/ -run AntiRetarget -v`).
+2. Sanity-check `DeviceDurableID`↔`ResolveDurableDevice` scheme round-trip on the real demo USB (`felhom-usb`) so a genuine customer wipe still resolves+matches (read-only `DeviceDurableID` on the live device; do NOT wipe).
+3. Confirm the `409 Conflict` refusal surfaces sensibly in the controller UI (controller `agentapi` maps the format response — verify it doesn't treat 409 as a hard error that hides the reason).
+4. Bump agent version + CHANGELOG, build, golden-rebake, re-provision per the agent deploy runbook.
+5. Live-validate one happy customer-confirmed wipe on a demo scratch drive (re-resolve matches → mkfs) and confirm the audit log line shows the re-resolved device == the confirmed durable id.

CLAUDE.md

@@ -0,0 +1,111 @@
+# CLAUDE.md — `felhom-agent`
+
+> Stable orientation only — **current state lives in `CONTEXT.md` and the top of `CHANGELOG.md`**,
+> never here. Cross-repo conventions (artifact taxonomy, access, clean-tree gate, secrets,
+> CHANGELOG/REPORT): workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`. Path-scoped detail:
+> `.claude/rules/`.
+
+## What this repo is
+
+The operator-tier **host agent**, one per Proxmox host, owning **all** Proxmox interaction:
+provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a
+narrow per-guest local API. It is the **most privilege-sensitive component in the system**.
+
+- Renamed from `proxmox-controller`.
+- **Distinct from `felhom-controller`** — that is the *in-guest* controller, Docker-only, holding no
+  Proxmox credentials. Do not confuse them.
+- **Control plane, not data plane:** if the agent dies, apps keep serving; only management degrades.
+- Pure Go stdlib + `golang.org/x/crypto`. No web frameworks.
+
+## Doing X → read Y
+
+| Doing | Read |
+|---|---|
+| writing any new code | `REUSE.md` — helpers, format-safety guards, traps, seams |
+| needing current state / open threads | `CONTEXT.md` + the top `CHANGELOG.md` entry |
+| Proxmox, reconcile or signed jobs | loads itself: `.claude/rules/proxmox.md` |
+| local API, authz or guest hooks | loads itself: `.claude/rules/localapi.md` |
+| backup, PBS or DR | loads itself: `.claude/rules/backup.md` |
+| storage or escrow | loads itself: `.claude/rules/storage.md` |
+| writing a health check | loads itself: `.claude/rules/health-checks.md` |
+| **release, build, publish, deploy, verify a version** | the **`felhom-build-deploy`** skill — **never hand-roll it** |
+| writing or reviewing a test, fixing a bug | the **`felhom-testing`** skill |
+| host addresses, break-glass, node facts | `felhom.eu/documentation/operations/nodes.md` — never restate them |
+| which box may I break | `felhom.eu/documentation/runbooks/target-selection.md` |
+| what version is live anywhere | ask the hub (`/hosts`, `/configs`) or the box — **never a doc** |
+| the authoritative design | `felhom.eu/documentation/architecture/03-host-agent.md` (locked) |
+
+## The root-CLI fence — API-first, exactly three exceptions
+
+This is in the core because breaching it is how this component stops being auditable.
+
+- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
+  keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors.
+- **`Client` never shells out; `Privileged` never makes HTTP calls** — asserted by `routing_test.go`.
+  Adding a method to `proxmox.Privileged` breaks the fence; use `proxmox.Runner` plus a new sudoers
+  `Cmnd_Alias` and `validate.go`-style checks (`REUSE.md` §3).
+- **Destructive ops go through the reconcile gate / signed-jobs path.** Never call
+  `Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc — that skips classification, signature,
+  per-guest serialization and crash recovery.
+- **Ownership must be PROVEN, never assumed.** A raw `ListLXC` list is not "guests the agent owns";
+  intersect with `Client.Pool` membership and fail safe on a read failure (audit A1).
+
+## Gates — ONE entry point
+
+**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
+repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
+`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
+sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
+that is all of them. **A missing gate is a FAILURE, never a skip.**
+
+**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
+**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
+when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the session
+report when you use it** — CI re-runs the same entry point on every push and **emails the operator on
+failure**, so a bypass is noticed even though it is not blocked (R-168, CLOSED 2026-08-02).
+
+<!--
+WHY ONE ENTRY POINT (2026-08-02, R-29): a census of all gates across the four repos found every check
+a CLAUDE.md names was passing, and two of the four nobody is told to run were failing. This repo was
+the extreme case — nothing ran against it at all, and 90 cited paths were checked by no one.
+-->
+
+## Live validation — the fence
+
+Exercise the **SERVER-SIDE PIPELINE** a real user triggers, end-to-end. **The forbidden shortcut is
+BYPASSING it** — the F9 episode was a raw guest-attach with hand-set state, and it proved nothing.
+
+`claude-in-chrome` is NOT available on DooPlex. Invoking the exact endpoint the UI invokes is an
+acceptable proxy — **say which method was used**. Low-level mechanism tests where the direct call IS
+the mechanism are exempt.
+
+## Conventions
+
+- **Trunk-based — no branches.** All shippable work commits directly to `main`; `main` equals what is
+  deployed. Report-only artifacts (audits, findings, fixspecs) go to `felhom.eu/documentation/`.
+- **Unattended escape hatch:** if a fix cannot be cleanly verified and shipped, **revert and report**
+  — never park it on a branch.
+- **Logging**: the slog logger fans out to journald (configured level) plus the always-DEBUG
+  `applog.Ring` (remote pulls). English, keys-never-values, durations on outcomes. Full rules:
+  `felhom.eu/documentation/runbooks/logging-conventions.md`.
+- Update `REUSE.md` in the same commit that adds, changes or deprecates a shared helper or pattern.
+
+## End-of-session checklist
+
+- **`CHANGELOG.md`** (cumulative, newest on top) and **`REPORT.md`** (overwritten with this run only)
+  — in every repo touched.
+- **`CONTEXT.md`** — decisions, state, what is next.
+- **`REUSE.md`** — if a shared helper or pattern moved.
+- **A finding goes in `felhom.eu/documentation/backlog/OPEN-ITEMS.md` first**, never only in a report
+  or an audit.
+- **Confirm your own last push's CI run went green, by run ID** — CI mails on failure, which is a PUSH
+  signal; this is the PULL check that catches a lost or unread mail. An unchecked green is an
+  assumption, not an observation.
+
+**A gate ships with a decoy test that has been seen to fail (R-421).** A decoy is the LABEL without
+the FACT — a directory with the right name and no bake log, a note whose prose mentions the marker it
+lacks. `scripts/decoy_coverage_gate.py` refuses a new gate that has neither a decoy nor a named
+exemption carrying its row. The four shapes, the 2026-09-01 sweep that fooled 16 of 29 gates, and the
+decoys withdrawn as illegitimate: `documentation/audits/AUDIT-gate-decoys-2026-09-01.md` and
+`felhom-controller/.claude/rules/gates.md`. **Scope is a fact too** — prefer `os.walk` over
+`os.listdir`, and a glob over a hand-maintained list.

CONTEXT.md

@@ -0,0 +1,521 @@
+# CONTEXT — felhom-agent working state
+
+
+> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
+> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
+> `spaceFits` — a vzdump to a LOCAL target needs free ≥ newest archive of that guest × 1.25 + 1 GiB (PVE prunes
+> only after success); a shortfall is a named skip (`BackupSkipNoSpacePrefix`), fail-open on PBS / first backup /
+> unknown usage. **Free space comes from `NodeStorage` (`GET /nodes/<n>/storage`) — `ListStorage` (`GET /storage`)
+> has NO usage**; the first build read it and a real vzdump started in its own live test (aborted, no archive).
+> demo-hp: `local_backup_retention` 1 (operator option A, saved `agent.json.pre-a4-retention`). Peti's box: nothing.
+
+> **2026-09-24 — v0.133.0 RELEASED, NOT DELIVERED (R-672, R-673).** Restore-test space preflight
+> (`reconcile/restoretest_space.go`, `internal/restorespace`): uncompressed size from the vzdump log / PBS size,
+> × 1.2 + 5 GiB, thin metadata, off the tested guest's pool, unknown refuses, reported as `skipped` non-pass.
+> Janitor (`cmd/felhom-agent/janitor.go`) every 10 min: `Engine.RetryScratchTeardown` + stale-lock sweep under
+> the heavy-op gate. Thin pool ≥ 90 % → immediate report (hub v0.124.0 alarms). **Operator rulings 2026-09-24
+> (evening):** the scheduled restore-test is OFF on both demo hosts (`backup.restore_test_eval_interval_seconds:
+> -1` — note: 0 means the 6 h DEFAULT, only a negative disables) until v0.133.0 is delivered there; saved configs
+> `/etc/felhom-agent/agent.json.pre-r672`. demo-hp 9201 was repaired (stop, fsck, start; two Redis AOF tails cut).
+> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
+> entry = current); the end-of-task detail lives in `REPORT.md`.
+
+## R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked
+
+`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
+customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
+(`hub.Client.FetchIdentityEscrow` → `GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
+per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
+repository password plus its sha256.
+
+**Rules that must not erode:**
+- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
+  trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
+- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
+  no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
+- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
+  never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
+  log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
+- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
+  the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
+  at age's KDF, nothing written).
+- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
+  `main` → `runDaemon` → `buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
+  `Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
+  self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
+  Links 6 and 7 were two of this project's six built-but-never-wired instances.
+
+**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
+A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
+recovered password, reopens a repository, or restores a file.
+
+**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
+"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
+into the same bundle — anyone reading it would conclude the password was not there. It now names what
+THIS bundle carried and what it did not.
+
+## Current
+
+- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
+  `FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
+  storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
+  through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
+  archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
+  restore-testable and nothing said so.
+  - **The permission question is asked directly**, because unlike the listing it has a definite
+    answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
+    own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
+    composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
+    belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
+    degraded alert applies with **no hub change**.
+  - **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
+    403 — it carries the privileges INHERITED from the box-wide `/` grant
+    (`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
+    blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
+    changing that constant (`storeGrantRequiredPriv`, red-proved).
+  - **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
+    defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
+    does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
+    ok when it could not ask.
+  - **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
+    grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
+    and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
+    correctly picking the 08-02 archive (08-03 had not settled 24 h).
+  - **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
+    grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.
+
+- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
+  None touches data; all three cost attention, which every other signal depends on.
+  - **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
+    from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
+    timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
+    a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
+    twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
+    still read, and a record missing archive-or-tier is NOT reported), exposes
+    `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
+    newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
+    twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
+    "host-report gauge" in its doc comment and had **no caller** for weeks.
+  - **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
+    future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
+    tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
+  - **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
+    locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
+    made the gate correctly fail a correct release ~half the time. The old order's invariant is now
+    asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
+    a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
+  - **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
+    bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
+    `CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
+    identical. The verification command is in `CLAUDE.md`.
+
+- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
+  now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
+  `settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
+  weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
+  literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
+  tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
+  where it matters most. Red-proved at 0 runs over 5 simulated days.
+  - **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
+    keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
+    upgrade, deliberately.
+  - **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
+    (verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
+    naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
+    (default 24 h). It is NOT carried into the evaluation interval.
+  - **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
+    both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
+    evaluation interval is also its retry interval for a multi-GB restore.
+  - The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
+    able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
+    failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
+  - New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
+  - **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
+    DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
+    a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
+    evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
+    with its error in that same line.
+  - **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
+    encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
+    named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
+  - **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
+    `/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
+    host tier has never been restore-testable there, and the due-check cannot distinguish that from
+    a newborn tier.
+
+- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
+  New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
+  stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
+  `RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
+  lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
+  down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
+  invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
+  guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
+  instead of keeping a second private definition of "should be running". Retry bounded at 3
+  (1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
+  Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
+  proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
+
+- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
+  structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
+  restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
+  **(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
+  VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
+  **(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
+  PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
+  `/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
+  counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
+  `Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
+  `pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
+  (`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
+  consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
+  privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
+  binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
+  PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
+
+- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
+  self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
+  its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
+  F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
+  `sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
+  ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
+
+- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
+  `NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
+  backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
+  the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
+  deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
+  measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
+  **Size is the only tier-agnostic discriminator** — `verification` and `encrypted` are absent on
+  every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
+  either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
+  smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
+  test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
+  demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
+  observable. Deployed on both boxes. Detail: `REPORT.md`.
+  **Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
+  2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
+  accumulate. Filed as R-99 (LOW).
+
+- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
+  drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
+  `/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
+  `/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
+  **SMART-only** device from its containing filesystem (never touches backing/durable_id — the
+  removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
+  watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
+  `felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
+- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
+  `Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
+  enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
+  controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
+  ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
+  so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
+- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
+  Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
+  (`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
+  clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
+  ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
+  **The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
+  static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
+
+- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
+  felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
+  host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
+  `bind: cannot assign requested address` — the daemon exits ~1.1 s after start, systemd gave up after
+  4 retries, and a manual restart reproduces it exactly. Deployed binary is **0.90.0**. Fix needs
+  `listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest bootstrap endpoint (plus the
+  pinned leaf's SAN) → **Viktor GO**; re-pinning to another literal just re-breaks on the next lease.
+  Also re-observed each start: `pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw:
+  permission denied` (R-39-adjacent). Evidence + ranked findings:
+  `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
+
+- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
+  demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
+  v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)** —
+  self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
+  every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and
+  applies via PVE `SetConfig` — **live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
+  break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
+  after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
+  memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
+  (esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
+  trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
+  Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
+  `reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
+  (3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
+  pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
+  hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
+  agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.
+
+- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
+  Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)** —
+  closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
+  the root wrapper + re-reads instead of aborting before the grant (the demo's `felhom-offsite`
+  case). (2) **escrow config live-reload** — `/escrow/preflight`'s `pbs_storage_id` row now reads
+  the live agent.json (late-bound `CurrentPBSStorageID`) so a pbsdr-seeded id flips green with no
+  restart. (3) **agent-plane poke listener (Direction-2a)** — `internal/poke`: contentless UDP poke
+  bound to the box WG /32 (port **51822**), leading-edge debounced, fires the hub-loop out-of-band
+  trigger for an immediate desired-state cycle; enabled with `wg_tunnel.enabled`; first slice of
+  R-13. Red-proofs for all three (run-fail-revert). **ALL THREE LIVE LEGS PROVEN on the demo
+  (2026-07-17), demo now LIVE on 0.89.0:** Scenario 4 floor-driven A/B train 0.88→0.89 (operator
+  signed+enqueued the `agent_update` op — the vouch+floor alone does NOT trigger it; committed, no
+  rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → `pre-check 403 … self-granting
+  (R-22)` → `converged state=adopted` in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick
+  ~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.
+
+- **v0.88.0** (2026-07-13 eve) — **controller-driven escrow ceremony (agent half), LIVE on demo
+  host + drill VM (63/63 capabilities both).** `--output=json` machine mode (text mode
+  byte-identical; extraction into `escrowCeremony()`); the ONE fixed argv
+  (`escrow.CeremonyArgs()` — shared by the localapi exec + the `escrow-ceremony` capability
+  (Critical, pbs_dr-gated EXPLICIT) + the new `FELHOM_ESCROW` sudoers alias, three-way
+  pin-tested); localapi job endpoints (`POST /escrow/ceremony` single-flight 60 s,
+  status, ONE-SHOT claim → 410, 10-min TTL → `unclaimed_void`, `GET /escrow/preflight`).
+  R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely).
+  Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW
+  list-probe) → job ~4 s → hub blob `restic_pw_sha256` covering (repaired the spike's hash-less
+  blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard).
+  **OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got
+  direct deploys.** Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).
+
+- **v0.87.0** (2026-07-13) — **SystemDisks device-mapper walk (IA finding 2, MEDIUM):
+  legacy-boot hosts get a working drive wizard.** Operator ruling (approved 2026-07-13,
+  verbatim): *resolve device-mapper/raid parents — for the root filesystem's backing block
+  device, walk `/sys/block/<dev>/slaves` recursively down to physical disks; those, plus any
+  ESP holder when present, are system. Disks outside that set become wizard candidates (still
+  subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk
+  failure — it returns to being the error case, not the legacy-boot common case.* Implemented as
+  `physicalDisksOf`/`walkSlaves` + `HostReader.BlockSlaves` (one seam method); per-branch
+  conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test
+  `TestSystemDisks_WalkTopologies` (root-backing disk ALWAYS system — never weaken). §3 spike
+  transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda →
+  byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted →
+  torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also
+  surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4
+  502-through-CF + native-alert ban).
+
+- **v0.83.0** (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — **observability
+  pass** (pairs with controller v0.116.1 + hub v0.46.0). `applog.New` → `(logger, *Ring)`: slog
+  fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. `GET /debug/logs`
+  (local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG
+  middleware. Heartbeat log-pull: envelope `log_tail_requested` → next heartbeat ships `log_tail`
+  (128 KB, consume-once; failed push re-armed by the next envelope; `operator log pull served` INFO
+  on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome,
+  /proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount
+  unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration,
+  selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull +
+  health verdicts, desired/loop per-exchange DEBUG. Logging conventions:
+  `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: the hub-side live pull awaits
+  the operator's button click (hub UI password-gated); pre-existing lanresolver
+  permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.
+
+- **v0.77.0** (2026-07-09) — **fork-4: escrow the offsite restic repo password under R.** `IdentityBundle`
+  gains `ResticRepoPassword` (rides the existing age-under-R `WrapIdentityBundle` path — validated by the
+  custody spike `febdc56`). New `POST /escrow/stage-secret` (`withGuest`) transiently stages the
+  controller-pushed password (0600, never logged), which the `--selftest=escrow-create` ceremony
+  auto-injects into the bundle and then wipes. `AttachResticPassword`/`StagedResticPasswordPath`/
+  `WipeStagedResticPassword` added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject +
+  `DRResticCoord`). **NOT yet live-validated** — the supervised escrow ceremony is operator-run.
+
+- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
+  manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
+  full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
+  DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
+  all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
+  (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
+  `MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
+  parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
+  rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.
+
+- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
+  actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
+  COMPLETE explicit restore param set derived from the archive's embedded config (NEW
+  `Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
+  params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
+  without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
+  swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
+  seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
+  archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
+  overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
+  before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
+  while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
+  supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.
+
+- **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
+  bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
+  standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0
+  is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record
+  `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`). **Day-0 manifest vouch = operator
+  step** (agent 0.74.0 / golden 0.103.0).
+
+- **v0.74.0** (2026-07-07) — **campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom** (LIVE on
+  felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; `pct restore --pool`
+  sets it only at CREATE, so a restore-over-existing dropped 9201 from the `felhom` pool → no
+  `VM.Audit` → restore-test's *existing* `bindMountOverrides` never ran → "mp8 … only possible for
+  root". Fix: `Client.PoolAddVMID` + bring-up re-asserts membership post-restore (warn-not-fail).
+  Role/ACL + `bindMountOverrides` untouched (both correct). **Live restore-test PASSED for the first
+  time** once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore →
+  boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code.
+  OPEN: DR `bring-up -mode dr` bind-override gap (spike `SPIKE-dr-bindmount-source-2026-07-07.md`:
+  small known-constant override reusing `bindMountOverrides`; mp8/mp9 are structural constants).
+
+- **v0.73.0** (2026-07-06) — **F2 mount-role fallback CLOSED** (LIVE on felhom-pve). `roleForMountPath`
+  gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE
+  storage, so it fail-safe'd to `system` and the eject/decommission gates 403'd EVERY user-data drive
+  (campaign F2, `where=/mnt/teszt_enroll role=system`). Device-keyed classification + whole-disk containment
+  (`storage.SameWholeDisk`); Observe-error keeps the fail-safe BEFORE the fallback. Only `roleForMountPath`
+  touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart,
+  end==pre). OPEN follow-up: the `deviceRole`/`roleForMountPath` unification refactor (deferred).
+
+- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
+  `SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
+  AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
+  render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
+  `inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
+  RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
+  grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
+  felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
+  path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
+- **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
+  health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
+  Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make
+  `/run/sshd` boot-persistent AND auto-heal it every ~60s **agent-independently** (heals with the
+  agent stopped — proven live: `/run/sshd` removed → restored in 30.0s, `:22` back, no login).
+  `internal/mgmtplane` reports the additive `mgmt_plane` heartbeat stanza; hub v0.34.1 raises
+  `mgmt_plane_healed`. **NO unit declares `RuntimeDirectory=`** (the incident cause). H1 may now
+  assume `/run/sshd` is guaranteed present. Live on felhom-pve; rollback `.bak-0.70.0`.
+- **v0.70.0** (2026-07-05) — **agent self-update (operator-signed A/B slots + crash-loop
+  auto-rollback)** — TASK D1, provenance `SPIKE-agent-selfupdate-2026-07-05`. An operator-signed
+  `agent_update` op (version+sha256, sha is the only integrity root) rides the signed-jobs gate;
+  `internal/selfupdate.Executor` downloads+verifies+hands to `felhom-selfupdate-guarded apply` (root
+  re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a
+  60s dwell; a crash-looping binary is auto-reverted by `OnFailure=felhom-agent-rollback.service`
+  (first-crash trigger [SF-1]) with the tuned `[Unit]` start-limit (120s/4) as backstop. Host
+  artifacts + sudoers `FELHOM_SELFUPDATE` + `felhom-host-install.sh` day-0 install + report field
+  `selfupdate_pending`. Green tests + companions. **LIVE-VALIDATED on felhom-pve (2026-07-05): all 4
+  drills PASS** — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback
+  (0.70.2-crash → OnFailure → **~2s crash-to-recovered**, byte-identical revert, no loop), no-pending
+  guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the
+  hub `signed_jobs` queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested).
+  Box restored to canonical **v0.70.0** (host artifacts KEPT installed; scratch operator key REMOVED —
+  self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback
+  `felhom-agent.bak-0.69.0`. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update
+  auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent —
+  future: serialize the rollback oneshot). Detail: REPORT.md.
+
+- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
+  **Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
+  refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
+  wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
+  fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
+  logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
+  sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
+  no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
+  mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
+  on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
+  (`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
+  near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
+  — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
+  confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
+  OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
+  felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
+- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
+  **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
+  `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
+  `SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
+  killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
+  PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
+  offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
+  vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
+  not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
+  VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
+  OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
+  Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
+- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
+  wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
+  lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
+  loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
+  + FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
+  **namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
+  unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
+  per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
+  `demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
+  token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
+  **Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
+  (NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
+  DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
+  `backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
+  offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
+  a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
+  escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
+  `felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
+- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
+  resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
+  TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
+  (and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
+  outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
+  **Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
+  endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
+  (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
+  apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
+  true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
+  Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
+  optional future note — NOT needed for MTU correctness.
+- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
+  (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
+  the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
+  revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
+  `IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
+  gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
+  on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
+  10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
+  hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
+  on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
+  grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
+  publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
+- **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
+  findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
+  argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the
+  guide's manual D.1b update) and the golden now bakes a `felhom-controller-bootstrap.path` unit
+  (controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer
+  v1.9.1's reboot is a redundant belt, kept). **Golden 0.98.3** baked on the drill VM, clean-room
+  validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact
+  Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches
+  **agent 0.63.0 + golden 0.98.3** (the v0.63.0 vouch follow-up below is DONE). Fresh installs land
+  current and self-manage. NEW operator follow-up (SECURITY): the customer-config `git.token` has
+  Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
+- **v0.63.0** (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — **drill findings
+  B3 + B2 FIXED** (`DRILL-day0-cleanroom-2026-07-03.md`): `TokenStore.Lookup` reloads the append-only
+  store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install
+  `/controller/swap` 401 / manual restart; size short-circuit bounds the cost; behind the
+  `TokenAuthority` seam) + `guesthook.InstallSnippet` issues a fenced `mkdir -p /var/lib/vz/snippets`
+  first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained
+  exactly that one grant — **ship sudoers WITH the binary** (done on felhom-pve). Red-proofed both;
+  Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path.
+  **OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0** — until then fresh installs get
+  0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
+- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
+  pool-intersected (`staleLockController.Guests()` = `ListLXC` ∩ `Client.Pool("felhom")` members),
+  fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
+  "pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest` —
+  **deploy order on any box: rescope ACL first, then this agent.** Per
+  `SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
+- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
+- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
+- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
+  `felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`: random temp
+  staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO
+  re-checks (validated by `scripts/mkfs-guarded-harness.sh`), classifyClaim empty-lsblk fail-safe,
+  and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
+- Deployed on demo host `felhom-pve` (node `demo-felhom`), non-root `felhom-agent` service user,
+  pool-scoped token (`felhom` pool).
+
+## Open threads
+
+- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
+  server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
+- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
+  shared-box operator format gate.
+- BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
+- Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted
+  member; a live /dev re-enumeration race during a real format).

PLAN.md

@@ -0,0 +1,136 @@
+# Slice 6 Phase A — backup + restore-test orchestration (felhom-agent)
+
+## Context
+
+The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a
+signed-op gate (slice 4), but it has no guest-level **backup/restore** layer and — the point
+of this slice — no **self-restore-test**, which closes the "a backup you haven't restored
+isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the
+produced archive, a benign restore-to-a-**new** guest, and a journaled **self-restore-test**
+(restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash
+recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden
+base (§9) are **slice 7**; PBS/offsite/zero-knowledge (§8 offsite tier) are **Phase B**.
+
+Everything here is **benign** (backup, restore-to-new, scratch teardown): reuse the slice-4
+classifier/gate/journal — **no new destructive class, no new crypto**. Version → **v0.6.0-rc1**,
+stop at the checkpoint.
+
+**Locked decisions (from the task + clarifications):**
+- Backups are **crash-consistent only** (marked so in the report); app-consistency needs the
+  controller quiesce (slice 8).
+- Restore is **to a NEW guest only** (no overwrite anywhere this slice).
+- Restore-test verify = **net link-down (benign SetConfig) → boot → reaches `running`** (no
+  in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live
+  host; it is test-safety, **not** slice-7 identity reset.
+- Restore-test cadence **defaults ON at 24h**, configurable; runs only when a valid scratch
+  VMID band is set; also on-demand via the selftest harness.
+- Scratch VMID band **990000–990009** (lowest-free, excludes 9999/real guests); refuse to run
+  if unset/invalid.
+- **Local target only** this phase; `PBSSnapshot` stays a `struct{}` stub (Phase B).
+- Bulk volumes: **report the gap only** (which `backup=0` mountpoints the guest vzdump omits).
+
+## The crash-safe core (load-bearing)
+
+The restore-test is journaled as a **single** entry whose `VMID` is the scratch id and
+`Kind = "scratch_restore_test"` with a new `Scratch bool` flag. The entry is **terminal only
+after teardown** — NOT when the restore sub-task's UPID completes. `Recover` must special-case
+Scratch entries **before** the generic UPID-recheck path (else it would mark the entry
+`succeeded` because the restore task ran OK, and drop it while the guest still exists → leak).
+
+`RunRestoreTest` (pseudocode) — `defer` teardown so it runs on every path incl. failed verify:
+```
+vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999)        // refuse if band unset/invalid
+opID := "scratch-restore-"+vmid+"-"+seq
+append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted})  // BEFORE any mutation
+defer teardownScratch(opID, vmid)   // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal
+upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid)   // UPID = error-detection, NOT terminal
+for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"})  // benign
+Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout)
+res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err}
+return res    // defer tears down regardless
+```
+
+`Recover` scratch branch (added before the existing UPID logic):
+```
+if entry.Scratch {
+    guests, err := ListLXC; if err { Unresolved++; continue }              // can't decide -> leave in-flight
+    if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue }  // already gone
+    dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil)
+    if !dec.Allowed { Unresolved++; continue }                             // fail-safe (should be benign)
+    upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover
+    waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++
+}
+```
+Idempotent: crash mid-destroy → next `Recover` finds the guest gone → `ScratchClean`. Both
+teardown paths (normal + recovery) go through `gate.Authorize` (benign `ClassGuestDestroy` +
+`Provenance{AgentTaggedScratch:true}`) for the audit trail.
+
+## Files to create
+
+- **`internal/reconcile/restoretest.go`** (+ `restoretest_test.go`): `RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult` (engine method — it needs the journal/gate/queue internals), `RestoreTestResult` (reconcile-local data — must NOT return `hub.RestoreTest`, to avoid a reconcile→hub edge), `IntentForScratchDestroy(hostID, vmid)`, `pickScratchVMID`. Runs the sequence on the scratch VMID's `Queue` lane.
+- **`internal/backup/{doc,runner,store,schedule}.go`** (+ tests):
+  - `runner.go`: `BackupRunner.Backup(ctx, vmid) (hub.Backup, error)` = `Vzdump`+`WaitTask`+volid-resolve + bulk-gap from `GuestConfig.MountPoints()` `backup=0`; `LatestBackup(ctx, target) (volid, error)`; the `RestoreTestRunner` seam (satisfied by `*reconcile.Engine`); `RestoreTestResult → hub.RestoreTest` mapping.
+  - `store.go`: mutex-guarded in-memory latest-`Backup`-per-target + latest-`RestoreTest`, implementing the hub `BackupReporter`/`RestoreTestReporter` seams.
+  - `schedule.go`: cadence goroutine (default 24h; 0=disabled) → `LatestBackup` → `engine.RunRestoreTest` → write `Store`. No-ops cleanly when no backup exists yet.
+  - Imports `reconcile`+`hub`+`proxmox` (acyclic; hub imports neither).
+- **`configs/`**: example agent config with the `backup` block.
+
+## Files to modify
+
+- **`internal/proxmox/mutate.go`**: `DestroyLXC(ctx, vmid)` → `DELETE /nodes/{node}/lxc/{vmid}` with `purge=1&destroy-unreferenced-disks=1` via `dataString` (async→UPID); add `Notes string` to `VzdumpOptions` → `notes-template` param (probe `notes-template` vs `notes` on the demo PVE 9.2.2 first, read-only).
+- **`internal/proxmox/query.go`**: `LatestBackupVolID(ctx, store, vmid) (string, error)` — `StorageContent` filtered `Content=="backup" && VMID==vmid`, max `CTime`.
+- **`internal/reconcile/state.go`**: add `RestoreLXC`, `DestroyLXC`, `GuestStatus` to the `GuestAPI` interface (`*proxmox.Client` already satisfies all three).
+- **`internal/reconcile/journal.go`**: add `Scratch bool \`json:"scratch,omitempty"\``; carry it (and `Kind`) through `terminal()`.
+- **`internal/reconcile/recover.go`**: the scratch branch above; add `ScratchClean`/`ScratchDestroyed` to `RecoverResult`.
+- **`internal/reconcile/recover_test.go`** + **`engine_test.go`**: `fakeAPI` gains `RestoreLXC`/`DestroyLXC`/`GuestStatus` recorders; new `TestRecover_LeakedScratchDestroyed` (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), `…AlreadyGone` (ScratchClean), `…ListUnreadable` (Unresolved).
+- **`internal/hub/report.go`**: fill `Backup` + `RestoreTest` structs (see wire shapes); `PBSSnapshot` stays `struct{}`.
+- **`internal/hub/collect.go`**: add `BackupReporter { Backups(ctx) []Backup }` + `RestoreTestReporter { RestoreTests(ctx) []RestoreTest }` consumer seams (mirror `StorageObserver`); `collectBackups`/`collectRestoreTests` degrade nil/err → non-nil empty.
+- **`internal/hub/contract_test.go`** + **`internal/hub/testdata/host-report.golden.json`**: populated `backups[0]`/`restore_tests[0]`; assert their key sets (bidirectional, slice-5 pattern).
+- **`internal/config/config.go`**: `BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage}` + `RestoreTestCadence()` accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. `Config.Validate` stays proxmox-only.
+- **`cmd/felhom-agent/main.go`**: wire `backup.Store` into `NewCollector`; add the cadence goroutine alongside `engine.Run`/`loop.Run`/`watchdog.Run`; add `backup` + `restore-test` to `selftestFlag.Set` + the switch + a `-archive` flag; `runSelftestBackup`/`runSelftestRestoreTest` reuse the `runSelftestStorage` wiring (`NewGate(nil, hostID, SlogAudit{}, logger)` + journal + engine).
+- **felhom.eu/hub**: `hub/internal/api/handler.go` — add `hostBackup`/`hostRestoreTest` mirror structs to `hostReportPayload`, parse, persist via existing `report_json` (no new DB columns — slice-5 precedent), and **log a FAILED restore-test prominently** (`[WARN]`, the loudest DR signal). `hub/internal/api/testdata/host-report.golden.json` byte-identical with the agent golden; `host_test.go` adds `TestHostBackup_GoldenContract`/`TestHostRestoreTest_GoldenContract` (bidirectional key-set, slice-5 pattern).
+- **`CHANGELOG.md`** (prepend v0.6.0-rc1), **`REPORT.md`** (overwrite), `CLAUDE.md` current-state line.
+
+## Proposed wire shapes (draft — must land byte-identical in both repos)
+
+```go
+type Backup struct {
+    TargetID         string   `json:"target_id"`          // backup storage name
+    VMID             int      `json:"vmid"`
+    Archive          string   `json:"archive"`            // produced volid
+    Mode             string   `json:"mode"`               // snapshot|stop
+    CrashConsistent  bool     `json:"crash_consistent"`   // always true this slice
+    SizeBytes        int64    `json:"size_bytes"`
+    Success          bool     `json:"success"`
+    Error            string   `json:"error,omitempty"`
+    StartedAt        string   `json:"started_at"`         // RFC3339
+    DurationSeconds  float64  `json:"duration_seconds"`
+    UncoveredVolumes []string `json:"uncovered_volumes"`  // backup=0 mountpoints (bulk gap)
+}
+type RestoreTest struct {
+    SourceArchive   string  `json:"source_archive"`
+    SourceTier      string  `json:"source_tier"`     // "local" (pbs = Phase B)
+    ScratchVMID     int     `json:"scratch_vmid"`
+    Pass            bool    `json:"pass"`
+    Verified        string  `json:"verified"`        // "boot+running"
+    Error           string  `json:"error,omitempty"`
+    TestedAt        string  `json:"tested_at"`       // RFC3339
+    DurationSeconds float64 `json:"duration_seconds"`
+}
+type PBSSnapshot struct{} // Phase B stub
+```
+
+## Verification
+
+- `go test ./...` (local, Windows) + `go test -race ./...` on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded.
+- New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign `ClassCreate` passes the gate; restore-test end-to-end against fakes incl. **(a) teardown-on-failed-verify** and **(b) journal-recovery cleanup** (extend the slice-4 recover test: in-flight Scratch → `Recover` destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests.
+- Build the linux binary on 192.168.0.180, relay to `felhom-pve` (see memory `demo-felhom-live-agent`: `MSYS_NO_PATHCONV=1`, config at `/root/.config/felhom-agent/agent.json`, mode=direct, served-cert pin `BA:7C:99…`).
+- **Live on the demo** (the checkpoint validation): `--selftest=backup -vmid <small stopped guest>` to a local target (e.g. `felhom-usb` or `local`, content=backup) → print the `Backup` record; then `--selftest=restore-test -archive <volid>` → restore into a 990000-band scratch guest, net-link-down, boot, verify `running`, teardown → print the `RestoreTest` record. Confirm no leaked scratch guest remains (`pct list`), and that a simulated mid-test crash + restart triggers `Recover` teardown.
+
+## Push & checkpoint
+
+Push **v0.6.0-rc1** to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps
+runbook if needed), update CHANGELOG/REPORT, then **stop and await validation** (restore-test
+teardown + recovery, the benign classifications, reporting) + the live demo restore-test.
+**Phase B (next): PBS** — datastore on the USB, zero-knowledge key custody, restore-from-PBS,
+PBS integrity-verify as the lighter frequent check.

README.md

@@ -0,0 +1,350 @@
+# felhom-agent
+
+The **host agent** for the Felhom platform — the operator-tier component that runs on each
+Proxmox host and owns *all* Proxmox interaction (provision/restore guests, host storage,
+backups, host+tunnel monitoring, hub control loop, per-guest local API). Design:
+[`felhom.eu/documentation/architecture/03-host-agent.md`](https://gitea.dooplex.hu/admin/felhom.eu/raw/branch/main/documentation/architecture/03-host-agent.md).
+
+> **Status — slice 1 of N.** This repo currently contains the project scaffold and the
+> **`internal/proxmox`** interaction layer (the typed library every other module will call to
+> talk to Proxmox), plus a runnable read-only `--selftest`. **No** reconcile loop, hub client,
+> signing, or storage/backup orchestration yet — those are later slices.
+
+Module: `gitea.dooplex.hu/admin/felhom-agent` · binary: `felhom-agent` · Go 1.24.
+
+## Layout
+
+```
+cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
+internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
+internal/config/       # JSON config + env overrides (secrets never logged)
+internal/log/          # slog setup
+internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
+internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
+internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
+internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
+internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
+configs/agent.example.json
+```
+
+## The `capability` package — privileged self-check (v0.44.0)
+
+The non-root agent depends on a fixed set of `sudo -n` grants (`configs/felhom-agent.sudoers`). When
+the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke
+unnoticed until a user hit them. `internal/capability` makes that loud:
+
+- **`Manifest()` is the source of truth** for the required `(binary, representative-arg)` vectors,
+  seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When
+  you add a privileged `runner.Run` call that needs a NEW grant, add it to BOTH the sudoers file AND
+  the manifest — `manifest_test.go` asserts **manifest ⊆ sudoers** at build time (with a red-proof
+  that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
+- **`Prober.Probe`** lists each vector with `sudo -n -l` (a policy LIST — never executes, safe for
+  mkfs/pct) + an `os.Stat` existence check → an `ok`/`degraded` snapshot. It runs at startup (logged
+  loud) and on every collect, riding the hub report as `HostReport.Capabilities`; the hub
+  (`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
+  — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
+
+**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
+(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
+**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
+`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
+grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
+target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
+dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
+
+## The `storage` package — observe + watchdog (slice 5)
+
+Read-only this slice (no hub desired-state until slice 10):
+
+- **Observer** builds the host-report's `storage_targets` from `ListStorage`/`NodeStorage`
+  joined with non-privileged host reads (`/proc/mounts`, `/dev/disk/by-uuid`,
+  `/sys/.../rotational`). It reports each target's `durable_id` (the DR-load-bearing
+  re-attach key: fs-UUID for usb/local-dir, `server:export` for nfs/cifs,
+  `repo+fingerprint` for pbs, `vg/pool` for lvmthin), state, usage, a rotational
+  **class hint** (never authoritative — class is hub-owned), and the lvmthin **thin-pool
+  data fill** (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
+- **Watchdog** is the third daemon goroutine: a fast poll (seconds) over the *known*
+  target set that detects an `attached↔disconnected` transition and fires a **debounced,
+  out-of-band** host-report so the hub learns of a USB drop in seconds rather than at the
+  ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in
+  Phase B). The `HostReader` seam keeps it root-free and unit-testable with no real devices.
+
+The reported `StorageTarget` shape is a cross-repo contract duplicated in `felhom.eu/hub`;
+`internal/hub/testdata/host-report.golden.json` is byte-identical with the hub's copy and a
+bidirectional key-set test guards drift.
+
+### DR recipe — the storage/guest/PBS half (v0.38.0)
+
+The host-report carries an additive `dr_recipe` section (`internal/hub/dr_recipe.go`) — the agent half
+of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`). It is the non-secret
+**re-provision scaffolding** that complements escrow (keys) + PBS/restic (bytes): `guests[]` sizing,
+`drives[]` (user-data drives by `durable_id` → role → mount → intent), `pve_storage[]` (the
+`storage.cfg` defs), and `pbs` coordinates. Built by the pure `BuildDRRecipeHostHalf` from facts the
+report already collects — **no new reads**. **Boundary:** every field is an identifier/intent/size/
+coordinate — never a key/password/token/hash/`ENC:`; the PBS key + restic password stay in escrow.
+`recipe_version=1`, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with
+the controller's app half into one customer recipe.
+
+### The privileged `HostOps` surface (slice 5 Phase B)
+
+The write side — the one place the agent steps outside its Proxmox API token into OS-root —
+is isolated behind the `HostOps` seam (`hostops.go`): production `SudoHostOps` shells out via
+a narrow **sudoers allowlist** (`configs/felhom-agent.sudoers`) with **fixed argument vectors
+and no shell**; tests use a fake (no real root in the suite).
+
+- **Persistent mounts** are **systemd `.mount` units keyed by fs-UUID**
+  (`What=/dev/disk/by-uuid/<UUID>`, enabled so they survive reboot) — not raw fstab or a
+  transient `mount`. Benign re-mount is idempotent; **detach** (stop+disable) is destructive
+  and routes through the gate.
+- **Host-reboot remount re-resolution (v0.37.0):** at startup (before binding drives into the
+  guest) and on the periodic tick, `ReassertEnrolledMounts` re-asserts every enrolled `.mount`
+  unit that isn't currently mounted: it re-resolves the drive's `uuid:<fs-uuid>` durable id to
+  its **current** `/dev` node by re-scanning `/dev/disk/by-uuid` (`ResolveStorageDevice`, never a
+  cached node) and re-runs the idempotent `enable --now`. This re-enables a unit a prior detach
+  left `disabled` AND tolerates kernel re-enumeration moving a drive's letter (`/dev/sdb`→`sdc`)
+  — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
+- **Every argument is validated before any command is constructed** (`validate.go`): UUIDs
+  against a strict hex regex, mount paths confined + traversal-checked, SMART devices
+  whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in
+  `validate_test.go` proves a hostile UUID / path / device is refused with **zero** exec.
+- **SMART** (`smart.go`) fills `StorageTarget.smart` via `smartctl -a -j` — SATA *and* NVMe
+  attribute sets, degrading to `UNKNOWN` for devices that expose no SMART (e.g. a USB bridge).
+  **`lvs`** fills the lvmthin thin-pool **metadata** fill (metadata exhaustion corrupts a pool
+  like data exhaustion).
+- The **watchdog** gains a benign **re-mount response**: when a known mount-backed target's
+  device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed
+  through the gate as benign. The **disk-grow executor** (`pct resize`, **grow-only**) lands
+  in `internal/reconcile` as a benign action; **destructive storage ops** (detach/wipe/
+  data-losing-resize) construct a `ClassStorageWipe`/`ClassDecommission` intent bound to the
+  storage **target identity** and go through the slice-4 gate (built + tested, inert live).
+
+### `--selftest=storage` (live storage harness)
+
+Runs standalone on the Proxmox host (no hub needed):
+- bare: an **observe pass** printing the full `StorageTarget` table incl. the SMART summary
+  and thin-pool data+metadata fill.
+- `-watch <dur>` (e.g. `--selftest=storage -watch 3m`): runs the watchdog verbose for the
+  window with the **re-mount response live**, so an operator can physically cycle a drive and
+  watch detect → report → re-mount in the logs.
+
+## The `proxmox` package — model
+
+Two backends, one fixed routing policy (the fence is structural — `Client` never shells out,
+`Privileged` never makes an HTTP call; asserted in `routing_test.go`):
+
+| | Backend | Used for |
+|---|---|---|
+| **API (default)** | `proxmox.Client` | everything the scoped **FelhomAgent** token can do |
+| **root-CLI (fenced)** | `proxmox.Privileged` | the **three** proven OS-root exceptions only |
+
+Grounded entirely in the spike findings (`felhom.eu/documentation/proxmox-platform.md`,
+`tests/phase{0,1-2,3}-findings.md`). Every mutating API op is **async**: it returns a UPID and
+the caller `WaitTask`s until the task stops, then asserts `exitstatus == "OK"` — authorization
+can surface at task execution, not the HTTP POST (phase1-2 §1.3).
+
+### Public surface
+
+`Client` (API):
+
+- Read: `Version`, `Nodes`, `NodeStatus`, `ListLXC`, `GuestStatus`, `GuestConfig`,
+  `ListStorage`, `NodeStorage`, `StorageContent`.
+- Async mutating (return UPID): `RestoreLXC` (primary create path), `Vzdump`, `Snapshot`,
+  `Rollback`, `DeleteSnapshot`, `SetConfig`, `Start`, `Stop`.
+- Tasks: `WaitTask`, `TaskStatusOnce`, `TaskLogTail`.
+- Errors: `*APIError` (parses the offending privilege from a 403), `*TaskError` (parses it from
+  a failed task `exitstatus`).
+
+`Privileged` (fenced root-CLI) — each method documents *why it can't be the API*:
+
+- `CreateGoldenLXC` — `pct create` with `keyctl=1` (root@pam-only; the only root-fenced create —
+  the per-customer path provisions by **restore**, which preserves keyctl).
+- `MountUSBByUUID` — host mount-by-UUID (not a Proxmox API op).
+- `SMART`, `Sensors` — hardware reads (not API-exposed).
+
+### API-vs-root routing table
+
+See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary: the entire guest
+lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
+`keyctl` create, host mounts, and SMART/sensors (phase3 §B3).
+
+### Controller swap (agentic controller update, Phase 1 — v0.42.0)
+
+The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
+replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
+pre-pulls the target image then calls the agent:
+
+- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
+  (`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
+  write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll
+  the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
+  guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
+  - **(v0.47.0) Verify hardening (F1):** the health poll reads `{{.RestartCount}}` (a 4th inspect
+    field) — `running && RestartCount>0` is **not** healthy (a process that already crash-restarted
+    isn't stably up), and a **no-healthcheck** image must report ok on 3 **consecutive** polls (a
+    stability dwell) before acceptance; a real `healthy` result is trusted immediately. This closes the
+    hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass
+    the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the
+    `docker inspect -f *` grant (the `*` spans the extended template) are unchanged.
+- `GET /controller/swap/status` → `{state, current, previous, target, error}`.
+
+The agent is **external** to the controller container, so it survives the controller being killed
+mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
+seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.
+
+## Agent self-update (operator-signed, A/B slots, crash-loop auto-rollback — v0.70.0, TASK D1)
+
+The agent updates ITSELF the same way it swaps the controller: **the thing that performs rollback is
+never the thing being updated.** For the agent that means systemd + an ~80-line root shell wrapper
+(`configs/felhom-selfupdate-guarded`) that changes almost never; the Go binary is what flips.
+
+**Trust model.** An update is an **operator-signed `agent_update` op** delivered through the existing
+signed-jobs pipeline (same LOCKED authz gate as `storage_wipe`/`decommission`). The signed params pin
+the exact **version + sha256**, so the pinned sha is the *only* integrity root — **neither a
+compromised hub (dumb transport) nor a compromised Gitea (dumb storage) can substitute a binary.**
+The operator signs offline with `felhom-opsign -op agent_update -agent-version <v> -sha256 <hex>`.
+
+**The flow** (`internal/selfupdate/` = the Go half; the wrapper = the root half):
+
+1. The control loop sees a pending signed op → the gate verifies it (pinned-key SSHSIG → namespace →
+   allow-list → crypto → host → time → **durable nonce-burn**) → the `agent_update` executor runs.
+2. Executor downloads the binary for the signed version from the config'd artifact host
+   (`selfupdate.url_template`, `{version}` interpolated) to `/var/lib/felhom-agent/selfupdate/`,
+   verifies its sha256 against the **signed** value (mismatch → refuse, remove, agent untouched),
+   and hands it to `sudo -n felhom-selfupdate-guarded apply <staged> <sha>`. The job is completed on
+   the hub **after verify+download, before apply** (the nonce is already burned — a queued job would
+   only re-fetch and no-op on the spent nonce; a failed/rolled-back update is visible via the report).
+3. The wrapper (as root) **re-verifies** the sha, confines the staged path to the staging dir, asserts
+   same-filesystem (the atomic-rename guarantee), snapshots the current binary to `.prev`, atomically
+   `mv`s the new binary into place, writes a `pending.json` marker, `reset-failed`s, and schedules a
+   **detached** restart (`systemd-run --on-active=2s … systemctl restart felhom-agent`, so the caller
+   survives to log the handoff).
+4. The **new** binary boots; after it has run cleanly for a dwell (`selfupdate.dwell_seconds`, default
+   60) *and* core init is done, `internal/selfupdate.Manager` calls the wrapper's `commit` (clears the
+   marker; `.prev` retained as a manual net). A pending marker naming a *different* version than the
+   running binary is **not** committed — loud WARN, marker left so the report shows why (a human
+   decides).
+5. **Crash-loop auto-rollback (the safety property).** If the new binary crashes, systemd's
+   `OnFailure=felhom-agent-rollback.service` (the `felhom-agent-limits.conf` drop-in) runs the
+   wrapper's `rollback`: pending marker present → restore `.prev` byte-identical → clear marker →
+   restart → the old binary is back **within seconds of the first crash**. On systemd 257 `OnFailure=`
+   fires on *every* crash, so rollback triggers at the first one; the marker-guard makes every later
+   fire (and any crash with no update in flight) a harmless no-op. The tuned start-limit
+   (`[Unit] StartLimitIntervalSec=120 + StartLimitBurst=4`) is the terminal **backstop** (e.g. an
+   environmental crash loop of the known-good binary → terminal `failed` ≈20s → the hub's
+   `host_staleness` dead-man's-switch alerts the operator).
+
+**Design provenance:** every systemd behaviour above is empirically validated in
+`felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md` (the SF-findings). The host
+report carries `selfupdate_pending` (+ version) so a runs-but-never-commits binary is visible even
+though it never crashes. v1 scope: no hub-floor auto-update, no auto-retry of a failed update, no
+pending-timeout auto-rollback (a stuck-but-alive binary is caught by `host_staleness`).
+
+### TLS trust
+
+The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in
+config: `ca_file` (PEM, full verify), `fingerprint` (SHA-256 of the host leaf cert — pinned
+exact-cert match; the `/nodes` API returns each node's `ssl_fingerprint` to pin), or the
+explicitly-named `insecure_skip_verify` (off by default; selftest-against-127.0.0.1 only).
+
+## Provisioning the token (out-of-band, operator side)
+
+The agent only **consumes** a privilege-separated API token; role setup is a provisioning step.
+The role must be granted on **both the user AND the token** for the same path, or the
+intersection is empty and every call 403s (phase1-2 §1.2):
+
+```bash
+pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
+  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
+  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
+  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
+pveum user add felhom-agent@pve
+pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
+pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
+pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent
+```
+
+(`VM.Config.CPUMemory` is **not** a real privilege; `SDN.Use` **is** required for bridge use.)
+
+## Run
+
+```bash
+go build ./...
+# read-only health check against the host:
+./felhom-agent --config configs/agent.example.json --selftest
+# or via env (keeps the secret off disk):
+FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
+FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
+FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
+FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
+  ./felhom-agent --selftest
+```
+
+`--selftest` (read-only) loads config, builds the API client, and runs the read queries (version,
+nodes, node status, guests, storage), printing a short health report. It mutates nothing and says
+so cleanly if the token/endpoint isn't configured.
+
+`--selftest=task --vmid N` (explicitly gated) exercises `WaitTask` on a **reversible** op
+(snapshot → rollback → delete-snapshot) against guest `N`. Default `--selftest` never mutates.
+
+`--selftest=bring-up|provision` accept an optional operator **CPU/RAM cap**: `-cores N` and
+`-memory M` (MiB). Both default to `0` = keep the golden's baked sizes. When set, the cap is written
+into the SAME pre-start config PUT as the identity reset (via `BringUpSpec.Cores/MemoryMB`), so the
+guest never boots uncapped — useful when the appliance shares a host with other guests.
+
+**Pool-scoped restore (v0.53.0):** every restore (provision bring-up, DR, and restore-test) allocates
+the guest INTO the `felhom` PVE pool (`reconcile.DefaultPool`; `RestoreLXCOptions.Pool` → `pct restore
+--pool`). This is what lets the agent token be scoped to `/pool/felhom` + `/storage/<targets>` instead
+of `/` (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under
+that scoped token (`VM.Allocate`+`Pool.Allocate` at `/pool/felhom`). Layout + validation:
+`felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`.
+
+**Drive discovery + tracking (v0.55.0, Impl-2a):** `GET /disks/candidates` lists host disks the Impl-1
+filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced
+from the intent registry + Felhom `.mount` units (`RegistryKnownTargets`), NOT PVE storages — so a drive
+enrolled with no PVE dir-storage is still health-tracked (`Observe()` stays for real PVE storages). The
+controller wizard consuming this is Impl-2b.
+
+**Format safety (v0.54.0, Impl-1; hardened v0.61.0, audit D1/D2/D3):** `Format` (mkfs) is gated by a
+mandatory **unclaimed-disk guard** (`internal/storage/claim.go`) — it refuses any device not provably
+free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read
+error, and on an empty/target-absent lsblk topology — D2), independent of `DataBearing`. Below the
+agent, mkfs runs ONLY through `configs/felhom-mkfs-guarded.sh` (the sole mkfs the sudoers permits),
+which re-checks the catastrophic cases as root: system disk, LVM PV (absolute-path pvs), foreign
+mount, read-only device, and any LVM/ZFS/mdraid/LUKS/swap member signature (D1 — validated by
+`scripts/mkfs-guarded-harness.sh`, a loop-device + recorder harness). The blank-format local-API path
+binds to the device's durable id and anti-retarget re-resolves before mkfs, same as the confirmed
+wipe (D3, AGENT-001's benign-branch twin). The pool-scoped token does NOT touch mkfs (sudo op) — the
+filter + wrapper are the guard. See `SPIKE-drive-enrollment-2026-07-01.md` and
+`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`.
+
+## Process model
+
+Native Go binary, systemd service, **non-root** `felhom-agent` service user holding the scoped token,
+with a **narrow sudoers allowlist** for the fenced host-root ops. `privileged.mode: "sudo"` matches
+this; `"direct"` is for dev/CI where the agent is already root.
+
+The canonical artifacts (BUNDLE slice):
+- **`configs/felhom-agent.service`** — the canonical unit (`User=felhom-agent`,
+  `ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`). It deliberately sets
+  **no** `NoNewPrivileges` and **no** mount-namespacing hardening (`ProtectHome`/`PrivateTmp`/…): the
+  first would block the `sudo` the agent needs, the second would put the agent in a private mount
+  namespace so its `mount --bind` drive enrollments wouldn't propagate into the running guest. The
+  security boundary is the sudoers allowlist, not systemd sandboxing.
+- **`configs/felhom-agent.sudoers`** → `/etc/sudoers.d/felhom-agent` (0440, `visudo -cf`-validated).
+- **`scripts/publish-agent.sh`** publishes the binary to Gitea as a generic package
+  (`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`), printing the sha256 the operator
+  records in the hub artifact manifest.
+
+**Install is automated.** The host-bootstrap script (`felhom.eu/scripts/felhom-host-install.sh`) fetches
+the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user +
+binary + sudoers + unit + config — no manual agent install step.
+
+## Test
+
+```bash
+go vet ./... && go test ./...
+```
+
+Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, `WaitTask`
+(running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response
+decoding against the captured live shapes, and the API-vs-root routing fence.

REPORT-storage-split-spike.md

@@ -0,0 +1,90 @@
+# Findings — OS/Docker-data storage-split spike (Part B)
+
+**Date:** 2026-06-13 · **Host:** demo-felhom (PVE 9.2.2) · **Throwaway:** unprivileged LXC 9300 (created, exercised, destroyed) · **Builds nothing** — this report gates the provisioning spec.
+
+**Goal:** prove the mechanics of splitting the controller guest's **OS rootfs** from its **Docker/data** onto separate `local-lvm` volumes, and the live-migration path, before writing the provisioning spec. Target shape to validate: a small OS rootfs (~32 GB default) + a separate large Docker-data volume (~256 GB), sizes configurable at install.
+
+**Method:** all risky steps were run on a throwaway unprivileged LXC (9300) replicating guest 9201's config (`unprivileged: 1`, `features nesting=1,keyctl=1`, rootfs on `local-lvm`), never on 9201's live data. Docker 29.5.3 — **identical to production 9201** (verified: both `overlayfs` driver + `io.containerd.snapshotter.v1`).
+
+---
+
+## Verdict summary
+
+| Item | Verdict |
+|---|---|
+| B1 — second volume + Docker `data-root` on unprivileged LXC | ✅ **Works.** overlayfs on the ext4 `local-lvm` mountpoint; no idmap/ownership problem; survives reboot. |
+| B2 — live move-then-verify migration | ✅ **Works**, copy-not-move is the safety net. Caveat: rsync must exist + its exit code must be checked. |
+| B3 — vzdump/PBS coverage of the new mountpoint | ⚠️ **NOT by default — contradicts the spec's premise.** Extra CT mountpoints are **excluded** from vzdump unless `backup=1` is set **and the CT restarted**. |
+| B4 — live resize vs fresh-install carving | ✅ Live grow is online/non-disruptive (`pct resize`). Fresh-install carving = installer `hdsize`/`maxroot`/`maxvz` (host-level, documented below). |
+| B5 — provisioning seams | ✅ Mapped: golden rootfs size + `BringUpSpec.RootfsGrowGB`/`Mounts`; `bringup.go:313` is the exact spot that today omits `backup=1`. |
+
+---
+
+## B1 — Second volume + Docker `data-root` on an unprivileged LXC ✅
+
+- Added a second `local-lvm` mountpoint: `pct set 9300 -mp0 local-lvm:16,mp=/mnt/dockerdata`. PVE created `vm-9300-disk-1`, formatted it **ext4**, mounted it; in-container it is `root:root 0755` — **no idmap/ownership fix needed** (PVE maps the unprivileged offset for a fresh volume automatically).
+- Pointed Docker at it via `/etc/docker/daemon.json` `{"data-root":"/mnt/dockerdata"}` → after `systemctl restart docker`, `docker info` reports `Docker Root Dir: /mnt/dockerdata`, `Storage Driver: overlayfs`.
+- **overlay works on the mountpoint's filesystem** (ext4): pulled an image + created a named volume + wrote to both the volume and the container's overlay layer. Image/volume bytes landed on `/mnt/dockerdata` (228K → 64M) while the rootfs `/var/lib/docker` stayed flat (232K).
+- **Survives reboot:** after `pct reboot`, `data-root` is still `/mnt/dockerdata`, the volume's marker file persists, the image is present.
+
+**Gotcha (Docker 29 / containerd-snapshotter):** the daemon already auto-starts at boot, so `systemctl start docker` is a no-op and won't pick up a freshly-written `daemon.json` — use `systemctl restart docker`. Also note the driver is the new **`overlayfs`** (containerd-snapshotter), not legacy `overlay2` — relevant to B2.
+
+## B2 — Live migration sequence (move-then-verify) ✅
+
+Proven on the throwaway with seeded **running** apps (postgres with a known row `migrate-survive` in a named volume + nginx). The safe sequence:
+
+1. `docker stop <apps>` 2. `systemctl stop docker docker.socket` 3. **copy, not move:** `rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/dockerdata/` 4. write `data-root` → `/mnt/dockerdata` in `daemon.json` 5. `systemctl start docker` 6. **verify** images + `docker volume ls` + start apps + query the DB row 7. **only then** reclaim the old location (`rm -rf /var/lib/docker/*`).
+
+Result: post-switch the `pgdata` volume was present, both apps booted, and the DB row was intact. Reclaiming the old path while apps ran on the new `data-root` did not disturb them — the new `data-root` is fully self-sufficient.
+
+**Two hard lessons (both nearly caused a silent false-positive):**
+- **`rsync` was not installed** on the fresh guest; the first attempt's `rsync … | grep …` swallowed the `127` exit (pipe returns grep's status, `set -e` doesn't fire), so the migration "completed" having copied **nothing**. The migration script MUST install rsync and **check rsync's own exit code**, not a piped one.
+- With **containerd-snapshotter** the image content store sits under `data-root/containerd…`; switching `data-root` to an empty volume made images *appear* present (they resolve from the snapshotter's own addressing) while volumes/containers were empty — a misleading partial state. The reliable approach is to **rsync the entire `data-root` with docker stopped** (captures the containerd store + volumes + containers) and verify functional integrity (volume + DB row + boot), or treat images as re-pullable and migrate only the volumes (the irreplaceable data). The copy-not-move discipline meant the source was always intact and recoverable — I reverted `data-root` and the original apps + DB row came straight back.
+
+## B3 — Backup coverage after the split ⚠️ (the load-bearing finding)
+
+**The spec's premise is wrong.** It assumed the new mountpoint "should be [captured], as a CT mountpoint unless `backup=0`." The opposite is true on PVE 9.2.2:
+
+```
+# mp0 with NO backup flag:
+INFO: excluding volume mount point mp0 ('/mnt/dockerdata') from backup (disabled)
+```
+
+Additional LXC mountpoints are **excluded from vzdump by default**. They are included **only** when `backup=1` is set on the mountpoint. And setting it on a running CT is a **pending change** (`[pve:pending]`) — vzdump kept excluding mp0 until the **CT was restarted**, after which:
+
+```
+# mp0 with backup=1, post-restart:
+INFO: including mount point mp0 ('/mnt/dockerdata') in backup
+```
+
+Confirmed end-to-end: the resulting archive contains `./mnt/dockerdata/volumes/pgdata/_data/…` — the DB data is in PBS/vzdump only once `backup=1` is active.
+
+**Implication:** if the OS/data split lands without `backup=1` on the docker-data mountpoint, **every named-volume database silently falls out of the PBS whole-guest snapshot** — exactly the class of silent-default trap the spike exists to catch (cf. the PBS `ignore-verified` default). This is the single must-fix for the provisioning spec.
+
+Honest note (as the spec asked): because **both** volumes stay in PBS once `backup=1` is set, the backup **size does not shrink** — the win of the split is independent sizing/growth and a smaller, faster-to-restore OS rootfs, **not** a smaller backup. (Excluding images from PBS would shrink it but would drop DB coverage — out of scope; not done.)
+
+## B4 — Resize paths: live vs fresh install ✅
+
+- **Live (within the existing 349 GB `local-lvm`), non-disruptive:** `pct resize 9300 rootfs +2G` and `pct resize 9300 mp0 +4G` both grew the volume **and** the in-guest ext4 online (7.8→9.8 GB rootfs, 16→20 GB docker-data) with the CT running and the DB still queryable — no reboot, no downtime. Adding a brand-new mountpoint (`pct set -mpN`) is also non-disruptive to the rootfs, but the **mount itself only appears after a CT restart** on an unprivileged guest (same pending-activation behaviour seen with `backup=1` and with drive binds — consistent with the agent's existing "activate at next boot" note). Grow-only is safe; shrinking LVM-thin volumes is not supported online and was not attempted. The host root (`local`) was **not** repartitioned on the live system.
+- **Fresh install (host-level carving — this is where "configurable at install" lives):** the Proxmox ISO installer's *Advanced LVM options* expose `hdsize` (total of the disk PVE uses), `maxroot` (cap on the host `/` = `pve/root`), `maxvz` (cap on the `data` thin pool = `local-lvm`; set to 0 / leave headroom to keep space unallocated), `minfree` (reserved free space in the VG), and `swapsize`. These decide how the SSD is split between a small **host** root and a large **`local-lvm` thin pool**. Note the distinction the spec's target shape blurs: those installer knobs size the **host's** local-vs-thinpool; the **guest's** 32 GB-rootfs + 256 GB-docker-data split is sized at *provision* time via `pct` (rootfs size + an additive mountpoint), carved out of the thin pool — not by the installer. On the demo, the thin pool already has ~350 GB free, so per-guest carving needs no host repartition.
+
+## B5 — Provisioning integration points ✅
+
+Where the guest is sized today, and where the split + configurable sizes slot in:
+
+- **Golden rootfs size — `configs/build-golden.sh:40`:** `--rootfs "${ROOTFS_STORAGE}:8"` — the OS rootfs is **hardcoded at 8 GB**. The golden also `docker pull`s the controller + app images into `/var/lib/docker` **on the rootfs** (lines ~71–92), so the baked-image set is bounded by this size. `ROOTFS_STORAGE` is the 3rd script arg (default `local-lvm`). The golden runs the same docker-ce as production (→ 29.5.3 overlayfs/containerd-snapshotter).
+- **Provision bring-up — `internal/reconcile/bringup.go`:** `BringUpSpec` already carries the right seams: `RestoreStorage` (rootfs target storage), `RootfsGrowGB` (grow-only rootfs resize, applied as its own `ResizeLXC` call at line ~205), and `Mounts []GuestMount{Storage,SizeGB,MountPoint}` (additive `mpN`, attached in `buildConfigParams` at line ~313). So a docker-data volume is a natural `GuestMount`, and the OS rootfs can be grown per-customer via `RootfsGrowGB`.
+- **🔴 Exact code spot for the B3 fix — `bringup.go:313`:** today it builds `fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)` — **no `backup=` flag**, so any additive mount (including a future docker-data volume) is created `backup=0` = **excluded from PBS**. The docker-data mount MUST be attached with `,backup=1` (and the provision flow must account for the restart-to-activate behaviour). `GuestMount` should gain a `Backup bool` (or always-on for the data mount).
+- **Where the size config comes from:** currently caller-provided — the `--selftest=provision`/bring-up path and `cfg.Backup.RestoreStorage`. The in-code marker (`bringup.go:49-50`) states slice 10 wires the **hub storage manifest** into `Mounts`; that is the right home for per-customer sizes, consistent with bootstrap-v2 (the controller already pulls customer-scoped config from the hub). So "configurable at install" = a per-customer field in the hub desired-state → `RootfsGrowGB` + `GuestMount.SizeGB`.
+
+---
+
+## Open questions for the provisioning spec to resolve
+
+1. **Where does `data-root` get redirected for a *fresh* guest?** The golden bakes images into `/var/lib/docker` on the rootfs. Mounting an empty data volume **over** `/var/lib/docker` hides the baked images. Options: (a) bake `daemon.json` `data-root=/mnt/dockerdata` into the golden and attach the (empty) data mount before first Docker start, re-pulling images on first boot; (b) bake the split into the golden itself (data volume present at golden-build time, images baked onto it); (c) first-boot migration step. Pick one; (b) keeps the baked-image fast-start, (a) is simplest but loses it.
+2. **`backup=1` is mandatory on the docker-data mount** (B3). Encode it at `bringup.go:313` and decide whether the OS rootfs alone (no app data) is worth a separate lighter backup cadence.
+3. **Restart-to-activate** (B3/B4): attaching a mount or flipping `backup=1` on a running unprivileged guest is pending until reboot. Provisioning attaches mounts *before* first start, so this is a non-issue at provision time, but any *post-hoc* migration of an existing customer guest must schedule a reboot.
+4. **Default sizes & source:** confirm 32 GB rootfs / 256 GB docker-data defaults, and wire them through the hub storage manifest (per-customer), sized to the actual SSD via the installer carving facts in B4.
+5. **Backup size does not shrink** (B3): set expectations — the split is for independent sizing/growth and a smaller OS-rootfs restore, not a smaller PBS footprint.
+
+**Nothing was shipped.** Throwaway LXC 9300 and its test archives were destroyed after the spike.

REPORT.md

@@ -0,0 +1,24 @@
+# REPORT — agent v0.135.0: the restore test proves only backups of a guest (R-689, 2026-09-27)
+
+**Baseline:** `main` `7403c2a838db`, v0.134.0 on both demo hosts. **Commits:** `d4be12c` (fix + tests), `9ff937d`
+(CHANGELOG, after the release). **Released** by `scripts/release-agent.sh`: tag `v0.135.0`, sha256
+`ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (operator).
+
+**Tests:** full suite rc=0; agent gates OK after the release. `TestR689_TheRestoreTestNeverPicksTheGolden` red-proofed
+(without the check it picks `local:backup/felhom-golden-0.236.0.tar.zst`); `TestR689_GuestBackupArchiveShapes`; two older
+picker fixtures moved to real archive names.
+
+**Delivered** by signed `agent_update` (felhom-opsign, key `felhom-op-1`): demo-felhom committed 12:22:53 CEST, demo-hp
+12:29:02 CEST (`felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`).
+
+**Then v0.136.0 (`16dbc83`, sha256 `2eb0b5eb…fa6d755`, NOT vouched):** the scheduler's read-only verdict on demo-hp
+(`-selftest=restore-test-due`, `D1/D1-selftest-due-demo-hp.txt`) skipped the golden but picked a leftover archive of
+guest 9100 (deleted in August). The guest must now exist; red-proof `D1/RP-r689-deleted-guest.txt`. Signed-delivered to
+both hosts. The verdict after 0.136.0: `D1/D1-selftest-due-after-0136.txt`.
+
+**Then v0.137.0 (`3ef095f`):** 0.136.0's lookup met PVE's 403 "permission denied" (the token sees only its pool), not
+"does not exist", so the local tier read UNKNOWN every evaluation (`D1/D1-selftest-due-after-0136.txt`) — a regression of
+0.136.0, fixed: a guest the agent cannot read is not one it manages. Red-proof `D1/RP-r689-acl.txt`; verified read-only on
+demo-hp with the pre-release binary before the release (`D1/D1-selftest-due-0137-pre.txt`): both leftovers skipped, the
+off-site tier due on 9201, the local tier waiting for today's 9201 archive to settle. Three agent releases in one session
+— each the smallest fix of what the box showed.

REUSE.md

@@ -0,0 +1,205 @@
+# REUSE.md — felhom-agent
+
+> Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid.
+> Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper.
+> Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
+
+## 1. Canonical helpers (MUST reuse — do not reinvent)
+
+### Allowlisted exec / privileged surface (sudoers)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Runner` / `ExecRunner.Run`, `RunStdin` | internal/proxmox/privileged.go | `Run(ctx, name, args...) (stdout, stderr []byte, err)` | ALL host command exec (direct or `sudo -n` prefix) | Arg vectors only, never a shell string; `capBuf` caps output at 1 MiB |
+| `Privileged` (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors) | internal/proxmox/privileged.go | methods on `*Privileged` | the 3 fenced root-CLI exceptions ONLY | Do NOT add methods — fence is structural (`routing_test.go` asserts it) |
+| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
+| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
+| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
+| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
+
+### Disk / format safety (role gates, durable IDs, format guards)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `SudoHostOps.Format` | internal/storage/hostops.go | `Format(ctx, device, fstype) error` | THE only mkfs path | Guards, in order: `ValidateBlockDevice` + `ValidateFSType` → mandatory `deviceUnclaimed` (claim.go) → exec `felhom-mkfs-guarded` (sudoers allowlists ONLY the wrapper, not raw mkfs) |
+| `SudoHostOps.InspectDevice` + `DeviceProbe.DataBearing` | internal/storage/hostops.go | `InspectDevice(ctx, device) (DeviceProbe, error)` | data-bearing verdict from the AGENT's own read | Fail-safe: `Probed=false` ⇒ DataBearing=true; blkid output is evidence, lsblk is read-success authority |
+| `classifyClaim` / `SudoHostOps.deviceUnclaimed` | internal/storage/claim.go | `classifyClaim(claimFacts) (unclaimed bool, reason string)` | "is this disk provably free to format" | Pure function of `gatherClaimFacts`; ANY read error/ambiguity/empty-lsblk ⇒ CLAIMED (audit D2); Felhom's own `/mnt/felhom-drives` mounts are not a foreign claim |
+| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
+| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
+| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
+| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
+| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
+| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
+| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
+| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
+| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
+| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
+| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
+| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
+
+### Mount lifecycle (host + guest binds)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
+| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
+| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
+| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
+| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
+| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
+| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
+| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
+| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
+| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
+| `ClassifyNetVerifyFailure` | internal/storage/netverify.go | `ClassifyNetVerifyFailure(journalTail, tcpReachable) (code, hint)` | NAS verify failure categories | String-based BY DESIGN (every mount failure is rc=32); substrings verbatim from SPIKE-nas-verify Q4; `nfs_export` merges not-found/not-permitted (NFSv4 identical) |
+
+### Durable stores (atomic state)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
+| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
+| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
+| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
+| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
+| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
+
+### Local-API plumbing
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Server.withGuest` | internal/localapi/server.go | `withGuest(fn(w, r, vmid)) http.HandlerFunc` | bearer auth + self-scoping for EVERY route | Token→vmid is the authority; explicit `?vmid=` only a consistency check (403 on mismatch) |
+| `Server.scopedFromBody` | internal/localapi/server.go | `scopedFromBody(w, bodyVMID, tokenVMID, path) bool` | POST-body vmid self-scope check | Call right after decode; false = already 403'd |
+| `decodeBody` / `writeOK` / `writeErr` / `writeStatus` | internal/localapi/server.go | `writeStatus(w, code, ok, data, errMsg)` | ALL local-API JSON I/O | Do not hand-roll response envelopes |
+| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
+| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
+| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
+| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
+| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
+
+### Proxmox client / hub / PBS / provisioning
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
+| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
+| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+| `Client.PoolAddVMID` | internal/proxmox/mutate.go | `PoolAddVMID(ctx, pool, vmid) error` | re-assert pool membership after a restore-over-existing (campaign-2 R2) | SYNC (no UPID, don't WaitTask); PVE `PUT /pools` is additive (merge, not replace) — `delete=1` removes; idempotent (already-member swallowed); needs `Pool.Allocate` at `/pool/<pool>`. `pct restore --pool` sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore |
+| `reconcile.PreflightRestoreSpace` + `restorespace.Provider` (v0.133.0) | internal/reconcile/restoretest_space.go, internal/restorespace/restorespace.go | `PreflightRestoreSpace(ctx, space, policy, archive, rawCfg, configured) SpaceVerdict` | ANY step that restores or copies a guest onto a storage — size it first | The restored size is UNCOMPRESSED (vzdump log "Total bytes written" / PBS snapshot size) — never the archive FILE (6.9 GB file → 22.6 GB restore, R-672); an unknown refuses; eligibility needs Datastore.AllocateSpace on `/storage/<id>` specifically (the `/` grant answers every path) |
+| `Engine.RetryScratchTeardown` + the daemon janitor (v0.133.0) | internal/reconcile/restoretest_retry.go, cmd/felhom-agent/janitor.go | `RetryScratchTeardown(ctx) ScratchRetryResult` | retrying a leftover on a TIMER instead of only at start | Never `Recover` on a timer — it also resolves generic in-flight ops; a periodic sweep that unlocks guests holds the one-heavy-op gate (`InFlight.TryAcquire`) |
+| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
+| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
+| `httpx.NewTransport` | internal/httpx/transport.go | `NewTransport(tlsCfg, idleConnTimeout) *http.Transport` | **EVERY** hand-rolled `http.Transport` in this repo — pbs, hub and proxmox all pin TLS, so none can use `http.DefaultTransport` | **R-344: never inline `&http.Transport{TLSClientConfig: ...}` again.** A composite literal takes `IdleConnTimeout` **zero, which means retain idle connections FOREVER** — `http.DefaultTransport` sets 90s and a literal does not inherit it. Combined with a client rebuilt per cycle and dropped (`pbsTargetsFromPVE`), that stranded **388 sockets on ep0 in 46 h**, held open on BOTH sides. `idleConnTimeout <= 0` means **use the default**, never "no timeout". Returns a **FRESH** transport every call — a shared one would pool connections across differently pinned endpoints. Pinned by `internal/pbs/client_leak_test.go` (server-side connection counting) + `internal/httpx/transport_test.go` |
+| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
+| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 60–3600 s |
+| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
+| `buildBringUpConfig` island NIC | internal/reconcile/bringup.go | (pure) `BringUpSpec{IslandBridge,IslandGuestAddr}` → `params["net1"]` | R-50 island control plane | When BOTH island fields are set (from `cfg.LocalAPI`), attaches a static `net1=name=eth1,bridge=<vmbr9>,ip=<.2/30>` (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in `LocalAPIConfig.Validate`. The guestnet healer is eth0-only (`parseMode` is dev-scoped) so it never touches the static island NIC |
+| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
+| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
+| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
+| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
+| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
+| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
+| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
+| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
+| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
+| `wgtunnel.Manager.TunnelConvergence` / `reconcile.Engine.LastResult` | internal/wgtunnel/manager.go, internal/reconcile/engine.go | `TunnelConvergence() (desired, operational bool)`; `LastResult() (Result, bool)` | cached convergence snapshots for the fast-tick | TunnelConvergence is refreshed at the END of every `Apply` (its own cadence) so the fast-tick never execs `wg`/`systemctl`; LastResult is mutex-recorded per `reconcileOnce`, ok=false until the first pass |
+
+## 2. Canonical patterns (copy structure from THE named file)
+
+| Pattern | Canonical file | Key traits |
+|---|---|---|
+| Validate-then-exec privileged op | internal/storage/hostops.go (`EnsureMount`) | validate EVERY arg (validate.go) → render → stage in agent dir → root `install` → `systemctl`; refuse before any command exists |
+| Fail-safe pure classifier over gathered facts | internal/storage/claim.go (`classifyClaim` over `claimFacts`) | pure function ⇒ fixture-testable; any error/ambiguity refuses; gather separated from verdict |
+| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
+| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
+| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
+| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
+| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
+| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
+| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
+| Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) |
+| Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash |
+| Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted |
+| Root-file install via random temp | internal/localapi/intermediary.go (`installSharedParentUnit`) | `stageTemp` (os.CreateTemp) → sudoers-globbed `install -m` → pinned destination |
+| Detached destructive job + restart recovery | internal/localapi/formatjob.go | persist `running` → run off baseCtx → record outcome; recovery re-resolves durable id, never a path |
+| Signed-op verify pipeline | internal/authz/verifier.go (`Verify`) | armor→namespace→key-material allowlist→crypto over RAW bytes→blob→target→window→nonce LAST |
+| Resilient daemon loop | internal/hub/loop.go (`Loop.Run`) | ctx-cancel returns nil; errors logged and loop continues |
+| Guarded-below-the-agent wrapper | configs/felhom-mkfs-guarded.sh | root re-checks catastrophic cases (system disk/LVM PV/foreign mount/RO/member FS) even against an agent bug |
+
+## 3. Dangerous lookalikes — do NOT reuse
+
+| Trap | Why it bites | Use instead |
+|---|---|---|
+| Acting on the caller's `req.Device` (or any remembered /dev path) after inspection | classify→mkfs TOCTOU (AGENT-001): /dev re-enumeration retargets the node to a different physical disk | `Server.reresolveWipe`/`reresolveBlank` → format the RE-RESOLVED device |
+| Exec'ing raw `mkfs.*` (incl. `Binaries.MkfsExt4/MkfsXfs`) | sudoers no longer allowlists raw mkfs; bypasses the claim filter + wrapper re-checks | `SudoHostOps.Format` (→ `felhom-mkfs-guarded`) |
+| `DiskInfo.DurableID` (`uuid:`) as a wipe-confirmation id | wipe gate resolves `byid:`/`byuuid:` — a `uuid:` id is a `binding_mismatch` (F20-BUG2) | `DiskInfo.WipeDurableID` / `storage.DeviceDurableID` |
+| `ResolveDurableDevice` for enrolled-storage remounts (or vice versa) | schemes differ: wipe = `byid:`/`byuuid:`, storage = `uuid:` — each refuses the other | `ResolveStorageDevice` for mounts; `ResolveDurableDevice` for wipes |
+| `authz.MemoryNonceStore` on a real host | replay protection dies on restart | `authz.FileNonceStore` |
+| Adding methods to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers Cmnd_Alias + validate.go-style checks |
+| Calling `Client.DestroyLXC`/`Vzdump`/`SetConfig` outside the gate/queue/journal | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
+| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can brick guest boot (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
+| `isHostMountpoint` to reconcile bind state | boolean can't converge stacked double-binds (the /mnt doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
+| Acting on a raw `ListLXC` list as if it were "guests the agent owns" | audit A1 (pre-v0.62.0 the stale-lock reaper did exactly this — contained only by the pool-scoped token) | ownership must be PROVEN: intersect with `Client.Pool` membership like `staleLockController.Guests()` (v0.62.0), fail-safe on read failure |
+
+## 4. Seams & interfaces (testing + cross-repo)
+
+| Interface | Defined in | Implemented by | Fakes/tests at |
+|---|---|---|---|
+| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
+| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
+| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
+| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
+| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
+| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
+| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
+| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
+| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
+| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
+| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
+| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
+| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
+| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
+| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
+| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
+| `hub.AddressEnumerator` (v0.119.0) | internal/hub/hostaddr.go | **defaults to the REAL `systemInterfaces`** when `Collector.addrEnum` is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record) | internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from `ip -o addr show` on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. `filterHostAddresses` keeps GLOBAL UNICAST only: one predicate that drops loopback, `fe80::/10`, and `169.254/16` — the last being the R-50 island literal, identical on every box and actively misleading if surfaced |
+| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
+| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
+| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go |
+| `applog.Ring` (always-DEBUG capture ring) + fan-out `applog.New → (logger, ring)` | internal/log/log.go | wired in cmd/felhom-agent/main.go → `localapi.Options.LogRing` + `Loop.SetLogTailSource(ring.Lines)` | internal/log/log_test.go; localapi/debuglogs_test.go — v0.83.0; the byte-capped `Lines` is the heartbeat tail source |
+| `pbsdr.StorageReader` / `SecretConsumer` / `Manager.probeFP` (func seam) | internal/pbsdr/manager.go | `*proxmox.Client`; `*hub.Client`; `pbs.ProbeFingerprint` | `fakeStorage`/`fakeConsumer`/`fakeRunner` internal/pbsdr/manager_test.go (argv+stdin recorder) |
+| `capability.Runner` | internal/capability/probe.go | `*proxmox.ExecRunner` (RunnerDirect) | `fakeRunner` internal/capability/probe_test.go |
+| Cross-repo: local API ↔ controller | internal/localapi/server.go routes; contract seeded by internal/provision/doc.go (`bootstrap.json`: endpoint + leaf fingerprint + token) | felhom-controller's agentapi client | pin = served leaf cert (memory gotcha) |
+| Cross-repo: agent ↔ hub | internal/hub/report.go (`HostReport`), `ControlEnvelope`; POST `/api/v1/host-report` | hub mirrors structs field-for-field | new event/report fields need hub-side ingest changes |
+| Cross-repo: shipped host artifacts | configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, configs/felhom-pbs-apply, shared-parent script/unit (inline in internal/localapi/intermediary.go) | deployed WITH the binary | sudoers globs must match `stageTemp` patterns + staging dirs exactly |
+| Operator signing | internal/authz (OpBlob, SSHSIG) | cmd/felhom-opsign (offline CLI) | blob/verify tests in internal/authz |
+
+## 5. Extension points (where new features plug in)
+
+- **Local-API route**: add to `Server.Handler` (internal/localapi/server.go) wrapped in `s.withGuest`; new deps go into `Options` as OPTIONAL fields degrading to "not configured".
+- **New signed-job verb**: implement `signedjobs.Executor` (return `ErrNoExecutor` for foreign ops) and append to the `signedjobs.ExecutorChain` in cmd/felhom-agent/main.go; add the op class + role scoping in internal/reconcile (classify.go, gate.go `roleAuthorizes`).
+- **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
+- **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
+- **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo).
+- **DR-recipe section (host-half)**: add the field to `DRRecipeHostHalf` (internal/hub/dr_recipe.go) **AND** to
+  the hub's `hostHalfShape` + `AssembledRecipe` (felhom.eu `hub/internal/store/dr_recipe.go`). Those two
+  hub structs are **ALLOW-LISTS**: a section only the agent knows about is stored intact and silently
+  dropped before any operator sees it — that is R-122, which cost `offsite_restic` its entire existence.
+  Then update BOTH copies of `testdata/host-report.golden.json` (byte-identical, cross-repo) and extend
+  `TestAssembleDRRecipe_CarriesEveryEmittedSection`. **A recipe field that cannot be resolved records an
+  explicit unknown with a reason — never a default, an empty string, or a placeholder** (`DRState*` /
+  `DRReason*`); a recipe read during a rebuild must not present a guess as a fact.
+- **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go.
+- **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go.
+- **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output).
+
+## 6. Known duplication (observed — NOT fixed)
+
+- Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go).
+- Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map).
+- **SMART device resolution (v0.95.0):** `smartDeviceFor` (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (`dmWholeDisk` in internal/storage/smartdev.go, via `/sys/block/<dm>/slaves`, `sysBlockRoot` test seam). `storage.SmartReader.SMARTForBacking` is the shared read the localapi `/disks` union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-`local` SMART device comes from `containingMountDevice` (SMART-only; never feeds backing/durable_id).
+- Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
+- `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go).
+- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go). **In localapi they were unified in v0.117.0**: `isHostMountpoint` and `countHostMounts` are now one-liners over `hostMountEntries`, the single parser that also yields devno/fstype/super-options for `bindLiveness`.
+- Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 1–3 (internal/signedjobs/wipe.go) across packages.
+- `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins.

RUNBOOK-provisioning-storage.md

@@ -0,0 +1,129 @@
+# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
+
+How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
+separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
+(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
++ 50 GB user-data**, rest of the thin pool free for other guests.
+
+## The design (locked)
+
+Three thin volumes are baked into the golden and grown at provision:
+
+| mp slot | Mount | Holds | Golden default | Provision grow knob |
+|---------|-------|-------|----------------|---------------------|
+| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
+| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
+| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
+
+- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
+  if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
+  unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
+- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
+  (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
+  partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
+  protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
+  baked log rotation), not by placement.
+- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
+  decommissions — it is part of the golden's baked layout. The controller auto-discovers
+  `<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
+  mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
+  the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
+  the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
+  use).
+- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
+  default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
+  data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
+
+## Golden build (per controller-image baseline)
+
+`configs/build-golden.sh` builds the template guest with the split baked in:
+
+```
+# defaults shown; override via env
+OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
+  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
+```
+
+- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
+  + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+  + `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
+- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
+  grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
+  mount; the golden does NOT pre-create it.
+- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
+  classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
+  AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
+  or mp1** (the B3 trap).
+- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
+  image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
+  the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
+  (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
+  overlay + volumes) under data-root = the data volume, which is what the split + the controller's
+  `statfs("/")` prevention guard both require.
+
+## Provision (per customer guest)
+
+The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
+baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
+are never shadowed:
+
+```
+felhom-agent --selftest=provision \
+  -archive <golden-volid> -vmid <VMID> \
+  -customer-id <id> -hub-password <retrieval-passphrase> \
+  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
+  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
+  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
+  -sysdata-grow <N>  # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
+  -sysdata-mount mp1 # the golden's user-data slot (default mp1)
+```
+
+In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
+— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
+controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
+the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
+mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
+USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
+
+**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to
+**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back
+on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) —
+a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the
+drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind.
+(Guests provisioned before v0.41.0 need a one-time `pct set <vmid> -onboot 1`.)
+
+**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
+home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
+the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
+
+### Default sizing guidance
+- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
+  (`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
+  other guests.
+- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
+  for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
+  written.
+
+## Fresh PVE host install — thin-pool carving (host-level, do this at install)
+
+This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
+carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:
+
+- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
+- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
+- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
+  rest, OR cap it to reserve unallocated VG space.
+- `minfree` — reserved free space in the VG (thin-pool safety).
+- `swapsize` — host swap.
+
+**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
+time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
+~350 GB free).
+
+## In-place migration of an EXISTING guest (NOT used by this slice)
+
+This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
+used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
+upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
+rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
+the new mount and a CT restart to activate it are required (spike B3/B4).

cmd/felhom-agent/archivetier_test.go

@@ -0,0 +1,23 @@
+package main
+
+import "testing"
+
+// R-82 live regression (2026-07-26): the restore-test derived its tier from the CONFIGURED default
+// target instead of the archive's own storage. Restoring a `felhom-pbs:` archive on a box whose
+// primary target is "local" was classified "local" → the 10-minute local wait instead of the
+// generous PBS one → the wait expired mid-restore at 600s against a 14.46 GB WAN restore, teardown
+// fired at a still-restoring guest, and the scratch leaked.
+func TestArchiveStorageID(t *testing.T) {
+	cases := []struct{ in, want string }{
+		{"felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", "felhom-pbs"},
+		{"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst", "local"},
+		{"", ""},
+		{"no-prefix", ""},
+		{":leading-colon", ""}, // i>0 guard: a leading colon is not a storage id
+	}
+	for _, c := range cases {
+		if got := archiveStorageID(c.in); got != c.want {
+			t.Fatalf("archiveStorageID(%q) = %q, want %q", c.in, got, c.want)
+		}
+	}
+}

cmd/felhom-agent/escrow_contract_test.go

@@ -0,0 +1,36 @@
+package main
+
+import (
+	"encoding/json"
+	"reflect"
+	"sort"
+	"testing"
+)
+
+// TestEscrowUploadContract pins the agent→hub escrow wire shape. It MUST match the hub's ingest
+// struct (felhom-hub api.escrowUploadRequest). Cross-repo, no shared module — this is the agent
+// half of the contract guard; the hub has the mirror in its own test.
+func TestEscrowUploadContract(t *testing.T) {
+	b, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t", ResticPwSHA256: "h"})
+	var m map[string]any
+	if err := json.Unmarshal(b, &m); err != nil {
+		t.Fatal(err)
+	}
+	got := make([]string, 0, len(m))
+	for k := range m {
+		got = append(got, k)
+	}
+	sort.Strings(got)
+	want := []string{"blob_b64", "created_at", "key_fingerprint", "posture", "restic_pw_sha256"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("escrow wire contract drift: got %v want %v (must match the hub ingest struct)", got, want)
+	}
+	// SLICE 3: no staged password folded in → the hash field is OMITTED on the wire (the hub stores NULL →
+	// the controller never matches → stays pending; correct — the blob doesn't cover the key).
+	b2, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t"})
+	var m2 map[string]any
+	_ = json.Unmarshal(b2, &m2)
+	if _, present := m2["restic_pw_sha256"]; present {
+		t.Fatal("restic_pw_sha256 must be omitted when no staged password was sealed")
+	}
+}

cmd/felhom-agent/escrow_recover_wiring_test.go

@@ -0,0 +1,188 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"strings"
+	"testing"
+)
+
+// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
+// grepping for a string.
+//
+// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
+// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
+// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
+// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
+// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
+// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
+// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
+
+func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
+	t.Helper()
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
+	if err != nil {
+		t.Fatalf("parsing main.go: %v", err)
+	}
+	return fset, f
+}
+
+// callsWithin returns the set of function names called (directly, by identifier or selector) inside
+// the named top-level function.
+func callsWithin(f *ast.File, fnName string) map[string]bool {
+	out := map[string]bool{}
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			ce, ok := n.(*ast.CallExpr)
+			if !ok {
+				return true
+			}
+			switch fn := ce.Fun.(type) {
+			case *ast.Ident:
+				out[fn.Name] = true
+			case *ast.SelectorExpr:
+				if x, ok := fn.X.(*ast.Ident); ok {
+					out[x.Name+"."+fn.Sel.Name] = true
+				}
+				out[fn.Sel.Name] = true
+			}
+			return true
+		})
+	}
+	return out
+}
+
+// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
+func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
+	_, f := parseMain(t)
+
+	// 1. main() reaches runDaemon.
+	if !callsWithin(f, "main")["runDaemon"] {
+		t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
+	}
+	// 2. runDaemon reaches buildLocalAPIServer.
+	if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
+		t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
+	}
+
+	// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
+	//    an escrow.OffsiteKeyRecoverer is constructed there.
+	var optionsHasField, recovererConstructed bool
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			cl, ok := n.(*ast.CompositeLit)
+			if !ok {
+				return true
+			}
+			sel, ok := cl.Type.(*ast.SelectorExpr)
+			if !ok {
+				return true
+			}
+			pkg, _ := sel.X.(*ast.Ident)
+			if pkg == nil {
+				return true
+			}
+			switch pkg.Name + "." + sel.Sel.Name {
+			case "localapi.Options":
+				for _, el := range cl.Elts {
+					kv, ok := el.(*ast.KeyValueExpr)
+					if !ok {
+						continue
+					}
+					if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
+						optionsHasField = true
+					}
+				}
+			case "escrow.OffsiteKeyRecoverer":
+				recovererConstructed = true
+			}
+			return true
+		})
+	}
+	if !recovererConstructed {
+		t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
+			"production assembly point (the built-but-never-wired shape, seventh instance)")
+	}
+	if !optionsHasField {
+		t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
+			"exists and the route would answer 503 forever")
+	}
+}
+
+// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
+// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
+// key is used.
+func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
+	fset, f := parseMain(t)
+	var fetchUsesHubClient bool
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			ce, ok := n.(*ast.CallExpr)
+			if !ok {
+				return true
+			}
+			sel, ok := ce.Fun.(*ast.SelectorExpr)
+			if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
+				return true
+			}
+			if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
+				fetchUsesHubClient = true
+			} else {
+				t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
+					fset.Position(ce.Pos()))
+			}
+			return true
+		})
+	}
+	if !fetchUsesHubClient {
+		t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
+			"not wired, or it uses a client whose credentials are not this host's")
+	}
+}
+
+// The route itself must be registered on the local API. A handler with no route is the same defect
+// one layer down, and it has shipped here before.
+func TestRecoverRouteIsRegistered(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parsing localapi/server.go: %v", err)
+	}
+	var registered bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		ce, ok := n.(*ast.CallExpr)
+		if !ok || len(ce.Args) < 2 {
+			return true
+		}
+		sel, ok := ce.Fun.(*ast.SelectorExpr)
+		if !ok || sel.Sel.Name != "HandleFunc" {
+			return true
+		}
+		lit, ok := ce.Args[0].(*ast.BasicLit)
+		if !ok {
+			return true
+		}
+		if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
+			registered = true
+		}
+		return true
+	})
+	if !registered {
+		t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
+			"exists and nothing can reach it")
+	}
+}

cmd/felhom-agent/guesthook_bulletproof_test.go

@@ -0,0 +1,60 @@
+package main
+
+import (
+	"context"
+	"testing"
+	"time"
+)
+
+// F10/rc255 (CAMPAIGN-3): a guest-hook phase body that PANICS must never crash the process — the hook
+// must return cleanly so the guest start proceeds (a nonzero exit blocks the start). runHookPhase
+// recovers the panic and returns.
+func TestRunHookPhase_PanicRecovered(t *testing.T) {
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		runHookPhase("9201", "pre-start", 5*time.Second, func(context.Context) {
+			panic("simulated heal panic (e.g. a future Heal bug)")
+		})
+	}()
+	select {
+	case <-done:
+		// returned cleanly — the guest start would proceed
+	case <-time.After(3 * time.Second):
+		t.Fatal("runHookPhase did not return after a panicking body (would have crashed the hook)")
+	}
+}
+
+// A phase body that overruns the timeout must be abandoned — the hook returns rather than hanging the
+// PVE start task. (The body's context is cancelled; the hook does not wait for the body to notice.)
+func TestRunHookPhase_TimeoutReturns(t *testing.T) {
+	bodyCtxCancelled := make(chan struct{}, 1)
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		runHookPhase("9201", "post-start", 20*time.Millisecond, func(ctx context.Context) {
+			<-ctx.Done() // simulate a body that respects cancellation eventually
+			bodyCtxCancelled <- struct{}{}
+		})
+	}()
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("runHookPhase did not return after the timeout (would hang the guest start)")
+	}
+	select {
+	case <-bodyCtxCancelled:
+		// the body's context was cancelled at the deadline — the intended signal
+	case <-time.After(time.Second):
+		t.Fatal("the phase body's context was not cancelled at the timeout")
+	}
+}
+
+// A body that errors (returns normally, no panic) is fine — the hook returns cleanly.
+func TestRunHookPhase_NormalBodyReturns(t *testing.T) {
+	ran := false
+	runHookPhase("9201", "pre-start", time.Second, func(context.Context) { ran = true })
+	if !ran {
+		t.Fatal("the phase body must run")
+	}
+}

cmd/felhom-agent/guesthook_wiring_test.go

@@ -0,0 +1,33 @@
+package main
+
+import (
+	"context"
+	"testing"
+)
+
+// The hook WIRING red-proof target: `guest-hook <vmid> post-start` must invoke the network
+// reassert with the vmid; pre-start and unknown phases must NOT. (Companion red-proof: remove the
+// PhasePostStart case from runGuestHook → the invoked assertion fails.)
+func TestRunGuestHook_PostStartInvokesNetworkReassert(t *testing.T) {
+	orig := postStartNetworkReassertFn
+	t.Cleanup(func() { postStartNetworkReassertFn = orig })
+
+	var gotVMIDs []string
+	postStartNetworkReassertFn = func(_ context.Context, vmid string) {
+		gotVMIDs = append(gotVMIDs, vmid)
+	}
+
+	runGuestHook([]string{"9201", "post-start"})
+	if len(gotVMIDs) != 1 || gotVMIDs[0] != "9201" {
+		t.Fatalf("post-start must invoke the network reassert with vmid 9201, got %v", gotVMIDs)
+	}
+
+	// pre-start must not touch the network reassert (it is the placeholder-heal phase; the heal
+	// no-ops on a nonexistent config path and never blocks).
+	runGuestHook([]string{"9201", "pre-start"})
+	// unknown phases are ignored entirely.
+	runGuestHook([]string{"9201", "pre-stop"})
+	if len(gotVMIDs) != 1 {
+		t.Fatalf("only post-start may invoke the network reassert, got %v", gotVMIDs)
+	}
+}

cmd/felhom-agent/guestnet_wiring_test.go

@@ -0,0 +1,100 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"testing"
+)
+
+// R-54 §9 rule 6 — the seam-discipline test, and the one this repo has the most reason to write:
+// v0.91.0 shipped the PBS auth-probe seam with `main.go` never calling `SetAuthSink`, every unit
+// test green because they all injected the seam directly. The guestnet watchdog has the identical
+// shape (a component + a reporter seam + a goroutine), so its wiring is asserted here rather than
+// trusted.
+//
+// This walks the AST rather than grepping the source: a commented-out call still satisfies a
+// substring match (found while red-proofing the controller's twin of this test), and a comment is
+// not a caller.
+func TestMainWiresGuestNetWatchdog(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+
+	var constructed, reporterWired, started bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.CallExpr:
+			switch fn := node.Fun.(type) {
+			case *ast.SelectorExpr:
+				switch fn.Sel.Name {
+				case "New":
+					// guestnet.New(...)
+					if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "guestnet" {
+						constructed = true
+					}
+				case "SetGuestNetReporter":
+					reporterWired = true
+				}
+			}
+		case *ast.GoStmt:
+			if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
+				if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
+					started = true
+				}
+			}
+		}
+		return true
+	})
+
+	if !constructed {
+		t.Error("main.go never calls guestnet.New — the watchdog does not exist at runtime")
+	}
+	if !reporterWired {
+		t.Error("main.go never calls collector.SetGuestNetReporter — the guest_net stanza would " +
+			"never reach the hub (the exact v0.91.0 inert-seam defect)")
+	}
+	if !started {
+		t.Error("main.go never starts the watchdog with `go gnWatchdog.Watch(ctx)` — it would be " +
+			"constructed, reported on, and never probe anything")
+	}
+}
+
+// The watchdog must NOT join the errc fan-out: a guest-network watchdog that can terminate the
+// agent turns a customer's DHCP problem into an operator-plane outage. If it is ever changed to
+// `errc <- ...`, the drain bound at the bottom of main() also has to change — this catches the
+// first half of that mistake.
+func TestGuestNetWatchdogIsNotInTheErrcFanout(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+	bad := false
+	ast.Inspect(f, func(n ast.Node) bool {
+		send, ok := n.(*ast.SendStmt)
+		if !ok {
+			return true
+		}
+		if ch, ok := send.Chan.(*ast.Ident); !ok || ch.Name != "errc" {
+			return true
+		}
+		call, ok := send.Value.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
+			if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
+				bad = true
+			}
+		}
+		return true
+	})
+	if bad {
+		t.Fatal("the guestnet watchdog was added to the errc fan-out — a watchdog over customer " +
+			"guests must never be able to bring the agent down (and the drain bound in main() " +
+			"would now be off by one, hanging shutdown)")
+	}
+}

cmd/felhom-agent/janitor.go

@@ -0,0 +1,81 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// janitorInterval is how often the leftovers of an interrupted restore-test or backup are retried
+// (R-672 rule 3, R-673). Both used to be resolved ONLY at agent start: on 2026-09-24 a failed scratch
+// teardown kept a full thin pool full for 2.5 h, and a stale `snapshot-delete` lock blocked 9201's
+// whole-box backups for five hours — each cleared within a minute of an agent restart.
+const janitorInterval = 10 * time.Minute
+
+// janitorDeps are the janitor's seams (tests drive one pass with fakes).
+type janitorDeps struct {
+	retryScratch func(ctx context.Context) reconcile.ScratchRetryResult
+	staleLocks   func(ctx context.Context) // localapi Server.RecoverStaleLockedGuests; nil when the local API is off
+	heavy        *backup.InFlight
+	record       func(hub.RestoreTest)
+	now          func() time.Time
+	logger       *slog.Logger
+}
+
+// janitorPass is one pass. The stale-lock sweep runs only while holding the one-heavy-operation gate, so
+// no agent backup can START between its "no vzdump is running" check and its unlock (at start-up the
+// sweep ran before the backup loop existed; on a timer that ordering must be made, not assumed). A busy
+// gate skips the sweep this pass — the next pass retries.
+func janitorPass(ctx context.Context, d janitorDeps) {
+	if d.retryScratch != nil {
+		r := d.retryScratch(ctx)
+		if r.Examined > 0 {
+			d.logger.Info("janitor: restore-test scratch retry pass", "examined", r.Examined,
+				"destroyed", r.Destroyed, "already_gone", r.Clean, "failed", r.Failed)
+		}
+		for _, vmid := range r.GaveUp {
+			// The operator is told through the existing restore-test failure path: the hub raises
+			// restore_test_failed (operator) once per distinct archive — this record's archive names
+			// the stuck scratch guest.
+			if d.record != nil {
+				d.record(hub.RestoreTest{
+					SourceArchive: fmt.Sprintf("scratch-teardown:%d", vmid),
+					ScratchVMID:   vmid,
+					Pass:          false,
+					Error: fmt.Sprintf("restore-test scratch guest %d could not be torn down after %d retries — it holds its disks; remove it by hand (pct destroy %d) after checking what keeps it busy",
+						vmid, reconcile.MaxTeardownTries, vmid),
+					TestedAt: d.now().UTC().Format(time.RFC3339),
+				})
+			}
+		}
+	}
+	if d.staleLocks != nil {
+		release, busy, ok := d.heavy.TryAcquire("stale-lock-sweep")
+		if !ok {
+			d.logger.Info("janitor: stale-lock sweep deferred — a heavy operation is in flight", "busy", busy)
+			return
+		}
+		defer release()
+		d.staleLocks(ctx)
+	}
+}
+
+// runJanitor runs janitorPass every janitorInterval until ctx ends.
+func runJanitor(ctx context.Context, d janitorDeps) {
+	d.logger.Info("janitor: starting (restore-test scratch retry + stale-lock sweep)", "interval", janitorInterval)
+	t := time.NewTicker(janitorInterval)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			janitorPass(ctx, d)
+		}
+	}
+}

cmd/felhom-agent/janitor_test.go

@@ -0,0 +1,60 @@
+package main
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-672 / R-673 (v0.133.0): one janitor pass, driven with fakes.
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// A scratch the engine gave up on reaches the hub as a failed restore-test record naming it — the
+// existing operator path (restore_test_failed). Never a pass.
+func TestJanitor_GaveUpIsReportedAsAFailure(t *testing.T) {
+	var got []hub.RestoreTest
+	janitorPass(context.Background(), janitorDeps{
+		retryScratch: func(context.Context) reconcile.ScratchRetryResult {
+			return reconcile.ScratchRetryResult{Examined: 1, Failed: 1, GaveUp: []int{990000}}
+		},
+		heavy: &backup.InFlight{}, record: func(r hub.RestoreTest) { got = append(got, r) },
+		now: time.Now, logger: quiet(),
+	})
+	if len(got) != 1 || got[0].Pass || got[0].ScratchVMID != 990000 || !strings.Contains(got[0].Error, "990000") {
+		t.Fatalf("records = %+v — want one FAILED record naming scratch 990000", got)
+	}
+}
+
+// R-673: the stale-lock sweep runs only while holding the one-heavy-operation gate, so no agent backup can
+// start between its "no vzdump running" check and its unlock.
+//
+// COMPANION RED-PROOF (REPORT): drop the TryAcquire → "the sweep ran while a backup held the gate".
+func TestJanitor_StaleLockSweepWaitsForTheHeavyGate(t *testing.T) {
+	heavy := &backup.InFlight{}
+	swept := 0
+	d := janitorDeps{staleLocks: func(context.Context) { swept++ }, heavy: heavy, now: time.Now, logger: quiet()}
+	release, _, ok := heavy.TryAcquire("backup")
+	if !ok {
+		t.Fatal("setup")
+	}
+	janitorPass(context.Background(), d)
+	if swept != 0 {
+		t.Fatal("the sweep ran while a backup held the gate")
+	}
+	release()
+	janitorPass(context.Background(), d)
+	if swept != 1 {
+		t.Fatalf("swept %d times with the gate free — want 1", swept)
+	}
+	if _, _, ok := heavy.TryAcquire("after"); !ok {
+		t.Fatal("the sweep did not release the gate")
+	}
+}

cmd/felhom-agent/restoretest_due_wiring_test.go

@@ -0,0 +1,156 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"testing"
+)
+
+// R-86 Scenario I — the seam-discipline test for the due-check.
+//
+// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
+// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
+// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
+// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
+// tested, never called — and this repo has shipped it four times.
+//
+// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
+// match, and a comment is not a caller.
+func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var settlePicker, oldPicker, settleWired, evalInterval bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.SelectorExpr:
+			// runner.PickSettledRestoreCandidateOn passed as a value (not called).
+			switch node.Sel.Name {
+			case "PickSettledRestoreCandidateOn":
+				settlePicker = true
+			case "PickRestoreCandidateOn":
+				oldPicker = true
+			}
+		case *ast.KeyValueExpr:
+			key, ok := node.Key.(*ast.Ident)
+			if !ok {
+				return true
+			}
+			if key.Name == "Settle" {
+				settleWired = true
+			}
+		case *ast.CallExpr:
+			if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
+				evalInterval = true
+			}
+		}
+		return true
+	})
+
+	if !settlePicker {
+		t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
+			"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
+	}
+	if oldPicker {
+		t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
+			"two pickers means the one under test is not the one running")
+	}
+	if !settleWired {
+		t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
+			"and every freshly-landed archive would be an immediate candidate")
+	}
+	if !evalInterval {
+		t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
+			"the retired cadence knob")
+	}
+}
+
+// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
+// RUN, and the shared heavy-operation gate is still handed to the scheduler.
+func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var inFlightWired, specIsAFunc bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		kv, ok := n.(*ast.KeyValueExpr)
+		if !ok {
+			return true
+		}
+		key, ok := kv.Key.(*ast.Ident)
+		if !ok {
+			return true
+		}
+		switch key.Name {
+		case "InFlight":
+			inFlightWired = true
+		case "Spec":
+			// A FuncLit means it is evaluated per run; anything else is a frozen value.
+			if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
+				specIsAFunc = true
+			}
+		}
+		return true
+	})
+
+	if !inFlightWired {
+		t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
+			"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
+	}
+	if !specIsAFunc {
+		t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
+			"(the tier and its timeout evaluated once at daemon start, forever)")
+	}
+}
+
+func parseMainForWiring(t *testing.T) *ast.File {
+	t.Helper()
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+	return f
+}
+
+// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
+//
+// This test exists because the method it feeds is the project's own cautionary tale:
+// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
+// was written and **had no caller at all** — a seam built, documented and never connected, found
+// only when a live restore-test's PASS reached no host-report. The fix must not become the next
+// instance, so the wiring is asserted rather than trusted.
+//
+// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
+// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
+func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var wired, feedsState bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		sel, ok := call.Fun.(*ast.SelectorExpr)
+		if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
+			return true
+		}
+		wired = true
+		// ...and it must be fed the PERSISTED state, not the in-memory store.
+		if len(call.Args) == 1 {
+			if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
+				feedsState = true
+			}
+		}
+		return true
+	})
+
+	if !wired {
+		t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
+			"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
+	}
+	if wired && !feedsState {
+		t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
+			"that does NOT survive a restart, so wiring it here would fix nothing")
+	}
+}

cmd/felhom-agent/seam_witnesses.go

@@ -0,0 +1,28 @@
+package main
+
+import (
+	"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
+)
+
+// COMPILE-TIME WITNESSES for OPTIONAL interfaces that are satisfied by a RUNTIME type assertion.
+//
+// WHY THIS FILE EXISTS. `localapi.BackupArchiveLister` is asserted at server.go's `newestArchiveOn`
+// via `tier.Service.(BackupArchiveLister)`. A failed assertion does not error — it degrades to
+// `archiveAbsent`, i.e. the pre-R-84 "ask the in-memory record only" behaviour. That degrade is
+// SILENT and it is behaviour-relevant: it is exactly the R-84 bug (a cold store after a restart
+// reading as "no backup ever") coming back, with nothing in any log to say so.
+//
+// The precedent is not hypothetical. During R-88 Part 2 the controller's `quiesceBackend` stopped
+// satisfying `quiesce.TieredBackend` when a signature changed, and `go build` AND `go vet` both
+// passed — because the interface is only ever asserted at runtime. Every box would have degraded to
+// the single-tier path, losing R-82's multi-tier backups, with no error anywhere. It was caught by
+// accident.
+//
+// A witness costs one line and converts that class of failure from a silent production degrade into
+// a compile error.
+//
+// THIS DOES NOT MAKE THE INTERFACE REQUIRED. The optionality is deliberate — it is what lets a
+// BackupService without a lister still work. The witness pins the IMPLEMENTATION (this concrete type
+// really does satisfy it), not the CONTRACT.
+var _ localapi.BackupArchiveLister = (*backup.BackupRunner)(nil)

cmd/felhom-agent/storegrant_test.go

@@ -0,0 +1,417 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"go/ast"
+	"io"
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
+)
+
+// R-185 — a tier the box cannot READ must say so.
+//
+// THE OBSERVATION (demo-felhom, 2026-08-03, reproduced at the start of this session): root lists
+// three archives on `felhom-backup`; the agent's own token gets `{"data":[]}` from the same
+// endpoint; and `local`, which has the grant, lists through that same token. The token is the
+// variable, not the storage.
+//
+// The defect is NOT the missing grant — that is one command. It is that an empty content listing is
+// what a FORBIDDEN tier and a NEWBORN tier both return, so the box could not tell them apart and
+// said nothing. These tests pin the distinction.
+
+// permAnswer is the shape /access/permissions really returns, taken from the live measurement:
+// an UNGRANTED path answers with the privileges inherited from the box-wide grant — NOT empty, and
+// NOT a 403.
+var (
+	permGranted   = map[string]int{"Datastore.Allocate": 1, "Datastore.AllocateSpace": 1}
+	permUngranted = map[string]int{"Sys.Audit": 1, "SDN.Use": 1, "Datastore.Audit": 1}
+)
+
+// probeWith calls the PRODUCTION decision with a permissions answer. **Naming the seam:** everything
+// below is true up to `storeGrantVerdict`; that the live call feeds it the real API answer is what
+// Part 0's measurement established and what the live run on the box demonstrates. An earlier draft
+// of this file re-implemented the branch here — it passed, and would have kept passing while
+// production diverged, which is the hollow shape this project keeps catching in its own tests.
+func probeWith(privs map[string]int, targetID string, critical bool) capability.Status {
+	return storeGrantVerdict(targetID, critical, privs, nil)
+}
+
+// ── SCENARIO A — a forbidden storage is REPORTED, not passed over ────────────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-03): delete the store-grant probes from `probeAll` in
+// main.go — i.e. restore `append(capProber.Probe(ctx), poolReadStatus(ctx, px))` — and
+// TestMainWiresTheStoreGrantProbe fails with "main.go never calls storeGrantStatuses". That is
+// today's behaviour on the live box: complete silence about a tier it cannot read.
+func TestStoreGrant_ForbiddenStorageIsDegradedAndNamed(t *testing.T) {
+	s := probeWith(permUngranted, "felhom-backup", true)
+
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("a storage the agent may not read must be DEGRADED, not %q — silence is the defect", s.Status)
+	}
+	if !s.Critical {
+		t.Fatal("it must be CRITICAL: the hub alerts only on critical, so a non-critical entry is the same silence with extra steps")
+	}
+	if !strings.Contains(s.Reason, "felhom-backup") {
+		t.Fatalf("the reason must NAME the storage — 'a grant is missing' costs a diagnosis at 07:00; got %q", s.Reason)
+	}
+	if !strings.Contains(s.Reason, "FelhomAgentStore") {
+		t.Fatalf("the reason must name the ROLE to grant, so the fix is in the alert; got %q", s.Reason)
+	}
+}
+
+// THE TRAP THE LIVE MEASUREMENT CAUGHT, pinned so it cannot be re-introduced: the ungranted answer
+// is not empty and not a 403 — it carries the INHERITED box-wide privileges. A probe that asked
+// "did the path come back?" or "does it have Datastore.Audit?" would report the blinded storage
+// healthy.
+func TestStoreGrant_InheritedPrivilegesAreNotAGrant(t *testing.T) {
+	if len(permUngranted) == 0 {
+		t.Fatal("fixture wrong: the ungranted answer is NOT empty — that is the whole trap")
+	}
+	if permUngranted["Datastore.Audit"] != 1 {
+		t.Fatal("fixture wrong: the ungranted path DOES carry Datastore.Audit, inherited box-wide")
+	}
+	if s := probeWith(permUngranted, "felhom-backup", true); s.Status != capability.StatusDegraded {
+		t.Fatalf("checking for the wrong privilege reports a blinded storage healthy; got %q", s.Status)
+	}
+	// ...and the privilege actually checked is the one whose absence was measured to blind listing.
+	if storeGrantRequiredPriv != "Datastore.AllocateSpace" {
+		t.Fatalf("the probed privilege changed to %q — re-measure before trusting it", storeGrantRequiredPriv)
+	}
+}
+
+// ── SCENARIO B — a newborn tier is still silent ──────────────────────────────────────────────
+//
+// A storage the agent IS allowed to read but which simply holds no archives yet is HEALTHY. The
+// probe must not look at content at all, or every freshly provisioned box alarms and the signal dies.
+//
+// COMPANION RED-PROOF (observed): make the probe degrade on an empty content listing instead of on
+// the permission — a granted-but-empty storage then reports degraded, i.e. every newborn box alarms.
+func TestStoreGrant_GrantedButEmptyIsHealthy(t *testing.T) {
+	s := probeWith(permGranted, "felhom-pbs", true)
+	if s.Status != capability.StatusOK {
+		t.Fatalf("a readable tier is healthy whether or not it holds archives yet; got %q (%s)", s.Status, s.Reason)
+	}
+	if s.Reason != "" {
+		t.Fatalf("a healthy probe carries no reason; got %q", s.Reason)
+	}
+}
+
+// ── SCENARIO C — the two states are distinguishable at a glance ──────────────────────────────
+func TestStoreGrant_ForbiddenAndNewbornAreDistinguishable(t *testing.T) {
+	forbidden := probeWith(permUngranted, "felhom-backup", true)
+	newborn := probeWith(permGranted, "felhom-pbs", true)
+
+	if forbidden.Status == newborn.Status {
+		t.Fatalf("the two states must differ — today both read as 'no settled archive yet'; got %q for both", forbidden.Status)
+	}
+	if forbidden.Name == newborn.Name {
+		t.Fatalf("each tier needs its own capability id, or one tier's fault hides another's; got %q twice", forbidden.Name)
+	}
+}
+
+// §8.3, weighed once and pinned: a box with NO dedicated target ("local" — host-install's own
+// DEGRADED fallback) must not turn an ordinary configuration into an operator page. It is still
+// probed and still reported; only the paging differs.
+func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
+	if storeGrantCritical("local") {
+		t.Fatal("a box whose backup target is the 'local' fallback must not page the operator about " +
+			"an ordinary, documented configuration")
+	}
+	for _, dedicated := range []string{"felhom-backup", "felhom-pbs", "some-nvme"} {
+		if !storeGrantCritical(dedicated) {
+			t.Fatalf("a DEDICATED target that cannot be read is user-facing and must be critical; %q was not", dedicated)
+		}
+	}
+	// The fallback is still reported — silence for it would be the original defect, scoped smaller.
+	if s := probeWith(permUngranted, "local", storeGrantCritical("local")); s.Status != capability.StatusDegraded {
+		t.Fatalf("the fallback target must still report degraded when unreadable; got %q", s.Status)
+	}
+}
+
+// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
+// probe, because it looks like coverage.
+func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
+	s := storeGrantStatus(context.Background(), nil, "felhom-backup", true, nil)
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
+	}
+	if s.Reason == "" {
+		t.Fatal("it must say why it could not ask")
+	}
+}
+
+// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
+//
+// This project's "built but never wired" count reached six last week. The fix for a SILENCE must not
+// itself be silent. AST, not grep: a commented-out call still contains the string.
+func TestMainWiresTheStoreGrantProbe(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var wired bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" {
+			wired = true
+		}
+		return true
+	})
+	if !wired {
+		t.Error("main.go never calls storeGrantStatuses — the probe would exist and report to nobody, " +
+			"which is precisely the silence R-185 is about")
+	}
+}
+
+// ── R-190 — the grant repairs itself, and the repair is VISIBLE ──────────────────────────────
+//
+// R-190 is a storage grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24,
+// with a host reinstall, logged `pveum` activity and cluster-log entries all ruled out. The cause is
+// open; the resilience is not conditional on it.
+//
+// The half that matters is the RECORD. R-190's own words: the probe sees the state, nothing sees the
+// transition. A self-repair that leaves only "ok" behind destroys the only evidence a loss happened,
+// so a recurring loss becomes undetectable forever — strictly worse than the fault it fixes.
+
+// fakeRepairRunner records wrapper invocations and can be made to fail.
+type fakeRepairRunner struct {
+	calls [][]string
+	fail  bool
+}
+
+func (f *fakeRepairRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	f.calls = append(f.calls, append([]string{name}, args...))
+	if f.fail {
+		return nil, []byte("pveum: refused"), errors.New("exit status 2")
+	}
+	return nil, nil, nil
+}
+
+func newRepairer(f *fakeRepairRunner) *storeGrantRepairer {
+	return &storeGrantRepairer{run: f.Run, log: slog.New(slog.NewTextHandler(io.Discard, nil))}
+}
+
+// ── SCENARIO F — the repair is BOUNDED ───────────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-04): make mayAttempt always return true (drop the
+// storeGrantRepairMinInterval check) →
+//
+//	--- FAIL: TestGrantRepair_IsBounded
+//	    storegrant_test.go: a repair must not run on every cycle; 5 cycles produced 5 attempt(s)
+//
+// which is a re-grant every report cycle, forever, against a fault an ACL cannot fix. Restored.
+func TestGrantRepair_IsBounded(t *testing.T) {
+	f := &fakeRepairRunner{}
+	r := newRepairer(f)
+	// Jittered, so the series never lands exactly on the interval boundary — a perfectly regular
+	// series is how a threshold test passes its own mutation, which has happened here before.
+	base := time.Date(2026, 8, 4, 9, 17, 43, 0, time.UTC)
+	offsets := []time.Duration{0, 13*time.Minute + 7*time.Second, 27*time.Minute + 51*time.Second,
+		41*time.Minute + 19*time.Second, 55*time.Minute + 3*time.Second}
+	attempts := 0
+	for _, off := range offsets {
+		if r.mayAttempt("felhom-backup", base.Add(off)) {
+			attempts++
+		}
+	}
+	if attempts != 1 {
+		t.Fatalf("a repair must not run on every cycle; %d cycles produced %d attempt(s) within %s",
+			len(offsets), attempts, storeGrantRepairMinInterval)
+	}
+	// ...and once the interval has genuinely passed, it may try again — a bound is not a ban.
+	if !r.mayAttempt("felhom-backup", base.Add(storeGrantRepairMinInterval+2*time.Minute+11*time.Second)) {
+		t.Fatal("after the interval a repair must be allowed again — otherwise one failure disables the repair forever")
+	}
+	// A DIFFERENT tier is not throttled by this one's attempt.
+	if !r.mayAttempt("felhom-pbs", base.Add(time.Minute)) {
+		t.Fatal("the bound must be per tier — one tier's attempt must not suppress another's")
+	}
+}
+
+// A nil repairer (or one with no runner) never attempts, and never panics.
+func TestGrantRepair_NilIsSafe(t *testing.T) {
+	var r *storeGrantRepairer
+	if r.mayAttempt("felhom-backup", time.Now()) {
+		t.Fatal("a nil repairer must never claim an attempt")
+	}
+	if (&storeGrantRepairer{}).mayAttempt("felhom-backup", time.Now()) {
+		t.Fatal("a repairer with no runner must never claim an attempt")
+	}
+}
+
+// The repair calls the EXISTING wrapper verb, with the storage id — no new privileged surface.
+func TestGrantRepair_CallsTheExistingWrapperVerb(t *testing.T) {
+	f := &fakeRepairRunner{}
+	r := newRepairer(f)
+	if err := r.repair(context.Background(), "felhom-backup"); err != nil {
+		t.Fatalf("repair should succeed with a healthy runner: %v", err)
+	}
+	if len(f.calls) != 1 {
+		t.Fatalf("exactly one wrapper invocation expected; got %d", len(f.calls))
+	}
+	got := f.calls[0]
+	want := []string{"/usr/local/sbin/felhom-backup-target-apply", "grant", "felhom-backup"}
+	if len(got) != len(want) {
+		t.Fatalf("wrapper argv = %v, want %v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("wrapper argv = %v, want %v — the sudoers vector is `grant *`; anything else is a policy change", got, want)
+		}
+	}
+}
+
+// A repair that FAILS must surface the failure, not swallow it (Scenario E's precondition).
+func TestGrantRepair_FailureIsReturned(t *testing.T) {
+	f := &fakeRepairRunner{fail: true}
+	if err := newRepairer(f).repair(context.Background(), "felhom-backup"); err == nil {
+		t.Fatal("a failed wrapper run must return its error — a repair that cannot run must never read as done")
+	}
+}
+
+// ── SCENARIO D (the half that matters) — the REPAIR MUST BE VISIBLE ──────────────────────────
+//
+// A repair that leaves only "ok" behind is worse than the fault: the tier works, and the fact that a
+// permission vanished is gone with it. R-190 exists because nothing saw the transition.
+//
+// The channel is the hub's EXISTING ok→degraded→ok edge (§8.5) — nothing new was built. That only
+// works if the agent deliberately reports ONE degraded cycle after repairing, and if the explanation
+// rides the field the hub actually puts in the operator's e-mail. The hub's message is built from the
+// capability NAME and FEATURE (`internal/monitor/host_capability.go` emitTransition) — **not** from
+// Reason — so the Feature must carry it.
+//
+// COMPANION RED-PROOF (observed 2026-08-04): after a successful repair, report ok instead —
+//
+//	s.Status = capability.StatusOK; s.Feature unchanged
+//
+// →	--- FAIL: TestGrantRepair_ARepairedGrantIsReportedAsATransition
+//
+//	storegrant_test.go: a self-repair must still report DEGRADED for one cycle so the hub raises
+//	its edge; got "ok" — the loss would be invisible
+//
+// i.e. exactly the silence R-190 is about. Restored.
+func TestGrantRepair_ARepairedGrantIsReportedAsATransition(t *testing.T) {
+	// THE PRODUCTION verdict, not a copy of it. An earlier draft of this test built the Status
+	// itself and asserted its own construction — it would have passed while production reported ok,
+	// which is precisely the silence being guarded against.
+	if pre := probeWith(permUngranted, "felhom-backup", true); pre.Status != capability.StatusDegraded {
+		t.Fatalf("precondition: a missing grant is degraded; got %q", pre.Status)
+	}
+	s := storeGrantRepairedVerdict("felhom-backup", true)
+
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("a self-repair must still report DEGRADED for one cycle so the hub raises its edge; "+
+			"got %q — the loss would be invisible", s.Status)
+	}
+	// The hub e-mails the FEATURE text. If the explanation is not there, the operator is told a
+	// capability was degraded and never learns it repaired itself or that anything vanished.
+	for _, want := range []string{"MISSING", "RESTORED", "felhom-backup", "R-190"} {
+		if !strings.Contains(s.Feature, want) {
+			t.Fatalf("the Feature text is what the hub puts in the operator's e-mail; it must contain %q. Got: %s", want, s.Feature)
+		}
+	}
+	if !s.Critical {
+		t.Fatal("the transition must be CRITICAL or the hub does not alert on it at all")
+	}
+}
+
+// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
+//
+// The wrapper's `grant` verb is itself a "built but never wired" example: it exists, is
+// sudoers-permitted for any id, and had only ever been called at storage CREATION. The repair must
+// not become the seventh instance. AST, not grep — a commented-out call still contains the string.
+func TestMainWiresTheGrantRepair(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var built, passed bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.CompositeLit:
+			if id, ok := node.Type.(*ast.Ident); ok && id.Name == "storeGrantRepairer" {
+				built = true
+			}
+		case *ast.CallExpr:
+			if id, ok := node.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" && len(node.Args) == 4 {
+				if a, ok := node.Args[3].(*ast.Ident); ok && a.Name == "grantRepairer" {
+					passed = true
+				}
+			}
+		}
+		return true
+	})
+	if !built {
+		t.Error("main.go never constructs a storeGrantRepairer — nothing would ever repair a lost grant")
+	}
+	if !passed {
+		t.Error("storeGrantStatuses is not passed the repairer — the probe would detect the loss and " +
+			"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
+	}
+}
+
+// The transition must survive a probe that is NOT the one feeding the hub.
+//
+// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
+// production while every unit test passed: `probeAll` is called independently by the self-check LOG
+// and by the collector building a host-report. The repairing call was the log's; the report three
+// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
+// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
+//
+// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
+//
+//	--- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
+//	    storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
+//	    the host-report would carry ok and the operator would never learn the grant vanished
+//
+// Restored.
+func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
+	r := newRepairer(&fakeRepairRunner{})
+	// Jittered, never landing on the window boundary.
+	repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
+	r.noteRepaired("felhom-backup", repairedAt)
+
+	// The DECISION a later probe makes — the production function, not the helper it calls. An
+	// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
+	// the latch's USE left the helper untouched.
+	healthy := probeWith(permGranted, "felhom-backup", true)
+	if healthy.Status != capability.StatusOK {
+		t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
+	}
+	got := storeGrantHealthyVerdict("felhom-backup", true,
+		healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
+	if got.Status != capability.StatusDegraded {
+		t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
+			"would carry ok and the operator would never learn the grant vanished", got.Status)
+	}
+	if !strings.Contains(got.Feature, "RESTORED") {
+		t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
+	}
+	// Outside the window it reports plain ok again.
+	late := storeGrantHealthyVerdict("felhom-backup", true,
+		healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
+	if late.Status != capability.StatusOK {
+		t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
+			"would be its own false alarm", late.Status)
+	}
+	if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
+		t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
+	}
+	// ...and it clears on its own rather than latching a box degraded forever.
+	if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
+		t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
+	}
+	// It is per tier.
+	if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
+		t.Fatal("one tier's repair must not latch another tier's status")
+	}
+	// The window MUST exceed the report interval — the property, asserted rather than assumed.
+	if storeGrantRepairReportWindow <= 15*time.Minute {
+		t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
+			"be missed entirely", storeGrantRepairReportWindow)
+	}
+}

cmd/felhom-opsign/main.go

@@ -0,0 +1,222 @@
+// Command felhom-opsign is the OPERATOR's offline signing CLI for destructive ops (slice 10B,
+// decision (a): offline operator key + signing CLI, hardware-key-ready).
+//
+// It constructs the canonical OpBlob bytes by REUSING internal/authz.CanonicalBlob — the exact
+// production path the agent's verifier authenticates over — so signer and verifier can never drift.
+// It signs that canonical message with the operator's key via `ssh-keygen -Y sign`, which makes it
+// hardware-ready: an `sk-`/YubiKey key works through ssh-keygen unchanged. The output is a signed-op
+// envelope { op_blob_b64, sig_armored } to hand to the hub's jobs queue (optionally uploaded with
+// --upload). This CLI touches ONLY the operator's signing key — never the hub's or agent's keys.
+//
+// Example — sign a data-bearing wipe (closing the 8C pending_signature gap):
+//
+//	felhom-opsign -op storage_wipe -host demo-felhom-01 \
+//	   -durable-id byid:wwn-0x5000c500abcd1234 -fstype ext4 \
+//	   -key-id ops-key-1 -key ~/.ssh/felhom_op_ed25519 -ttl 30m
+package main
+
+import (
+	"bytes"
+	"crypto/rand"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/authz"
+)
+
+func main() {
+	if err := run(); err != nil {
+		fmt.Fprintln(os.Stderr, "felhom-opsign:", err)
+		os.Exit(1)
+	}
+}
+
+func run() error {
+	var (
+		op        = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
+		host      = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
+		guest     = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
+		keyID     = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
+		paramsRaw = flag.String("params", "", "op params as JSON (overrides -durable-id/-fstype)")
+		durableID = flag.String("durable-id", "", "storage_wipe: the DURABLE device id (byid:…|byuuid:…); decommission: the drive's STORAGE durable-id (e.g. uuid:<fs-uuid>)")
+		fstype    = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
+		agentVer  = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
+		sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
+		keyFile   = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
+		ttl       = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
+		nonce     = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
+		uploadURL = flag.String("upload", "", "optional hub base URL to POST the signed op to the jobs queue")
+		hubKey    = flag.String("hub-key", "", "hub bearer key for --upload (operator's hub key — NOT a signing key)")
+		out       = flag.String("o", "", "write the envelope JSON to this file (default: stdout)")
+	)
+	flag.Parse()
+
+	if *op == "" || *host == "" || *keyID == "" || *keyFile == "" {
+		return fmt.Errorf("-op, -host, -key-id and -key are required")
+	}
+
+	// Params: explicit JSON, or built from the convenience flags.
+	params := strings.TrimSpace(*paramsRaw)
+	if params == "" {
+		switch *op {
+		case "storage_wipe":
+			if *durableID == "" {
+				return fmt.Errorf("storage_wipe needs -durable-id (byid:…|byuuid:…) — a path-only binding is refused by the agent")
+			}
+			pj, _ := json.Marshal(map[string]string{"durable_id": *durableID, "fstype": *fstype})
+			params = string(pj)
+		case "decommission":
+			// Decommission binds to the drive's STORAGE durable-id (the watchdog's key, e.g.
+			// "uuid:<fs-uuid>"), NOT the device-level byid:/byuuid: scheme. The agent records this
+			// id into the intent map, so it must match what the storage observer reports.
+			if *durableID == "" {
+				return fmt.Errorf("decommission needs -durable-id (the drive's storage durable-id, e.g. uuid:<fs-uuid>)")
+			}
+			pj, _ := json.Marshal(map[string]string{"durable_id": *durableID})
+			params = string(pj)
+		case "agent_update":
+			// The agent downloads the binary for -agent-version and verifies it against -sha256.
+			// The sha is the ONLY integrity root, so both are mandatory and the sha is strict-validated.
+			if *agentVer == "" || *sha256Hex == "" {
+				return fmt.Errorf("agent_update needs -agent-version and -sha256 (the pinned binary hash)")
+			}
+			if !isHex64(*sha256Hex) {
+				return fmt.Errorf("agent_update -sha256 must be 64 lowercase hex chars (got %d)", len(*sha256Hex))
+			}
+			pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
+			params = string(pj)
+		default:
+			params = "{}"
+		}
+	}
+
+	n := *nonce
+	if n == "" {
+		var b [16]byte
+		if _, err := rand.Read(b[:]); err != nil {
+			return fmt.Errorf("generating nonce: %w", err)
+		}
+		n = hex.EncodeToString(b[:])
+	}
+
+	now := time.Now().UTC()
+	issued := now
+	expires := now.Add(*ttl)
+
+	// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+	blob, err := authz.CanonicalBlob(*op, *host, *guest, *keyID, n, params, issued, expires)
+	if err != nil {
+		return err
+	}
+
+	sigArmored, err := signWithSSHKeygen(blob, *keyFile)
+	if err != nil {
+		return err
+	}
+
+	env := map[string]string{
+		"op_blob_b64": base64.StdEncoding.EncodeToString(blob),
+		"sig_armored": sigArmored,
+	}
+	envJSON, _ := json.Marshal(env)
+
+	// Emit the envelope (stdout or file). Also print the human summary to stderr (never the key).
+	fmt.Fprintf(os.Stderr, "signed: op=%s host=%s guest=%q key_id=%s nonce=%s expires=%s\n",
+		*op, *host, *guest, *keyID, n, expires.Format(time.RFC3339))
+	if *out != "" {
+		if err := os.WriteFile(*out, envJSON, 0o600); err != nil {
+			return err
+		}
+		fmt.Fprintln(os.Stderr, "wrote envelope to", *out)
+	} else {
+		fmt.Println(string(envJSON))
+	}
+
+	if *uploadURL != "" {
+		if *hubKey == "" {
+			return fmt.Errorf("--upload needs --hub-key (the operator's hub bearer key)")
+		}
+		if err := upload(*uploadURL, *hubKey, *host, envJSON); err != nil {
+			return fmt.Errorf("upload to hub: %w", err)
+		}
+		fmt.Fprintln(os.Stderr, "uploaded signed op to the hub jobs queue")
+	}
+	return nil
+}
+
+// isHex64 reports whether s is exactly 64 lowercase hex chars (a sha256 hex digest).
+func isHex64(s string) bool {
+	if len(s) != 64 {
+		return false
+	}
+	for _, c := range s {
+		if (c < '0' || c > '9') && (c < 'a' || c > 'f') {
+			return false
+		}
+	}
+	return true
+}
+
+// signWithSSHKeygen signs `message` with `ssh-keygen -Y sign -n <namespace>`, the hardware-ready
+// path (sk-/YubiKey keys work unchanged). It writes the message to a temp file, runs ssh-keygen,
+// and reads the armored SSHSIG it produces. The namespace is the agent's FIXED domain separator.
+func signWithSSHKeygen(message []byte, keyFile string) (string, error) {
+	dir, err := os.MkdirTemp("", "felhom-opsign-")
+	if err != nil {
+		return "", err
+	}
+	defer os.RemoveAll(dir)
+	msgPath := filepath.Join(dir, "op.blob")
+	if err := os.WriteFile(msgPath, message, 0o600); err != nil {
+		return "", err
+	}
+	cmd := exec.Command("ssh-keygen", "-Y", "sign", "-n", authz.Namespace, "-f", keyFile, msgPath)
+	var stderr bytes.Buffer
+	cmd.Stderr = &stderr
+	if err := cmd.Run(); err != nil {
+		return "", fmt.Errorf("ssh-keygen -Y sign: %w: %s", err, strings.TrimSpace(stderr.String()))
+	}
+	sig, err := os.ReadFile(msgPath + ".sig")
+	if err != nil {
+		return "", fmt.Errorf("reading produced signature: %w", err)
+	}
+	return string(sig), nil
+}
+
+// upload POSTs the signed-op envelope to the hub's jobs queue (POST /api/v1/admin/hosts/{id}/jobs).
+// The queued blob is base64(envelope JSON); the hub stores it opaquely (it cannot forge or open it).
+func upload(baseURL, hubKey, hostID string, envJSON []byte) error {
+	body, _ := json.Marshal(map[string]string{
+		"blob_b64": base64.StdEncoding.EncodeToString(envJSON),
+	})
+	// 10A's enqueue lives under /admin/hosts/{id}/jobs (operator/global key). The queued blob is
+	// base64(envelope JSON); the hub stores it opaquely.
+	url := strings.TrimRight(baseURL, "/") + "/api/v1/admin/hosts/" + hostID + "/jobs"
+	req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Authorization", "Bearer "+hubKey)
+	req.Header.Set("Content-Type", "application/json")
+	hc := &http.Client{Timeout: 15 * time.Second}
+	resp, err := hc.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
+	}
+	return nil
+}

cmd/felhom-opsign/main_test.go

@@ -0,0 +1,24 @@
+package main
+
+import "testing"
+
+// TASK D1 Group D — agent_update opsign param validation. isHex64 is the sha gate the CLI applies
+// before it will build an agent_update envelope (the agent re-validates too, but a bad sha should
+// never even be signed).
+func TestIsHex64(t *testing.T) {
+	good := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" // 64 'a'
+	if !isHex64(good) {
+		t.Errorf("isHex64(%q) = false, want true", good)
+	}
+	for name, bad := range map[string]string{
+		"too short":     "abcdef",
+		"too long":      good + "a",
+		"uppercase hex": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+		"non-hex char":  "g" + good[1:],
+		"empty":         "",
+	} {
+		if isHex64(bad) {
+			t.Errorf("%s: isHex64(%q) = true, want false", name, bad)
+		}
+	}
+}

configs/agent.example.json

@@ -0,0 +1,63 @@
+{
+  "proxmox": {
+    "endpoint": "https://127.0.0.1:8006",
+    "node": "demo-felhom",
+    "token": "felhom-agent@pve!agent=REPLACE_WITH_SECRET",
+    "tls": {
+      "ca_file": "",
+      "fingerprint": "BA:7C:99:7D:45:D0:67:91:E2:F2:72:74:6E:D6:9F:83:51:D1:61:E5:C3:BD:F6:A0:B8:0B:E3:D8:DB:89:5B:CF",
+      "insecure_skip_verify": false
+    }
+  },
+  "privileged": {
+    "mode": "sudo",
+    "sudo_path": "sudo"
+  },
+  "authz": {
+    "nonce_store_path": "/var/lib/felhom-agent/nonces.log",
+    "signers": [
+      {
+        "key_id": "felhom-op-1",
+        "role": "operational",
+        "public_key": "ssh-ed25519 AAAA... felhom-op-1"
+      },
+      {
+        "key_id": "felhom-recovery-1",
+        "role": "recovery",
+        "public_key": "ssh-ed25519 AAAA... felhom-recovery-1"
+      }
+    ]
+  },
+  "hub": {
+    "url": "https://hub.felhom.eu",
+    "host_id": "demo-host-01",
+    "api_key": "REPLACE_WITH_PER_HOST_HUB_KEY",
+    "poll_seconds": 900,
+    "timeout_seconds": 30,
+    "ca_file": ""
+  },
+  "backup": {
+    "local_backup_target": "local",
+    "restore_storage": "local-lvm",
+    "restore_test_cadence_seconds": 0,
+    "scratch_vmid_min": 990000,
+    "scratch_vmid_max": 990009,
+    "pbs_verify_cadence_seconds": 0,
+    "pbs_secret_dir": "/etc/pve/priv/storage",
+    "backup_cadence_seconds": 0
+  },
+  "local_api": {
+    "enable": true,
+    "listen_addr": "169.254.253.1:8443",
+    "cert_file": "/var/lib/felhom-agent/local-api.crt",
+    "key_file": "/var/lib/felhom-agent/local-api.key",
+    "token_store": "/var/lib/felhom-agent/local-tokens.log",
+    "island_bridge": "vmbr9",
+    "island_guest_addr": "169.254.253.2/30"
+  },
+  "lan_resolver": {
+    "enable": true,
+    "host_ip": "192.168.0.162"
+  },
+  "log_level": "info"
+}

configs/build-golden.sh

@@ -0,0 +1,451 @@
+#!/usr/bin/env bash
+# build-golden.sh — build the Felhom golden base LXC archive (slice 7).
+#
+# Produces a minimal Debian + Docker, unprivileged, nesting=1,keyctl=1, overlayfs LXC, baked
+# identity-clean, and archives it for a token-restore by the bring-up reconcile job
+# (internal/reconcile/bringup.go). Run as root@pam on a Proxmox host (the keyctl=1 feature flag
+# is root-only — phase3 #1; this is the ONE root step, off the per-customer path).
+#
+# Grounded by documentation/tests/slice7-bringup-spike-findings.md (commit 3342993):
+#   - F3: removing the SSH host keys does NOT auto-regenerate them on Debian (pct restore runs no
+#     keygen hook), so a baked, Condition-gated first-boot unit regenerates them — keeping the
+#     agent's front half host-side-only. The gate (ConditionPathExists=!…) makes it fire on a
+#     provision (golden, keys absent) and no-op on a DR restore (customer backup, keys present),
+#     symmetric with machine-id.
+#   - machine-id: truncated; systemd regenerates it on first boot for free (no unit needed).
+#
+# Slice 8A — the golden now also BAKES the in-guest controller (decision: image baked at golden
+# build on the trusted host, so NO registry credential ever enters a customer guest at deploy) and
+# a controller-bootstrap unit that, on boot, deploys the baked image from the agent-populated
+# config mount (/etc/felhom-bootstrap/bootstrap.json) — no docker login/pull at deploy. Refreshing
+# the golden bumps the controller baseline; controller self-update covers in-between drift.
+#
+# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE
+#        CONTROLLER_IMAGE is REQUIRED (no default) — pass the released controller tag explicitly,
+#        e.g. gitea.dooplex.hu/admin/felhom-controller:0.98.3.
+# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
+# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
+#
+# OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
+#
+# The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
+# NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
+#
+#     /var/lib/felhom/docker     --bind-->  /var/lib/docker    (Docker's data-root)
+#     /var/lib/felhom/sys_drive  --bind-->  /mnt/sys_drive     (the controller's system_data_path)
+#
+# WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
+# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
+# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
+# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
+#
+# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
+# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
+# a different documented guarantee:
+#   * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
+#     stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
+#     destroys every local recovery unit on the box;
+#   * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
+#     container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
+#     /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
+#     felhom-data-namespace mounts" would become false.
+# The neutral mount breaks neither, for one extra path and one extra fstab line.
+#
+# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
+# agent-recoverable if the data volume fills (the controller's prevention layer, and since
+# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
+# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
+# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
+# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
+set -euo pipefail
+
+# Script provenance — logged into every bake transcript next to the baked controller tag, so an
+# archive can always be traced to the script that produced it. Bump on any behavior change.
+GOLDEN_SCRIPT_VERSION="3.0.0"
+
+VMID="${1:-9100}"
+TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
+ROOTFS_STORAGE="${3:-local-lvm}"
+ARCHIVE_STORAGE="${4:-local}"
+BRIDGE="${5:-vmbr0}"
+# CONTROLLER_IMAGE is MANDATORY — no default. The hand-bumped default rotted twice (0.43.0 →
+# 0.85.1 → stale again): each time, a fresh provision would have booted a pre-floor controller
+# needing a manual install-day update (drill finding B5). A required argument cannot rot, and
+# auto-resolving "latest" could bake a tag the hub manifest never vouched — so the caller states
+# the released controller tag explicitly at every rebuild.
+CONTROLLER_IMAGE="${6:-}"
+if [ -z "$CONTROLLER_IMAGE" ]; then
+  echo "[golden] FATAL: CONTROLLER_IMAGE (argument 6) is required — pass the released controller tag explicitly." >&2
+  echo "Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE" >&2
+  echo "  e.g.: build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3" >&2
+  exit 1
+fi
+REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
+# OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
+#
+# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
+# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
+# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
+# per-customer target (bringup.go DataVolGrowGB).
+OS_SIZE_GB="${OS_SIZE_GB:-32}"
+# 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
+# did before the merge. It is deliberately NOT a per-customer size: provision grows it.
+GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
+# The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
+GOLDEN_VOLUME_MP="/var/lib/felhom"
+
+echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
+echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
+# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
+# whole point of R-165 is that it stops existing rather than being made bigger.
+pct create "$VMID" "$TEMPLATE" \
+  --hostname felhom-golden --unprivileged 1 \
+  --features nesting=1,keyctl=1 \
+  --rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
+  --mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
+  --net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
+
+echo "[golden] starting + installing Docker (official repo, trixie channel) …"
+pct start "$VMID"
+# wait for DHCP/DNS
+for i in $(seq 1 30); do
+  if pct exec "$VMID" -- getent hosts download.docker.com >/dev/null 2>&1; then break; fi
+  sleep 1
+done
+pct exec "$VMID" -- bash -c '
+  set -e
+  export DEBIAN_FRONTEND=noninteractive
+  apt-get update -qq
+  apt-get install -y -qq ca-certificates curl >/dev/null
+  install -m0755 -d /etc/apt/keyrings
+  curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
+  echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable" \
+    > /etc/apt/sources.list.d/docker.list
+  apt-get update -qq
+  apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
+'
+echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
+# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
+# /var/lib/containerd — which is NOT /var/lib/docker, so it would stay on the OS rootfs and the split
+# would only move named volumes, defeating the whole point (validated: images landed on the rootfs).
+# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
+# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
+# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
+# DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
+# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
+# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
+# still needs no override because the path is unchanged. Log caps kill the most common runaway.
+pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
+{
+  "features": { "containerd-snapshotter": false },
+  "log-driver": "json-file",
+  "log-opts": { "max-size": "10m", "max-file": "3" }
+}
+JSON'
+echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
+# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
+# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
+# round would hide those files under the bind and silently ship a golden whose baked images are on the
+# rootfs — the exact failure class the assertions below exist to catch.
+#
+# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
+# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
+# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
+pct exec "$VMID" -- bash -c "
+  set -e
+  systemctl stop docker docker.socket containerd 2>/dev/null || true
+  mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
+  if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
+    cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
+    rm -rf /var/lib/docker/*
+  fi
+  mkdir -p /var/lib/docker /mnt/sys_drive
+  printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker'    >> /etc/fstab
+  printf '%s /mnt/sys_drive none bind 0 0\n'  '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
+  systemctl daemon-reload
+  mount /var/lib/docker
+  mount /mnt/sys_drive
+  systemctl start containerd
+"
+
+echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
+# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
+# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
+pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm hello-world >/dev/null && echo "  docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"'
+# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
+# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
+pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
+# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
+# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
+# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
+# its baked images there.
+pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo "  /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
+# ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
+# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
+# (clearing that warning is the whole point of the volume).
+pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo "  /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
+# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
+# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
+# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
+# there, and a shared pool neither `df` can see coming.
+pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo "  both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
+
+echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
+# docker login is used ONCE here on the trusted build host, then logged out before archiving so
+# the credential is NEVER baked into the golden. The IMAGE is what gets baked (in Docker storage).
+if [ -n "${REGISTRY_USER:-}" ] && [ -n "${REGISTRY_TOKEN:-}" ]; then
+  pct exec "$VMID" -- bash -c "systemctl start docker; sleep 1; echo '$REGISTRY_TOKEN' | docker login '$REGISTRY_HOST' -u '$REGISTRY_USER' --password-stdin >/dev/null"
+fi
+pct exec "$VMID" -- bash -c "docker pull '$CONTROLLER_IMAGE'"
+# Record the baked image ref for the bootstrap unit (so the unit needs no login/pull).
+pct exec "$VMID" -- bash -c "printf '%s\n' '$CONTROLLER_IMAGE' > /etc/felhom-controller-image"
+
+# Bake EVERY controller-managed infra image so enabling an infra stack on a fresh box is near-instant
+# instead of a multi-minute silent registry pull.
+#
+# THE LIST COMES FROM THE CONTROLLER WE JUST PULLED, not from a copy here. This script used to carry
+# a hand-maintained array of three tags with a comment telling the reader to keep it in sync with the
+# controller's internal/infra constants. It drifted the moment a fourth stack was added: felhom-samba
+# was never added here, so the golden baked 3 of 4 and turning on Megosztás pulled from the registry
+# with zero UI feedback — observed live, twice. Asking the binary (`--print-infra-images`, backed by
+# infra.Images()) makes golden-vs-controller drift structurally impossible: the golden bakes exactly
+# what THIS controller version will request.
+echo "[golden] asking the controller which infra images it manages …"
+INFRA_LIST="$(pct exec "$VMID" -- bash -c "docker run --rm --entrypoint /usr/local/bin/felhom-controller '$CONTROLLER_IMAGE' --print-infra-images 2>/dev/null" || true)"
+mapfile -t INFRA_IMAGES < <(printf '%s\n' "$INFRA_LIST" | grep -E '^[a-z0-9._/-]+:[A-Za-z0-9._-]+$' || true)
+if [ "${#INFRA_IMAGES[@]}" -eq 0 ]; then
+  # Controllers older than v0.147.0 have no --print-infra-images. Fall back to the historical list so
+  # an old-controller bake still works — but say so loudly, because the fallback is exactly the
+  # drift-prone thing this change removed, and it CANNOT know about stacks added after it was written.
+  echo "[golden] WARN: '$CONTROLLER_IMAGE' does not support --print-infra-images (pre-0.147.0)."
+  echo "[golden] WARN: falling back to the historical 3-image list. felhom-samba will NOT be baked,"
+  echo "[golden] WARN: so enabling Megosztás on this golden will pull at runtime. Bake >=0.147.0."
+  INFRA_IMAGES=(
+    "traefik:v3.6.7"
+    "cloudflare/cloudflared:2026.6.0"
+    "gtstef/filebrowser:1.3.3-stable"
+  )
+fi
+echo "[golden] baking infra images (${#INFRA_IMAGES[@]}): ${INFRA_IMAGES[*]} …"
+for img in "${INFRA_IMAGES[@]}"; do
+  # Hard gate: fail loudly BEFORE pulling if a pinned tag doesn't resolve (a bad pin otherwise fails
+  # mid-bake with a confusing error).
+  pct exec "$VMID" -- bash -c "docker manifest inspect '$img' >/dev/null 2>&1" \
+    || { echo "[golden] FATAL: pinned infra image does not resolve: $img"; exit 1; }
+  pct exec "$VMID" -- bash -c "docker pull '$img'"
+done
+
+# Log out LAST — not right after the controller pull. felhom-samba lives on the same private registry
+# as the controller, so the infra loop above needs the credential; logging out first made that pull
+# 401. The credential is still never archived: the logout + config.json removal happen before the
+# guest is stopped and templated.
+pct exec "$VMID" -- bash -c "docker logout '$REGISTRY_HOST' >/dev/null 2>&1 || true; rm -f /root/.docker/config.json"
+pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
+  || { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
+
+echo "[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …"
+pct push "$VMID" /dev/stdin /usr/local/sbin/felhom-controller-bootstrap.sh --perms 700 <<'BOOTSH'
+#!/bin/bash
+# felhom controller-bootstrap (slice 8A): the host agent's back-half populated the read-only
+# config mount /etc/felhom-bootstrap; this golden-baked oneshot deploys the BAKED controller image
+# with that config. NO docker login / NO docker pull — the image is already in this golden's Docker
+# storage (and self-update handles version drift). Host-side only; the agent never enters the guest.
+set -euo pipefail
+CFG=/etc/felhom-bootstrap/bootstrap.json
+[ -r "$CFG" ] || { echo "[ctrl-bootstrap] no $CFG — not provisioned, nothing to do"; exit 0; }
+IMAGE=$(cat /etc/felhom-controller-image 2>/dev/null || true)
+[ -n "$IMAGE" ] || { echo "[ctrl-bootstrap] FATAL: /etc/felhom-controller-image missing"; exit 1; }
+# Per-guest container hostname (slice base-infra/3A): derive from the bootstrap's customer.id so the
+# controller's os.Hostname() (its hub-reported hostname) is the customer id, not the Docker container
+# ID. Portable, dependency-free parse (NO jq in the golden) — bootstrap.json has exactly one "id" key
+# (customer.id). Falls back to no --hostname if the parse yields nothing (fail-safe).
+CUSTOMER_ID=$(sed -n 's/.*"id"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$CFG" | head -1)
+# SECURITY: $CUSTOMER_ID is interpolated into `docker run` — reject anything that isn't a DNS-safe
+# label so a malformed/hostile customer.id can't smuggle extra docker flags (e.g. --privileged, -v).
+# Then pass it via a quoted array (never word-split). Empty/invalid → no --hostname (fail-safe).
+case "$CUSTOMER_ID" in
+  ""|*[!a-zA-Z0-9._-]*|-*) CUSTOMER_ID="" ;;
+esac
+HOSTNAME_ARGS=()
+[ -n "$CUSTOMER_ID" ] && HOSTNAME_ARGS=(--hostname "$CUSTOMER_ID")
+echo "[ctrl-bootstrap] deploying $IMAGE from $CFG (hostname=${CUSTOMER_ID:-<unset>})"
+docker rm -f felhom-controller >/dev/null 2>&1 || true
+# Section-G fix (base-infra slice): the controller writes app/infra compose stacks under
+# /opt/docker/stacks INSIDE the container, but `docker compose up` is executed by the GUEST daemon
+# (shared socket), which resolves every relative bind source on the GUEST filesystem. Without a
+# SAME-PATH host bind for /opt/docker/stacks, the daemon can't see those bind sources and silently
+# creates empty dirs → every bind-mounted stack (base infra AND customer apps) breaks. A named volume
+# would NOT fix this (it resolves to /var/lib/docker/volumes/...). Pre-create the dir, then same-path bind.
+mkdir -p /opt/docker/stacks
+# slice 10 P2: the controller must SEE enrolled external data drives, which the agent binds into the
+# guest at /mnt/<name>. Make /mnt a SHARED mount so those binds (and later host-remount self-heal)
+# propagate into the controller container, then bind /mnt :rslave (host->container propagation ONLY —
+# the container can't mutate the guest mount tree). This is the ONE bind the 8C de-privileging left
+# out; scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the
+# customer's other on-drive data. rbind preserves existing submounts; re-running re-shares (safe).
+mkdir -p /mnt
+mountpoint -q /mnt || mount --rbind /mnt /mnt
+mount --make-rshared /mnt
+# Otherwise still DE-PRIVILEGED: disk EXECUTION (scan/format/mount) stays the agent's — NO --privileged,
+# no /dev, no /etc/fstab. Bootstrap config (ro), data volume, stacks dir (same-path), the /mnt :rslave
+# view, and the docker socket. The controller reaches the agent's local API for disk management.
+# R-523: `always`, not `unless-stopped`. It covers ONE extra case only — a Docker daemon restart after
+# the container was stopped by hand. Neither policy restarts a container that `docker kill`/`docker
+# stop` ended (measured 2026-09-15, Docker 29.8.0, evidence-p1fixes-2026-09-15/A1); the host agent's
+# controller supervisor (felhom-agent v0.131.0, internal/localapi/controllersupervisor.go) covers that.
+docker run -d --name felhom-controller --restart always "${HOSTNAME_ARGS[@]}" \
+  -e FELHOM_BOOTSTRAP_PATH=/etc/felhom-bootstrap/bootstrap.json \
+  -v /etc/felhom-bootstrap:/etc/felhom-bootstrap:ro \
+  -v felhom-controller-data:/opt/docker/felhom-controller \
+  -v /opt/docker/stacks:/opt/docker/stacks \
+  -v /mnt:/mnt:rslave \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  "$IMAGE"
+echo "[ctrl-bootstrap] controller started"
+BOOTSH
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-controller-bootstrap.service <<UNIT
+[Unit]
+Description=Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)
+After=docker.service network-online.target
+Wants=docker.service network-online.target
+ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/felhom-controller-bootstrap.sh
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-controller-bootstrap.service'
+
+# B1 (DRILL-day0-cleanroom-2026-07-03 R6/B1): the service's ConditionPathExists is evaluated only
+# when the service is STARTED — i.e. at boot via multi-user.target — but the agent back-half
+# hot-plugs the bootstrap mount into the ALREADY-RUNNING guest, so on a provision the boot-time
+# start races the mount and loses on slow hardware. This path unit watches for bootstrap.json and
+# starts the service when it APPEARS — covering the provision hot-plug without a reboot. The boot
+# case is still served by the enabled service itself; RemainAfterExit=yes on the service means the
+# path unit cannot re-trigger it in a loop once it has run.
+echo "[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …"
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-controller-bootstrap.path <<UNIT
+[Unit]
+Description=Watch for the agent-populated bootstrap config; start the controller bootstrap when it appears
+
+[Path]
+PathExists=/etc/felhom-bootstrap/bootstrap.json
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-controller-bootstrap.path'
+
+echo "[golden] baking the first-boot SSH host-key regeneration unit (F3) …"
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-regen-hostkeys.service <<UNIT
+[Unit]
+Description=Regenerate SSH host keys on first boot if absent
+ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
+DefaultDependencies=no
+After=local-fs.target
+Before=ssh.service sshd.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/ssh-keygen -A
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-regen-hostkeys.service'
+
+echo "[golden] identity-clean + minimize …"
+pct exec "$VMID" -- bash -c '
+  systemctl stop docker containerd 2>/dev/null || true
+  apt-get clean; rm -rf /var/lib/apt/lists/*
+  rm -f /etc/ssh/ssh_host_*            # regenerated on first boot by the baked unit (F3)
+  truncate -s 0 /etc/machine-id        # systemd regenerates on first boot (free)
+  rm -f /var/lib/dbus/machine-id; ln -sf /etc/machine-id /var/lib/dbus/machine-id
+  rm -rf /var/log/*; : > /root/.bash_history
+  rm -f /etc/hostname                  # set per-guest at provision (host-side token config)
+'
+
+echo "[golden] stop + archive …"
+pct stop "$VMID"
+# --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
+# INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
+# exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
+# Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
+# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
+vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
+if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
+  exit 1
+fi
+# ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
+# archive". After the merge there is no mp1 — so the same failure now looks like the volume being
+# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
+# inclusion line names the volume's actual mount path.
+if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
+  grep -iE "mount point" /tmp/golden-vzdump.log || true
+  exit 1
+fi
+# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
+# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
+if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
+  exit 1
+fi
+grep -q "including mount point mp0" /tmp/golden-vzdump.log \
+  || echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
+
+VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
+echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
+
+#-------------------------------------------------------------------------------
+# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
+#-------------------------------------------------------------------------------
+# The host-install script fetches the golden from Gitea
+# (/api/packages/admin/generic/felhom-golden/<golden-version>/golden.tar.zst) and verifies its sha256
+# against the hub-vouched artifact manifest before importing it. <golden-version> = the BAKED
+# controller version (so the golden's published version tracks what it ships). Publishing is OPT-IN:
+# only runs when GITEA_USER + GITEA_TOKEN (or REGISTRY_USER/REGISTRY_TOKEN) are set. The local-golden
+# auto-discovery in the host-install script stays as a fallback either way.
+GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
+GITEA_OWNER="${GITEA_OWNER:-admin}"
+PUB_USER="${GITEA_USER:-${REGISTRY_USER:-}}"
+PUB_TOKEN="${GITEA_TOKEN:-${REGISTRY_TOKEN:-}}"
+# Golden version = the baked controller tag's version (strip the image path + any leading 'v').
+GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"; GOLDEN_VERSION="${GOLDEN_VERSION#v}"
+
+if [ -z "$VOLID" ]; then
+  echo "[golden] WARN: could not resolve the archive volid — skipping Gitea publish."
+elif [ -z "$PUB_USER" ] || [ -z "$PUB_TOKEN" ]; then
+  echo "[golden] Gitea publish SKIPPED (set GITEA_USER+GITEA_TOKEN or REGISTRY_USER+REGISTRY_TOKEN to enable)."
+  echo "[golden]   would publish version=$GOLDEN_VERSION from volid $VOLID"
+else
+  # Resolve the archive's on-disk path (pvesm path turns a volid into a filesystem path).
+  ARCHIVE_PATH="$(pvesm path "$VOLID" 2>/dev/null || true)"
+  if [ -z "$ARCHIVE_PATH" ] || [ ! -f "$ARCHIVE_PATH" ]; then
+    echo "[golden] WARN: cannot resolve archive path for $VOLID — skipping publish."
+  else
+    GOLDEN_SHA256="$(sha256sum "$ARCHIVE_PATH" | awk '{print $1}')"
+    PUB_URL="${GITEA_BASE}/api/packages/${GITEA_OWNER}/generic/felhom-golden/${GOLDEN_VERSION}/golden.tar.zst"
+    echo "[golden] publishing golden ($(wc -c < "$ARCHIVE_PATH") bytes, sha256 ${GOLDEN_SHA256:0:16}…) → $PUB_URL"
+    # Delete-then-PUT so re-publishing the same version overwrites cleanly (idempotent).
+    dcode="$(curl -fsS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X DELETE "$PUB_URL" 2>/dev/null || true)"
+    echo "[golden]   pre-delete existing: HTTP ${dcode} (404/204 expected)"
+    ucode="$(curl -sS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X PUT --upload-file "$ARCHIVE_PATH" "$PUB_URL")"
+    if [ "$ucode" = "201" ] || [ "$ucode" = "200" ]; then
+      echo "[golden]   upload OK (HTTP $ucode)"
+      echo "GOLDEN_VERSION=${GOLDEN_VERSION}"
+      echo "GOLDEN_SHA256=${GOLDEN_SHA256}"
+      echo "[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden ${GOLDEN_VERSION} / ${GOLDEN_SHA256}"
+    else
+      echo "[golden] WARN: golden upload failed (HTTP $ucode) — the local archive is still usable via auto-discovery."
+    fi
+  fi
+fi
+
+echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"

configs/felhom-agent-limits.conf

@@ -0,0 +1,19 @@
+# felhom-agent-limits.conf — start-limit + rollback-trigger drop-in for felhom-agent.service
+# (TASK D1). Install as /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf
+# and `systemctl daemon-reload`.
+#
+# Values are the SPIKE-agent-selfupdate-2026-07-05 tuned set, verbatim [SF-2]: with the unit's
+# Restart=on-failure + RestartSec=5s and systemd 257's compiled defaults (10s/5), a crash-looping
+# binary NEVER trips the start limit and loops forever (35 starts/180s measured). 120s/4 gives a
+# terminal `failed` ≈20s after the first crash.
+#
+# PLACEMENT TRAP [SF-3]: these keys MUST be in [Unit]. On systemd 257 a [Service] placement is
+# HALF-APPLIED — StartLimitBurst is accepted while StartLimitIntervalSec is silently ignored
+# (journal warning only). Never split them; never put them in [Service].
+#
+# OnFailure fires on EVERY crash on systemd 257 [SF-1] — see the comment block in
+# felhom-agent-rollback.service for why that is safe (pending-marker guard).
+[Unit]
+StartLimitIntervalSec=120
+StartLimitBurst=4
+OnFailure=felhom-agent-rollback.service

configs/felhom-agent-rollback.service

@@ -0,0 +1,21 @@
+# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping agent
+# self-update (TASK D1; SPIKE-agent-selfupdate-2026-07-05).
+#
+# Install as /etc/systemd/system/felhom-agent-rollback.service. It is referenced by the
+# felhom-agent-limits.conf drop-in's OnFailure= line.
+#
+# THE PER-CRASH-FIRING REALITY [SF-1]: on systemd 257, OnFailure= fires on EVERY crash of the main
+# unit — even while it is merely `activating (auto-restart)`, long before (and regardless of) the
+# terminal start-limit `failed` state. During one crash incident this unit therefore runs MANY
+# times. That is safe BY DESIGN: the wrapper's rollback verb is pending-marker-guarded — the first
+# fire after a bad update reverts the binary and clears the marker; every later fire (and any fire
+# with no update in flight at all) is a logged no-op that touches nothing. Consequence: a bad
+# update is rolled back at the FIRST crash (~seconds), not after the start-limit burst — the tuned
+# start-limit in the drop-in is the terminal BACKSTOP (e.g. an environmental crash loop of the
+# known-good binary), not the rollback trigger.
+[Unit]
+Description=Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/felhom-selfupdate-guarded rollback

configs/felhom-agent.service

@@ -0,0 +1,50 @@
+# felhom-agent.service — canonical systemd unit for the Felhom host agent.
+#
+# Install as /etc/systemd/system/felhom-agent.service (the host-bootstrap script installs this from
+# the Gitea-published binary; previously this unit was hand-made on each host). The agent runs as the
+# NON-root `felhom-agent` service user (the documented production model — README "Process model";
+# `privileged.mode: "sudo"`) and shells the few host-root ops out via `sudo -n` against the fixed-arg
+# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the
+# sudoers `felhom-agent ALL=(root) NOPASSWD: …` MUST name the SAME user.
+#
+# Paths MUST match what the sudoers / host-install script expect:
+#   binary  /usr/local/bin/felhom-agent
+#   config  /etc/felhom-agent/agent.json   (0600 felhom-agent:felhom-agent — secrets live here)
+#   state   /var/lib/felhom-agent          (nonces, local-api cert/key/tokens, staged units, guests)
+#
+# === DELIBERATELY NO SANDBOXING — read before adding any hardening directive ===
+#
+# 1. NoNewPrivileges is NOT set. It is INCOMPATIBLE with the agent's privilege model: it blocks the
+#    setuid `sudo` the agent relies on for EVERY host-root op (mount, format, pct, dnsmasq …), so the
+#    agent would silently lose all privileged capability. The narrow surface comes from the sudoers
+#    fixed-arg allowlist + the agent's in-process fine validation (internal/storage/validate.go), NOT
+#    from NoNewPrivileges. Do not add it.
+#
+# 2. NO mount-namespacing hardening (ProtectHome, ProtectSystem, PrivateTmp, ReadOnlyPaths,
+#    ProtectControlGroups, …). Any of these give the unit a PRIVATE mount namespace — and the agent's
+#    intermediary-mount drive model does `mount --make-shared /mnt/felhom-drives` + `mount --bind` and
+#    relies on those propagating into the RUNNING customer guest. In a private namespace the binds
+#    would be invisible to the host/guest and every external-drive enrollment would silently break.
+#    The agent MUST share the host mount namespace. The security boundary is the sudoers allowlist.
+
+[Unit]
+Description=Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)
+Documentation=https://gitea.dooplex.hu/admin/felhom-agent
+After=network-online.target pve-cluster.service pveproxy.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=felhom-agent
+Group=felhom-agent
+ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
+Restart=on-failure
+RestartSec=5s
+
+# State dir: created 0750 felhom-agent:felhom-agent on start if absent (local-api cert/key/tokens,
+# nonces, staged .mount units, per-guest bootstrap dirs all live here).
+StateDirectory=felhom-agent
+StateDirectoryMode=0750
+
+[Install]
+WantedBy=multi-user.target

configs/felhom-agent.sudoers

@@ -0,0 +1,308 @@
+# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7).
+#
+# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with
+# `visudo -cf`. The agent runs as the non-root `felhom-agent` service user and shells out via
+# `sudo -n` with FIXED argument vectors (no shell). The fine-grained validation is done IN
+# the agent BEFORE exec (internal/storage/validate.go): UUIDs against a strict hex regex,
+# mount paths confined+traversal-checked, SMART devices whitelisted to raw disks, LVM names
+# charset-checked. These sudoers wildcards are the COARSE allowlist; the agent is the fine
+# gate, so a wildcard can never be abused by a value the agent didn't already validate.
+#
+# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). Adjust
+# for your distro (Debian/PVE shown). A missing/declined entry degrades the agent with a
+# warning (SMART→UNKNOWN, mount→logged error), it does not crash.
+
+Cmnd_Alias FELHOM_MOUNT = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.mount, \
+  /usr/bin/systemctl daemon-reload, \
+  /usr/bin/systemctl enable --now -- *.mount, \
+  /usr/bin/systemctl disable -- *.mount, \
+  /usr/bin/systemctl stop -- *.mount
+
+Cmnd_Alias FELHOM_DISK = \
+  /usr/sbin/smartctl -a -j /dev/sd[a-z]*, \
+  /usr/sbin/smartctl -a -j /dev/nvme[0-9]*n[0-9]*, \
+  /usr/sbin/smartctl -a -j /dev/vd[a-z]*, \
+  /usr/sbin/smartctl -a -j /dev/hd[a-z]*, \
+  /usr/sbin/lvs --reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- *, \
+  /usr/sbin/pvs --reportformat json --noheadings -o pv_name, \
+  /usr/sbin/zpool status -P
+
+# Provisioning back-half (slice 8A, doc 03 §6): populate a guest's bootstrap config mount
+# host-side (internal/provision). These are host-root ops the API token cannot do — a bind mount
+# is root@pam-only, and the chown maps the 0600 bootstrap.json to the unprivileged-LXC guest-root
+# (uid/gid 100000, spike gotcha 1). The host dir is AGENT-OWNED state under /var/lib/felhom-agent/
+# (the wildcard only ever names a path the agent itself created), and the bootstrap file the agent
+# writes there is the only thing these touch. ':' is escaped per sudoers grammar.
+Cmnd_Alias FELHOM_PROVISION = \
+  /usr/bin/chown -R 100000\:100000 /var/lib/felhom-agent/guests/*, \
+  /usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*, \
+  /usr/sbin/pct set [0-9]* -onboot 1
+
+# Disk inspection + format (slice 8C + Impl-1). blkid/lsblk read the device's data-bearing evidence
+# (the agent decides data-bearing-ness from THIS, never the caller's claim). Format goes ONLY through
+# felhom-mkfs-guarded (Impl-1 Part B): raw mkfs.* is NO LONGER allowlisted, so even a bad agent cannot
+# mkfs the OS disk — the wrapper re-checks the catastrophic cases (system disk / LVM PV / foreign mount)
+# as root and refuses, and the agent's unclaimed-disk filter (claim.go) is the primary guard above it.
+Cmnd_Alias FELHOM_FORMAT = \
+  /usr/sbin/blkid -p -o export /dev/*, \
+  /usr/bin/lsblk -J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/*, \
+  /usr/local/sbin/felhom-mkfs-guarded /dev/* *
+
+# LAN split-horizon resolver (internal/lanresolver): the agent manages a host-side dnsmasq that
+# answers *.<customer-domain> with each guest's live LAN IP. install only ever writes felhom-*.conf
+# drop-ins (from agent-written /tmp temp files); the two `pct exec` reads are FIXED command vectors
+# (the guest's eth0 IPv4 + the controller's pulled controller.yaml for the domain) — NOT a general
+# `pct exec`. systemctl is scoped to the dnsmasq unit only. The agent never edits /etc/resolv.conf.
+Cmnd_Alias FELHOM_DNSMASQ = \
+  /usr/bin/apt-get install -y -q dnsmasq, \
+  /usr/bin/install -m 0644 /tmp/felhom-resolver-*.conf /etc/dnsmasq.d/felhom-*.conf, \
+  /usr/bin/systemctl enable --now dnsmasq, \
+  /usr/bin/systemctl reload dnsmasq, \
+  /usr/bin/systemctl restart dnsmasq, \
+  /usr/bin/rm -f /etc/dnsmasq.d/felhom-*.conf, \
+  /usr/sbin/pct exec [0-9]* -- ip -4 -o addr show dev eth0, \
+  /usr/sbin/pct exec [0-9]* -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
+
+# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook
+# wrapper is installed once into the PVE snippets dir (from an agent-written /tmp file) and registered
+# per-guest; decommission/eject DELETE the dead mountpoint slot so a missing bind source can't brick the
+# guest at next boot (the B3 C1 fix). The agent fine-validates the vmid (numeric) + slot (mp[0-9]+) and
+# the snippet path is fixed — the wildcards are the coarse allowlist. The install SOURCE is a
+# random-named agent temp (os.CreateTemp, audit B1 — a fixed /tmp name was a local TOCTOU), hence the
+# glob; the DESTINATION stays pinned. The `mkdir -p` creates the snippets dir on a FRESH box —
+# `install` won't create parents, so without it the hook install failed silently on Day-0 boxes
+# (B2, DRILL-day0-cleanroom-2026-07-03; fixed agent v0.63.0).
+Cmnd_Alias FELHOM_GUESTHOOK = \
+  /usr/bin/mkdir -p /var/lib/vz/snippets, \
+  /usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-*.sh /var/lib/vz/snippets/felhom-guest-hook.sh, \
+  /usr/sbin/pct set [0-9]* --hookscript local\:snippets/felhom-guest-hook.sh, \
+  /usr/sbin/pct set [0-9]* --delete mp[0-9]*, \
+  /usr/sbin/pct reboot [0-9]*
+
+# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
+# /mnt/felhom-drives (self-bind + make-shared + a boot-persistence systemd unit) and binds/unbinds each
+# drive's felhom-data namespace UNDERNEATH it so the change propagates into the running guest live (no
+# pct, no reboot). The agent fine-validates the drive name + confines paths before any exec; the trailing
+# `*` (matching the comma-laden mp spec) mirrors the existing FELHOM_PROVISION pattern.
+# `lxc-info -n <vmid> -p -H` resolves the guest init PID for the GuestSeesMount / bound_under_parent check
+# (a READ — the drive-gate's "is the drive live in the guest?" signal); WITHOUT it the non-root agent gets
+# an empty PID and reports every drive absent (multi-drive flapping, audit 2026-06-29). `make-private`
+# isolates the parent's peer group on FIRST setup only (EnsureSharedParent guards on mountpoint, so it
+# never re-churns a live parent); without it the parent stays in root's group and submounts double.
+Cmnd_Alias FELHOM_INTERMEDIARY = \
+  /usr/bin/mkdir -p /mnt/felhom-drives, \
+  /usr/bin/mkdir -p /mnt/felhom-drives/*, \
+  /usr/bin/mkdir -p /mnt/*/felhom-data, \
+  /usr/bin/chown 100000\:100000 /mnt/*/felhom-data, \
+  /usr/bin/mount --bind /mnt/felhom-drives /mnt/felhom-drives, \
+  /usr/bin/mount --make-shared /mnt/felhom-drives, \
+  /usr/bin/mount --make-private /mnt/felhom-drives, \
+  /usr/bin/mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*, \
+  /usr/bin/umount /mnt/felhom-drives/*, \
+  /usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-*.sh /usr/local/sbin/felhom-shared-parent.sh, \
+  /usr/bin/install -m 0644 -- /tmp/felhom-shared-parent-*.service /etc/systemd/system/felhom-shared-parent.service, \
+  /usr/bin/systemctl enable felhom-shared-parent.service, \
+  /usr/bin/lxc-info -n [0-9]* -p -H, \
+  /usr/sbin/pct set [0-9]* -mp8 /mnt/felhom-drives*
+
+# Controller-swap / managed auto-update (Option A, non-root). The agent owns the in-guest controller
+# image SWAP (it survives the controller being killed mid-swap): read the baked image ref, check the
+# pre-pulled target is present, rewrite /etc/felhom-controller-image, restart the bootstrap unit,
+# health-check, roll back on failure. Each grant is bounded — NO general `pct exec` and NO `bash -c`:
+#   cat <fixed file>            — read the current image ref (read-only)
+#   docker image inspect *      — is the pre-pulled target present? (read-only)
+#   docker inspect -f *         — container running/health/image (read-only; `*` spans the -f template
+#                                 + container across spaces, spike-confirmed)
+#   systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
+#   tee <FIXED image file>      — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
+#                                 the agent strict-validates the ref (controllerImageRe) before the write.
+# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
+Cmnd_Alias FELHOM_CONTROLLERSWAP = \
+  /usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image, \
+  /usr/sbin/pct exec [0-9]* -- docker image inspect *, \
+  /usr/sbin/pct exec [0-9]* -- docker inspect -f *, \
+  /usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service, \
+  /usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image
+
+# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
+# agent clears the STALE lock at startup (only when no vzdump is in-flight). `pct unlock` is the one op
+# with no API equivalent (snapshot-delete + start go through the API token); the agent fine-validates the
+# vmid (numeric) before exec — the `[0-9]*` is the coarse allowlist.
+Cmnd_Alias FELHOM_STALELOCK = \
+  /usr/sbin/pct unlock [0-9]*
+
+# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
+# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
+# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
+# out of reach (403 VM.Allocate) holding its disks until a human removes it.
+#
+# TWO API-SIDE FIXES WERE TRIED AND BOTH REFUTED LIVE on 2026-07-28, which is why this grant exists:
+#   1. Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
+#      VM.Allocate on the VM being added — pool membership cannot bootstrap its own authority.
+#   2. Grant FelhomAgentGuest per-path at /vms/990000..990009. Durable for exactly one use per slot:
+#      PVE's own destroy path calls `AccessControl::remove_vm_access($vmid)` (LXC.pm:906), which DELETES
+#      every ACL at /vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it
+#      authorises, so after ten teardowns the band is ungranted and the defect returns.
+#
+# WHY THIS IS THE TIGHTEST AVAILABLE FENCE, not a widening: sudo matches the vmid LITERALLY, so
+# `99000[0-9]` is exactly the ten-slot scratch band the restore-test picks from — nothing else. There is
+# no `[0-9]*` coarse allowlist here on purpose: unlike `pct unlock`, this op DESTROYS, so the band must
+# be in the policy and not merely validated in the agent. Even a compromised agent asking for
+# `pct destroy 9201` is refused by sudo itself. Unlike an ACL, a sudoers rule is not consumed by use.
+# The agent re-checks the band in code before exec (defence in depth); this is the outer fence.
+Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
+  /usr/sbin/pct destroy 99000[0-9] --purge
+
+# Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share
+# HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates
+# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
+# A NAS is NOT a drive — no durable-id, no SMART, no wipe; these grants only install/enable/remove the
+# unit pair. The agent fine-validates every value (share name, server, export, uid/gid, creds path) before
+# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the trailing globs are the
+# COARSE allowlist. The `.mount` install/enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
+# `.automount` variants + the unit-file removal. The unit FILE name is the systemd-escaped mountpoint,
+# which always begins `mnt-felhom` (the mountpoint is /mnt/felhom-drives/<name>), so the rm glob is scoped
+# to felhom mount units only. mkdir of the mountpoint reuses FELHOM_INTERMEDIARY's /mnt/felhom-drives/*.
+# CAMPAIGN-3 additions (loud, per the no-widening rule):
+#  - `systemctl reset-failed -- mnt-felhom*`: F10 (CRITICAL) — a NAS automount that hit
+#    mount-start-limit-hit during an outage was re-armable by NO platform path; the reassert now
+#    reset-failed's the stuck unit before `enable --now` (which the start-limit otherwise refuses),
+#    and RemoveNetworkMount clears failed-state residue (F2). Scoped to felhom mount units (the unit
+#    name is the systemd-escaped mountpoint, always beginning `mnt-felhom`). reset-failed only clears
+#    a unit's failed latch — it cannot start/stop/alter anything.
+#  - `rmdir /mnt/felhom-drives/*`: F1 — remove the now-empty mountpoint dir a removed share leaves
+#    behind (the campaign accumulated 10 stub-shaped leftovers). rmdir ONLY (never rm -rf): it refuses
+#    a non-empty dir, so unexpected data is preserved, not destroyed — a fail-safe grant.
+Cmnd_Alias FELHOM_NETMOUNT = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.automount, \
+  /usr/bin/systemctl enable --now -- *.automount, \
+  /usr/bin/systemctl disable -- *.automount, \
+  /usr/bin/systemctl stop -- *.automount, \
+  /usr/bin/systemctl reset-failed -- mnt-felhom*, \
+  /usr/bin/rmdir /mnt/felhom-drives/*, \
+  /usr/bin/rm -f /etc/systemd/system/mnt-felhom*
+
+# Offsite WG tunnel (S3, doc 06 §3.3). The agent manages wg-quick@wg-felhom as an agent-managed
+# host service (the dnsmasq/lanresolver shape): conf staged in the agent-owned StateDir (never
+# /tmp), installed 0600 to the FIXED destination, unit enable/restart/disable. The ONLY wg read
+# is `latest-handshakes` — `wg show <if> dump` is FORBIDDEN everywhere (its interface line
+# carries the PRIVATE KEY; the S1 session-log incident). Both install paths are FIXED (no glob):
+# the agent has exactly one tunnel conf to manage.
+Cmnd_Alias FELHOM_WG = \
+  /usr/bin/apt-get install -y -q wireguard-tools, \
+  /usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf, \
+  /usr/bin/systemctl enable --now wg-quick@wg-felhom, \
+  /usr/bin/systemctl restart wg-quick@wg-felhom, \
+  /usr/bin/systemctl disable --now wg-quick@wg-felhom, \
+  /usr/bin/wg show wg-felhom latest-handshakes
+
+# Agent self-update (TASK D1, SPIKE-agent-selfupdate-2026-07-05). The agent downloads the
+# operator-SIGNED binary (sha256 pinned in the signed op — neither hub nor Gitea compromise can
+# substitute it), verifies the sha in-process, then hands off to the guarded wrapper, which
+# RE-verifies the sha as root, confines the staged path to /var/lib/felhom-agent/selfupdate/,
+# performs the A/B flip (atomic same-fs rename, .prev retained) and schedules a detached restart.
+# The apply args are a COARSE glob (spike S4b: sudoers fnmatch makes a [a-f0-9]* sha pattern
+# first-char-only anyway) — the wrapper's own sha re-verify + path confinement is the real gate.
+# `rollback` is normally run by felhom-agent-rollback.service (root, OnFailure=), not via sudo;
+# granting it here keeps the verb probe-able (capability self-check) and operator-invokable.
+Cmnd_Alias FELHOM_SELFUPDATE = \
+  /usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/* *, \
+  /usr/local/sbin/felhom-selfupdate-guarded commit, \
+  /usr/local/sbin/felhom-selfupdate-guarded rollback
+
+# Dedicated OOB sshd (TASK H1). The agent manages felhom-sshd like wg-felhom/dnsmasq: it RENDERS the
+# config (Port from its claim) + the operator's authorized_keys, validates with `sshd -t`, and reloads
+# (never restart-on-change [SF-2]). Both install SOURCES are the agent-owned staged files under
+# StateDir; both DESTINATIONS are FIXED. `sshd -t/-T` are the validate/discover reads. The
+# systemctl verbs are SCOPED to felhom-sshd only. reset-failed precedes a deliberate restart [SF-5].
+# NOTHING here can touch the stock sshd, :22, or /etc/ssh.
+Cmnd_Alias FELHOM_SSHD = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config, \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op /etc/felhom-sshd/authorized_keys/felhom-op, \
+  /usr/sbin/sshd -t -f /var/lib/felhom-agent/felhom-sshd/sshd_config, \
+  /usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config, \
+  /usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config, \
+  /usr/bin/systemctl enable --now felhom-sshd, \
+  /usr/bin/systemctl reload felhom-sshd, \
+  /usr/bin/systemctl restart felhom-sshd, \
+  /usr/bin/systemctl reset-failed felhom-sshd, \
+  /usr/bin/wg show wg-felhom latest-handshakes
+
+# PBS DR tier apply (slice 2, SPIKE-pbs-tier-provisioning-2026-07-10 §2b). Storage-entry
+# lifecycle is /storage-ROOT-gated in the PVE API (spike Probe 1: create/modify/delete all check
+# Datastore.Allocate on /storage), so the agent token cannot do it — this wrapper is the pinned
+# vector. THE SET-ONLY LAW: the wrapper contains NO deletion path (entry deletion destroys the
+# client encryption key = un-decryptable backups); verbs are create/reconcile/grant only. The
+# token secret rides the wrapper's STDIN — sudo logs argv, so it must never appear here. The
+# agent fine-validates every field (charset + descriptor equality) before exec; these globs are
+# the coarse allowlist.
+#
+# `read` (R-39 leg b, agent v0.91.0) is the ONE added verb. It prints a token secret to stdout and
+# performs no mutation. It exists because the agent writes that file through this wrapper but could
+# never read it back (/etc/pve/priv is 0700 root:www-data), leaving its PBS verify loop permanently
+# blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory
+# and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to
+# STDOUT, never argv — sudo logs argv.
+# E-2a: the backup-target storage shim. Creating a PVE storage needs Datastore.Allocate at /storage
+# and the grant needs Permissions.Modify -- the agent holds NEITHER by design (blast-radius
+# containment; Permissions.Modify would let it rewrite its own authority). Both live behind this
+# fixed-vocabulary root shim instead, exactly like the mkfs and pbs-apply wrappers. The wrapper has
+# NO storage-removal path, enforces is_mountpoint 1, and refuses a target on the root device.
+Cmnd_Alias FELHOM_BACKUPTARGET = \
+  /usr/local/sbin/felhom-backup-target-apply create *, \
+  /usr/local/sbin/felhom-backup-target-apply grant *
+
+Cmnd_Alias FELHOM_PBSDR = \
+  /usr/local/sbin/felhom-pbs-apply create *, \
+  /usr/local/sbin/felhom-pbs-apply reconcile *, \
+  /usr/local/sbin/felhom-pbs-apply grant *, \
+  /usr/local/sbin/felhom-pbs-apply read *
+
+# OOB nft belt (TASK H1). The STATIC table `inet felhom_oob` is installed once by host-install; the
+# agent mutates ONLY its two SETS — @operator_ips (the operator /32) + @ssh_port (the claimed port).
+# SET ELEMENTS ONLY [trap 4]: NO `nft add rule`, NO `nft -f`, NO `flush ruleset/table` — a rule grant
+# would let the agent firewall anything. The agent fine-validates every element (netip / int range)
+# before exec; the trailing wildcards are the coarse allowlist (values only).
+Cmnd_Alias FELHOM_OOB = \
+  /usr/sbin/nft list set inet felhom_oob operator_ips, \
+  /usr/sbin/nft list set inet felhom_oob ssh_port, \
+  /usr/sbin/nft flush set inet felhom_oob operator_ips, \
+  /usr/sbin/nft flush set inet felhom_oob ssh_port, \
+  /usr/sbin/nft add element inet felhom_oob operator_ips *, \
+  /usr/sbin/nft add element inet felhom_oob ssh_port *
+
+# Escrow ceremony (controller-driven, TASK 2026-07-13; mechanics validated by
+# SPIKE-controller-escrow-2026-07-13). ONE fixed argv — sudoers matches the argument vector
+# byte-for-byte (spike §2.2): any alteration (value, extra flag, order, config path) is refused.
+# --config pinned: env_reset strips FELHOM_AGENT_CONFIG and the pin closes alternate-config
+# injection. R rides the subprocess stdout pipe only; sudo logs argv = secrets-free. The argv
+# MUST stay byte-identical to escrow.CeremonyArgs() (internal/escrow/ceremony.go) — the
+# capability manifest entry + TestManifestCoveredBySudoers lock the three copies together.
+Cmnd_Alias FELHOM_ESCROW = \
+  /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
+
+# Node self-heal (CAMPAIGN-3 Part 6, F12-class defense in depth). The ONE fixed unit the appliance
+# watchdog may (re)start when a boot leaves networking down — the exact command the morning recovery
+# ran by hand after the F12 host loss. FIXED unit, no glob: this grant alone cannot harm — starting
+# networking.service is precisely what the boot should have done. The remedy is ALSO code-gated on
+# deployment_mode="appliance" (the Manager refuses to invoke it on a byo host); the sudoers grant is
+# the coarse floor, the mode gate is the fine one.
+Cmnd_Alias FELHOM_SELFHEAL = \
+  /usr/bin/systemctl start networking.service
+
+# Guest-network watchdog (internal/guestnet, R-54). The guest's DHCP client is unsupervised — when it
+# died on 2026-07-20 the box lost its address ~80 minutes later and went off the internet for 1h15m
+# (INCIDENT-guest-dhclient-killed-2026-07-20). Four FIXED read vectors plus ONE fixed heal vector; the
+# heal is the incident's own restored invocation, byte for byte. This is NOT a general `pct exec`: every
+# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
+# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
+# and the same command must not be granted twice under two names.
+Cmnd_Alias FELHOM_GUESTNET = \
+  /usr/sbin/pct exec [0-9]* -- ip route show default, \
+  /usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
+  /usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
+  /usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
+
+felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN

configs/felhom-backup-target-apply

@@ -0,0 +1,111 @@
+#!/bin/bash
+#===============================================================================
+# felhom-backup-target-apply — the ONLY path the felhom-agent sudoers permits for creating the
+# whole-guest backup TARGET storage and granting the agent access to it (E-2a).
+#
+# WHY A WRAPPER AT ALL. Creating a PVE storage needs `Datastore.Allocate` at `/storage`, and the ACL
+# grant needs `Permissions.Modify`. The agent holds NEITHER by design — its token is scoped per
+# storage path for blast-radius containment, and `Permissions.Modify` would let it rewrite its own
+# authority. Widening the PVE role to make the move possible would trade the entire containment model
+# for one feature. So the privileged half lives here: a minimal, auditable root shim with a fixed
+# vocabulary, exactly like felhom-mkfs-guarded and felhom-pbs-apply.
+#
+# THE NO-DELETE LAW (inherited from felhom-pbs-apply, same reasoning class). This wrapper contains NO
+# storage-removal path of any kind. `pvesm remove` on a dir storage does not delete the archives, but
+# it DOES silently orphan a configured backup tier, and a "cleanup" verb here would be reachable by
+# any bug in the agent. Retiring a target is a deliberate operator op, not this tool. Grep-assertable;
+# do not add one.
+#
+# THE TWO LAWS E-1 PAID FOR ON LIVE HARDWARE, both enforced here rather than trusted to the caller:
+#
+#   F-1  the storage path must BE the drive's own mountpoint. A subdirectory fails the agent's
+#        exactMount check, so the target reports `disconnected` FOREVER and its durable id degrades
+#        off the filesystem UUID. Enforced: `mountpoint -q` must pass on the exact path given.
+#
+#   F-2  --is_mountpoint 1 is not optional. Without it, an unplugged or late-mounting drive leaves a
+#        bare directory on the ROOT filesystem and vzdump writes the whole-guest backup onto the
+#        system drive — the exact device the whole change exists to escape — while PVE reports the
+#        storage `active` and advertises the root filesystem's free space. Proven live: the unguarded
+#        form had already created dump/ on pve-root. Hardcoded below; not a caller-supplied flag.
+#
+# Ops (all non-secret; nothing here touches a credential, so nothing arrives on stdin):
+#   create <id> <mountpoint>
+#           Create a `dir` storage with content=backup at <mountpoint>, is_mountpoint 1.
+#           IDEMPOTENT: an existing entry with the SAME path is accepted (re-run safe, and the
+#           installer re-run path depends on it). An existing entry with a DIFFERENT path is REFUSED
+#           — silently repointing a live backup target is the failure this whole arc closes.
+#   grant  <id>
+#           The dual grant: FelhomAgentStore on /storage/<id> to the agent user AND token (privsep
+#           intersection — a token's rights are the intersection, so granting one is granting neither).
+#           Without it every backup 403s on first run (E-1 finding F-3, found by the first real backup).
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-backup-target-apply: REFUSED: $*" >&2; exit 1; }
+
+op="${1:-}"; id="${2:-}"
+[[ -n "$op" && -n "$id" ]] || die "usage: felhom-backup-target-apply <create|grant> <storage-id> [mountpoint]"
+
+# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
+[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
+
+STORECFG=/etc/pve/storage.cfg
+
+# current_path_of <id> — the configured `path` of dir storage <id>, or "" when absent/not-a-dir.
+current_path_of() {
+  awk -v want="dir: $1" '
+    $0 == want { found=1; next }
+    found && /^[a-z]+: / { exit }
+    found && $1 == "path" { print $2; exit }
+  ' "$STORECFG" 2>/dev/null || true
+}
+
+case "$op" in
+create)
+  [[ $# -eq 3 ]] || die "create takes <id> <mountpoint>"
+  mp="$3"
+  # Absolute, normalized, no traversal, no shell metacharacters. The value reaches pvesm and the
+  # filesystem, so it is validated here rather than assumed well-formed.
+  [[ "$mp" = /* ]]            || die "mountpoint must be absolute ($mp)"
+  [[ "$mp" != *".."* ]]       || die "mountpoint must not contain .. ($mp)"
+  [[ "$mp" =~ ^[A-Za-z0-9/_.-]+$ ]] || die "mountpoint has unexpected characters ($mp)"
+  [[ "$mp" != "/" ]]          || die "refusing / as a backup target"
+
+  # F-1 + F-2, checked as one: the path must BE a mountpoint right now. A bare directory here is
+  # precisely the silent-retarget shape, and is_mountpoint would make PVE refuse it later anyway —
+  # better to refuse now, with a reason, than to create a storage that can never activate.
+  mountpoint -q "$mp" || die "$mp is not a mountpoint — the backup target must be the drive's OWN mountpoint (F-1), and an unmounted path would silently retarget onto the system drive (F-2)"
+
+  # Never the system disk: a target on the root filesystem is not drive-loss protection, it is the
+  # thing we are escaping. The root device and the candidate's device are compared, not their paths.
+  root_dev="$(findmnt -no SOURCE / 2>/dev/null || true)"
+  mp_dev="$(findmnt -no SOURCE "$mp" 2>/dev/null || true)"
+  [[ -n "$mp_dev" ]] || die "could not resolve the backing device of $mp"
+  [[ "$mp_dev" != "$root_dev" ]] || die "$mp is backed by the ROOT device ($root_dev) — a backup target there protects against corruption only, never drive loss"
+
+  existing="$(current_path_of "$id")"
+  if [[ -n "$existing" ]]; then
+    if [[ "$existing" == "$mp" ]]; then
+      echo "felhom-backup-target-apply: storage $id already exists at $mp — nothing to do (idempotent)" >&2
+      exit 0
+    fi
+    die "storage $id already exists at $existing — refusing to repoint it at $mp (a live backup target is never silently moved)"
+  fi
+
+  # is_mountpoint 1 is HARDCODED (F-2). content=backup only: this storage exists for vzdump archives
+  # and must never become a place guests are allocated on.
+  pvesm add dir "$id" --path "$mp" --content backup --is_mountpoint 1 >&2
+  echo "felhom-backup-target-apply: created dir storage $id at $mp (content=backup, is_mountpoint 1)" >&2
+  ;;
+grant)
+  [[ $# -eq 2 ]] || die "grant takes only <id>"
+  # BOTH, always. A privsep token's rights are the intersection of the user's and the token's ACLs,
+  # so granting one of the two grants nothing usable.
+  pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
+  pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
+  echo "felhom-backup-target-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
+  ;;
+*)
+  die "unknown op ($op)"
+  ;;
+esac

configs/felhom-localapi-firewall.example

@@ -0,0 +1,49 @@
+# felhom-agent local API — host firewall narrowing (doc 03 §6; R-50 island update 2026-07-25)
+#
+# Defense-in-depth for the per-guest local API (the controller→agent channel). The PER-GUEST BEARER
+# TOKEN + the served-leaf pin are the authorization gate; a firewall rule is only an ADDITIONAL layer
+# limiting who can even open the port.
+#
+# ===  R-50 ISLAND INSTALL (the default on a fresh appliance)  =================================
+# The agent binds local_api.listen_addr on the HOST-INTERNAL island bridge — 169.254.253.1:8443 on
+# vmbr9, a bridge with NO physical port (bridge-ports none). That bind is the security win:
+#   * Nothing listens on the LAN IP at all, so no LAN host (or off-site attacker on the LAN) can
+#     reach the local API — the LAN:8443 surface is CLOSED by the bind, not by a rule.
+#   * vmbr9 has no uplink, so 169.254.253.1:8443 is reachable ONLY from the one guest wired to the
+#     /30 (169.254.253.2) — the controller. The portless bridge is the isolation.
+# So on an island install NO firewall rule is required for exposure; the topology provides it. If you
+# want belt-and-suspenders, restrict the port to the island bridge (it changes nothing, since nothing
+# off-bridge can route to a portless bridge anyway):
+#
+#   nft add rule inet filter input iifname != "vmbr9" ip daddr 169.254.253.1 tcp dport 8443 drop
+#
+# Verify: from the guest, a TLS connect to 169.254.253.1:8443 succeeds; there is no LAN listener to
+# probe (`ss -lnt 'sport = :8443'` shows only the island IP).
+#
+# ===  LEGACY LAN BIND (byo, --no-island, or an explicit --bridge-ip)  =========================
+# When the agent still binds a LAN bridge IP (e.g. 192.168.0.162:8443), the port is exposed to the
+# whole LAN and the subnet-narrowing rule below is worth applying. Replace the bridge IP, port, and
+# the guest bridge subnet with this host's values.
+#
+# Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept:
+#
+#   nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
+#       ip saddr != 192.168.0.0/24 drop
+#   nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
+#       ip saddr 192.168.0.0/24 accept
+#
+# Option B — iptables:
+#
+#   iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT
+#   iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP
+#
+# Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled.
+# Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw):
+#
+#   [RULES]
+#   IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog
+#   IN DROP   -dport 8443 -proto tcp -log nolog
+#
+# Apply at HOST SETUP — a host-level packet-filter change, intentionally OUTSIDE the agent's
+# 3-exception privileged fence (the agent never mutates the host firewall at runtime). The token +
+# leaf-pin still gate the request regardless of which bind is in force.

configs/felhom-mgmt-watchdog.service

@@ -0,0 +1,19 @@
+# felhom-mgmt-watchdog.service (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.service.
+#
+# LAYER 2 (auto-heal without login): a DUMB oneshot that runs /usr/local/sbin/felhom-mgmt-watchdog,
+# triggered every ~60s by felhom-mgmt-watchdog.timer. It recreates a missing /run/sshd and clears a
+# start-limited stock sshd — so a management-plane lockout self-corrects in ≤1 tick with NOBODY logged
+# in and WITHOUT the felhom-agent binary running (the agent only OBSERVES/reports; the heal must not
+# depend on it — SPIKE-felhom-sshd-2026-07-05 finding #9 / TASK G1 trap 1).
+#
+# CRITICAL: this unit MUST NOT declare `RuntimeDirectory=` — that directive (with value `sshd`) is the
+# incident it exists to fix. It writes only to /run (dir + heal marker) as root; no state dir needed.
+[Unit]
+Description=Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)
+# Ordering only — the timer drives cadence; no hard dependency so it runs even in degraded boots.
+After=systemd-tmpfiles-setup.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/felhom-mgmt-watchdog
+# Deliberately NO RuntimeDirectory=, NO PrivateTmp, NO sandboxing that would hide /run/sshd from it.

configs/felhom-mgmt-watchdog.sh

@@ -0,0 +1,68 @@
+#!/bin/sh
+# felhom-mgmt-watchdog — the DUMB, agent-INDEPENDENT management-plane healer (TASK G1).
+#
+# WHY THIS EXISTS: /run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by
+# every sshd on the host. If it goes missing, sshd's listener stays up but every new session RESETS
+# right after SSH2_MSG_KEXINIT — a management-plane lockout on an otherwise-healthy box
+# (SPIKE-felhom-sshd-2026-07-05 §8, caused live by a second unit's `RuntimeDirectory=sshd` cleanup).
+# The tmpfiles.d entry (felhom-privsep.tmpfiles) PREVENTS the known cause; this script AUTO-HEALS any
+# novel cause with ZERO login and ZERO dependency on the felhom-agent binary (so it self-corrects even
+# when the agent is down — the whole point). It is run by felhom-mgmt-watchdog.timer every ~60s.
+#
+# CONTRACT (all four are load-bearing):
+#   1. STAT-FIRST / IDEMPOTENT — a HEALTHY /run/sshd is NEVER touched (no mkdir/chmod/chown call), so
+#      steady state is zero mutation and the dir's mtime is preserved. Only a MISSING or
+#      wrong-mode/owner dir is corrected.
+#   2. reset-failed the STOCK sshd ONLY when it is in the `failed` state (a start-limit lockout) —
+#      never otherwise, and NEVER `restart` it (mkdir alone restores it; PID stays, spike-proven).
+#   3. HEAL MARKER — writes an RFC3339 UTC timestamp to $MARKER only on a real heal, so the agent can
+#      REPORT the condition to the hub (a recurring clobber must surface BEFORE it becomes a lockout).
+#   4. NO `RuntimeDirectory=` anywhere in this feature (that IS the incident cause) — the unit that
+#      runs this script must not declare one either.
+#
+# Touches host /run + the stock sshd unit ONLY — no guests (pool-scoping is moot here). Fail-safe:
+# any single step failing is logged and does not abort the others (`|| true`); the next tick retries.
+
+set -u
+
+PRIVSEP=/run/sshd
+MARKER=/run/felhom-mgmt-watchdog.healed
+SSHD_UNIT=ssh.service   # Debian/PVE stock sshd unit name (sshd.service is an alias)
+healed=0
+
+log() { logger -t felhom-mgmt-watchdog "$*" 2>/dev/null || true; }
+
+# 1. Privsep dir — stat-first: create only when missing; correct mode/owner only when wrong.
+if [ ! -d "$PRIVSEP" ]; then
+	if mkdir -p "$PRIVSEP" && chown root:root "$PRIVSEP" && chmod 0755 "$PRIVSEP"; then
+		healed=1
+		log "recreated missing privsep dir $PRIVSEP (0755 root:root)"
+	else
+		log "ERROR: failed to recreate $PRIVSEP"
+	fi
+else
+	mode=$(stat -c %a "$PRIVSEP" 2>/dev/null || echo "")
+	owner=$(stat -c %U:%G "$PRIVSEP" 2>/dev/null || echo "")
+	if [ "$mode" != "755" ]; then
+		chmod 0755 "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP mode ($mode -> 755)"
+	fi
+	if [ "$owner" != "root:root" ]; then
+		chown root:root "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP owner ($owner -> root:root)"
+	fi
+fi
+
+# 2. Stock sshd — clear a start-limit lockout ONLY when the unit is genuinely `failed`. Never restart
+#    it (unnecessary — a recreated privsep dir is picked up by the next fork; the listener never died).
+if systemctl is-failed --quiet "$SSHD_UNIT" 2>/dev/null; then
+	if systemctl reset-failed "$SSHD_UNIT" 2>/dev/null; then
+		healed=1
+		log "reset-failed $SSHD_UNIT (was in the failed state)"
+	fi
+fi
+
+# 3. Heal marker — only on a real heal, so the agent reports the condition (hub raises a warning).
+if [ "$healed" = "1" ]; then
+	date -u +%Y-%m-%dT%H:%M:%SZ > "$MARKER" 2>/dev/null || true
+fi
+
+exit 0

configs/felhom-mgmt-watchdog.timer

@@ -0,0 +1,18 @@
+# felhom-mgmt-watchdog.timer (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.timer.
+#
+# Drives felhom-mgmt-watchdog.service on a ~60s cadence (the auto-heal tick). OnBootSec fires shortly
+# after boot (belt-and-suspenders with the tmpfiles layer); OnUnitActiveSec=60s gives the ≤1-tick
+# heal budget the G1 acceptance drill measures. Persistent=true runs a missed tick immediately after a
+# resume/late boot. The oneshot is idempotent, so a fast cadence never churns a healthy host.
+[Unit]
+Description=Felhom management-plane watchdog timer (~60s privsep-dir auto-heal tick)
+
+[Timer]
+OnBootSec=30s
+OnUnitActiveSec=60s
+AccuracySec=5s
+Persistent=true
+Unit=felhom-mgmt-watchdog.service
+
+[Install]
+WantedBy=timers.target

configs/felhom-mkfs-guarded.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+#===============================================================================
+# felhom-mkfs-guarded — the ONLY mkfs path the felhom-agent sudoers permits (Impl-1 Part B,
+# SPIKE-drive-enrollment-2026-07-01 §SQ3). Defense-in-depth BELOW the agent: even a buggy or
+# compromised agent cannot mkfs a catastrophic target through this — it re-checks, as root, the
+# cheap catastrophic cases (OS/system disk, LVM physical volume, a foreign mount, a read-only
+# device, and any LVM/ZFS/mdraid/LUKS/swap member signature — audit D1) and refuses.
+#
+# The agent's full unclaimed-disk filter (internal/storage/claim.go) is the PRIMARY guard; this
+# wrapper is a deliberately minimal, auditable second gate. It is NOT the place for the full filter.
+#
+# Usage:  felhom-mkfs-guarded <device> <fstype:ext4|xfs>
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-mkfs-guarded: REFUSED: $*" >&2; exit 1; }
+
+dev="${1:-}"; fstype="${2:-}"
+[[ -n "$dev" && -n "$fstype" ]] || die "usage: felhom-mkfs-guarded <device> <fstype>"
+
+# The device must be a REAL block-device node under /dev — no symlink (e.g. /dev/disk/by-*), no traversal.
+[[ "$dev" == /dev/* ]]  || die "device must be under /dev ($dev)"
+[[ "$dev" != *..* ]]    || die "path traversal ($dev)"
+[[ -b "$dev" ]]         || die "not a block device ($dev)"
+[[ ! -L "$dev" ]]       || die "device must be a real node, not a symlink ($dev)"
+
+# Whole-disk of the target (a partition's parent, else the disk itself).
+pk="$(lsblk -ndo PKNAME "$dev" 2>/dev/null || true)"
+whole="$dev"; [[ -n "$pk" ]] && whole="/dev/$pk"
+
+# 1) OS/system disk — does the target's whole-disk back /, /boot or /boot/efi?
+while read -r src mnt _rest; do
+  case "$mnt" in
+    /|/boot|/boot/efi)
+      spk="$(lsblk -ndo PKNAME "$src" 2>/dev/null || true)"
+      swhole="$src"; [[ -n "$spk" ]] && swhole="/dev/$spk"
+      [[ "$swhole" == "$whole" || "$src" == "$dev" || "$src" == "$whole" ]] && die "system/OS disk ($dev backs $mnt)"
+      ;;
+  esac
+done < /proc/mounts
+
+# 2) LVM physical volume anywhere on the target disk or its partitions. pvs is resolved by ABSOLUTE
+#    path (audit D1: `command -v pvs` silently skipped this check when pvs wasn't on the caller's
+#    PATH); if neither candidate exists, check 5's LVM2_member FSTYPE loop still catches a PV
+#    independently — pvs-absent never silently drops LVM detection.
+pvsbin=""
+for c in /usr/sbin/pvs /sbin/pvs; do
+  [[ -x "$c" ]] && { pvsbin="$c"; break; }
+done
+if [[ -n "$pvsbin" ]]; then
+  while read -r pv; do
+    pv="${pv//[[:space:]]/}"; [[ -z "$pv" ]] && continue
+    pvpk="$(lsblk -ndo PKNAME "$pv" 2>/dev/null || true)"
+    pvwhole="$pv"; [[ -n "$pvpk" ]] && pvwhole="/dev/$pvpk"
+    [[ "$pvwhole" == "$whole" ]] && die "device holds an LVM physical volume ($pv)"
+  done < <("$pvsbin" --noheadings -o pv_name 2>/dev/null || true)
+fi
+
+# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
+#    /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
+while read -r mp; do
+  [[ -z "$mp" ]] && continue
+  case "$mp" in
+    /mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
+    *) die "device (or a partition) is mounted at $mp ($dev)" ;;
+  esac
+done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
+
+# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
+wbase="${whole#/dev/}"
+rof="/sys/block/$wbase/ro"
+if [[ -r "$rof" ]]; then
+  ro="$(cat "$rof" 2>/dev/null || true)"
+  [[ "$ro" == "1" ]] && die "read-only device ($whole)"
+fi
+
+# 5) member/active FSTYPEs anywhere on the target disk or its partitions (audit D1). Mirrors
+#    claim.go memberFSTypes exactly: a member of LVM/ZFS/mdraid/LUKS or active-swap signature is
+#    always a claim, never a plain formattable data disk. This also independently catches an LVM PV
+#    when pvs is not installed (check 2's belt-and-suspenders).
+while read -r fst; do
+  [[ -z "$fst" ]] && continue
+  case "$fst" in
+    LVM2_member|zfs_member|linux_raid_member|crypto_LUKS|swap)
+      die "device holds a $fst signature ($whole)" ;;
+  esac
+done < <(lsblk -nro FSTYPE "$whole" 2>/dev/null || true)
+
+# Passed the catastrophic checks → format. exec so the mkfs exit status is the wrapper's.
+case "$fstype" in
+  ext4) exec /usr/sbin/mkfs.ext4 -F "$dev" ;;
+  xfs)  exec /usr/sbin/mkfs.xfs  -f "$dev" ;;
+  *)    die "unsupported fstype ($fstype)" ;;
+esac

configs/felhom-oob-nft.service

@@ -0,0 +1,19 @@
+# felhom-oob-nft.service (TASK H1) — install as /etc/systemd/system/felhom-oob-nft.service.
+#
+# Loads the STATIC felhom_oob belt table on boot (the agent then fills its sets each tick). Oneshot +
+# RemainAfterExit so `systemctl status` reflects "loaded". Independent of pve-firewall/nftables.service
+# (the table is a self-contained `inet felhom_oob`, additive — it never flushes the ruleset).
+[Unit]
+Description=Felhom OOB belt — load the static felhom_oob nft table (H1)
+After=network-pre.target
+Before=network.target felhom-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/sbin/nft -f /etc/felhom-oob.nft
+# On stop, remove the table (best-effort) so an uninstall/disable leaves nft clean.
+ExecStop=-/usr/sbin/nft delete table inet felhom_oob
+
+[Install]
+WantedBy=multi-user.target

configs/felhom-oob.nft

@@ -0,0 +1,37 @@
+#!/usr/sbin/nft -f
+# felhom OOB belt — STATIC table (TASK H1). Installed by host-install; loaded on boot by
+# felhom-oob-nft.service. The agent mutates ONLY the two SETS (@operator_ips, @ssh_port) via the
+# narrow FELHOM_OOB sudoers — it NEVER touches the rules [trap 4], so the enforcement semantics are
+# fixed here and cannot be changed by the agent.
+#
+# Enforcement (scoped to iifname "wg-felhom" — the offsite tunnel — so :22 and every other interface
+# are UNTOUCHED [SF-3/OF-5]):
+#   - ESTABLISHED,RELATED replies always pass (covers the PBS path + reply traffic; PMTU-safe).
+#   - felhom-sshd's port is reachable from the operator /32 (in @operator_ips) over the tunnel ONLY.
+#   - any other tunnel source to that port is DROPPED at the host (defense-in-depth vs the endpoint).
+#   - the port is unreachable OFF the tunnel entirely.
+# Empty sets (before the agent's first sync) = the port rules match nothing = no enforcement yet
+# (felhom-sshd is not started until the agent renders its config either). priority -5 sits just above
+# the default; policy accept so this table only ever ADDS drops for the felhom-sshd port.
+#
+# Idempotent load: create-then-delete-then-define (a re-run/boot yields a clean table with EMPTY sets;
+# the agent refills them within one tick).
+
+table inet felhom_oob
+delete table inet felhom_oob
+
+table inet felhom_oob {
+	set operator_ips {
+		type ipv4_addr
+	}
+	set ssh_port {
+		type inet_service
+	}
+	chain input {
+		type filter hook input priority -5; policy accept;
+		ct state established,related accept
+		iifname "wg-felhom" tcp dport @ssh_port ip saddr @operator_ips accept
+		iifname "wg-felhom" tcp dport @ssh_port drop
+		tcp dport @ssh_port iifname != "wg-felhom" drop
+	}
+}

configs/felhom-op.sudoers

@@ -0,0 +1,23 @@
+# felhom-op operator sudoers (TASK H1) — install as /etc/sudoers.d/felhom-op (0440 root:root).
+#
+# The DEFAULT operator identity (login via felhom-sshd only; key in /etc/felhom-sshd/authorized_keys/
+# felhom-op, outside ~/.ssh, so the customer's sshd never honours it [SF-3]). felhom-op is a normal
+# login user with NO ambient privilege — it escalates ONLY through these explicit repair verbs, so a
+# leaked operator key's blast radius is this verb set, not the box (locked decision §6). Deep repairs
+# that genuinely need root (pct, deeper systemctl) remain available via `root` as a tunnel-gated
+# break-glass account (PermitRootLogin prohibit-password on felhom-sshd), but the day-to-day identity
+# is felhom-op.
+Cmnd_Alias FELHOM_OP_REPAIR = \
+  /usr/bin/mkdir -p /run/sshd, \
+  /usr/bin/systemctl reset-failed wg-quick@wg-felhom, \
+  /usr/bin/systemctl restart wg-quick@wg-felhom, \
+  /usr/bin/systemctl start felhom-agent, \
+  /usr/bin/systemctl restart felhom-agent, \
+  /usr/bin/systemctl reset-failed felhom-sshd, \
+  /usr/bin/systemctl restart felhom-sshd, \
+  /usr/sbin/pct list, \
+  /usr/sbin/pct start [0-9]*, \
+  /usr/sbin/pct stop [0-9]*, \
+  /usr/sbin/pct unlock [0-9]*
+
+felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR

configs/felhom-pbs-apply

@@ -0,0 +1,159 @@
+#!/bin/bash
+#===============================================================================
+# felhom-pbs-apply — the ONLY storage-entry path the felhom-agent sudoers permits for the PBS DR
+# tier (slice 2; SPIKE-pbs-tier-provisioning-2026-07-10 §2b). The guarded-mkfs shape: the agent's
+# in-process validation is the PRIMARY gate (descriptor-field charset + equality checks BEFORE
+# exec); this wrapper is the minimal, auditable second gate as root.
+#
+# THE SET-ONLY LAW (spike §4, data-loss class): `pvesm` entry deletion DESTROYS the client
+# encryption key file (<id>.enc = K) — un-decryptable backups. This wrapper therefore contains
+# NO deletion path of any kind, and a re-apply is `pvesm set`-only. Grep-assertable; do not add
+# a "cleanup" verb here, ever. Deprovision is a deliberate future operator op, not this tool.
+#
+# SECRET DISCIPLINE (spike §2b): sudo logs its full argv to auth.log → the PBS token secret
+# arrives on STDIN, never as an argument to this wrapper. Inside, it is passed to pvesm's
+# --password (root-local, transient ps exposure — the accepted spike posture); it is never
+# echoed, never written anywhere except by pvesm itself (the 0600 .pw store).
+#
+# Ops (non-secret args on argv):
+#   create    <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>
+#             secret on stdin (required). Creates the pbs entry with --encryption-key autogen
+#             (K born at /etc/pve/priv/storage/<id>.enc), then places .pw/.enc copies in
+#             <secret-dir> when it differs (the §4b WARN-fix dir; escrow-create's PBSEncKeyPath
+#             must find K there — spike §4 flag).
+#   reconcile <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>
+#             secret on stdin (optional; empty = no credential change). `pvesm set` ONLY:
+#             server/fingerprint (+ --password when a secret is fed, e.g. after a hub re-issue).
+#             NOTE datastore is deliberately NOT settable, and namespace/token-id are accepted
+#             for validation parity but NOT applied — tenancy identity is adopt-only (the
+#             demo's live entry must never be repointed at a different namespace).
+#   read      <id> <secret-dir>
+#             R-39 leg (b): print the storage's token secret to STDOUT and nothing else.
+#             The non-root agent WRITES this file through this wrapper but could never read it
+#             back (/etc/pve/priv is 0700 root:www-data and there is no read verb), so its
+#             15-minute PBS verify loop was permanently blind to the one failure it exists to
+#             catch — an `applied` tier authenticating 401. This verb is that missing read.
+#             It is deliberately the narrowest thing that works: no network, no mutation, no
+#             logging of the value, one file, prefix-asserted under the given secret dir.
+#
+#   grant     <id>
+#             The Part-0-evidenced dual-grant: FelhomAgentStore on /storage/<id> to the agent
+#             user AND token (privsep intersection). Datastore.Audit reads ride the base role.
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-pbs-apply: REFUSED: $*" >&2; exit 1; }
+
+op="${1:-}"; id="${2:-}"
+[[ -n "$op" && -n "$id" ]] || die "usage: felhom-pbs-apply <create|reconcile|grant|read> <storage-id> ..."
+
+# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
+[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
+
+STORECFG=/etc/pve/storage.cfg
+PRIVDIR=/etc/pve/priv/storage
+
+entry_exists() { grep -Eq "^pbs: ${id}\$" "$STORECFG"; }
+
+val_server()    { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9.-]{0,62}$ ]] || die "bad server ($1)"; }
+val_datastore() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad datastore ($1)"; }
+val_ns()        { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad namespace ($1)"; }
+val_tok()       { [[ "$1" =~ ^[A-Za-z0-9_.-]+@[A-Za-z0-9]+![A-Za-z0-9_.-]+$ ]] || die "bad token id ($1)"; }
+val_fp()        { [[ "$1" =~ ^([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}$ ]] || die "bad fingerprint"; }
+val_sdir() {
+  case "$1" in
+    /etc/pve/priv/storage) : ;;
+    /var/lib/felhom-agent/*) [[ "$1" != *..* ]] || die "secret-dir traversal" ;;
+    *) die "secret-dir must be $PRIVDIR or under /var/lib/felhom-agent ($1)" ;;
+  esac
+}
+
+read_secret() { # → SECRET (may be empty when optional)
+  local s
+  s=$(head -c 256 || true)
+  s="${s%$'\n'}"; s="${s%$'\r'}"
+  printf '%s' "$s"
+}
+
+place_copies() { # secret-dir — the §4b WARN-fix placement (non-root agent can't read /etc/pve/priv)
+  local sdir="$1"
+  [[ "$sdir" == "$PRIVDIR" ]] && return 0
+  install -d -o felhom-agent -g felhom-agent -m 0700 "$sdir"
+  [[ -f "$PRIVDIR/$id.pw" ]] && install -o felhom-agent -g felhom-agent -m 0600 "$PRIVDIR/$id.pw" "$sdir/$id.pw"
+  # K's copy: escrow-create stats PBSEncKeyPath(<secret-dir>/<id>.enc) — the spike §4 flag.
+  [[ -f "$PRIVDIR/$id.enc" ]] && install -o root -g felhom-agent -m 0640 "$PRIVDIR/$id.enc" "$sdir/$id.enc"
+  return 0
+}
+
+case "$op" in
+create)
+  [[ $# -eq 8 ]] || die "create needs 7 args: <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>"
+  server="$3"; datastore="$4"; ns="$5"; tok="$6"; fp="$7"; sdir="$8"
+  val_server "$server"; val_datastore "$datastore"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
+  entry_exists && die "entry $id already exists (reconcile is the re-apply path — set-only law)"
+  SECRET=$(read_secret)
+  [[ -n "$SECRET" ]] || die "create requires the token secret on stdin"
+  pvesm add pbs "$id" \
+    --server "$server" --datastore "$datastore" --namespace "$ns" \
+    --username "$tok" --password "$SECRET" --fingerprint "$fp" \
+    --content backup --encryption-key autogen >&2
+  [[ -f "$PRIVDIR/$id.enc" ]] || die "pvesm add succeeded but K ($PRIVDIR/$id.enc) was not born"
+  place_copies "$sdir"
+  echo "felhom-pbs-apply: created $id (K born; encryption-key autogen)" >&2
+  ;;
+reconcile)
+  [[ $# -eq 7 ]] || die "reconcile needs 6 args: <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>"
+  server="$3"; ns="$4"; tok="$5"; fp="$6"; sdir="$7"
+  val_server "$server"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
+  entry_exists || die "entry $id does not exist (create is the fresh path)"
+  SECRET=$(read_secret)
+  # R-39 (2026-07-18): NEVER pass --server to `pvesm set`. PVE treats `server` as a CREATE-ONLY
+  # parameter and rejects the ENTIRE call — "can't change value of fixed parameter 'server'" —
+  # even when the value is byte-identical to the stored one. That made every reconcile exit 255,
+  # so each hub-re-issued one-time secret was consumed-then-burned and the tier stayed pinned to
+  # a revoked credential (401 forever). Proven live on the N100 demo host: `pvesm set <id>
+  # --server <same> --fingerprint <same>` -> rejected; the same call without --server -> rc 0.
+  # The server address is immutable by construction (relocating a PBS endpoint needs a fresh
+  # create), so there is nothing here to reconcile. Guarded by
+  # TestReconcileNeverPassesServerToPvesmSet.
+  args=(--fingerprint "$fp")
+  [[ -n "$SECRET" ]] && args+=(--password "$SECRET")
+  pvesm set "$id" "${args[@]}" >&2
+  place_copies "$sdir"
+  echo "felhom-pbs-apply: reconciled $id (set-only; tenancy identity untouched)" >&2
+  ;;
+read)
+  # R-39(b): the missing read path. Prints the secret to STDOUT, nothing else — no stderr note (it
+  # would be the only verb whose success line could be confused with the value), no mutation.
+  #
+  # Traversal is refused three times over, because this is the one verb that EXFILTRATES a file and
+  # its argv is attacker-shaped if the agent is ever compromised:
+  #   1. `id` already matched ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ at the top — it cannot start with a dot
+  #      and cannot contain a slash, so "../../etc/shadow" never reaches here;
+  #   2. val_sdir pins the directory to PRIVDIR or under /var/lib/felhom-agent, rejecting "..";
+  #   3. the RESOLVED path is prefix-asserted under that directory below, so even a future change to
+  #      either grammar cannot walk out.
+  [[ $# -eq 3 ]] || die "read needs 2 args: <id> <secret-dir>"
+  sdir="$3"
+  val_sdir "$sdir"
+  target="$sdir/$id.pw"
+  # Belt: resolve and re-check the prefix (guards a symlinked <id>.pw pointing outside the dir).
+  resolved=$(readlink -f -- "$target" 2>/dev/null || true)
+  [[ -n "$resolved" ]] || die "secret file not found ($target)"
+  case "$resolved" in
+    "$sdir"/*) : ;;
+    *) die "resolved secret path escapes the secret dir" ;;
+  esac
+  [[ -f "$resolved" ]] || die "secret file not found ($target)"
+  cat -- "$resolved"
+  ;;
+grant)
+  [[ $# -eq 2 ]] || die "grant takes only <id>"
+  pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
+  pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
+  echo "felhom-pbs-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
+  ;;
+*)
+  die "unknown op ($op)"
+  ;;
+esac

configs/felhom-privsep.tmpfiles

@@ -0,0 +1,15 @@
+# felhom privsep-dir persistence (TASK G1) — install as /etc/tmpfiles.d/felhom-privsep.conf.
+#
+# LAYER 1 (prevent): make OpenSSH's shared privilege-separation directory /run/sshd boot-persistent
+# and owned by NO systemd unit's lifecycle. This closes the exact incident cause from
+# SPIKE-felhom-sshd-2026-07-05 §8: a second sshd unit declaring `RuntimeDirectory=sshd` had systemd
+# REMOVE the shared /run/sshd on that unit's stop/failure, taking the stock sshd on :22 down with it
+# (sessions reset right after SSH2_MSG_KEXINIT). A tmpfiles.d entry recreates the dir at every boot
+# independently of any unit, so no unit's RuntimeDirectory cleanup can be the sole owner.
+#
+# systemd-tmpfiles is idempotent: `systemd-tmpfiles --create` re-run is a no-op on an existing,
+# correct dir (it only creates/fixes, never churns). Complemented at runtime by the
+# felhom-mgmt-watchdog timer (layer 2), which re-heals a dir removed AFTER boot.
+#
+# Type d = create the directory if absent (leaves an existing one, only fixing mode/owner).
+d /run/sshd 0755 root root -

configs/felhom-selfupdate-guarded

@@ -0,0 +1,139 @@
+#!/bin/sh
+# felhom-selfupdate-guarded — the ROOT half of the agent's A/B self-update (TASK D1).
+#
+# Install as /usr/local/sbin/felhom-selfupdate-guarded (0755 root:root). The non-root agent invokes
+# `apply`/`commit` via `sudo -n` (FELHOM_SELFUPDATE alias); `rollback` is ALSO the ExecStart of
+# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping update.
+#
+# Design provenance: SPIKE-agent-selfupdate-2026-07-05 (its SF-findings are cited inline). The core
+# principle: the thing that performs rollback is never the thing being updated — this wrapper +
+# systemd change almost never; the agent binary is what flips.
+#
+# Trust model: the agent verifies the download against the OPERATOR-SIGNED sha before staging; this
+# wrapper RE-verifies the same sha as root (defense in depth — the sudoers arg glob is coarse, the
+# sha check here is the real gate). Path confinement: apply only ever reads from the agent's own
+# staging dir and only ever writes the fixed live path + its siblings. NO env-overridable paths —
+# path-fixedness IS the security property (a test-mode override would be a root escalation hole).
+#
+# Verbs:
+#   apply <staged> <sha256>  stage-verify → .prev → atomic flip → pending marker → detached restart
+#   commit                   clear the pending marker (idempotent; .prev retained as a manual net)
+#   rollback                 pending-guarded revert to .prev + restart (no pending → exit 0 no-op)
+set -u
+
+BIN=/usr/local/bin/felhom-agent
+PREV=$BIN.prev
+STAGING=/var/lib/felhom-agent/selfupdate
+PENDING=$STAGING/pending.json
+UNIT=felhom-agent.service
+
+# Every refusal/decision goes to stderr AND the journal (strict rule 10).
+log() { echo "felhom-selfupdate-guarded: $*" >&2; logger -t felhom-selfupdate-guarded -- "$*" 2>/dev/null || true; }
+
+case "${1:-}" in
+
+apply)
+    staged=${2:-}; want=${3:-}
+    # [SF-7] entry sweep: a kill between staging-copy and mv leaves an orphaned temp — harmless,
+    # but sweep it so temps can never accumulate.
+    rm -f "$BIN".new.*
+
+    if [ -z "$staged" ] || [ -z "$want" ]; then
+        log "refusing apply: usage: apply <staged> <sha256>"
+        exit 2
+    fi
+    # Root-side path confinement: the staged binary MUST live in the agent's staging dir.
+    case "$staged" in
+        "$STAGING"/*) ;;
+        *) log "refusing apply: staged path outside $STAGING: $staged"; exit 1 ;;
+    esac
+    case "$staged" in
+        *..*) log "refusing apply: staged path contains '..'"; exit 1 ;;
+    esac
+    [ -f "$staged" ] || { log "refusing apply: staged file missing: $staged"; exit 1; }
+    # The sha must be 64 lowercase hex chars — anything else is refused before any hashing.
+    case "$want" in
+        *[!0-9a-f]*) log "refusing apply: sha256 is not lowercase hex"; exit 1 ;;
+    esac
+    [ "${#want}" -eq 64 ] || { log "refusing apply: sha256 must be 64 hex chars (got ${#want})"; exit 1; }
+
+    # [SF-7] sha-verify FIRST — before .prev, before any mutation (spike S3a companion ordering).
+    got=$(sha256sum "$staged" | awk '{print $1}')
+    if [ "$got" != "$want" ]; then
+        log "refusing apply: sha mismatch (got=$got want=$want)"
+        exit 1
+    fi
+
+    # Same-fs assert (§8): the atomic-rename guarantee only holds within one filesystem.
+    if [ "$(stat -c %d "$staged")" != "$(stat -c %d /usr/local/bin)" ]; then
+        log "refusing apply: staging and /usr/local/bin are on different filesystems — atomic rename impossible"
+        exit 1
+    fi
+
+    old_ver=$("$BIN" --version 2>/dev/null | awk '{print $2}')
+    [ -n "$old_ver" ] || old_ver=unknown
+    # The staged filename is felhom-agent-<version> (executor contract) — version without executing.
+    new_ver=$(basename "$staged"); new_ver=${new_ver#felhom-agent-}
+
+    cp -p "$BIN" "$PREV" || { log "apply failed: cannot snapshot current binary to .prev"; exit 1; }
+    # Stage a root-owned 0755 copy next to the live path, then ATOMIC same-fs rename.
+    if ! cp "$staged" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
+        rm -f "$BIN.new.$$"; log "apply failed: staging copy"; exit 1
+    fi
+    mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "apply failed: atomic rename"; exit 1; }
+
+    # Pending marker: written AFTER the flip — its existence means "an update is awaiting commit",
+    # which is exactly the rollback unit's trigger condition.
+    printf '{"old_version":"%s","new_version":"%s","sha256":"%s","applied_at":"%s"}\n' \
+        "$old_ver" "$new_ver" "$want" "$(date -Is)" > "$PENDING" \
+        || { log "apply failed: cannot write pending marker"; exit 1; }
+
+    # [SF-4/5] deliberate restarts consume start-limit budget — clear the counter first.
+    systemctl reset-failed "$UNIT" 2>/dev/null || true
+    # [SF-6] the spike's S2b winner, verbatim: detached transient timer OUTSIDE the agent's cgroup,
+    # so the sudo/agent caller survives to log the handoff and the restart cannot be torn down
+    # by its own requester dying.
+    systemd-run --on-active=2s --timer-property=AccuracySec=100ms systemctl restart "$UNIT" \
+        || { log "apply: flip done but detached restart scheduling FAILED — restart $UNIT manually"; exit 1; }
+    log "applied $new_ver (prev $old_ver, sha $want); detached restart scheduled"
+    ;;
+
+commit)
+    if [ ! -f "$PENDING" ]; then
+        log "commit: no pending — no-op"
+        exit 0
+    fi
+    # .prev is deliberately RETAINED (spike S3d) — a manual safety net until the next apply.
+    rm -f "$PENDING" || { log "commit failed: cannot remove pending marker"; exit 1; }
+    log "committed (pending cleared, .prev retained)"
+    ;;
+
+rollback)
+    # [SF-1] On systemd 257 OnFailure= fires on EVERY crash, so this verb runs MANY times per
+    # incident — the pending-guard makes every fire after the first a harmless no-op, and makes a
+    # crash with NO update in flight touch nothing at all (spike S1d/S3e).
+    if [ ! -f "$PENDING" ]; then
+        log "rollback: no pending update — no-op"
+        exit 0
+    fi
+    [ -f "$PREV" ] || { log "rollback FAILED: pending exists but no .prev binary"; exit 1; }
+    rm -f "$BIN".new.*
+    if ! cp "$PREV" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
+        rm -f "$BIN.new.$$"; log "rollback FAILED: staging copy"; exit 1
+    fi
+    mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "rollback FAILED: atomic rename"; exit 1; }
+    # Clear pending BEFORE the restart: once the binary is reverted, later OnFailure fires must
+    # no-op (the guard above) instead of re-copying .prev forever.
+    rm -f "$PENDING"
+    # [SF-4/5] the crash burst has been eating the start-limit budget — reset before starting.
+    systemctl reset-failed "$UNIT" 2>/dev/null || true
+    # Direct restart is correct HERE: this caller is the rollback oneshot, OUTSIDE the agent cgroup.
+    systemctl restart "$UNIT" || { log "rollback: binary reverted but restart FAILED"; exit 1; }
+    log "rolled back to previous binary and restarted $UNIT"
+    ;;
+
+*)
+    log "usage: felhom-selfupdate-guarded apply <staged> <sha256> | commit | rollback"
+    exit 2
+    ;;
+esac

configs/felhom-sshd.service

@@ -0,0 +1,28 @@
+# felhom-sshd.service (TASK H1) — install as /etc/systemd/system/felhom-sshd.service.
+#
+# The dedicated OOB sshd instance: a SECOND sshd on a claimed non-22 port, COEXISTING with the
+# customer's/stock sshd on :22 (never touched). Config is AGENT-RENDERED at /etc/felhom-sshd/sshd_config
+# (Port from the agent's claim); the agent reloads on change (never restart-on-change [SF-2]).
+#
+# CRITICAL [SF-1]: this unit MUST NOT declare `RuntimeDirectory=` — that directive (value `sshd`)
+# removed the SHARED /run/sshd and broke the stock sshd live (SPIKE-felhom-sshd §8). /run/sshd is
+# guaranteed by G1 (tmpfiles + watchdog); the ExecStartPre mkdir is a harmless extra guarantee. The
+# host-install RuntimeDirectory guard will REJECT this unit if the directive is ever added.
+[Unit]
+Description=Felhom OOB sshd (dedicated instance, agent-managed; H1)
+After=network.target
+
+[Service]
+Type=notify
+ExecStartPre=-/usr/bin/mkdir -p /run/sshd
+ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
+ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd/sshd_config
+# Config-change path: validate then HUP — a bad reload is REFUSED and the running daemon SURVIVES
+# [SF-2]. The agent always writes→sshd -t→reload; it never `restart`s on a config change.
+ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
+ExecReload=/bin/kill -HUP $MAINPID
+KillMode=process
+Restart=on-failure
+
+[Install]
+WantedBy=multi-user.target

go.mod

@@ -0,0 +1,7 @@
+module gitea.dooplex.hu/admin/felhom-agent
+
+go 1.25.0
+
+require golang.org/x/crypto v0.52.0
+
+require golang.org/x/sys v0.45.0 // indirect

go.sum

@@ -0,0 +1,6 @@
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
+golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=

internal/authz/blob.go

@@ -0,0 +1,81 @@
+package authz
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// CanonicalBlob builds the canonical OpBlob bytes (phase4 §2 field order: keys sorted at
+// every level, no insignificant whitespace, no trailing newline, UTF-8). This is the SINGLE
+// production source of the signed bytes — the operator signing CLI (cmd/felhom-opsign) and the
+// in-Go test minting both call it, so the signer can NEVER drift from what the verifier expects
+// (the verifier authenticates over the RAW received bytes, so these bytes ARE the contract).
+//
+// params is canonicalized internally (parsed + re-marshaled → object keys sorted, whitespace
+// stripped) so the same op+params always yields identical bytes; "" → "{}". Returns an error
+// only when params is not valid JSON.
+func CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) ([]byte, error) {
+	params := strings.TrimSpace(paramsJSON)
+	if params == "" {
+		params = "{}"
+	}
+	var pv interface{}
+	if err := json.Unmarshal([]byte(params), &pv); err != nil {
+		return nil, fmt.Errorf("authz: params is not valid JSON: %w", err)
+	}
+	pc, err := json.Marshal(pv) // Go marshals object keys sorted, compact
+	if err != nil {
+		return nil, fmt.Errorf("authz: canonicalizing params: %w", err)
+	}
+	return []byte(fmt.Sprintf(
+		`{"expires_at":%q,"issued_at":%q,"key_id":%q,"nonce":%q,"op":%q,"params":%s,"target":{"guest_id":%q,"host_id":%q}}`,
+		expires.UTC().Format(time.RFC3339), issued.UTC().Format(time.RFC3339),
+		keyID, nonce, op, pc, guestID, hostID)), nil
+}
+
+// Target binds an op to a specific box (and optionally a guest) — the anti-retarget
+// field. The §7 reference omitted the json tags; production needs them so the
+// signed canonical bytes decode correctly.
+type Target struct {
+	HostID  string `json:"host_id"`
+	GuestID string `json:"guest_id"`
+}
+
+// OpBlob is the canonical signed object (phase4 §2). The signature covers the
+// EXACT bytes of this object's canonical JSON (keys sorted at every level, no
+// insignificant whitespace, no trailing newline, UTF-8) — produced by the
+// operator CLI / hub, verified here over the raw received bytes.
+type OpBlob struct {
+	Op        string          `json:"op"`
+	Target    Target          `json:"target"`
+	Params    json.RawMessage `json:"params"`
+	Nonce     string          `json:"nonce"`
+	IssuedAt  time.Time       `json:"issued_at"`
+	ExpiresAt time.Time       `json:"expires_at"`
+	KeyID     string          `json:"key_id"`
+}
+
+// VerifiedOp is the authenticated, parsed op returned on success — everything the
+// reconcile layer (slice 4) needs to route and execute, not just the op string.
+type VerifiedOp struct {
+	Op     string // the operation, e.g. "guest_destroy"
+	HostID string // target host (== this agent's host)
+	// GuestID is non-empty for a guest-scoped op; the caller routes by it. "" =
+	// host-scoped op. The verifier does NOT need to know all guest ids.
+	GuestID   string
+	Params    json.RawMessage
+	Nonce     string
+	IssuedAt  time.Time
+	ExpiresAt time.Time
+
+	// KeyID is the blob's self-declared key id — ADVISORY / audit only, never an
+	// authz input. Authz is the key-material allow-list match (Signer below).
+	KeyID string
+	// Signer is the allow-listed key whose material matched the signature.
+	Signer AllowedSigner
+	// KeyIDMatchesSigner is false when the blob's advisory KeyID disagrees with
+	// the matched signer's id (a benign audit signal, not a rejection).
+	KeyIDMatchesSigner bool
+}

internal/authz/doc.go

@@ -0,0 +1,37 @@
+// Package authz is the control-plane-authorization layer: it verifies
+// operator-signed destructive ops before the agent executes them. It is what the
+// reconcile loop (slice 4) calls to gate destructive desired-state deltas and
+// signed one-shot jobs (03 §4, 04). The signing mechanism is proven (Phase 4,
+// 14/14) — this package is its production form: a key-type-agnostic SSHSIG
+// verifier, the full anti-replay/authorization pipeline, and a durable,
+// crash-safe nonce store.
+//
+// # Mechanism (LOCKED — do not redesign)
+//
+//   - SSHSIG via golang.org/x/crypto/ssh; no hand-rolled crypto, no raw-Ed25519
+//     fallback. pub.Verify dispatches on the key's own algorithm, so the same path
+//     accepts ed25519 / sk-ssh-ed25519 (FIDO2) / rsa / ecdsa — a hardware operator
+//     key later is a box no-op (Phase 4 §5/§6, doc 04 §7).
+//   - Fixed namespace felhom-op-v1 (package constant, never caller-supplied).
+//   - The verifier verifies over the RAW received blob bytes and never
+//     canonicalizes — the canonical form (sorted-key, whitespace-free JSON) is the
+//     signer's contract, shared by the hub and the felhom-sign CLI.
+//
+// # Pipeline order (load-bearing — Verify)
+//
+//	parse armor → namespace → parse pubkey → allow-list (by key MATERIAL, not
+//	key_id) → crypto verify → parse blob → target → time window → nonce LAST
+//
+// Each post-crypto stage rejects even with an otherwise-valid signature. The nonce
+// is recorded last, so an invalid signature can never consume a nonce. key_id is
+// advisory/audit only — authz is the key-material allow-list match.
+//
+// # Shared-contract dependency (flag for later, not built here)
+//
+// Signatures only verify if the op-generator (hub) and the felhom-sign CLI produce
+// BYTE-IDENTICAL canonical JSON (keys sorted at every level, no insignificant
+// whitespace, no trailing newline, UTF-8 — Phase 4 §2). The verifier deliberately
+// does NOT re-canonicalize, so a divergence between those two producers surfaces as
+// a crypto failure here. A shared canonicalizer that both import would be the right
+// home for that contract; it is out of scope for this slice.
+package authz

internal/authz/errors.go

@@ -0,0 +1,27 @@
+package authz
+
+import "errors"
+
+// Typed rejection sentinels — one per pipeline stage so the reconcile layer can
+// distinguish "rejected" (a real signed op that failed a check) from "malformed"
+// from a future "not yet signed". All are errors.Is-friendly: Verify wraps them
+// with %w plus context.
+var (
+	// ErrMalformed: the armor/SSHSIG/blob could not be parsed (not a rejection of
+	// a well-formed op — bad input).
+	ErrMalformed = errors.New("authz: malformed signature or blob")
+	// ErrNamespace: SSHSIG namespace != the fixed felhom-op-v1 domain separator.
+	ErrNamespace = errors.New("authz: namespace mismatch")
+	// ErrUnknownSigner: the signing key's material is not in the pinned allow-list.
+	ErrUnknownSigner = errors.New("authz: signer not in allowed set")
+	// ErrBadSignature: cryptographic verification failed (tamper / wrong key).
+	ErrBadSignature = errors.New("authz: signature did not verify")
+	// ErrTarget: target.host_id is not this box.
+	ErrTarget = errors.New("authz: target mismatch")
+	// ErrExpired: now > expires_at.
+	ErrExpired = errors.New("authz: op expired")
+	// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
+	ErrNotYetValid = errors.New("authz: op not yet valid")
+	// ErrReplay: the nonce was already recorded in the window.
+	ErrReplay = errors.New("authz: replay (nonce already seen)")
+)

internal/authz/mint_test.go

@@ -0,0 +1,103 @@
+package authz
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/pem"
+	"testing"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// Test helpers that MINT armored SSHSIGs in-Go (hermetic) — the inverse of the
+// production framing. They reuse the production signedData()/sshsigBlob so a test
+// can never drift from the verifier's notion of the signed bytes.
+
+// canonicalBlob delegates to the production CanonicalBlob (so the in-Go test minting can never
+// drift from the real signed-bytes path). Panics on a params error — tests pass valid JSON.
+func canonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) []byte {
+	b, err := CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON, issued, expires)
+	if err != nil {
+		panic(err)
+	}
+	return b
+}
+
+// mintArmor builds an armored SSHSIG over message, using sign to produce the inner
+// ssh.Signature over the recomputed SSHSIG signed-data.
+func mintArmor(t *testing.T, pubMarshaled []byte, namespace, hashName string, message []byte, sign func([]byte) ssh.Signature) []byte {
+	t.Helper()
+	sb := &sshsigBlob{Version: 1, PublicKey: string(pubMarshaled), Namespace: namespace, Reserved: "", HashAlgo: hashName}
+	signed, err := signedData(sb, message)
+	if err != nil {
+		t.Fatalf("signedData: %v", err)
+	}
+	sig := sign(signed)
+	sb.Signature = string(ssh.Marshal(&sig))
+	raw := append([]byte(sshsigMagic), ssh.Marshal(sb)...)
+	return pem.EncodeToMemory(&pem.Block{Type: "SSH SIGNATURE", Bytes: raw})
+}
+
+// newEd25519Signer returns an ssh.PublicKey + a sign closure for a fresh ed25519 key.
+func newEd25519Signer(t *testing.T) (ssh.PublicKey, func([]byte) ssh.Signature) {
+	t.Helper()
+	pub, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sshPub, err := ssh.NewPublicKey(pub)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sign := func(signed []byte) ssh.Signature {
+		return ssh.Signature{Format: ssh.KeyAlgoED25519, Blob: ed25519.Sign(priv, signed)}
+	}
+	return sshPub, sign
+}
+
+// newSyntheticSKSigner emulates a FIDO2 sk-ssh-ed25519@openssh.com key with NO
+// hardware (Phase 4 §5). It builds a spec-faithful sk public key and an sk-format
+// signature: ed25519 over sha256(application)‖flags‖counter‖sha256(signed_data),
+// sig.Blob = the raw ed25519 signature, sig.Rest = flags‖counter. It must verify
+// through the UNCHANGED Verify path.
+func newSyntheticSKSigner(t *testing.T) (ssh.PublicKey, func([]byte) ssh.Signature) {
+	t.Helper()
+	edPub, edPriv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	const application = "ssh:"
+	skBlob := ssh.Marshal(struct {
+		Name        string
+		KeyBytes    []byte
+		Application string
+	}{"sk-ssh-ed25519@openssh.com", []byte(edPub), application})
+	skPub, err := ssh.ParsePublicKey(skBlob)
+	if err != nil {
+		t.Fatalf("parse synthetic sk pubkey: %v", err)
+	}
+	if skPub.Type() != "sk-ssh-ed25519@openssh.com" {
+		t.Fatalf("sk pubkey type = %q", skPub.Type())
+	}
+
+	sign := func(signed []byte) ssh.Signature {
+		const flagUserPresence = byte(0x01) // required, else Verify rejects
+		const counter = uint32(1)
+		appDigest := sha256.Sum256([]byte(application))
+		dataDigest := sha256.Sum256(signed)
+		// original = appDigest ‖ flags ‖ counter(BE) ‖ dataDigest (x/crypto layout)
+		var original []byte
+		original = append(original, appDigest[:]...)
+		original = append(original, flagUserPresence)
+		original = binary.BigEndian.AppendUint32(original, counter)
+		original = append(original, dataDigest[:]...)
+		edSig := ed25519.Sign(edPriv, original)
+		// sig.Rest = skFields{Flags, Counter} = flags ‖ counter(BE)
+		rest := append([]byte{flagUserPresence}, binary.BigEndian.AppendUint32(nil, counter)...)
+		return ssh.Signature{Format: "sk-ssh-ed25519@openssh.com", Blob: edSig, Rest: rest}
+	}
+	return skPub, sign
+}

internal/authz/noncestore.go

@@ -0,0 +1,195 @@
+package authz
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sync"
+	"time"
+)
+
+// MemoryNonceStore is a non-durable NonceStore for tests. Replay protection does
+// NOT survive process restart — never use it on a real host.
+type MemoryNonceStore struct {
+	mu   sync.Mutex
+	seen map[string]time.Time
+}
+
+// NewMemoryNonceStore builds an empty in-memory store.
+func NewMemoryNonceStore() *MemoryNonceStore {
+	return &MemoryNonceStore{seen: make(map[string]time.Time)}
+}
+
+// SeenOrRecord reports whether nonce was already recorded, recording it if not.
+func (m *MemoryNonceStore) SeenOrRecord(nonce string, exp time.Time) bool {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if _, ok := m.seen[nonce]; ok {
+		return true
+	}
+	m.seen[nonce] = exp
+	return false
+}
+
+// FileNonceStore is the durable, crash-safe NonceStore for the host. Mechanism:
+// an fsync'd append-only JSONL log with an in-memory index, periodic compaction,
+// and expiry-only pruning.
+//
+// Durability guarantee: a nonce is on disk AND fsync'd before SeenOrRecord returns
+// false, so the caller acting on a verified op always does so AFTER the durable
+// record. A crash between verify and execute therefore drops the op (fail-safe
+// direction) and never enables a replay. Replay protection survives restarts: the
+// log is replayed into the index on Open.
+//
+// Pruning: a nonce is dropped only after its exp (compaction), never before —
+// pruning before expiry would reopen the replay window. (An expired nonce can't be
+// replayed anyway: the time-window check rejects an expired op before the nonce
+// check, so pruning is housekeeping, not an authz hole.)
+//
+// Concurrency: a single mutex guards the file handle and index (single-process; the
+// agent is concurrent — 03 §10).
+type FileNonceStore struct {
+	mu           sync.Mutex
+	path         string
+	f            *os.File
+	idx          map[string]time.Time
+	sinceCompact int
+	now          func() time.Time
+
+	// CompactEvery is the append count that triggers a compaction (default 1000).
+	CompactEvery int
+}
+
+type nonceRecord struct {
+	Nonce string    `json:"n"`
+	Exp   time.Time `json:"e"`
+}
+
+// OpenFileNonceStore opens (or creates) the durable store at path, replaying any
+// existing log into the index.
+func OpenFileNonceStore(path string) (*FileNonceStore, error) {
+	s := &FileNonceStore{
+		path:         path,
+		idx:          make(map[string]time.Time),
+		now:          func() time.Time { return time.Now().UTC() },
+		CompactEvery: 1000,
+	}
+	if err := s.load(); err != nil {
+		return nil, err
+	}
+	f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+	if err != nil {
+		return nil, err
+	}
+	s.f = f
+	syncDir(filepath.Dir(path)) // make a freshly-created file's dir entry durable
+	return s, nil
+}
+
+func (s *FileNonceStore) load() error {
+	b, err := os.ReadFile(s.path)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil
+	}
+	if err != nil {
+		return err
+	}
+	for _, line := range bytes.Split(b, []byte("\n")) {
+		line = bytes.TrimSpace(line)
+		if len(line) == 0 {
+			continue
+		}
+		var r nonceRecord
+		if json.Unmarshal(line, &r) != nil {
+			continue // skip a torn trailing line from a crash mid-append
+		}
+		s.idx[r.Nonce] = r.Exp
+	}
+	return nil
+}
+
+// SeenOrRecord durably records an unseen nonce before returning false. On any I/O
+// failure it returns true (fail-safe: the op is NOT executed rather than risk an
+// unrecorded nonce enabling a later replay).
+func (s *FileNonceStore) SeenOrRecord(nonce string, exp time.Time) bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if _, ok := s.idx[nonce]; ok {
+		return true
+	}
+	rec, _ := json.Marshal(nonceRecord{Nonce: nonce, Exp: exp})
+	rec = append(rec, '\n')
+	if _, err := s.f.Write(rec); err != nil {
+		return true
+	}
+	if err := s.f.Sync(); err != nil {
+		return true
+	}
+	s.idx[nonce] = exp
+	s.sinceCompact++
+	s.maybeCompact()
+	return false
+}
+
+// Close releases the file handle.
+func (s *FileNonceStore) Close() error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.f != nil {
+		return s.f.Close()
+	}
+	return nil
+}
+
+// maybeCompact rewrites the log keeping only non-expired entries once enough
+// appends have accrued. Caller holds the mutex. Compaction is housekeeping: the
+// recorded nonce is already durable, so a compaction failure never fails the op.
+func (s *FileNonceStore) maybeCompact() {
+	if s.CompactEvery <= 0 || s.sinceCompact < s.CompactEvery {
+		return
+	}
+	s.sinceCompact = 0
+	now := s.now()
+
+	live := make(map[string]time.Time, len(s.idx))
+	var buf bytes.Buffer
+	for n, e := range s.idx {
+		if e.Before(now) {
+			continue // prune AFTER expiry only — safe
+		}
+		live[n] = e
+		rec, _ := json.Marshal(nonceRecord{Nonce: n, Exp: e})
+		buf.Write(rec)
+		buf.WriteByte('\n')
+	}
+
+	tmp := s.path + ".tmp"
+	if err := os.WriteFile(tmp, buf.Bytes(), 0o600); err != nil {
+		return // keep using the existing handle; nonce already durable
+	}
+	if tf, err := os.OpenFile(tmp, os.O_WRONLY, 0o600); err == nil {
+		_ = tf.Sync()
+		_ = tf.Close()
+	}
+	if s.f != nil {
+		_ = s.f.Close()
+	}
+	if err := os.Rename(tmp, s.path); err != nil {
+		s.f, _ = os.OpenFile(s.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+		return
+	}
+	syncDir(filepath.Dir(s.path))
+	s.f, _ = os.OpenFile(s.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+	s.idx = live
+}
+
+// syncDir best-effort fsyncs a directory so a create/rename is durable.
+func syncDir(dir string) {
+	if d, err := os.Open(dir); err == nil {
+		_ = d.Sync()
+		_ = d.Close()
+	}
+}

internal/authz/noncestore_test.go

@@ -0,0 +1,95 @@
+package authz
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestMemoryNonceStore(t *testing.T) {
+	m := NewMemoryNonceStore()
+	exp := time.Now().Add(time.Hour)
+	if m.SeenOrRecord("a", exp) {
+		t.Fatal("first record should be unseen")
+	}
+	if !m.SeenOrRecord("a", exp) {
+		t.Fatal("second record should be seen")
+	}
+	if m.SeenOrRecord("b", exp) {
+		t.Fatal("distinct nonce should be unseen")
+	}
+}
+
+func TestFileNonceStore_RecordAndReload(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	exp := refNow.Add(time.Hour)
+
+	s1, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if s1.SeenOrRecord("dead", exp) {
+		t.Fatal("first record should be unseen")
+	}
+	if err := s1.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Reopen: the recorded nonce must still be seen (durable across restart).
+	s2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if !s2.SeenOrRecord("dead", exp) {
+		t.Fatal("nonce not durable across reopen")
+	}
+}
+
+func TestFileNonceStore_CompactionPrunesExpiredOnly(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	s, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	s.now = func() time.Time { return refNow }
+	s.CompactEvery = 2 // force a compaction after two appends
+
+	s.SeenOrRecord("expired", refNow.Add(-time.Hour)) // exp in the past
+	s.SeenOrRecord("live", refNow.Add(time.Hour))     // triggers compaction
+	if err := s.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Reopen: the live nonce survived, the expired one was pruned (housekeeping;
+	// an expired op is rejected by the time check before the nonce check anyway).
+	s2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if !s2.SeenOrRecord("live", refNow.Add(time.Hour)) {
+		t.Error("live nonce should have survived compaction")
+	}
+	if s2.SeenOrRecord("expired", refNow.Add(-time.Hour)) {
+		t.Error("expired nonce should have been pruned (was still present)")
+	}
+}
+
+func TestFileNonceStore_SkipsTornLine(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	// a valid record line + a torn/garbage trailing line from a hypothetical crash
+	content := `{"n":"good","e":"` + refNow.Add(time.Hour).Format(time.RFC3339Nano) + `"}` + "\n" + `{"n":"tor`
+	if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	s, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatalf("open with torn line should not fail: %v", err)
+	}
+	defer s.Close()
+	if !s.SeenOrRecord("good", refNow.Add(time.Hour)) {
+		t.Error("valid record before the torn line should have loaded")
+	}
+}

internal/authz/signer_test.go

@@ -0,0 +1,36 @@
+package authz
+
+import (
+	"os"
+	"testing"
+)
+
+func TestNewAllowedSigner(t *testing.T) {
+	line, err := os.ReadFile("testdata/operator.pub")
+	if err != nil {
+		t.Fatal(err)
+	}
+	s, err := NewAllowedSigner("felhom-op-1", RoleOperational, string(line))
+	if err != nil {
+		t.Fatalf("NewAllowedSigner: %v", err)
+	}
+	if s.KeyID != "felhom-op-1" || s.Role != RoleOperational || s.PublicKey == nil {
+		t.Errorf("signer = %+v", s)
+	}
+	if s.PublicKey.Type() != "ssh-ed25519" {
+		t.Errorf("key type = %q", s.PublicKey.Type())
+	}
+}
+
+func TestNewAllowedSigner_BadRole(t *testing.T) {
+	line, _ := os.ReadFile("testdata/operator.pub")
+	if _, err := NewAllowedSigner("k", "bogus", string(line)); err == nil {
+		t.Fatal("invalid role should error")
+	}
+}
+
+func TestNewAllowedSigner_BadLine(t *testing.T) {
+	if _, err := NewAllowedSigner("k", RoleOperational, "not a key"); err == nil {
+		t.Fatal("malformed key line should error")
+	}
+}

internal/authz/sshsig.go

@@ -0,0 +1,81 @@
+package authz
+
+import (
+	"crypto/sha256"
+	"crypto/sha512"
+	"encoding/pem"
+	"fmt"
+	"hash"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// SSHSIG framing — ported verbatim-in-spirit from phase4-signing-findings.md §7.
+// The only manual work is SSHSIG *framing*; all crypto and key-type dispatch is
+// x/crypto/ssh's (pub.Verify dispatches on the key's own algorithm, which is what
+// makes the verifier key-type-agnostic — ed25519 / sk-ssh-ed25519 / rsa / ecdsa).
+// No hand-rolled crypto.
+
+const sshsigMagic = "SSHSIG"
+
+// sshsigBlob is the binary SSHSIG body (after the 6-byte magic). Field order is
+// the SSH wire order — do not reorder.
+type sshsigBlob struct {
+	Version   uint32
+	PublicKey string
+	Namespace string
+	Reserved  string
+	HashAlgo  string
+	Signature string
+}
+
+func hashByName(n string) (hash.Hash, error) {
+	switch n {
+	case "sha256":
+		return sha256.New(), nil
+	case "sha512":
+		return sha512.New(), nil
+	}
+	return nil, fmt.Errorf("%w: unsupported SSHSIG hash %q", ErrMalformed, n)
+}
+
+// parseArmoredSSHSIG decodes the `-----BEGIN SSH SIGNATURE-----` armor into the
+// SSHSIG body: pem.Decode → strip the literal 6-byte magic (not length-prefixed)
+// → ssh.Unmarshal.
+func parseArmoredSSHSIG(armored []byte) (*sshsigBlob, error) {
+	block, _ := pem.Decode(armored)
+	if block == nil || block.Type != "SSH SIGNATURE" {
+		return nil, fmt.Errorf("%w: not an SSH SIGNATURE armor", ErrMalformed)
+	}
+	if len(block.Bytes) < len(sshsigMagic) || string(block.Bytes[:len(sshsigMagic)]) != sshsigMagic {
+		return nil, fmt.Errorf("%w: missing SSHSIG magic", ErrMalformed)
+	}
+	var sb sshsigBlob
+	if err := ssh.Unmarshal(block.Bytes[len(sshsigMagic):], &sb); err != nil {
+		return nil, fmt.Errorf("%w: %v", ErrMalformed, err)
+	}
+	if sb.Version != 1 {
+		return nil, fmt.Errorf("%w: bad SSHSIG version %d", ErrMalformed, sb.Version)
+	}
+	return &sb, nil
+}
+
+// signedData recomputes the bytes the signature actually covers, per the SSHSIG
+// spec: "SSHSIG" || ssh.Marshal(namespace, reserved, hash_algorithm, H(message)),
+// where H is the named hash. The message is the RAW received blob bytes — the
+// verifier never canonicalizes (the canonical form is the signer's contract).
+func signedData(sb *sshsigBlob, msg []byte) ([]byte, error) {
+	h, err := hashByName(sb.HashAlgo)
+	if err != nil {
+		return nil, err
+	}
+	h.Write(msg)
+	md := h.Sum(nil)
+	body := ssh.Marshal(struct {
+		Namespace string
+		Reserved  string
+		HashAlgo  string
+		Hash      []byte
+	}{sb.Namespace, sb.Reserved, sb.HashAlgo, md})
+	return append([]byte(sshsigMagic), body...), nil
+}

internal/authz/testdata/op_blob.json

@@ -0,0 +1 @@
+{"expires_at":"2026-06-09T00:00:00Z","issued_at":"2026-06-08T00:00:00Z","key_id":"felhom-op-1","nonce":"a1b2c3d4e5f60718293a4b5c6d7e8f90","op":"guest_destroy","params":{"purge":true},"target":{"guest_id":"9001","host_id":"demo-felhom"}}

internal/authz/testdata/op_blob.sig

@@ -0,0 +1,6 @@
+-----BEGIN SSH SIGNATURE-----
+U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgNXOOuMvD3Fh9MJYspBRWLXyQAd
+WVeBICspeB9eL1xfIAAAAMZmVsaG9tLW9wLXYxAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz
+aC1lZDI1NTE5AAAAQG+bj+GNodNw7cfGYg3HWTDyJiu3g/5Aez1xlZQ540JGUIG9FV7vv8
+wrgN0r+rNh+ytEAM6UTOyI7g3LOjuVJgY=
+-----END SSH SIGNATURE-----

internal/authz/testdata/operator.pub

@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDVzjrjLw9xYfTCWLKQUVi18kAHVlXgSArKXgfXi9cXy felhom-op-1

internal/authz/verifier.go

@@ -0,0 +1,191 @@
+package authz
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// Namespace is the FIXED SSHSIG domain separator. It is a package constant, never
+// caller-supplied (phase4 §2.2): a signature minted for any other namespace must
+// not verify.
+const Namespace = "felhom-op-v1"
+
+// DefaultClockSkew tolerates operator/host clock drift on the not-yet-valid check
+// only (issued_at may be up to this far in the future). Expiry is NOT extended —
+// the validity window stays an honest upper bound.
+const DefaultClockSkew = 2 * time.Minute
+
+// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
+type KeyRole string
+
+const (
+	// RoleOperational signs ordinary destructive ops (the "master stamp").
+	RoleOperational KeyRole = "operational"
+	// RoleRecovery is the cold key; authorizes ONLY key-rotation/break-glass ops.
+	// Role-scoping is enforced by the consuming layer (slice 4), not here.
+	RoleRecovery KeyRole = "recovery"
+)
+
+// AllowedSigner is one pinned operator public key.
+type AllowedSigner struct {
+	KeyID     string
+	Role      KeyRole
+	PublicKey ssh.PublicKey // parsed; allow-list match is by PublicKey.Marshal()
+	Comment   string        // from the authorized_keys line, if any
+}
+
+// NewAllowedSigner parses a standard authorized_keys line ("ssh-ed25519 AAAA… [comment]"
+// or "sk-ssh-ed25519@openssh.com AAAA… …") into an AllowedSigner with the given id+role.
+func NewAllowedSigner(keyID string, role KeyRole, authorizedKeyLine string) (AllowedSigner, error) {
+	pub, comment, _, _, err := ssh.ParseAuthorizedKey([]byte(authorizedKeyLine))
+	if err != nil {
+		return AllowedSigner{}, fmt.Errorf("authz: parsing pinned key %q: %w", keyID, err)
+	}
+	if role != RoleOperational && role != RoleRecovery {
+		return AllowedSigner{}, fmt.Errorf("authz: pinned key %q has invalid role %q", keyID, role)
+	}
+	return AllowedSigner{KeyID: keyID, Role: role, PublicKey: pub, Comment: comment}, nil
+}
+
+// NonceStore records seen nonces for anti-replay. SeenOrRecord reports whether the
+// nonce was already recorded; if not, it records it (durably, in the host impl)
+// before returning false. See noncestore.go.
+type NonceStore interface {
+	SeenOrRecord(nonce string, exp time.Time) (seen bool)
+}
+
+// Verifier authenticates operator-signed destructive ops. Construct with New.
+type Verifier struct {
+	signers []AllowedSigner
+	store   NonceStore
+	hostID  string
+
+	// ClockSkew tolerance for the not-yet-valid check (default DefaultClockSkew).
+	ClockSkew time.Duration
+	// Logger, if set, emits a warning when a blob's advisory key_id disagrees with
+	// the matched signer. Never affects the verdict.
+	Logger *slog.Logger
+
+	now func() time.Time // injectable for tests
+}
+
+// New builds a Verifier over the pinned signer set, a nonce store, and this box's
+// host id. allowedSigners is a set (single signer today; quorum is just sizing).
+func New(signers []AllowedSigner, store NonceStore, hostID string) *Verifier {
+	return &Verifier{
+		signers:   signers,
+		store:     store,
+		hostID:    hostID,
+		ClockSkew: DefaultClockSkew,
+		now:       func() time.Time { return time.Now().UTC() },
+	}
+}
+
+// Verify runs the LOCKED pipeline (phase4 §4 / doc 04 §2.3) and returns the
+// authenticated op. Order is load-bearing and each post-crypto stage rejects even
+// with an otherwise-valid signature:
+//
+//	parse armor → namespace → parse pubkey → allow-list (by key MATERIAL, not
+//	key_id) → crypto verify (over the RAW received blob bytes) → parse blob →
+//	target → time window → nonce SeenOrRecord (LAST)
+//
+// The nonce is recorded last, so an invalid signature can never consume a nonce
+// (DoS / replay-priming safe). Errors wrap the typed sentinels in errors.go.
+func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
+	// 1. parse armor
+	sb, err := parseArmoredSSHSIG(sigArmored)
+	if err != nil {
+		return nil, err
+	}
+
+	// 2. namespace (fixed domain separator)
+	if sb.Namespace != Namespace {
+		return nil, fmt.Errorf("%w: got %q want %q", ErrNamespace, sb.Namespace, Namespace)
+	}
+
+	// 3. parse the embedded public key
+	pub, err := ssh.ParsePublicKey([]byte(sb.PublicKey))
+	if err != nil {
+		return nil, fmt.Errorf("%w: parsing signature public key: %v", ErrMalformed, err)
+	}
+
+	// 4. allow-list match by KEY MATERIAL (pub.Marshal equality) — NOT by key_id
+	matched, ok := v.matchSigner(pub)
+	if !ok {
+		return nil, ErrUnknownSigner
+	}
+
+	// 5. crypto verify over the RAW received bytes (never re-serialized)
+	signed, err := signedData(sb, blob)
+	if err != nil {
+		return nil, err
+	}
+	var inner ssh.Signature
+	if err := ssh.Unmarshal([]byte(sb.Signature), &inner); err != nil {
+		return nil, fmt.Errorf("%w: %v", ErrMalformed, err)
+	}
+	if err := pub.Verify(signed, &inner); err != nil { // dispatches on the key's algorithm
+		return nil, fmt.Errorf("%w: %v", ErrBadSignature, err)
+	}
+
+	// 6. parse the (now authenticated) blob bytes
+	var op OpBlob
+	if err := json.Unmarshal(blob, &op); err != nil {
+		return nil, fmt.Errorf("%w: decoding op blob: %v", ErrMalformed, err)
+	}
+
+	// 7. target binding — host must be this box. guest_id is surfaced, not matched
+	// here (the verifier doesn't enumerate guests; the caller routes by it).
+	if op.Target.HostID != v.hostID {
+		return nil, fmt.Errorf("%w: blob host_id=%q this=%q", ErrTarget, op.Target.HostID, v.hostID)
+	}
+
+	// 8. time window (clock-skew tolerance on not-yet-valid only)
+	now := v.now()
+	if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
+		return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
+	}
+	if now.After(op.ExpiresAt) {
+		return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
+	}
+
+	// 9. nonce LAST — only now is it durably recorded.
+	if v.store.SeenOrRecord(op.Nonce, op.ExpiresAt) {
+		return nil, fmt.Errorf("%w: nonce %s", ErrReplay, op.Nonce)
+	}
+
+	// advisory key_id audit (never a verdict input)
+	keyIDMatches := op.KeyID == matched.KeyID
+	if !keyIDMatches && v.Logger != nil {
+		v.Logger.Warn("authz: blob key_id does not match the matched signer (advisory)",
+			"blob_key_id", op.KeyID, "matched_signer", matched.KeyID)
+	}
+
+	return &VerifiedOp{
+		Op:                 op.Op,
+		HostID:             op.Target.HostID,
+		GuestID:            op.Target.GuestID,
+		Params:             op.Params,
+		Nonce:              op.Nonce,
+		IssuedAt:           op.IssuedAt,
+		ExpiresAt:          op.ExpiresAt,
+		KeyID:              op.KeyID,
+		Signer:             matched,
+		KeyIDMatchesSigner: keyIDMatches,
+	}, nil
+}
+
+func (v *Verifier) matchSigner(pub ssh.PublicKey) (AllowedSigner, bool) {
+	pm := pub.Marshal()
+	for _, s := range v.signers {
+		if s.PublicKey != nil && bytes.Equal(s.PublicKey.Marshal(), pm) {
+			return s, true
+		}
+	}
+	return AllowedSigner{}, false
+}

internal/authz/verify_test.go

@@ -0,0 +1,248 @@
+package authz
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// fixed reference instant used across in-Go tests (deterministic time window).
+var refNow = time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
+
+func atRefNow(v *Verifier) *Verifier { v.now = func() time.Time { return refNow }; return v }
+
+// rejects asserts a Verify error matches the expected sentinel.
+func rejects(t *testing.T, err, want error) {
+	t.Helper()
+	if !errors.Is(err, want) {
+		t.Fatalf("want %v, got %v", want, err)
+	}
+}
+
+// signerSet builds a one-key operational allow-list around an ssh.PublicKey.
+func signerSet(pub ssh.PublicKey, keyID string) []AllowedSigner {
+	return []AllowedSigner{{KeyID: keyID, Role: RoleOperational, PublicKey: pub}}
+}
+
+// validBlob is an op blob valid at refNow.
+func validBlob(host, guest, keyID, nonce string) []byte {
+	return canonicalBlob("guest_destroy", host, guest, keyID, nonce, `{"purge":true}`,
+		refNow.Add(-time.Hour), refNow.Add(time.Hour))
+}
+
+// --- Real OpenSSH interop: committed ssh-keygen fixture ---
+
+func TestVerify_RealSSHKeygenFixture(t *testing.T) {
+	blob := readFile(t, "testdata/op_blob.json")
+	sig := readFile(t, "testdata/op_blob.sig")
+	pubLine := readFile(t, "testdata/operator.pub")
+
+	signer, err := NewAllowedSigner("felhom-op-1", RoleOperational, string(pubLine))
+	if err != nil {
+		t.Fatalf("NewAllowedSigner: %v", err)
+	}
+	v := New([]AllowedSigner{signer}, NewMemoryNonceStore(), "demo-felhom")
+	v.now = func() time.Time { return time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC) } // inside fixture window
+
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("real fixture did not verify: %v", err)
+	}
+	if op.Op != "guest_destroy" || op.HostID != "demo-felhom" || op.GuestID != "9001" {
+		t.Errorf("unexpected op: %+v", op)
+	}
+	if op.KeyID != "felhom-op-1" || !op.KeyIDMatchesSigner {
+		t.Errorf("key_id audit wrong: %q matches=%v", op.KeyID, op.KeyIDMatchesSigner)
+	}
+}
+
+// --- Happy path (in-Go ed25519) ---
+
+func TestVerify_HappyPath(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-happy-0001")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+
+	v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("Verify: %v", err)
+	}
+	if op.Op != "guest_destroy" || op.Signer.KeyID != "op" {
+		t.Errorf("op = %+v", op)
+	}
+}
+
+// --- Per-stage rejection, each with an otherwise-valid signature ---
+
+func TestVerify_RejectsPerStage(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	other, _ := newEd25519Signer(t)
+
+	t.Run("wrong namespace", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-ns-1")
+		sig := mintArmor(t, pub.Marshal(), "felhom-op-wrong", "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrNamespace)
+	})
+
+	t.Run("signer not in set", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-unk-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(other, "other"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrUnknownSigner)
+	})
+
+	t.Run("tampered blob (crypto)", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-tamper-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		tampered := append([]byte{}, blob...)
+		tampered[len(tampered)-2] = '!' // mutate inside the JSON
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(tampered, sig)
+		rejects(t, err, ErrBadSignature)
+	})
+
+	t.Run("retargeted host", func(t *testing.T) {
+		blob := validBlob("other-host", "9001", "op", "n-target-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrTarget)
+	})
+
+	t.Run("expired", func(t *testing.T) {
+		blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
+			refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrExpired)
+	})
+
+	t.Run("not yet valid", func(t *testing.T) {
+		blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-nyv-1", "{}",
+			refNow.Add(time.Hour), refNow.Add(2*time.Hour))
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrNotYetValid)
+	})
+
+	t.Run("replay", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-replay-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		if _, err := v.Verify(blob, sig); err != nil {
+			t.Fatalf("first use: %v", err)
+		}
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrReplay)
+	})
+}
+
+// --- THE anti-replay invariant: an invalid-sig attempt must NOT burn the nonce ---
+
+func TestVerify_InvalidSigDoesNotBurnNonce(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	store := NewMemoryNonceStore()
+	const nonce = "n-not-burned-cafe"
+
+	blobV := validBlob("demo-felhom", "9001", "op", nonce)
+	validSig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blobV, sign)
+
+	// Attacker reuses the SAME nonce but a signature that fails crypto (valid key,
+	// signed over different bytes) — passes namespace + allow-list, fails at the
+	// crypto stage, which is BEFORE the nonce stage.
+	badSig := mintArmor(t, pub.Marshal(), Namespace, "sha512", []byte(`{"different":"bytes"}`), sign)
+
+	v := atRefNow(New(signerSet(pub, "op"), store, "demo-felhom"))
+	if _, err := v.Verify(blobV, badSig); !errors.Is(err, ErrBadSignature) {
+		t.Fatalf("invalid attempt: want ErrBadSignature, got %v", err)
+	}
+	// The genuine valid op with the same nonce must still succeed — proving the
+	// failed attempt did NOT burn the nonce (nonce-recorded-last).
+	if _, err := v.Verify(blobV, validSig); err != nil {
+		t.Fatalf("valid op after invalid attempt should succeed, got %v", err)
+	}
+}
+
+// --- Persistence across restart (durable nonce store) ---
+
+func TestVerify_ReplayRejectedAcrossRestart(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-persist-1")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+	path := filepath.Join(t.TempDir(), "nonces.log")
+
+	store1, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	v1 := atRefNow(New(signerSet(pub, "op"), store1, "demo-felhom"))
+	if _, err := v1.Verify(blob, sig); err != nil {
+		t.Fatalf("first use: %v", err)
+	}
+	if err := store1.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Fresh store + verifier over the SAME path — simulates an agent restart.
+	store2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store2.Close()
+	v2 := atRefNow(New(signerSet(pub, "op"), store2, "demo-felhom"))
+	_, err = v2.Verify(blob, sig)
+	rejects(t, err, ErrReplay)
+}
+
+// --- Key-type-agnostic: synthetic FIDO2 sk-ssh-ed25519 through the unchanged path ---
+
+func TestVerify_KeyTypeAgnostic_SK(t *testing.T) {
+	skPub, skSign := newSyntheticSKSigner(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-sk-1")
+	sig := mintArmor(t, skPub.Marshal(), Namespace, "sha512", blob, skSign)
+
+	v := atRefNow(New(signerSet(skPub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("sk verify through unchanged path failed: %v", err)
+	}
+	if op.Op != "guest_destroy" {
+		t.Errorf("op = %q", op.Op)
+	}
+}
+
+// --- Byte-exactness: a re-serialized blob is NOT re-canonicalized (fails crypto) ---
+
+func TestVerify_ByteExactNoRecanonicalization(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-bytes-1")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+
+	// Same fields, different whitespace + key order — what a non-identical producer
+	// canonicalizer would emit. The verifier verifies raw bytes, so this fails crypto.
+	reserialized := []byte(`{ "op":"guest_destroy", "target":{"host_id":"demo-felhom","guest_id":"9001"}, "params":{"purge":true}, "nonce":"n-bytes-1", "issued_at":"` +
+		refNow.Add(-time.Hour).Format(time.RFC3339) + `", "expires_at":"` + refNow.Add(time.Hour).Format(time.RFC3339) + `", "key_id":"op" }`)
+
+	v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	_, err := v.Verify(reserialized, sig)
+	rejects(t, err, ErrBadSignature)
+}
+
+func readFile(t *testing.T, path string) []byte {
+	t.Helper()
+	b, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return b
+}

internal/backup/archive_completeness_test.go

@@ -0,0 +1,243 @@
+package backup
+
+import (
+	"bytes"
+	"context"
+	"log/slog"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// F-CRIT-2 (Campaign 8): a failed backup must not look like a fresh one.
+//
+// Every fixture below is a VERBATIM shape captured from the live PVE API on 2026-07-28
+// (`pvesh get /nodes/<node>/storage/<store>/content`), not a hand-invented struct. That matters:
+// the `unparseable` path in this package went untested for months behind a JSON shape that did not
+// match production, and the whole point of this fix is that presence != validity.
+
+// phantomEntry is the artefact a PBS daemon killed mid-upload leaves behind: listed as a restorable
+// backup, 1 byte, NEWEST, and carrying no `verification`/`encrypted`/`notes` at all because it has
+// no manifest (`index.json.blob` is absent on disk).
+func phantomEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z",
+		Content: "backup",
+		Format:  "pbs-ct",
+		Size:    1,
+		CTime:   1785216674,
+		VMID:    9201,
+	}
+}
+
+// goodPBSEntry is a real, complete offsite snapshot (demo-hp, 2026-07-28T03:40:42Z).
+func goodPBSEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "felhom-pbs:backup/ct/9201/2026-07-28T03:40:42Z",
+		Content: "backup",
+		Format:  "pbs-ct",
+		Size:    4353457559,
+		CTime:   1785210042,
+		VMID:    9201,
+	}
+}
+
+// goodLocalEntry is a real, complete LOCAL vzdump (demo-hp). Note it legitimately has no
+// `verification` and no `encrypted` on the wire — a dir storage has no such concept — which is
+// exactly why those fields must never be used as completeness discriminators.
+func goodLocalEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "local:backup/vzdump-lxc-9201-2026_07_28-07_29_54.tar.zst",
+		Content: "backup",
+		Format:  "tar.zst",
+		Size:    1590431865,
+		CTime:   1785216594,
+		VMID:    9201,
+	}
+}
+
+func runnerWithContent(t *testing.T, buf *bytes.Buffer, content []proxmox.StorageContent) *BackupRunner {
+	t.Helper()
+	lg := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
+	return NewBackupRunner(&fakeBackupAPI{content: content}, "felhom-pbs", proxmox.ModeSnapshot, "", "", lg)
+}
+
+// Group A — the phantom must NOT set tier freshness, even though it is the newest entry.
+//
+// RED-PROOF: restore the old predicate in NewestArchiveTime
+// (`if e.Content == "backup" && e.VMID == vmid && e.CTime > best`) → the phantom's ctime
+// (1785216674) wins over the good snapshot's (1785210042) and this test fails with
+// "got 1785216674, want 1785210042" — i.e. the exact F-CRIT-2 defect.
+func TestNewestArchiveTime_PhantomIsNotCounted(t *testing.T) {
+	var buf bytes.Buffer
+	// phantom deliberately listed FIRST and is also the newest by ctime.
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
+
+	got, found, err := r.NewestArchiveTime(context.Background(), 9201)
+	if err != nil {
+		t.Fatalf("NewestArchiveTime: %v", err)
+	}
+	if !found {
+		t.Fatal("found=false — the GOOD snapshot must still be counted; rejecting everything is the thrash path")
+	}
+	if got.Unix() != goodPBSEntry().CTime {
+		t.Errorf("freshness came from the wrong entry: got ctime %d, want %d (the good snapshot)", got.Unix(), goodPBSEntry().CTime)
+	}
+	if got.Unix() == phantomEntry().CTime {
+		t.Error("the 1-byte manifest-less phantom set tier freshness — this is F-CRIT-2")
+	}
+}
+
+// Group A — with ONLY a phantom present the tier must report "no backup", not a fresh one.
+// That is what lets the controller see age_state=absent and fire its first-backup valve.
+func TestNewestArchiveTime_OnlyPhantomReportsNotFound(t *testing.T) {
+	var buf bytes.Buffer
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry()})
+
+	_, found, err := r.NewestArchiveTime(context.Background(), 9201)
+	if err != nil {
+		t.Fatalf("NewestArchiveTime: %v", err)
+	}
+	if found {
+		t.Error("found=true with only a phantom present — the tier would report fresh and go silent for a full cadence")
+	}
+}
+
+// Group B — THE SCENARIO-D GUARD. A valid snapshot on EITHER tier must still be counted.
+//
+// This is what makes Group A safe. A filter that is too aggressive does not merely lose safety
+// margin: the tier reports absent on every poll, backs up every cycle, and the R-88 breaker cannot
+// save it because those backups SUCCEED. That is a continuous multi-GB write loop across the fleet.
+//
+// RED-PROOF: make archivePlausiblyComplete return `false, "reject everything"` unconditionally →
+// both subtests fail with found=false.
+func TestNewestArchiveTime_ValidSnapshotsAreStillCounted(t *testing.T) {
+	for _, tc := range []struct {
+		name  string
+		entry proxmox.StorageContent
+	}{
+		{"pbs offsite (has verification+encrypted on the wire)", goodPBSEntry()},
+		{"local dir vzdump (has NEITHER verification NOR encrypted — and must still count)", goodLocalEntry()},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			var buf bytes.Buffer
+			r := runnerWithContent(t, &buf, []proxmox.StorageContent{tc.entry})
+
+			got, found, err := r.NewestArchiveTime(context.Background(), 9201)
+			if err != nil {
+				t.Fatalf("NewestArchiveTime: %v", err)
+			}
+			if !found {
+				t.Fatalf("a REAL %s backup was rejected — this is the backup-thrash path, not extra safety", tc.name)
+			}
+			if got.Unix() != tc.entry.CTime {
+				t.Errorf("got ctime %d, want %d", got.Unix(), tc.entry.CTime)
+			}
+			if strings.Contains(buf.String(), "INCOMPLETE archive") {
+				t.Errorf("a valid archive was announced as incomplete:\n%s", buf.String())
+			}
+		})
+	}
+}
+
+// Group B — the smallest REAL backup measured anywhere on the fleet (612,397,450 B, a guest-9100
+// vzdump) must clear the floor with room to spare. If someone ever raises
+// minPlausibleArchiveBytes past this, that is the fleet-thrash bug and this test is the tripwire.
+func TestMinPlausibleArchiveBytes_LeavesHeadroomBelowTheSmallestRealBackup(t *testing.T) {
+	const smallestObservedRealBackup int64 = 612397450 // fleet survey 2026-07-28
+	if minPlausibleArchiveBytes >= smallestObservedRealBackup {
+		t.Fatalf("floor %d B is not below the smallest real backup ever observed (%d B) — this WILL reject real archives",
+			minPlausibleArchiveBytes, smallestObservedRealBackup)
+	}
+	if ratio := smallestObservedRealBackup / minPlausibleArchiveBytes; ratio < 100 {
+		t.Errorf("floor %d B leaves only %dx headroom below the smallest real backup (%d B) — too tight",
+			minPlausibleArchiveBytes, ratio, smallestObservedRealBackup)
+	}
+}
+
+// Group C — UNDECIDABLE ⇒ NOT COUNTED (the fail-safe direction).
+//
+// A zero/absent size is not evidence of a good backup; it is absence of evidence. Erring toward
+// "not fresh" costs one extra backup. Erring the other way is F-CRIT-2.
+//
+// RED-PROOF: flip the comparison in archivePlausiblyComplete to `e.Size > minPlausibleArchiveBytes
+// || e.Size == 0` (i.e. treat unknown as complete) → the size-0 case reports ok=true and this fails.
+func TestArchivePlausiblyComplete_UndecidableIsNotCounted(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		size int64
+	}{
+		{"the observed phantom", 1},
+		{"absent size field (unmarshals to 0)", 0},
+		{"just under the floor", minPlausibleArchiveBytes - 1},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			e := phantomEntry()
+			e.Size = tc.size
+			ok, why := archivePlausiblyComplete(e)
+			if ok {
+				t.Errorf("size %d counted as a complete backup — undecidable must fail safe", tc.size)
+			}
+			if why == "" {
+				t.Error("rejection carried no reason — a silent rejection is a new quiet path")
+			}
+		})
+	}
+	if ok, why := archivePlausiblyComplete(goodPBSEntry()); !ok {
+		t.Errorf("a real snapshot was rejected: %s", why)
+	}
+}
+
+// Group D — the rejection is announced ONCE per snapshot, not once per due-check.
+//
+// The due-check runs every 5 minutes and a phantom persists indefinitely (server-side prune does
+// not collect it), so per-poll logging would emit ~288 identical lines a day and bury the signal.
+//
+// RED-PROOF: delete the `if seen { return }` guard in warnRejectedArchiveOnce → this test reports
+// "logged 5 times, want 1".
+func TestNewestArchiveTime_RejectionLoggedOncePerSnapshot(t *testing.T) {
+	var buf bytes.Buffer
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
+
+	const polls = 5
+	for i := 0; i < polls; i++ {
+		if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
+			t.Fatalf("poll %d: %v", i, err)
+		}
+	}
+
+	n := strings.Count(buf.String(), "INCOMPLETE archive")
+	if n != 1 {
+		t.Errorf("rejection logged %d times across %d polls, want exactly 1:\n%s", n, polls, buf.String())
+	}
+	out := buf.String()
+	if !strings.Contains(out, phantomEntry().VolID) {
+		t.Errorf("the log line does not NAME the rejected snapshot:\n%s", out)
+	}
+	if !strings.Contains(out, "below the") {
+		t.Errorf("the log line does not say WHY it was rejected:\n%s", out)
+	}
+	if !strings.Contains(out, "level=WARN") {
+		t.Errorf("rejection was not logged at WARN:\n%s", out)
+	}
+}
+
+// Group D — a SECOND, distinct phantom is announced separately. The dedupe must be per snapshot,
+// not a one-shot latch that hides every later phantom.
+func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
+	var buf bytes.Buffer
+	second := phantomEntry()
+	second.VolID = "felhom-pbs:backup/ct/9201/2026-07-29T05:31:14Z"
+	second.CTime = phantomEntry().CTime + 86400
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), second, goodPBSEntry()})
+
+	for i := 0; i < 3; i++ {
+		if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
+			t.Fatalf("poll %d: %v", i, err)
+		}
+	}
+	if n := strings.Count(buf.String(), "INCOMPLETE archive"); n != 2 {
+		t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
+	}
+}

internal/backup/backup_test.go

@@ -0,0 +1,276 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"encoding/json"
+	"errors"
+	"io"
+	"log/slog"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// fakeBackupAPI is a synthetic BackupAPI.
+type fakeBackupAPI struct {
+	vzdumpUPID string
+	vzdumpErr  error
+	waitErr    error
+	cfg        proxmox.GuestConfig
+	goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
+	aclGuests  map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
+	cfgErr     error
+	content    []proxmox.StorageContent
+	contentErr error
+	storages   []proxmox.Storage // returned by ListStorage (the local-prune scope gate) — DEFINITIONS: like
+	// production's GET /storage, it never carries usage; ListStorage strips Avail/Used (R-685's live lesson)
+	nodeStorages []proxmox.Storage // returned by NodeStorage (GET /nodes/{node}/storage — WITH usage)
+	storageErr   error
+	vzdumps      []proxmox.VzdumpOptions
+	logLines     []string      // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
+	waitGate     chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
+}
+
+func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) {
+	f.vzdumps = append(f.vzdumps, o)
+	return f.vzdumpUPID, f.vzdumpErr
+}
+func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOptions) (proxmox.TaskStatus, error) {
+	if f.waitGate != nil {
+		<-f.waitGate
+	}
+	return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
+}
+func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
+	if f.aclGuests[vmid] {
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
+	}
+	if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
+	}
+	return f.cfg, f.cfgErr
+}
+func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
+	return f.content, f.contentErr
+}
+func (f *fakeBackupAPI) ListStorage(_ context.Context) ([]proxmox.Storage, error) {
+	out := make([]proxmox.Storage, len(f.storages))
+	for i, s := range f.storages {
+		s.Avail, s.Used, s.Total = 0, 0, 0 // GET /storage has no usage — a fake that had it hid R-685's defect
+		out[i] = s
+	}
+	return out, f.storageErr
+}
+func (f *fakeBackupAPI) NodeStorage(_ context.Context) ([]proxmox.Storage, error) {
+	if f.nodeStorages != nil {
+		return f.nodeStorages, f.storageErr
+	}
+	return f.storages, f.storageErr
+}
+func (f *fakeBackupAPI) TaskLogTail(_ context.Context, _ string, _ int) ([]string, error) {
+	return f.logLines, nil
+}
+
+// guestCfgWithMounts builds a GuestConfig whose Extra carries the given mpN strings.
+func guestCfgWithMounts(mps map[string]string) proxmox.GuestConfig {
+	extra := map[string]json.RawMessage{}
+	for k, v := range mps {
+		b, _ := json.Marshal(v)
+		extra[k] = b
+	}
+	return proxmox.GuestConfig{Extra: extra}
+}
+
+func TestBackup_SuccessResolvesArchiveAndBulkGap(t *testing.T) {
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:vzdump:1",
+		cfg: guestCfgWithMounts(map[string]string{
+			"mp0": "local-lvm:8,mp=/mnt/bulk,backup=0", // explicit opt-out → uncovered
+			"mp1": "local-lvm:4,mp=/mnt/db,backup=1",   // covered
+			"mp2": "local-lvm:2,mp=/mnt/scratch",       // UNSET → uncovered (opt-in default)
+		}),
+		content: []proxmox.StorageContent{
+			{VolID: "local:backup/old-9001.tar.zst", Content: "backup", VMID: 9001, Size: 100, CTime: 100},
+			{VolID: "local:backup/new-9001.tar.zst", Content: "backup", VMID: 9001, Size: 524288000, CTime: 200},
+			{VolID: "local:backup/other-9002.tar.zst", Content: "backup", VMID: 9002, Size: 7, CTime: 999},
+		},
+	}
+	r := NewBackupRunner(api, "local", "", "felhom test", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err != nil {
+		t.Fatalf("Backup: %v", err)
+	}
+	if !rec.Success || !rec.CrashConsistent {
+		t.Errorf("record = %+v, want success + crash_consistent", rec)
+	}
+	if rec.Archive != "local:backup/new-9001.tar.zst" || rec.SizeBytes != 524288000 {
+		t.Errorf("resolved wrong archive/size: %+v", rec)
+	}
+	if rec.Mode != string(proxmox.ModeSnapshot) {
+		t.Errorf("mode = %q, want snapshot (default)", rec.Mode)
+	}
+	// Bulk gap: mp0 (backup=0) AND mp2 (unset) are uncovered; mp1 (backup=1) is NOT.
+	if got := rec.UncoveredVolumes; len(got) != 2 || !has(got, "/mnt/bulk") || !has(got, "/mnt/scratch") {
+		t.Errorf("uncovered = %v, want [/mnt/bulk /mnt/scratch] (unset is uncovered too)", got)
+	}
+	if has(rec.UncoveredVolumes, "/mnt/db") {
+		t.Error("backup=1 mountpoint must NOT be reported uncovered")
+	}
+	// Notes-template threaded through.
+	if len(api.vzdumps) != 1 || api.vzdumps[0].Notes != "felhom test" {
+		t.Errorf("vzdump opts = %+v", api.vzdumps)
+	}
+}
+
+func TestBackup_ReportsActualModeFromTaskLog(t *testing.T) {
+	// Requested snapshot, but PVE used stop (stopped guest) — the report must reflect ACTUAL.
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:vzdump:1",
+		content:    []proxmox.StorageContent{{VolID: "v", Content: "backup", VMID: 9001, Size: 10, CTime: 1}},
+		logLines:   []string{"INFO: CT Name: spike", "INFO: backup mode: stop", "INFO: Finished"},
+	}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if rec.Mode != "stop" {
+		t.Errorf("mode = %q, want the ACTUAL %q from the task log (not the requested snapshot)", rec.Mode, "stop")
+	}
+}
+
+func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
+	api := &fakeBackupAPI{vzdumpErr: errors.New("vzdump boom")}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if rec.Success || rec.Error == "" {
+		t.Errorf("failed backup must produce a Success=false record with an Error: %+v", rec)
+	}
+}
+
+func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
+	const big = 4 << 30 // a plausible whole-guest archive
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		// R-689: real vzdump names with their vmid — only a backup OF A GUEST is a candidate.
+		{VolID: "local:backup/vzdump-lxc-9001-a.tar.zst", VMID: 9001, Content: "backup", CTime: 10, Size: big},
+		{VolID: "local:backup/vzdump-lxc-9001-b.tar.zst", VMID: 9001, Content: "backup", CTime: 99, Size: big},
+		{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
+	}}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	vol, err := r.PickRestoreCandidate(context.Background())
+	if err != nil || vol != "local:backup/vzdump-lxc-9001-b.tar.zst" {
+		t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
+	}
+	// no backups → "".
+	api.content = []proxmox.StorageContent{{VolID: "iso", Content: "iso"}}
+	if vol, _ := r.PickRestoreCandidate(context.Background()); vol != "" {
+		t.Errorf("no backup → empty, got %q", vol)
+	}
+}
+
+// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
+// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
+// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
+// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
+//
+// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
+// PickSettledRestoreCandidateOn and this fails with
+// `pick = "phantom" want the newest COMPLETE archive 'real'`.
+func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		{VolID: "felhom-pbs:backup/ct/9001/real", VMID: 9001, Content: "backup", CTime: 10, Size: 4 << 30},
+		{VolID: "felhom-pbs:backup/ct/9001/phantom", VMID: 9001, Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
+	}}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	vol, err := r.PickRestoreCandidate(context.Background())
+	if err != nil || vol != "felhom-pbs:backup/ct/9001/real" {
+		t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
+	}
+}
+
+// --- scheduler ---
+
+type fakeRTRunner struct {
+	res  reconcile.RestoreTestResult
+	runs int
+}
+
+func (f *fakeRTRunner) RunRestoreTest(_ context.Context, _ reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
+	f.runs++
+	return f.res
+}
+
+func TestScheduler_TickRunsAndRecords(t *testing.T) {
+	store := NewStore()
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick:   func(context.Context) (string, error) { return "vol", nil },
+		Store:  store,
+		Spec: func(context.Context, string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence: time.Hour,
+		Logger:  quiet(),
+	})
+	s.tick(context.Background())
+	if rt.runs != 1 {
+		t.Fatalf("tick should run the restore-test once, got %d", rt.runs)
+	}
+	got := store.RestoreTests(context.Background())
+	if len(got) != 1 || !got[0].Pass || got[0].SourceArchive != "vol" {
+		t.Fatalf("store should have the recorded result: %+v", got)
+	}
+}
+
+func TestScheduler_TickNoBackupNoOp(t *testing.T) {
+	rt := &fakeRTRunner{}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt, Pick: func(context.Context) (string, error) { return "", nil },
+		Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background())
+	if rt.runs != 0 {
+		t.Errorf("no backup available → no restore-test run, got %d", rt.runs)
+	}
+}
+
+func TestScheduler_DisabledRunReturnsOnCancel(t *testing.T) {
+	s := NewScheduler(SchedulerOptions{Cadence: 0, Logger: quiet()})
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- s.Run(ctx) }()
+	cancel()
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatalf("disabled scheduler Run should return nil on cancel, got %v", err)
+		}
+	case <-time.After(time.Second):
+		t.Fatal("disabled scheduler did not return on cancel")
+	}
+}
+
+func TestStore_Reporters(t *testing.T) {
+	s := NewStore()
+	if len(s.Backups(context.Background())) != 0 || len(s.RestoreTests(context.Background())) != 0 {
+		t.Fatal("empty store must report empty (non-nil) slices")
+	}
+}
+
+func has(ss []string, want string) bool {
+	for _, s := range ss {
+		if s == want {
+			return true
+		}
+	}
+	return false
+}

internal/backup/doc.go

@@ -0,0 +1,23 @@
+// Package backup is the guest-level backup + self-restore-test layer (doc 03 §8, slice 6
+// Phase A). It orchestrates a crash-consistent vzdump to a LOCAL target, resolves the
+// produced archive, and drives the self-restore-test (restore → boot → verify → teardown)
+// through the reconcile engine so it inherits the journal / per-guest serialization /
+// crash-safe recovery.
+//
+// Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): it reuses the
+// slice-4 classifier/gate/journal via reconcile — no new destructive class, no new crypto.
+// Restore is to a NEW guest only (no overwrite this slice). PBS / offsite / zero-knowledge
+// is Phase B.
+//
+// Layout:
+//   - runner.go   — BackupRunner: vzdump + archive-volid/size resolve + the bulk-volume gap;
+//     restore-candidate picker.
+//   - store.go    — in-memory latest-backup-per-target + latest-restore-test, implementing
+//     the hub BackupReporter / RestoreTestReporter seams (point-in-time state
+//     the collector reads; re-populated each cadence/selftest run).
+//   - schedule.go — the restore-test cadence goroutine (default 24h; disabled when 0).
+//
+// hub does NOT import this package (the report types live in hub; this package imports hub
+// for them, mirroring the slice-5 storage seam). This package may import reconcile + hub +
+// proxmox (acyclic).
+package backup

internal/backup/inflight.go

@@ -0,0 +1,69 @@
+package backup
+
+import "sync"
+
+// InFlight is the host-wide "one heavy guest operation at a time" gate.
+//
+// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
+// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
+// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
+// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
+// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
+// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
+// up recorded as failed.
+//
+// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
+// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
+//
+// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
+// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
+// coverage; cancelling a running backup costs the backup.
+//
+// CORRECTED 2026-07-28 (F-A1). That "DEFERS" was true of the restore-test caller and NOT of the
+// backup caller, and the comment did not say so. The controller's start path had no 409 branch, so
+// a refusal here was recorded as a tier FAILURE: the R-88 breaker armed and the operator was
+// emailed "Whole-guest backup FAILED" about a backup that was merely waiting its turn. Campaign 8
+// observed it on both demo boxes in the same minute.
+//
+// Fixed on the CONTROLLER side (v0.179.0), which is where the misreading lived — this gate's
+// behaviour was correct throughout and is unchanged. The controller now maps HTTP 409 to a
+// contention path: it defers the tier, keeps it DUE, and alarms only if contention outlives the
+// agent's own restore-test ceiling. Nothing here needs to change; the claim above is simply now
+// true of both callers.
+type InFlight struct {
+	mu   sync.Mutex
+	what string // "" = idle
+}
+
+// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
+// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
+func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
+	if g == nil {
+		// Not wired (older call sites, tests) → no gating, previous behaviour.
+		return func() {}, "", true
+	}
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.what != "" {
+		return nil, g.what, false
+	}
+	g.what = what
+	var once sync.Once
+	return func() {
+		once.Do(func() {
+			g.mu.Lock()
+			g.what = ""
+			g.mu.Unlock()
+		})
+	}, "", true
+}
+
+// Busy reports what currently holds the gate ("" = idle).
+func (g *InFlight) Busy() string {
+	if g == nil {
+		return ""
+	}
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	return g.what
+}

internal/backup/prune_test.go

@@ -0,0 +1,99 @@
+package backup
+
+import (
+	"context"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/config"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// localTarget is a non-PBS dir storage; pbsTarget is a PBS storage — for the scope gate.
+var (
+	localTargetStores = []proxmox.Storage{{Storage: "local", Type: "dir", Content: "backup"}}
+	pbsTargetStores   = []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", Content: "backup"}}
+)
+
+func okAPI(stores []proxmox.Storage) *fakeBackupAPI {
+	return &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: stores}
+}
+
+// TestPrune_LocalCarriesKeepLast: a LOCAL-target backup with retention carries `--prune-backups
+// keep-last=3` on the vzdump. Companion: the SAME runner built with no retention ("") emits NO prune
+// option — proving the flag only rides when retention is set (a no-prune build accumulates).
+func TestPrune_LocalCarriesKeepLast(t *testing.T) {
+	api := okAPI(localTargetStores)
+	r := NewBackupRunner(api, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201) // archive-resolution may fail in the fake; we assert the captured vzdump opts
+	if len(api.vzdumps) != 1 || api.vzdumps[0].PruneBackups != "keep-last=3" {
+		t.Fatalf("local backup must carry prune-backups keep-last=3, got %q", api.vzdumps[0].PruneBackups)
+	}
+
+	// COMPANION: no retention → no prune option (dumps would accumulate).
+	api2 := okAPI(localTargetStores)
+	r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "", quiet())
+	_, _ = r2.Backup(context.Background(), 9201)
+	if api2.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("a no-retention runner must NOT prune, got %q", api2.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_NeverPrunesPBS is the scope rule (§9): retention is NOT applied when the target is a PBS
+// storage (offsite retention is a separate lifecycle). Companion: the identical retention on a LOCAL
+// target IS applied — proving the gate keys on storage type, not luck.
+func TestPrune_NeverPrunesPBS(t *testing.T) {
+	api := okAPI(pbsTargetStores)
+	r := NewBackupRunner(api, "felhom-pbs", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if api.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("a PBS target must NEVER be pruned by the per-run flag, got %q", api.vzdumps[0].PruneBackups)
+	}
+
+	// COMPANION: same retention, local target → applied.
+	api2 := okAPI(localTargetStores)
+	r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r2.Backup(context.Background(), 9201)
+	if api2.vzdumps[0].PruneBackups != "keep-last=3" {
+		t.Fatalf("control: a local target with the same retention MUST be pruned, got %q", api2.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_FailSafeOnUnknownTarget: if the target's type can't be confirmed (lookup error / not in the
+// list), the run SKIPS pruning rather than risk pruning a PBS/unknown storage.
+func TestPrune_FailSafeOnUnknownTarget(t *testing.T) {
+	// target not present in the list → skip.
+	api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: localTargetStores}
+	r := NewBackupRunner(api, "some-other-store", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if api.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("an unknown target must skip pruning (fail-safe), got %q", api.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_KeepLastClamp (§7-B): a 0/negative/unset LocalBackupRetention clamps to ≥1 (default 3) so the
+// vzdump NEVER prunes the archive it just made. Companion: a no-clamp impl that returns 0 would emit
+// keep-last=0 → PVE prunes everything → FAILS the "≥1" assertion.
+func TestPrune_KeepLastClamp(t *testing.T) {
+	cases := []struct {
+		set  int
+		want int
+	}{
+		{0, 3},   // unset → default
+		{-5, 3},  // negative → default
+		{1, 1},   // honored
+		{3, 3},   // honored
+		{10, 10}, // honored
+	}
+	for _, c := range cases {
+		b := config.BackupConfig{LocalBackupRetention: c.set}
+		if got := b.KeepLast(); got != c.want {
+			t.Errorf("KeepLast(%d) = %d, want %d", c.set, got, c.want)
+		}
+		if b.KeepLast() < 1 {
+			t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
+		}
+	}
+	if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" {
+		t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec)
+	}
+}

internal/backup/r672_skip_test.go

@@ -0,0 +1,36 @@
+package backup
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-672 (v0.133.0): a restore-test the SPACE preflight refused is the test's RESULT — recorded for the
+// hub as pass=false with the reason, never dropped (a band skip still is) and never a pass.
+//
+// COMPANION RED-PROOF (REPORT): the scheduler's pre-v0.133.0 `if res.Skipped { return }` → "a space
+// refusal never reached the host report".
+func TestR672_SpaceSkipIsReportedNotDropped(t *testing.T) {
+	store := NewStore()
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Skipped: true,
+		SkipReason: "skipped: not enough space on local-lvm: restoring 21.1 GiB (vzdump log) needs 30.3 GiB free, has 21.6 GiB"}}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt, Pick: func(context.Context) (string, error) { return "vol", nil }, Store: store,
+		Spec: func(context.Context, string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background())
+	got := store.RestoreTests(context.Background())
+	if len(got) != 1 {
+		t.Fatalf("a space refusal never reached the host report: %+v", got)
+	}
+	if got[0].Pass || !got[0].Skipped || !strings.HasPrefix(got[0].Error, "skipped: not enough space") {
+		t.Fatalf("record = %+v — want pass=false, skipped, the reason as the error", got[0])
+	}
+}

internal/backup/r685_space_test.go

@@ -0,0 +1,83 @@
+package backup
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// R-685 (v0.134.0) — a whole-box backup that cannot fit its LOCAL target is a named skip BEFORE anything
+// starts, with the numbers in the record's Error, never a vzdump that fills the disk and fails.
+
+const gib = int64(1) << 30
+
+func spaceAPI(avail int64, lastArchive int64, typ string) *fakeBackupAPI {
+	api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1",
+		storages: []proxmox.Storage{{Storage: "local", Type: typ, Content: "backup", Avail: avail}}}
+	if lastArchive > 0 {
+		api.content = []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst",
+			Content: "backup", VMID: 9201, Size: lastArchive, CTime: 1790280000}}
+	}
+	return api
+}
+
+// TestR685_BackupThatCannotFitIsSkipped — demo-hp's shape: an 8.2 GB archive, 4 GiB free. No vzdump is
+// started; the record says why, with the numbers, under a stable prefix.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the spaceFits call from backup() — this test fails at "a vzdump
+// was started on a target that cannot hold it".
+func TestR685_BackupThatCannotFitIsSkipped(t *testing.T) {
+	api := spaceAPI(4*gib, 8182759056, "dir")
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	rec, err := r.Backup(context.Background(), 9201)
+	if len(api.vzdumps) != 0 {
+		t.Fatalf("a vzdump was started on a target that cannot hold it: %+v", api.vzdumps)
+	}
+	if err == nil || rec.Success || !strings.HasPrefix(rec.Error, BackupSkipNoSpacePrefix) {
+		t.Fatalf("want a named skip, got err=%v rec=%+v", err, rec)
+	}
+	for _, want := range []string{"4.0 GiB free", "7.6 GiB", "10.5 GiB"} {
+		if !strings.Contains(rec.Error, want) {
+			t.Errorf("the reason must carry the numbers (%q missing): %s", want, rec.Error)
+		}
+	}
+}
+
+// TestR685_BackupThatFitsRuns — the same archive with 16 GiB free (demo-hp after tonight's prune) runs.
+func TestR685_BackupThatFitsRuns(t *testing.T) {
+	api := spaceAPI(16*gib, 8182759056, "dir")
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if len(api.vzdumps) != 1 {
+		t.Fatalf("a backup that fits must run, vzdumps=%d", len(api.vzdumps))
+	}
+}
+
+// TestR685_FailsOpen — never refuse on what is not KNOWN: a PBS target, a first backup (no archive to
+// size from), an unknown free figure, or a storage list that cannot be read.
+func TestR685_FailsOpen(t *testing.T) {
+	cases := map[string]*fakeBackupAPI{
+		"pbs target":    spaceAPI(1*gib, 8*gib, "pbs"),
+		"first backup":  spaceAPI(1*gib, 0, "dir"),
+		"avail unknown": spaceAPI(0, 8*gib, "dir"),
+		"storage list error": func() *fakeBackupAPI {
+			a := spaceAPI(1*gib, 8*gib, "dir")
+			a.storageErr = context.DeadlineExceeded
+			return a
+		}(),
+	}
+	for name, api := range cases {
+		target := "local"
+		if name == "pbs target" {
+			api.storages[0].Storage = "felhom-pbs"
+			target = "felhom-pbs"
+		}
+		r := NewBackupRunner(api, target, proxmox.ModeSnapshot, "", "", quiet())
+		_, _ = r.Backup(context.Background(), 9201)
+		if len(api.vzdumps) != 1 {
+			t.Errorf("%s: the preflight must fail OPEN, but no vzdump ran", name)
+		}
+	}
+}

internal/backup/r689_golden_test.go

@@ -0,0 +1,112 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// R-689 (v0.135.0) — demo-hp keeps its golden template in `local:backup/`. It is content "backup",
+// 654 MB and so "plausibly complete", and it was the newest SETTLED entry: the restore test picked it
+// every 6 h and failed extractconfig with a 403 (measured 2026-09-24 10:36, 09-25 04:57 and 10:57),
+// while the guest's own archive — younger than the 24 h settle — went untested and nothing was proven.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the guestBackupArchive call from PickSettledRestoreCandidateOn —
+// this test then picks `local:backup/felhom-golden-0.236.0.tar.zst`.
+func TestR689_TheRestoreTestNeverPicksTheGolden(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790370000) // 2026-09-25 ~19:00Z
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		// the guest's real archive, settled (older than the cutoff below)
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 3*day},
+		// the golden: newer, settled, big, and NOT a backup of a guest
+		{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 2*day},
+		// a hand-copied tarball that PVE happens to attribute to a vmid — the name is not a vzdump's
+		{VolID: "local:backup/copy-of-9201.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 2*day},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst" {
+		t.Fatalf("picked %q — the restore test must prove a backup OF A GUEST", got)
+	}
+}
+
+func TestR689_GuestBackupArchiveShapes(t *testing.T) {
+	for _, c := range []struct {
+		e  proxmox.StorageContent
+		ok bool
+	}{
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst", VMID: 9201}, true},
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-qemu-300-2026_09_24-21_59_25.vma.zst", VMID: 300}, true},
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z", VMID: 9201}, true},
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
+		{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false},       // no vmid reported
+	} {
+		if ok, why := guestBackupArchive(c.e); ok != c.ok {
+			t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
+		}
+	}
+}
+
+// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
+// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
+// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
+func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
+	}
+	got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
+	if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
+		t.Fatalf("with 9201's archive settled the pick is %q", got)
+	}
+}
+
+// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
+// "nothing to prove".
+func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
+	api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
+		t.Fatal("a failed guest lookup read as a clean answer")
+	}
+}
+
+// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
+// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
+// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
+func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
+	}
+}

internal/backup/restoretest_due.go

@@ -0,0 +1,168 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"time"
+)
+
+// R-86 — a restore-test follows the BACKUP, not the clock.
+//
+// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
+//
+// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
+// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
+// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
+// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
+// sometimes twice on the same archive.
+//
+// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
+//
+// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
+// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
+// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
+// The naive rule silently switches restore-testing off for the tier that matters most, and it is
+// the version a reasonable person would write. It has a red-proof of its own
+// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
+//
+// The rule implemented here:
+//
+//	Let A = the newest archive on this tier that is at least `settle` old.
+//	The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
+//
+//	daily tier    → A is yesterday's archive; a new one settles each day  → proved once per day
+//	weekly tier   → A is last week's until the next settles               → proved once per week
+//	newborn tier  → A does not exist                                      → UNKNOWN, never a fault
+//
+// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
+// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
+// produce a cadence nobody can predict from either.
+//
+// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
+//
+// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
+// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
+// the trigger changed.
+
+// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
+// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
+type DueVerdict struct {
+	Target string // the tier's storage target id
+
+	// Due is true only when Archive is set and has not been proven.
+	Due bool
+	// Archive is the settled candidate A ("" when the tier holds none).
+	Archive string
+	// Landed is when A landed on the tier (zero when Archive is "").
+	Landed time.Time
+	// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
+	ProvenArchive string
+	// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
+	// NEVER reported as "not due", which would silently retire a tier the moment its storage
+	// stopped answering. Due stays false (we have no archive to test) and the error travels.
+	Err error
+	// Reason is the one-line human account of this verdict.
+	Reason string
+}
+
+// String renders a verdict for the operator log / selftest output.
+func (v DueVerdict) String() string {
+	return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
+}
+
+// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
+//
+// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
+// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
+// still never be starved — a tier that has waited longest is served first — and keeping it as the
+// order rather than as the trigger is the whole of this change.
+func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
+	if !s.rotating() {
+		return nil
+	}
+	order := s.tiers
+	if s.rtState != nil {
+		order = s.rtState.OldestFirst(s.tiers)
+	}
+	cutoff := s.settleCutoff()
+	out := make([]DueVerdict, 0, len(order))
+	for _, target := range order {
+		out = append(out, s.evaluateTier(ctx, target, cutoff))
+	}
+	return out
+}
+
+// settleCutoff is the newest landing time an archive may have and still count as settled.
+func (s *Scheduler) settleCutoff() time.Time {
+	if s.settle <= 0 {
+		return time.Time{} // no settle requirement configured → any archive is a candidate
+	}
+	return s.now().Add(-s.settle)
+}
+
+// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
+// unit-tested directly rather than inferred from whether a fake runner happened to be called.
+func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
+	v := DueVerdict{Target: target}
+	archive, landed, err := s.tierPick(ctx, target, cutoff)
+	if err != nil {
+		// UNKNOWN, never "not due", and never silent.
+		v.Err = err
+		v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
+		return v
+	}
+	v.Archive, v.Landed = archive, landed
+	if archive == "" {
+		v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
+		return v
+	}
+	proven, ok := "", false
+	if s.rtState != nil {
+		proven, ok = s.rtState.ProvenArchive(target)
+	}
+	v.ProvenArchive = proven
+	if ok && proven == archive {
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
+		return v
+	}
+	v.Due = true
+	switch {
+	case !ok && proven == "":
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
+	default:
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
+	}
+	return v
+}
+
+// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
+// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
+func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
+	return s.evaluateTier(ctx, target, s.settleCutoff())
+}
+
+// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
+//
+// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
+// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
+// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
+// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
+// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
+func (s *Scheduler) verdictSummary(ctx context.Context) string {
+	out := ""
+	for _, v := range s.EvaluateDue(ctx) {
+		if out != "" {
+			out += "; "
+		}
+		switch {
+		case v.Err != nil:
+			out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
+		default:
+			out += v.Target + ": " + v.Reason
+		}
+	}
+	if out == "" {
+		return "no tiers configured"
+	}
+	return out
+}

internal/backup/restoretest_due_test.go

@@ -0,0 +1,593 @@
+package backup
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-86 — the restore-test follows the BACKUP, not the clock.
+//
+// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
+// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
+// asserted "no error" would pass against a scheduler that never ran anything at all — which is
+// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
+// archive, and did a second evaluation correctly run NOTHING.
+
+// ── the fake tier storage ────────────────────────────────────────────────────────────────────
+
+// archiveStub is one archive on a tier: its volid and when it landed.
+type archiveStub struct {
+	volid  string
+	landed time.Time
+}
+
+// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
+// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
+// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
+// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
+// picker; that the real picker obeys the same rule is asserted there, not here.
+type tierStorage struct {
+	archives map[string][]archiveStub
+	err      map[string]error // target → lookup failure
+}
+
+func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
+	if e, ok := ts.err[target]; ok && e != nil {
+		return "", time.Time{}, e
+	}
+	var best archiveStub
+	for _, a := range ts.archives[target] {
+		if !notAfter.IsZero() && a.landed.After(notAfter) {
+			continue // not settled yet
+		}
+		if best.volid == "" || a.landed.After(best.landed) {
+			best = a
+		}
+	}
+	return best.volid, best.landed, nil
+}
+
+// dueHarness is a scheduler with a driven clock over a fake tier storage.
+type dueHarness struct {
+	s     *Scheduler
+	rr    *rotRunner
+	st    *RestoreTestState
+	ts    *tierStorage
+	clock time.Time
+	path  string
+}
+
+func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
+	t.Helper()
+	return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
+}
+
+func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
+	t.Helper()
+	h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
+	h.st = NewRestoreTestState(statePath)
+	h.s = NewScheduler(SchedulerOptions{
+		Runner: h.rr,
+		Store:  NewStore(),
+		Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence:  time.Hour,
+		Settle:   settle,
+		Logger:   quiet(),
+		Tiers:    tiers,
+		TierPick: ts.pick,
+		State:    h.st,
+		InFlight: &InFlight{},
+	})
+	h.s.now = func() time.Time { return h.clock }
+	return h
+}
+
+// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
+func (h *dueHarness) advance(step, total time.Duration) {
+	for elapsed := time.Duration(0); elapsed < total; elapsed += step {
+		h.clock = h.clock.Add(step)
+		h.s.tick(context.Background())
+	}
+}
+
+var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
+
+// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
+func dailyArchives(tier string, n int) []archiveStub {
+	out := make([]archiveStub, 0, n)
+	for d := 0; d < n; d++ {
+		out = append(out, archiveStub{
+			volid:  fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
+			landed: day0.AddDate(0, 0, d),
+		})
+	}
+	return out
+}
+
+// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
+//
+// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
+// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
+// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
+// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
+// the tier that matters most.
+//
+// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
+// was replaced by the naive age rule:
+//
+//   - if ok && proven == archive { … not due … }
+//   - if s.now().Sub(landed) < s.settle { … not due … }   // and the proven-archive check deleted
+//
+// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
+// old enough". Result:
+//
+//	--- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
+//	    restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
+//
+// Zero runs — restore-testing off. Restored immediately afterwards.
+func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
+	h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	// Five days, evaluated hourly.
+	h.advance(time.Hour, 5*24*time.Hour)
+
+	got := h.rr.seen()
+	if len(got) != 5 {
+		t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
+	}
+	// And each run must be on the archive that settled that day — day0's on day 1, and so on.
+	for i, a := range got {
+		want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
+		if a != want {
+			t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
+		}
+	}
+	// The newest archive is NEVER the one tested: it has not settled.
+	if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
+		t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
+	}
+}
+
+// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
+func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
+		{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
+		{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
+		{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
+	}}}
+	h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+
+	// Three weeks, evaluated every 6 hours — 84 evaluations.
+	h.advance(6*time.Hour, 21*24*time.Hour)
+
+	got := h.rr.seen()
+	want := []string{
+		"felhom-pbs:backup/ct/9201/w0",
+		"felhom-pbs:backup/ct/9201/w1",
+		"felhom-pbs:backup/ct/9201/w2",
+	}
+	if len(got) != len(want) {
+		t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
+		}
+	}
+}
+
+// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
+//
+// This is the defect a person actually notices: today every deploy restarts the ticker, so a
+// restore-test runs one interval after each deploy regardless of what has already been proven.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
+// ProvenArchive ignore the stored archive —
+//
+//   - if !ok || p.Archive == "" { return "", false }
+//   - return "", false     // per-tier time only, the pre-R-86 state
+//
+// →	--- FAIL: TestDue_RestartRunsNothing
+//
+//	restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
+//	produced 4 run(s)
+//
+// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
+// today's behaviour with the ticker's phase reset by the deploy. Restored.
+func TestDue_RestartRunsNothing(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
+	start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
+
+	h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
+	}
+
+	// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
+	total := 0
+	for i := 0; i < 2; i++ {
+		h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+		h2.s.tick(context.Background())
+		h2.s.tick(context.Background())
+		total += len(h2.rr.seen())
+	}
+	if total != 0 {
+		t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
+	}
+}
+
+// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
+func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	h.s.tick(context.Background()) // proves day0's archive
+	h.s.tick(context.Background()) // nothing new has settled → nothing
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
+	}
+
+	// Day 2, 03:00 — day1's archive has now settled.
+	h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
+	h.s.tick(context.Background())
+
+	got := h.rr.seen()
+	if len(got) != 2 {
+		t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
+	}
+	if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
+		t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
+	}
+}
+
+// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
+//
+//   - if rt.Pass && s.rtState != nil && target != "" {
+//   - if s.rtState != nil && target != "" {
+//
+// →	--- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
+//
+//	restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
+//	evaluations
+//
+// A single failure would have retired the archive as proven — a permanently broken DR tier looking
+// freshly verified, which is the loudest signal this system produces going silent. Restored.
+func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
+
+	for i := 0; i < 3; i++ {
+		h.s.tick(context.Background())
+	}
+
+	got := h.rr.seen()
+	if len(got) != 3 {
+		t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
+	}
+	if _, ok := h.st.ProvenArchive("local"); ok {
+		t.Fatal("a FAILED restore-test must not record the archive as proven")
+	}
+	if _, ok := h.st.LastSuccess("local"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+}
+
+// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
+func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{
+		"local":      {{volid: "local:backup/a.tar.zst", landed: day0}},
+		"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
+	}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+
+	// Both tiers are due at this instant.
+	due := h.s.EvaluateDue(context.Background())
+	if len(due) != 2 || !due[0].Due || !due[1].Due {
+		t.Fatalf("precondition: both tiers should be due; got %v", due)
+	}
+
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
+	}
+
+	// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
+	h.s.tick(context.Background())
+	got := h.rr.seen()
+	if len(got) != 2 || got[0] == got[1] {
+		t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
+	}
+}
+
+// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
+func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	gate := &InFlight{}
+	h.s.inFlight = gate
+	release, _, _ := gate.TryAcquire("backup:felhom-pbs")
+
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
+	}
+	if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
+		t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
+	}
+	release()
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("must resume once the gate frees; got %d run(s)", n)
+	}
+}
+
+// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
+func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
+	h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+
+	due := h.s.EvaluateDue(context.Background())
+	if len(due) != 1 {
+		t.Fatalf("want one verdict, got %v", due)
+	}
+	v := due[0]
+	if v.Due || v.Err != nil || v.Archive != "" {
+		t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
+	}
+	if v.Reason == "" {
+		t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
+	}
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
+	}
+}
+
+// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
+func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
+	h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
+		t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
+	}
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("nothing settled → no run; got %d", n)
+	}
+}
+
+// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
+// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
+// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
+func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
+	boom := errors.New("storage unreachable")
+	ts := &tierStorage{
+		archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
+		err:      map[string]error{"felhom-pbs": boom},
+	}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+
+	var pbs DueVerdict
+	for _, v := range h.s.EvaluateDue(context.Background()) {
+		if v.Target == "felhom-pbs" {
+			pbs = v
+		}
+	}
+	if pbs.Err == nil {
+		t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
+	}
+	if pbs.Due {
+		t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
+	}
+	if pbs.Reason == "" {
+		t.Fatal("the failure must be explained, not merely flagged")
+	}
+
+	// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
+	// tier its proof.
+	h.s.tick(context.Background())
+	if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
+		t.Fatalf("the readable tier must still be proved; got %v", got)
+	}
+}
+
+// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
+
+// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
+// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
+// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
+// invented by a migration.
+func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
+	if err := writeFileForTest(path, legacy); err != nil {
+		t.Fatal(err)
+	}
+
+	st := NewRestoreTestState(path)
+	if _, ok := st.ProvenArchive("local"); ok {
+		t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
+	}
+	at, ok := st.LastSuccess("local")
+	if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
+		t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
+	}
+	// Ordering still works off the legacy times.
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		t.Fatalf("oldest-first must still order legacy records; got %v", got)
+	}
+}
+
+// The new shape round-trips, archive and all.
+func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	now := time.Now().UTC().Truncate(time.Second)
+	st := NewRestoreTestState(path)
+	if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
+		t.Fatal(err)
+	}
+	re := NewRestoreTestState(path)
+	got, ok := re.ProvenArchive("felhom-pbs")
+	if !ok || got != "felhom-pbs:backup/ct/9201/x" {
+		t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
+	}
+	at, ok := re.LastSuccess("felhom-pbs")
+	if !ok || !at.Equal(now) {
+		t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
+	}
+}
+
+// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
+func writeFileForTest(path, content string) error {
+	return os.WriteFile(path, []byte(content), 0o600)
+}
+
+// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
+// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
+// quiet journal is equally consistent with a healthy loop and a dead goroutine.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
+// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
+// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
+func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{
+		"local":      {{volid: "local:backup/a.tar.zst", landed: day0}},
+		"felhom-pbs": nil, // no archive at all
+	}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+	// Prove the local tier so NOTHING is due.
+	if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
+		t.Fatal(err)
+	}
+
+	// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
+	// test would pass against a tick that never calls it.
+	var logbuf strings.Builder
+	h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
+	h.s.tick(context.Background())
+	got := logbuf.String()
+	for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
+		if !strings.Contains(got, want) {
+			t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
+		}
+	}
+}
+
+// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
+// reads as "nothing due" is the silence this rule exists to prevent.
+func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
+	ts := &tierStorage{
+		archives: map[string][]archiveStub{"local": nil},
+		err:      map[string]error{"felhom-pbs": errors.New("storage unreachable")},
+	}
+	h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+	got := h.s.verdictSummary(context.Background())
+	if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
+		t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
+	}
+}
+
+// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
+//
+// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
+// agent will not repeat the work. So the persisted record has to be able to become a host-report
+// entry — without inventing anything it does not know.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
+// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
+//
+//	--- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
+//	    restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
+//	    per-tier proof on it); got [{... SourceTier: ...}]
+//
+// Restored.
+func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
+	// file has ever had, which is what a real box carries after two upgrades.
+	legacy := `{
+	  "old-v1":     "2026-07-30T02:11:07Z",
+	  "old-v2":     {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
+	  "felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
+	}`
+	if err := writeFileForTest(path, legacy); err != nil {
+		t.Fatal(err)
+	}
+
+	got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
+	if len(got) != 1 {
+		t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
+	}
+	e := got[0]
+	if e.SourceTier != "pbs" {
+		t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
+	}
+	if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
+		t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
+	}
+	if e.TestedAt != "2026-08-03T13:25:14Z" {
+		t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
+	}
+	if e.Verified != "boot+running" {
+		t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
+	}
+	// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
+	if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
+		t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
+			e.DurationSeconds, e.ScratchVMID)
+	}
+	// The legacy records still serve the DUE-check, which is a separate question from reporting.
+	if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
+		t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
+	}
+}
+
+// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
+// the production path, not a hand-built fixture.
+func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+	// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
+	h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+		return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
+	}
+	h.s.tick(context.Background())
+
+	got := h.st.ProvenRestoreTests(context.Background())
+	if len(got) != 1 {
+		t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
+	}
+	if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
+		t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
+	}
+}
+
+// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
+func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
+	h.s.tick(context.Background())
+	if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
+		t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
+			"would outlive the fault; got %+v", got)
+	}
+}

internal/backup/restoretest_state.go

@@ -0,0 +1,289 @@
+package backup
+
+import (
+	"context"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"sort"
+	"sync"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
+//
+// R-85 (1.4). This one genuinely needs PERSISTENCE, unlike R-84 — and the difference is worth
+// stating, because the two look like the same problem and are not:
+//
+//   - R-84 (backup freshness) had a GROUND TRUTH to consult: the archive is still on the storage,
+//     so the agent could ask "when did a backup last land?" and never persist anything. That is
+//     strictly better, because a pruned archive correctly stops counting.
+//   - A restore-test leaves NO artifact — the scratch guest is destroyed as its final act. There is
+//     nothing to query. "Did we prove this tier restores?" exists only as remembered state, so it
+//     must be written down or it is lost.
+//
+// Why it must survive a restart: rotation is oldest-first (the operator ruling), so an in-memory map
+// would reset every tier to "never tested" on each restart. Ordering would then depend on map
+// iteration order, and one tier could be starved indefinitely while the other is re-tested — with
+// agent deploys as routine as they are, that is not a corner case.
+//
+// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
+// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
+// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
+// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
+// ever becoming due again. The rule earns its keep twice now.
+//
+// R-86 (1.2) — WHICH ARCHIVE, not just when.
+//
+// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
+// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
+// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
+// a tier is due once it holds a settled archive that has not been proven — so the identity of the
+// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
+//
+// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
+// here it records a result, but not WHICH result, and that is just as unable to answer the question
+// being asked of it.
+type RestoreTestState struct {
+	path string
+	mu   sync.Mutex
+	last map[string]provenTier // target id → what was last PROVEN on that tier
+}
+
+// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
+// and when.
+//
+// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
+// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
+// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
+// hub can stay ignorant of a real success until the next archive generation.
+//
+// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
+// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
+// at report-building time — a network call that can fail, on a path where failing means mis-labelling
+// a proof. Store what you knew when you knew it.
+type provenTier struct {
+	Archive  string    // volid of the archive that PASSED; "" = a legacy record with no archive
+	Tier     string    // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
+	Verified string    // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
+	At       time.Time // when that run passed (UTC)
+}
+
+// reportable reports whether this record can be re-reported to the hub as a restore-test result.
+//
+// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
+// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
+// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
+// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
+func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
+
+// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
+//
+//	v1 (pre-R-86)  "<target>": "<RFC3339>"                     — a time, no archive
+//	v2 (R-86)      "<target>": {archive, proven_at}            — due-check usable, not reportable
+//	v3 (R-189)     "<target>": {archive, tier, verified, …}    — both
+//
+// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
+// readers above test for — the migration needs no version number because the absence IS the answer.
+type provenTierJSON struct {
+	Archive  string `json:"archive"`
+	Tier     string `json:"tier,omitempty"`
+	Verified string `json:"verified,omitempty"`
+	ProvenAt string `json:"proven_at"`
+}
+
+// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
+// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
+// corrupt file from wedging the daemon.
+//
+// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
+// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
+// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
+// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
+// read a legacy time as proof of whatever archive happens to be current, which would mark an
+// unproven archive proven — inventing a guarantee out of a migration.
+func NewRestoreTestState(path string) *RestoreTestState {
+	s := &RestoreTestState{path: path, last: map[string]provenTier{}}
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return s
+	}
+	var raw map[string]json.RawMessage
+	if json.Unmarshal(data, &raw) != nil {
+		return s
+	}
+	for target, msg := range raw {
+		// Legacy shape: a bare RFC3339 string.
+		var legacy string
+		if json.Unmarshal(msg, &legacy) == nil {
+			if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
+				s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
+			}
+			continue
+		}
+		var cur provenTierJSON
+		if json.Unmarshal(msg, &cur) != nil {
+			continue // one unreadable entry must not lose the others
+		}
+		t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
+		if perr != nil {
+			continue
+		}
+		s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
+	}
+	return s
+}
+
+// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
+// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
+// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
+//
+// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
+// the next reader will notice failures are absent and try to "fix" it:
+//
+//	a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
+//	the system quietly less tested than it believes. It must survive a restart.
+//
+//	a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
+//	so a lost failure heals itself within one interval. Persisting it would do the opposite of
+//	helping: a healed tier would keep reporting a failure that is no longer true.
+func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
+	if target == "" {
+		return nil
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
+	return s.saveLocked()
+}
+
+// LastSuccess returns when this tier was last proven (ok=false = never).
+func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	p, ok := s.last[target]
+	return p.At, ok
+}
+
+// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
+// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
+// not this one has not been proven.
+func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	p, ok := s.last[target]
+	if !ok || p.Archive == "" {
+		return "", false
+	}
+	return p.Archive, true
+}
+
+// Snapshot returns a copy of the last-proven TIMES.
+//
+// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
+// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
+// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
+// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
+// only the times; if it acquires none, delete it rather than let it claim a purpose again.
+func (s *RestoreTestState) Snapshot() map[string]time.Time {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make(map[string]time.Time, len(s.last))
+	for k, v := range s.last {
+		out[k] = v.At
+	}
+	return out
+}
+
+// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
+//
+// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
+// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
+//
+//   - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
+//   - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
+//   - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
+//     is not a claim; a fabricated one would be.
+//
+// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
+// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
+// worse defect than the one this fixes.
+func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make([]hub.RestoreTest, 0, len(s.last))
+	for _, p := range s.last {
+		if !p.reportable() {
+			continue
+		}
+		out = append(out, hub.RestoreTest{
+			SourceArchive: p.Archive,
+			SourceTier:    p.Tier,
+			Pass:          true,
+			Verified:      p.Verified,
+			TestedAt:      p.At.UTC().Format(time.RFC3339),
+		})
+	}
+	// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
+	// iteration is randomised.
+	sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
+	return out
+}
+
+// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
+//
+// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
+// naturally prioritises a tier that has never been restore-tested at all — which on this fleet was
+// the offsite tier, unproven for its entire existence.
+//
+// Ties break on target id so the order is deterministic; without that, two tiers proven in the same
+// second would rotate by map iteration order, which is randomised in Go and would make the
+// behaviour untestable and occasionally starving.
+func (s *RestoreTestState) OldestFirst(targets []string) []string {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := append([]string(nil), targets...)
+	sort.SliceStable(out, func(i, j int) bool {
+		pi, oki := s.last[out[i]]
+		pj, okj := s.last[out[j]]
+		ti, tj := pi.At, pj.At
+		switch {
+		case !oki && !okj:
+			return out[i] < out[j] // both never proven → deterministic
+		case !oki:
+			return true // never proven wins
+		case !okj:
+			return false
+		case !ti.Equal(tj):
+			return ti.Before(tj)
+		default:
+			return out[i] < out[j]
+		}
+	})
+	return out
+}
+
+func (s *RestoreTestState) saveLocked() error {
+	raw := make(map[string]provenTierJSON, len(s.last))
+	for target, p := range s.last {
+		raw[target] = provenTierJSON{
+			Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
+			ProvenAt: p.At.UTC().Format(time.RFC3339),
+		}
+	}
+	data, err := json.MarshalIndent(raw, "", "  ")
+	if err != nil {
+		return err
+	}
+	if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
+		return err
+	}
+	tmp := s.path + ".tmp"
+	if err := os.WriteFile(tmp, data, 0o600); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, s.path)
+}

internal/backup/rotation_test.go

@@ -0,0 +1,354 @@
+package backup
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-85 Phase 2 — tier rotation, persisted state, and the one-heavy-operation gate.
+//
+// The failure this prevents is not hypothetical: demo-hp's DR tier reported `applied` with ZERO
+// snapshots for five days and nobody noticed, because the scheduler could only ever see the primary
+// tier. Rotation is what makes the offsite tier testable at all.
+
+// rotRunner records which archives it was asked to restore.
+type rotRunner struct {
+	mu       sync.Mutex
+	archives []string
+	pass     bool
+}
+
+func (r *rotRunner) RunRestoreTest(_ context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.archives = append(r.archives, spec.Archive)
+	return reconcile.RestoreTestResult{
+		Archive: spec.Archive, SourceTier: spec.SourceTier,
+		Pass: r.pass, Verified: "boot+running",
+	}
+}
+func (r *rotRunner) seen() []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]string(nil), r.archives...)
+}
+
+// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
+// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
+// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
+var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
+
+// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
+func archiveFor(m map[string]string) TierPicker {
+	return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
+		a := m[target]
+		if a == "" {
+			return "", time.Time{}, nil
+		}
+		return a, testLanded, nil
+	}
+}
+
+func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPicker, gate *InFlight) *Scheduler {
+	t.Helper()
+	return NewScheduler(SchedulerOptions{
+		Runner: rr,
+		Store:  NewStore(),
+		Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence:  time.Hour,
+		Logger:   quiet(),
+		Tiers:    []string{"local", "felhom-pbs"},
+		TierPick: pick,
+		State:    st,
+		InFlight: gate,
+	})
+}
+
+// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
+//
+// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
+// it was a faithful statement of the defect. It read:
+//
+//	4 ticks → 4 runs, and consecutive runs must hit different tiers
+//
+// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
+// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
+// newest settled archive is already proven is not re-tested just because time passed. So the
+// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
+// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
+//
+// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
+// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
+// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
+// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
+func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
+	}), &InFlight{})
+	s.now = func() time.Time { return time.Now().UTC() }
+
+	for i := 0; i < 4; i++ {
+		s.tick(context.Background())
+	}
+
+	got := rr.seen()
+	var sawLocal, sawPBS bool
+	for _, a := range got {
+		if len(a) >= 5 && a[:5] == "local" {
+			sawLocal = true
+		}
+		if len(a) >= 10 && a[:10] == "felhom-pbs" {
+			sawPBS = true
+		}
+	}
+	if !sawLocal || !sawPBS {
+		t.Fatalf("both tiers must be exercised; got %v", got)
+	}
+	// Exactly one run per tier: the archives never changed, so nothing became due a second time.
+	if len(got) != 2 {
+		t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
+	}
+	if got[0] == got[1] {
+		t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
+	}
+}
+
+// A tier with NO archive is skipped, not failed, and the other tier still runs. A brand-new offsite
+// tier legitimately has nothing to restore; turning that into a failure would make every fresh box
+// look broken for its first week.
+func TestRotation_EmptyTierSkippedNotFailed(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs": "", // provisioned but empty
+	}), &InFlight{})
+
+	s.tick(context.Background())
+	got := rr.seen()
+	if len(got) != 1 || got[0][:5] != "local" {
+		t.Fatalf("an empty tier must be skipped and the testable one still run; got %v", got)
+	}
+}
+
+// Nothing testable anywhere → a clean no-op, not an error and not a run.
+func TestRotation_NoArchivesAnywhereIsANoOp(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{}), &InFlight{})
+	s.tick(context.Background())
+	if got := rr.seen(); len(got) != 0 {
+		t.Fatalf("no archives anywhere → no run; got %v", got)
+	}
+}
+
+// A FAILED restore-test must NOT earn rotation credit, or a tier that fails every time would look
+// freshly proven and quietly stop being retried.
+func TestRotation_FailureEarnsNoCredit(t *testing.T) {
+	rr := &rotRunner{pass: false}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
+	}), &InFlight{})
+	s.tick(context.Background())
+	if _, ok := st.LastSuccess("local"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+	if _, ok := st.LastSuccess("felhom-pbs"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+}
+
+// ── SCENARIO E — rotation survives a restart ─────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed): make the state in-memory (construct a fresh
+// `NewRestoreTestState` on a DIFFERENT path for the second scheduler, i.e. lose the file) and this
+// fails with "after a restart the OTHER tier must be next; got felhom-pbs" — the same tier repeats
+// and the other is starved indefinitely, which with agent deploys as routine as they are is not a
+// corner case.
+func TestRotation_SurvivesRestart(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	picks := archiveFor(map[string]string{
+		"local":      "local:backup/x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
+	})
+
+	// First process: the OFFSITE tier is tested (never-proven sorts first).
+	rr1 := &rotRunner{pass: true}
+	st1 := NewRestoreTestState(path)
+	s1 := rotScheduler(t, rr1, st1, picks, &InFlight{})
+	s1.tick(context.Background())
+	first := rr1.seen()
+	if len(first) != 1 {
+		t.Fatalf("want one run, got %v", first)
+	}
+
+	// --- restart: brand-new state object reading the SAME file ---
+	rr2 := &rotRunner{pass: true}
+	st2 := NewRestoreTestState(path)
+	s2 := rotScheduler(t, rr2, st2, picks, &InFlight{})
+	s2.tick(context.Background())
+	second := rr2.seen()
+	if len(second) != 1 {
+		t.Fatalf("want one run after restart, got %v", second)
+	}
+
+	if second[0] == first[0] {
+		t.Fatalf("after a restart the OTHER tier must be next; got %s twice (rotation state was lost)", second[0])
+	}
+}
+
+// ── SCENARIO F — no collision with a backup ──────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed): drop the TryAcquire guard from `tick` and this fails with
+// "the restore-test must DEFER while a backup holds the gate; concurrent operations = 2" — the
+// count is the assertion, since "both completed" would pass against a fully concurrent
+// implementation.
+func TestRotation_DefersWhileABackupHoldsTheGate(t *testing.T) {
+	gate := &InFlight{}
+	release, _, ok := gate.TryAcquire("backup:felhom-pbs")
+	if !ok {
+		t.Fatal("precondition: the gate should have been free")
+	}
+	defer release()
+
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
+
+	s.tick(context.Background())
+
+	concurrent := 1 + len(rr.seen()) // the backup holding the gate, plus anything the tick started
+	if concurrent != 1 {
+		t.Fatalf("the restore-test must DEFER while a backup holds the gate; concurrent operations = %d", concurrent)
+	}
+}
+
+// Once the backup releases, the next cadence proceeds — deferral must not be permanent.
+func TestRotation_ResumesAfterTheGateFrees(t *testing.T) {
+	gate := &InFlight{}
+	release, _, _ := gate.TryAcquire("backup:local")
+
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
+
+	s.tick(context.Background())
+	if len(rr.seen()) != 0 {
+		t.Fatal("should have deferred while held")
+	}
+	release()
+	s.tick(context.Background())
+	if len(rr.seen()) != 1 {
+		t.Fatalf("must resume once the gate frees; got %v", rr.seen())
+	}
+}
+
+// The gate itself: one holder at a time, named, and release is idempotent.
+func TestInFlight_Semantics(t *testing.T) {
+	g := &InFlight{}
+	rel, busy, ok := g.TryAcquire("backup:local")
+	if !ok || busy != "" {
+		t.Fatalf("first acquire must succeed; ok=%v busy=%q", ok, busy)
+	}
+	if _, busy2, ok2 := g.TryAcquire("restore-test"); ok2 || busy2 != "backup:local" {
+		t.Fatalf("second acquire must fail and NAME the holder; ok=%v busy=%q", ok2, busy2)
+	}
+	rel()
+	rel() // idempotent — a double release must not free someone else's later claim
+	if g.Busy() != "" {
+		t.Fatalf("gate should be idle after release; busy=%q", g.Busy())
+	}
+	if _, _, ok3 := g.TryAcquire("restore-test"); !ok3 {
+		t.Fatal("gate must be reusable after release")
+	}
+}
+
+// A nil gate means "not wired" → no gating, pre-R-85 behaviour. Keeps every existing caller working.
+func TestInFlight_NilIsUngated(t *testing.T) {
+	var g *InFlight
+	rel, _, ok := g.TryAcquire("x")
+	if !ok {
+		t.Fatal("a nil gate must not block")
+	}
+	rel()
+	if g.Busy() != "" {
+		t.Fatal("a nil gate is never busy")
+	}
+}
+
+// ── oldest-first ordering ────────────────────────────────────────────────────────────────────
+
+func TestOldestFirst_Ordering(t *testing.T) {
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	now := time.Now().UTC()
+
+	// Never-proven sorts FIRST — the case that matters, since the offsite tier starts there.
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		// both never proven → deterministic tie-break by id
+		if got[0] != "felhom-pbs" && got[0] != "local" {
+			t.Fatalf("unexpected: %v", got)
+		}
+	}
+	_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
+	}
+	_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
+		t.Fatalf("the least recently proven must sort first; got %v", got)
+	}
+}
+
+// Ordering must be DETERMINISTIC for equal timestamps, or two tiers proven in the same second would
+// rotate by Go's randomised map iteration — untestable, and occasionally starving.
+func TestOldestFirst_DeterministicOnTies(t *testing.T) {
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	now := time.Now().UTC()
+	_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
+	_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
+	for i := 0; i < 20; i++ {
+		if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
+			t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
+		}
+	}
+}
+
+// The state file round-trips, and a corrupt file degrades to "nothing proven" rather than wedging.
+func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	now := time.Now().UTC().Truncate(time.Second)
+
+	st := NewRestoreTestState(path)
+	if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
+		t.Fatal(err)
+	}
+	reopened := NewRestoreTestState(path)
+	got, ok := reopened.LastSuccess("felhom-pbs")
+	if !ok || !got.Equal(now) {
+		t.Fatalf("state must round-trip; got %v ok=%v want %v", got, ok, now)
+	}
+
+	bad := filepath.Join(dir, "corrupt.json")
+	if err := os.WriteFile(bad, []byte("{{{not json"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	c := NewRestoreTestState(bad)
+	if _, ok := c.LastSuccess("felhom-pbs"); ok {
+		t.Fatal("a corrupt state file must degrade to 'nothing proven', not invent a timestamp")
+	}
+}

internal/backup/runner.go

@@ -0,0 +1,669 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// BackupAPI is the read+backup proxmox surface the runner needs. *proxmox.Client satisfies it.
+type BackupAPI interface {
+	Vzdump(ctx context.Context, opts proxmox.VzdumpOptions) (string, error)
+	WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
+	GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
+	StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
+	// ListStorage enumerates storages (name+type) — used to scope local-only retention (never prune PBS).
+	ListStorage(ctx context.Context) ([]proxmox.Storage, error)
+	// NodeStorage is GET /nodes/{node}/storage — the storages WITH live usage (avail/used). R-685's space
+	// preflight reads free space HERE: ListStorage (GET /storage) is the cluster DEFINITIONS and carries no
+	// usage at all — measured live 2026-09-24 on demo-hp, where reading it let a backup through.
+	NodeStorage(ctx context.Context) ([]proxmox.Storage, error)
+	// TaskLogTail reads trailing task-log lines — used to read the ACTUAL vzdump mode
+	// (PVE may downgrade a requested snapshot to stop for a stopped guest — spike B1).
+	TaskLogTail(ctx context.Context, upid string, limit int) ([]string, error)
+}
+
+// BackupRunner orchestrates a crash-consistent vzdump to a local target and reports the
+// result (incl. the bulk-volume gap). An agent-initiated vzdump is crash-consistent only
+// (no fsfreeze); the report says so.
+type BackupRunner struct {
+	api    BackupAPI
+	target string             // backup storage (content=backup)
+	mode   proxmox.BackupMode // default ModeSnapshot
+	notes  string             // optional notes-template
+	// retention is the per-run `--prune-backups` spec (e.g. "keep-last=3") applied to a LOCAL target after
+	// each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune
+	// (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target.
+	retention string
+	// waitTimeout bounds the WaitTask poll on this runner's vzdump. Per-TIER since R-82: 30m is
+	// right for a local vzdump and badly wrong for an offsite PBS upload (see the 2026-07-26 live
+	// failure recorded on config.BackupTargetConfig.WaitTimeoutSeconds). 0 → 30m (legacy).
+	waitTimeout time.Duration
+	// allowPBSPrune permits `--prune-backups` on a PBS-type target. OFF by default and ON only for
+	// an ADDITIONAL tier whose keep_last was set explicitly (operator ruling 2026-07-26: keep two
+	// weeks of weekly offsite backups).
+	//
+	// The blanket PBS refusal it replaces existed for a real reason and still applies to the
+	// PRIMARY tier: BackupTarget() DEFAULTS to "felhom-pbs" and KeepLast() DEFAULTS to 3, so a box
+	// with neither key set would silently prune its offsite DR to 3 restore points. An additional
+	// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
+	// there is a deliberate act.
+	allowPBSPrune bool
+	logger        *slog.Logger
+	now           func() time.Time
+	// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
+	// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
+	// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
+	// due-check is served from the local-API handler goroutines.
+	rejectedMu sync.Mutex
+	rejected   map[string]struct{}
+}
+
+// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
+// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
+// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
+func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner {
+	return NewBackupRunnerWithWait(api, target, mode, notes, retention, 0, logger)
+}
+
+// NewBackupRunnerWithWait is NewBackupRunner plus an explicit vzdump wait bound (0 → 30m).
+func NewBackupRunnerWithWait(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, logger *slog.Logger) *BackupRunner {
+	return NewBackupRunnerFull(api, target, mode, notes, retention, waitTimeout, false, logger)
+}
+
+// NewBackupRunnerFull is the full constructor. allowPBSPrune must be true ONLY for an additional
+// tier with an explicitly configured keep_last — see BackupRunner.allowPBSPrune.
+func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, allowPBSPrune bool, logger *slog.Logger) *BackupRunner {
+	if mode == "" {
+		mode = proxmox.ModeSnapshot
+	}
+	if logger == nil {
+		logger = slog.Default()
+	}
+	if waitTimeout <= 0 {
+		waitTimeout = 30 * time.Minute
+	}
+	return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention,
+		waitTimeout: waitTimeout, allowPBSPrune: allowPBSPrune, logger: logger,
+		now: func() time.Time { return time.Now().UTC() }}
+}
+
+// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
+// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
+// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
+// can't be confirmed (lookup error / not found), it SKIPS pruning rather than risk pruning PBS — the
+// host_disk + storage_fill detectors remain the safety net.
+func (r *BackupRunner) localPruneSpec(ctx context.Context) string {
+	if r.retention == "" {
+		return ""
+	}
+	stores, err := r.api.ListStorage(ctx)
+	if err != nil {
+		r.logger.Warn("backup: could not resolve target storage type — skipping local prune this run", "target", r.target, "err", err)
+		return ""
+	}
+	for _, s := range stores {
+		if s.Storage == r.target {
+			if s.Type == "pbs" && !r.allowPBSPrune {
+				// Not opted in → never prune the offsite DR (the pre-R-82 rule, and still the rule
+				// for the primary tier, whose target+retention both DEFAULT and could prune by
+				// accident).
+				return ""
+			}
+			return r.retention
+		}
+	}
+	r.logger.Warn("backup: target storage not found in list — skipping local prune this run (fail-safe)", "target", r.target)
+	return ""
+}
+
+// snapshotMarker is the vzdump task-log line that signals the storage snapshot has been created
+// and the backup is now reading from it — the point after which resuming the guest's app cannot
+// affect the backup (slice 8B.2; validated on PVE 9.2.2: `INFO: create storage snapshot 'vzdump'`).
+// It only appears in snapshot mode (stop mode takes no storage snapshot), so its presence ⟹
+// snapshot mode — the basis for the controller's early resume.
+const snapshotMarker = "create storage snapshot"
+
+// snapshotWatchInterval is how often watchForSnapshot polls the task log. A package var so tests
+// can shrink it (production: poll once a second — the marker appears in the first ~1s, §0).
+var snapshotWatchInterval = time.Second
+
+// Backup runs one vzdump of vmid to the local target and returns the report record.
+func (r *BackupRunner) Backup(ctx context.Context, vmid int) (hub.Backup, error) {
+	return r.backup(ctx, vmid, nil)
+}
+
+// BackupWithSnapshotHook is Backup plus an onSnapshot callback invoked ONCE, mid-backup, when the
+// storage snapshot has been taken (snapshot mode only) — the 8B.2 early-resume signal. In
+// stop/downgraded mode the marker never appears, so onSnapshot is never called (the caller then
+// resumes at completion). onSnapshot must be cheap + non-blocking (it runs on a watcher goroutine).
+func (r *BackupRunner) BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) {
+	return r.backup(ctx, vmid, onSnapshot)
+}
+
+// backup is the shared body. A failure is returned BOTH as an error and as a
+// Backup{Success:false,...} so the caller can record the failed attempt.
+func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) {
+	start := r.now()
+	rec := hub.Backup{
+		TargetID:        r.target,
+		VMID:            vmid,
+		Mode:            string(r.mode),
+		CrashConsistent: true, // always, this slice (no controller quiesce)
+		StartedAt:       start.Format(time.RFC3339),
+	}
+	if r.target == "" {
+		rec.Error = "no backup target configured"
+		return rec, fmt.Errorf("backup: %s", rec.Error)
+	}
+
+	// Bulk-volume gap: which mountpoints the vzdump will EXCLUDE (best-effort; a config-read
+	// failure just leaves the gap unknown, never fails the backup).
+	if cfg, err := r.api.GuestConfig(ctx, vmid); err == nil {
+		rec.UncoveredVolumes = uncoveredMountpoints(cfg.MountPoints())
+	} else {
+		r.logger.Warn("backup: could not read guest config for bulk-gap", "vmid", vmid, "err", err)
+		rec.UncoveredVolumes = []string{}
+	}
+
+	// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
+	// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
+	// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
+	if ok, why := r.spaceFits(ctx, vmid); !ok {
+		rec.Error = BackupSkipNoSpacePrefix + why
+		rec.DurationSeconds = time.Since(start).Seconds()
+		r.logger.Warn("backup SKIPPED by the space preflight (R-685) — nothing was started", "vmid", vmid, "target", r.target, "reason", why)
+		return rec, fmt.Errorf("backup: %s", rec.Error)
+	}
+
+	upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
+		VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
+		PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
+	})
+	if err != nil {
+		rec.Error = err.Error()
+		rec.DurationSeconds = time.Since(start).Seconds()
+		return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
+	}
+	if upid != "" {
+		// 8B.2: while the backup runs, watch the task log for the storage-snapshot marker and
+		// fire onSnapshot once (snapshot mode only) so the controller can resume its app early.
+		if onSnapshot != nil {
+			watchCtx, stopWatch := context.WithCancel(ctx)
+			defer stopWatch()
+			go r.watchForSnapshot(watchCtx, upid, onSnapshot)
+		}
+		if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
+			rec.Error = err.Error()
+			rec.DurationSeconds = time.Since(start).Seconds()
+			return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
+		}
+		// Report the ACTUAL mode PVE used (it may downgrade snapshot→stop for a stopped
+		// guest — spike B1), read from the task log; fall back to the requested mode.
+		if lines, err := r.api.TaskLogTail(ctx, upid, 200); err == nil {
+			if actual := parseBackupMode(lines); actual != "" {
+				rec.Mode = actual
+			}
+		}
+	}
+
+	// Resolve the produced archive (volid + size) — the task status carries no result volid.
+	vol, size, err := r.latestArchive(ctx, vmid)
+	if err != nil {
+		rec.Error = fmt.Sprintf("backup succeeded but archive not resolved: %v", err)
+		rec.DurationSeconds = time.Since(start).Seconds()
+		return rec, fmt.Errorf("backup: resolve archive vmid %d: %w", vmid, err)
+	}
+	rec.Archive = vol
+	rec.SizeBytes = size
+	rec.Success = true
+	rec.DurationSeconds = time.Since(start).Seconds()
+	r.logger.Info("backup: completed", "vmid", vmid, "target", r.target, "archive", vol, "size_bytes", size,
+		"uncovered_volumes", len(rec.UncoveredVolumes))
+	return rec, nil
+}
+
+// BackupSkipNoSpacePrefix starts a backup record's Error when the space preflight refused (R-685) — a
+// stable prefix the controller's page and the hub can key on.
+const BackupSkipNoSpacePrefix = "skipped: not enough space: "
+
+// Space preflight margins (R-685): the new archive is predicted as the newest archive of this guest on the
+// target × backupSpaceGrowth, plus backupSpaceFloorBytes of headroom for the host. MEASURED 2026-09-24:
+// demo-hp 9201's archives grew 5.8 → 6.2 → 6.9 → 7.6 GB in four nights (+10 % a night at worst), so 1.25
+// covers two nights' growth. PVE prunes old archives only AFTER a successful backup, so the free space
+// must hold the new archive while every kept one still exists.
+const (
+	backupSpaceGrowth     = 1.25
+	backupSpaceFloorBytes = int64(1) << 30
+)
+
+// spaceFits answers whether a new archive of vmid fits on a LOCAL (non-PBS) target. It FAILS OPEN — a
+// backup is the thing being protected, so an unreadable storage, an unknown type or a first backup (no
+// previous archive to size from) proceeds and says so; only a POSITIVE "it does not fit" refuses.
+func (r *BackupRunner) spaceFits(ctx context.Context, vmid int) (bool, string) {
+	// NodeStorage, never ListStorage: only the node view carries avail (see BackupAPI.NodeStorage).
+	stores, err := r.api.NodeStorage(ctx)
+	if err != nil {
+		r.logger.Warn("backup: space preflight could not read storage usage — proceeding (fail-open)", "target", r.target, "err", err)
+		return true, ""
+	}
+	var st *proxmox.Storage
+	for i := range stores {
+		if stores[i].Storage == r.target {
+			st = &stores[i]
+			break
+		}
+	}
+	if st == nil || st.Type == "pbs" || st.Avail <= 0 {
+		return true, "" // PBS dedups and has its own lifecycle; an unknown avail never refuses
+	}
+	_, last, err := r.latestArchive(ctx, vmid)
+	if err != nil || last <= 0 {
+		r.logger.Info("backup: space preflight has no previous archive to size from — proceeding", "vmid", vmid, "target", r.target)
+		return true, ""
+	}
+	need := int64(float64(last)*backupSpaceGrowth) + backupSpaceFloorBytes
+	if st.Avail >= need {
+		r.logger.Info("backup: space preflight passed", "vmid", vmid, "target", r.target, "last_archive_bytes", last, "need_bytes", need, "avail_bytes", st.Avail)
+		return true, ""
+	}
+	return false, fmt.Sprintf("%s has %s free; the last archive of guest %d was %s, so a new one needs about %s (old archives are removed only after a successful backup)",
+		r.target, humanGiB(st.Avail), vmid, humanGiB(last), humanGiB(need))
+}
+
+func humanGiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
+
+// watchForSnapshot polls the running backup's task log until it sees the storage-snapshot marker
+// (→ onSnapshot once) or the requested mode is reported as `stop` (→ downgraded; the marker will
+// never come, so stop watching) or ctx is cancelled (backup finished). Best-effort: a log-read
+// error is retried on the next tick; onSnapshot fires at most once.
+func (r *BackupRunner) watchForSnapshot(ctx context.Context, upid string, onSnapshot func()) {
+	ticker := time.NewTicker(snapshotWatchInterval)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			lines, err := r.api.TaskLogTail(ctx, upid, 200)
+			if err != nil {
+				continue
+			}
+			// A stop-mode (or downgraded) backup never creates a storage snapshot → never resume early.
+			if m := parseBackupMode(lines); m != "" && m != string(proxmox.ModeSnapshot) {
+				return
+			}
+			for _, ln := range lines {
+				if strings.Contains(ln, snapshotMarker) {
+					onSnapshot()
+					return
+				}
+			}
+		}
+	}
+}
+
+// PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
+// when there is none — the restore-test then no-ops cleanly.
+func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
+	return r.PickRestoreCandidateOn(ctx, r.target)
+}
+
+// PickRestoreCandidateOn is PickRestoreCandidate for an ARBITRARY tier's storage (R-85 1.2), so the
+// scheduler can rotate across tiers instead of only ever seeing this runner's own target.
+//
+// Contract preserved: "" + nil error when the storage holds no archive. **A tier with nothing to
+// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
+// that into a failure would make every fresh box look broken for its first week.
+func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
+	archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
+	return archive, err
+}
+
+// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
+// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
+// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
+// one-line call into this and its contract is untouched (one scan, one owner).
+//
+// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
+// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
+// restore-testing the archive a backup is still writing proves nothing about the backup that
+// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
+// on that: the tier is due when a settled archive exists that has not been proven.
+//
+// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
+// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
+// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
+// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
+// the archive generation rather than by the evaluation interval.
+//
+// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
+// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
+// turning that into a failure would make every fresh box look broken for its first week.
+func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
+	if target == "" {
+		return "", time.Time{}, nil
+	}
+	contents, err := r.api.StorageContent(ctx, target)
+	if err != nil {
+		return "", time.Time{}, err
+	}
+	var best string
+	var bestCTime int64 = -1
+	known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
+	for _, e := range contents {
+		if e.Content != "backup" {
+			continue
+		}
+		// R-689 (v0.135.0): only a backup OF A GUEST is a restore-test candidate. demo-hp keeps its golden
+		// template in `local:backup/` — content "backup", 654 MB, plausibly complete — and it was picked as
+		// the newest settled archive every 6 h and failed extractconfig (403) each time, while the guest's
+		// real archive went untested.
+		if ok, why := guestBackupArchive(e); !ok {
+			r.noteNotAGuestBackupOnce(e, why)
+			continue
+		}
+		// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
+		// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
+		// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
+		// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
+		if _, seen := known[e.VMID]; !seen {
+			_, err := r.api.GuestConfig(ctx, e.VMID)
+			switch {
+			case err == nil:
+				known[e.VMID] = true
+			case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
+				// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
+				// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
+				// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
+				// read is not one it manages; its archive is not a candidate.
+				known[e.VMID] = false
+			default:
+				return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
+			}
+		}
+		if !known[e.VMID] {
+			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
+			continue
+		}
+		if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
+			continue // not settled yet — a newer archive is not a reason to re-prove an older one
+		}
+		if ok, why := archivePlausiblyComplete(e); !ok {
+			r.warnRejectedArchiveOnce(e, why)
+			continue
+		}
+		if e.CTime > bestCTime {
+			bestCTime, best = e.CTime, e.VolID
+		}
+	}
+	if best == "" {
+		return "", time.Time{}, nil
+	}
+	return best, time.Unix(bestCTime, 0).UTC(), nil
+}
+
+// latestArchive finds the newest backup archive volid + size for vmid on the target.
+func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int64, error) {
+	contents, err := r.api.StorageContent(ctx, r.target)
+	if err != nil {
+		return "", 0, err
+	}
+	var vol string
+	var size, bestCTime int64 = 0, -1
+	for _, e := range contents {
+		if e.Content == "backup" && e.VMID == vmid && e.CTime > bestCTime {
+			bestCTime, vol, size = e.CTime, e.VolID, e.Size
+		}
+	}
+	if vol == "" {
+		return "", 0, fmt.Errorf("no backup archive found for vmid %d on %s", vmid, r.target)
+	}
+	return vol, size, nil
+}
+
+// NewestArchiveTime reports when this guest's newest backup archive LANDED ON THIS TARGET, from the
+// storage itself. ok=false means the target genuinely holds no archive for this guest.
+//
+// R-84: this is the cure for the redundant-backup-after-restart problem. The agent's backup Store is
+// in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due
+// reported "no successful backup recorded yet" and the controller dutifully took another one. On the
+// local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
+// every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
+// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
+// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
+//
+// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
+//
+//	smallest REAL backup anywhere on the fleet ...  612,397,450 B  (~584 MiB, a guest-9100 vzdump)
+//	demo-hp      local / PBS ..................... 1.59 GB / 4.35-4.37 GB
+//	demo-felhom  local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
+//	the phantom left by a PBS daemon killed mid-upload ....... 1 B
+//
+// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
+// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
+// archive — which is the property that matters, because a floor set too HIGH does not merely lose
+// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
+const minPlausibleArchiveBytes int64 = 1 << 20
+
+// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
+// why. Pure, so the contract is unit-testable without a storage.
+//
+// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
+// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
+// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
+//
+//   - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
+//     storage has no verification concept — and absent on a good PBS snapshot until verify-new
+//     catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
+//     one: continuous re-backup across the fleet.
+//   - `encrypted` fails the same way, and for the same reason.
+//   - `notes` happens to be present on both good tiers today only because the agent sets it; an
+//     archive written by any other path lacks it. Too fragile to gate freshness on.
+//
+// Size is the only signal that means the same thing on every tier.
+//
+// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
+// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
+// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
+// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
+func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
+	if e.Size < minPlausibleArchiveBytes {
+		return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
+			e.Size, minPlausibleArchiveBytes)
+	}
+	return true, ""
+}
+
+// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
+//
+// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
+// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
+// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
+// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
+// logging per poll would emit ~288 identical lines a day and bury the one that matters.
+func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
+	r.rejectedMu.Lock()
+	if r.rejected == nil {
+		r.rejected = map[string]struct{}{}
+	}
+	_, seen := r.rejected[e.VolID]
+	if !seen {
+		r.rejected[e.VolID] = struct{}{}
+	}
+	r.rejectedMu.Unlock()
+	if seen {
+		return
+	}
+	r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
+		"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
+}
+
+// demo-felhom in a single afternoon of deploys (2026-07-26).
+//
+// Asking the STORAGE rather than persisting the store is deliberate:
+//   - it is ground truth, not remembered state — if an archive was pruned or deleted it correctly
+//     stops counting, whereas a persisted record would keep claiming a backup that no longer exists;
+//   - it needs no new on-disk state and no migration;
+//   - it is the same source `latestArchive` already trusts to build the post-backup record.
+//
+// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
+// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
+// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
+// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
+// richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
+// synthesized record would put invented numbers into the host-report.
+func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
+	contents, err := r.api.StorageContent(ctx, r.target)
+	if err != nil {
+		return time.Time{}, false, err
+	}
+	var best int64 = -1
+	for _, e := range contents {
+		if e.Content != "backup" || e.VMID != vmid {
+			continue
+		}
+		if ok, why := archivePlausiblyComplete(e); !ok {
+			r.warnRejectedArchiveOnce(e, why)
+			continue
+		}
+		if e.CTime > best {
+			best = e.CTime
+		}
+	}
+	if best < 0 {
+		return time.Time{}, false, nil
+	}
+	return time.Unix(best, 0).UTC(), true, nil
+}
+
+// parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>`
+// (e.g. "INFO: backup mode: stop"). Returns "" if not found.
+func parseBackupMode(lines []string) string {
+	const marker = "backup mode:"
+	for _, ln := range lines {
+		if i := strings.Index(ln, marker); i >= 0 {
+			return strings.TrimSpace(ln[i+len(marker):])
+		}
+	}
+	return ""
+}
+
+// uncoveredMountpoints returns the mountpoint paths the guest vzdump EXCLUDES. LXC mount
+// points are OPT-IN to vzdump: a mpN with `backup=1` is covered; ANY other state — the
+// `backup=` token absent OR `backup=0` — is excluded. We deliberately treat unset as
+// uncovered (the safe DR direction: never imply an unprotected volume is backed up).
+func uncoveredMountpoints(mps map[string]string) []string {
+	var out []string
+	for key, cfg := range mps {
+		if mountpointCovered(cfg) {
+			continue
+		}
+		out = append(out, mountpointLabel(key, cfg))
+	}
+	sort.Strings(out)
+	if out == nil {
+		return []string{}
+	}
+	return out
+}
+
+// mountpointCovered reports whether a mpN config string is included in the vzdump — true
+// ONLY when it carries an explicit `backup=1`.
+func mountpointCovered(cfg string) bool {
+	for _, tok := range strings.Split(cfg, ",") {
+		if v, ok := strings.CutPrefix(tok, "backup="); ok {
+			return v == "1"
+		}
+	}
+	return false // no backup= token → opt-out by default → not covered
+}
+
+// mountpointLabel prefers the mp=/path token, falling back to the mpN key.
+func mountpointLabel(key, cfg string) string {
+	for _, tok := range strings.Split(cfg, ",") {
+		if p, ok := strings.CutPrefix(tok, "mp="); ok && p != "" {
+			return p
+		}
+	}
+	return key
+}
+
+// ToHubRestoreTest maps a reconcile restore-test result to the hub wire record (the backup
+// package owns the reconcile→hub mapping so reconcile need not import hub for the result).
+func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.RestoreTest {
+	rt := hub.RestoreTest{
+		SourceArchive:      res.Archive,
+		SourceTier:         res.SourceTier,
+		ScratchVMID:        res.ScratchVMID,
+		Pass:               res.Pass,
+		Verified:           res.Verified,
+		TestedAt:           testedAt.Format(time.RFC3339),
+		DurationSeconds:    res.Duration.Seconds(),
+		Warnings:           res.StartWarnings,
+		WarningsRecognized: res.WarningsRecognized,
+		MountParity:        res.MountParity,
+		MountInventory:     res.MountInventory,
+	}
+	if res.Err != nil {
+		rt.Error = res.Err.Error()
+	}
+	if res.SkipReason != "" { // R-672: the space preflight refused — reported, never a pass
+		rt.Pass = false
+		rt.Skipped = true
+		rt.Error = res.SkipReason
+	}
+	return rt
+}
+
+// guestBackupArchive reports whether a storage entry is a whole-guest backup of a known guest — a
+// `vzdump-<type>-<vmid>-…` file on a dir storage, or a `backup/{ct,vm}/<vmid>/<time>` snapshot on a PBS
+// datastore — whose vmid the storage itself reports. Anything else in a backup content type (a golden
+// template, a hand-copied tarball) is not a backup of a guest and is never restore-tested (R-689).
+// Pure, so the rule is unit-tested without a storage.
+func guestBackupArchive(e proxmox.StorageContent) (bool, string) {
+	if e.VMID <= 0 {
+		return false, "not a backup of a guest (the storage reports no vmid)"
+	}
+	vol := e.VolID
+	if i := strings.Index(vol, ":"); i >= 0 {
+		vol = vol[i+1:]
+	}
+	vol = strings.TrimPrefix(vol, "backup/")
+	vmid := strconv.Itoa(e.VMID)
+	switch {
+	case strings.HasPrefix(vol, "vzdump-lxc-"+vmid+"-"), strings.HasPrefix(vol, "vzdump-qemu-"+vmid+"-"):
+		return true, ""
+	case strings.HasPrefix(vol, "ct/"+vmid+"/"), strings.HasPrefix(vol, "vm/"+vmid+"/"):
+		return true, ""
+	}
+	return false, "not a vzdump archive or a PBS snapshot of guest " + vmid
+}
+
+// noteNotAGuestBackupOnce logs, once per volid, that a backup-content entry is not a restore-test
+// candidate because it is not a backup of a guest (R-689). INFO, not WARN: a golden template kept in
+// the backup directory is the operator's, and not a fault.
+func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why string) {
+	r.rejectedMu.Lock()
+	if r.rejected == nil {
+		r.rejected = map[string]struct{}{}
+	}
+	_, seen := r.rejected[e.VolID]
+	if !seen {
+		r.rejected[e.VolID] = struct{}{}
+	}
+	r.rejectedMu.Unlock()
+	if !seen {
+		r.logger.Info("backup: restore-test skips an entry that is not a backup of a guest",
+			"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
+	}
+}

internal/backup/runner_snapshot_test.go

@@ -0,0 +1,72 @@
+package backup
+
+import (
+	"context"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// 8B.2: in snapshot mode, the runner fires onSnapshot when the storage-snapshot marker appears in
+// the task log — mid-backup, before completion.
+func TestBackupWithSnapshotHook_FiresOnMarker(t *testing.T) {
+	old := snapshotWatchInterval
+	snapshotWatchInterval = 2 * time.Millisecond
+	defer func() { snapshotWatchInterval = old }()
+
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:backup",
+		waitGate:   make(chan struct{}), // hold the backup open so the watcher gets to poll
+		logLines:   []string{"INFO: backup mode: snapshot", "INFO: create storage snapshot 'vzdump'"},
+		content:    []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
+	}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+
+	var fired int32
+	done := make(chan struct{})
+	go func() {
+		_, _ = r.BackupWithSnapshotHook(context.Background(), 9001, func() { atomic.StoreInt32(&fired, 1) })
+		close(done)
+	}()
+
+	// the watcher should fire onSnapshot well before we release the backup
+	deadline := time.Now().Add(time.Second)
+	for time.Now().Before(deadline) && atomic.LoadInt32(&fired) == 0 {
+		time.Sleep(2 * time.Millisecond)
+	}
+	if atomic.LoadInt32(&fired) != 1 {
+		t.Fatal("onSnapshot did not fire on the storage-snapshot marker")
+	}
+	close(api.waitGate) // let the backup complete
+	<-done
+}
+
+// Stop/downgraded mode → no storage-snapshot marker → onSnapshot never fires (the 8B.2 fallback).
+func TestBackupWithSnapshotHook_StopMode_NeverFires(t *testing.T) {
+	old := snapshotWatchInterval
+	snapshotWatchInterval = 2 * time.Millisecond
+	defer func() { snapshotWatchInterval = old }()
+
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:backup",
+		waitGate:   make(chan struct{}),
+		logLines:   []string{"INFO: backup mode: stop"}, // downgraded; no snapshot marker
+		content:    []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
+	}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+
+	var fired int32
+	done := make(chan struct{})
+	go func() {
+		_, _ = r.BackupWithSnapshotHook(context.Background(), 9001, func() { atomic.StoreInt32(&fired, 1) })
+		close(done)
+	}()
+	time.Sleep(40 * time.Millisecond) // the watcher polls several times + sees stop mode → returns
+	if atomic.LoadInt32(&fired) != 0 {
+		t.Fatal("onSnapshot fired in stop mode (must not)")
+	}
+	close(api.waitGate)
+	<-done
+}

internal/backup/schedule.go

@@ -0,0 +1,299 @@
+package backup
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// RestoreTestRunner is the reconcile-engine seam the scheduler drives (*reconcile.Engine
+// satisfies it). Kept narrow so the scheduler is unit-testable with a fake.
+type RestoreTestRunner interface {
+	RunRestoreTest(ctx context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult
+}
+
+// CandidatePicker resolves the archive volid to restore-test (newest backup), or "" when
+// there is none yet (the tick then no-ops).
+type CandidatePicker func(ctx context.Context) (string, error)
+
+// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
+//
+// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
+// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
+// the resulting value reused for every run for the lifetime of the process. Two consequences:
+//   - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
+//   - it was a latent staleness bug in its own right — a storage-type or config change did not take
+//     effect until the daemon restarted.
+//
+// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
+// the configured target: deriving it from config is what produced the 600 s false failure when a
+// PBS archive was classified "local" and got the 10-minute local wait.
+type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
+
+// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
+// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
+// satisfies it. A zero notAfter means "no settle requirement".
+//
+// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
+// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
+// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
+// brand-new offsite tier legitimately has nothing to restore yet.
+type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
+
+// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
+// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
+// AND a valid scratch band is configured (validated by the caller before construction).
+type Scheduler struct {
+	runner RestoreTestRunner
+	pick   CandidatePicker
+	store  *Store
+	spec   SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
+	// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
+	// longer the thing that decides a test happens; see restoretest_due.go.
+	cadence time.Duration
+	// settle is how long an archive must have sat before it is a candidate (R-86).
+	settle time.Duration
+	logger *slog.Logger
+	now    func() time.Time
+
+	// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
+	// (single tier via `pick`), which keeps every existing caller and test working untouched.
+	tiers    []string          // configured tier target ids, primary first
+	tierPick TierPicker        // newest archive on a named tier
+	rtState  *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
+	inFlight *InFlight         // shared with the backup path — Scenario F
+}
+
+// SchedulerOptions configures a Scheduler.
+type SchedulerOptions struct {
+	Runner RestoreTestRunner
+	Pick   CandidatePicker
+	Store  *Store
+	// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
+	// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
+	Spec SpecBuilder
+	// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
+	// (R-86). 0 → disabled.
+	Cadence time.Duration
+	// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
+	// 0 → no settle requirement (any archive is a candidate).
+	Settle time.Duration
+	Logger *slog.Logger
+
+	// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
+	// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
+	// named tier; State persists last-successful-per-tier; InFlight is the shared one-heavy-op gate.
+	Tiers    []string
+	TierPick TierPicker
+	State    *RestoreTestState
+	InFlight *InFlight
+}
+
+// NewScheduler builds a Scheduler.
+func NewScheduler(opts SchedulerOptions) *Scheduler {
+	logger := opts.Logger
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Scheduler{
+		runner:   opts.Runner,
+		pick:     opts.Pick,
+		store:    opts.Store,
+		spec:     opts.Spec,
+		cadence:  opts.Cadence,
+		settle:   opts.Settle,
+		logger:   logger,
+		now:      func() time.Time { return time.Now().UTC() },
+		tiers:    append([]string(nil), opts.Tiers...),
+		tierPick: opts.TierPick,
+		rtState:  opts.State,
+		inFlight: opts.InFlight,
+	}
+}
+
+// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
+// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
+//
+// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
+// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
+// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
+//
+// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
+// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
+// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
+// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
+// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
+//
+// Returns nil on ctx cancellation.
+func (s *Scheduler) Run(ctx context.Context) error {
+	if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
+		s.logger.Info("backup: restore-test cadence disabled")
+		<-ctx.Done()
+		return nil
+	}
+	s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
+		"eval_interval", s.cadence, "settle", s.settle)
+	t := time.NewTicker(s.cadence)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err())
+			return nil
+		case <-t.C:
+			s.tick(ctx)
+		}
+	}
+}
+
+// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
+// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
+// is already in flight. Deterministic given s.now — tests call it directly.
+//
+// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
+// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
+func (s *Scheduler) tick(ctx context.Context) {
+	if s.spec == nil {
+		// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
+		// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
+		// spec must cost a restore-test, never the agent.
+		s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
+		return
+	}
+	// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
+	//
+	// It used to be the other way round, and correctly so: the gate was held for one heavy run a
+	// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
+	// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
+	// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
+	// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
+	// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
+	// not be able to manufacture that.
+	//
+	// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
+	// before anything heavy begins.
+	archive, target, err := s.pickForThisRun(ctx)
+	if err != nil {
+		s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
+		return
+	}
+	if archive == "" {
+		// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
+		//
+		// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
+		// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
+		// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
+		// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
+		// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
+		// nothing run last night?" is in the log rather than in a re-derivation.
+		s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
+		return
+	}
+
+	// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
+	// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
+	// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
+	// never cancel what is already running: a deferred restore-test costs hours of coverage, a
+	// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
+	release, busy, ok := s.inFlight.TryAcquire("restore-test")
+	if !ok {
+		s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
+			"busy", busy, "target", target, "archive", archive)
+		return
+	}
+	defer release()
+	// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
+	spec := s.spec(ctx, archive)
+	spec.Archive = archive
+	res := s.runner.RunRestoreTest(ctx, spec)
+	if res.Skipped && res.SkipReason == "" {
+		return // already logged by the engine (no free scratch VMID)
+	}
+	// R-672: a SPACE refusal is the test's result — reported (pass=false, the reason as the error),
+	// never dropped and never a pass. It earns no rotation credit, so the tier stays due.
+	rt := ToHubRestoreTest(res, s.now())
+	s.store.RecordRestoreTest(rt)
+	// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
+	// that fails every time would look freshly proven and quietly stop being retried.
+	if rt.Pass && s.rtState != nil && target != "" {
+		// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
+		// not-due until a NEWER archive settles, and what makes a proof survive a restart.
+		// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
+		// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
+		// this run was actually judged as, and deriving it later would need a storage lookup that
+		// can fail on the one path where failing means mislabelling a proof.
+		if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
+			s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
+		}
+	}
+	switch {
+	case !rt.Pass:
+		// A failing restore-test is the loudest DR signal there is.
+		s.logger.Error("backup: scheduled restore-test FAILED", "archive", rt.SourceArchive, "err", rt.Error)
+	case len(res.StartWarnings) == 0:
+		s.logger.Info("backup: scheduled restore-test passed", "archive", rt.SourceArchive, "duration_s", rt.DurationSeconds)
+	case res.WarningsRecognized:
+		// Passed; the only warnings are the known-benign (e.g. systemd-nesting) advisory.
+		s.logger.Info("backup: scheduled restore-test passed with warnings (recognized)",
+			"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
+	default:
+		// Passed liveness, but an UNRECOGNIZED start warning stood out — worth an operator look.
+		s.logger.Warn("backup: scheduled restore-test passed with UNRECOGNIZED warnings",
+			"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
+	}
+}
+
+// rotating reports whether multi-tier rotation is wired.
+func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
+
+// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
+// order.
+//
+// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
+// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
+// due?", and "nothing" is a normal, frequent and correct answer.
+//
+// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
+// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
+// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
+// that a test happens at all.
+//
+// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
+// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
+// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
+// storage being unreadable must not cost the other tier its proof, and must not be silent either.
+//
+// Returns ("", "", nil) when nothing anywhere is due.
+func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
+	if !s.rotating() {
+		// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
+		// identity and no persisted proof here, so there is nothing to compare an archive against
+		// and no due-check is possible. It runs on every evaluation, exactly as it always did.
+		a, perr := s.pick(ctx)
+		return a, "", perr
+	}
+	var firstErr error
+	for _, v := range s.EvaluateDue(ctx) {
+		if v.Err != nil {
+			s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
+				"target", v.Target, "err", v.Err)
+			if firstErr == nil {
+				firstErr = v.Err
+			}
+			continue
+		}
+		if !v.Due {
+			s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
+			continue
+		}
+		s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
+			"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
+		return v.Archive, v.Target, nil
+	}
+	if firstErr != nil {
+		return "", "", firstErr
+	}
+	return "", "", nil
+}

internal/backup/schedule_spec_test.go

@@ -0,0 +1,124 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-85 (1.1) — the spec is built PER RUN, never frozen at construction.
+//
+// It used to be an immediately-invoked function at daemon start, so storageTier() and
+// restoreTaskTimeout() were evaluated ONCE and the value reused for every run for the process
+// lifetime. That is what made an offsite restore-test impossible to schedule at all, and it was a
+// latent staleness bug besides: a storage-type or config change did not take effect until restart.
+
+type specSpy struct {
+	mu       sync.Mutex
+	calls    int
+	archives []string
+	tiers    []string // what the builder decided, per call
+}
+
+func (sp *specSpy) build(_ context.Context, archive string) reconcile.RestoreTestSpec {
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	sp.calls++
+	sp.archives = append(sp.archives, archive)
+	// Decide the tier from the ARCHIVE, exactly as main.go does (the v0.100.0 rule).
+	tier := "local"
+	if len(archive) > 10 && archive[:10] == "felhom-pbs" {
+		tier = "pbs"
+	}
+	sp.tiers = append(sp.tiers, tier)
+	return reconcile.RestoreTestSpec{
+		RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: tier,
+	}
+}
+
+// COMPANION RED-PROOF (observed): change Scheduler.spec back to a frozen
+// `reconcile.RestoreTestSpec` value captured at construction → this fails with
+// "the spec builder must run ONCE PER RUN, got 1 call(s) across 3 ticks", because a frozen value is
+// evaluated exactly once no matter how many ticks fire. Restored.
+func TestScheduler_SpecIsBuiltPerRun(t *testing.T) {
+	sp := &specSpy{}
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
+	n := 0
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick: func(context.Context) (string, error) {
+			n++
+			return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", n), nil
+		},
+		Store:   NewStore(),
+		Spec:    sp.build,
+		Cadence: time.Hour,
+		Logger:  quiet(),
+	})
+
+	for i := 0; i < 3; i++ {
+		s.tick(context.Background())
+	}
+
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	if sp.calls != 3 {
+		t.Fatalf("the spec builder must run ONCE PER RUN, got %d call(s) across 3 ticks", sp.calls)
+	}
+	// And it must see the archive THIS run picked — not a stale one.
+	for i, a := range sp.archives {
+		want := fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", i+1)
+		if a != want {
+			t.Fatalf("run %d: builder saw archive %q, want %q — the spec is not tracking the picked archive", i+1, a, want)
+		}
+	}
+}
+
+// The tier must follow the ARCHIVE across runs. A builder that saw only the configured target would
+// return the same tier every time — which is exactly the v0.100.0 defect that killed a 14.46 GB WAN
+// restore at the 10-minute local bound.
+func TestScheduler_SpecTierFollowsTheArchive(t *testing.T) {
+	sp := &specSpy{}
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
+	archives := []string{
+		"local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
+	}
+	i := 0
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick: func(context.Context) (string, error) {
+			a := archives[i%len(archives)]
+			i++
+			return a, nil
+		},
+		Store: NewStore(), Spec: sp.build, Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background())
+	s.tick(context.Background())
+
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	if len(sp.tiers) != 2 || sp.tiers[0] != "local" || sp.tiers[1] != "pbs" {
+		t.Fatalf("the tier must follow the archive per run; got %v", sp.tiers)
+	}
+}
+
+// A nil spec builder must SKIP loudly, not panic — a wiring bug costs a restore-test, never the
+// daemon goroutine.
+func TestScheduler_NilSpecSkipsInsteadOfPanicking(t *testing.T) {
+	rt := &fakeRTRunner{}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick:   func(context.Context) (string, error) { return "vol", nil },
+		Store:  NewStore(), Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background()) // must not panic
+	if rt.runs != 0 {
+		t.Fatalf("a nil spec must not run a restore-test; got %d run(s)", rt.runs)
+	}
+}

internal/backup/store.go

@@ -0,0 +1,75 @@
+package backup
+
+import (
+	"context"
+	"sync"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// Store holds the agent's LATEST backup result per target and the latest restore-test
+// result — the point-in-time state the host-report surfaces. It is updated by the backup
+// runner + the restore-test scheduler/selftest and read by the collector via the hub
+// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
+// collector vs scheduler access.
+//
+// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
+// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
+// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
+// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
+// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
+// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
+// consecutive host-reports carried `0 restore-tests`.
+//
+// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
+// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
+// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
+// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
+// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
+type Store struct {
+	mu       sync.Mutex
+	byTarget map[string]hub.Backup // latest backup per target id
+	lastTest *hub.RestoreTest
+}
+
+// NewStore builds an empty Store.
+func NewStore() *Store {
+	return &Store{byTarget: map[string]hub.Backup{}}
+}
+
+// RecordBackup stores the latest backup for its target.
+func (s *Store) RecordBackup(b hub.Backup) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.byTarget[b.TargetID] = b
+}
+
+// RecordRestoreTest stores the latest restore-test result.
+func (s *Store) RecordRestoreTest(r hub.RestoreTest) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	cp := r
+	s.lastTest = &cp
+}
+
+// Backups implements hub.BackupReporter — the latest backup per target (stable order by
+// target id is not guaranteed; the hub does not depend on order).
+func (s *Store) Backups(context.Context) []hub.Backup {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make([]hub.Backup, 0, len(s.byTarget))
+	for _, b := range s.byTarget {
+		out = append(out, b)
+	}
+	return out
+}
+
+// RestoreTests implements hub.RestoreTestReporter — the latest restore-test result (0 or 1).
+func (s *Store) RestoreTests(context.Context) []hub.RestoreTest {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.lastTest == nil {
+		return []hub.RestoreTest{}
+	}
+	return []hub.RestoreTest{*s.lastTest}
+}

internal/capability/manifest.go

@@ -0,0 +1,195 @@
+// Package capability is the agent's privileged-capability self-check (slice 1 of agent
+// self-health). It declares the MANIFEST — the (binary, representative-arg-vector) pairs the
+// non-root agent depends on running via `sudo -n` — and a PROBE that lists each against the live
+// sudoers policy (`sudo -n -l`, never executing) + checks the binary exists. The result is a
+// snapshot the agent attaches to its hub report; the hub owns the ok→degraded transition + alert.
+//
+// Why this exists: the 2026-06-28 root→non-root cutover dropped several grants from
+// configs/felhom-agent.sudoers (lxc-info, make-private, restart dnsmasq, …). Each broke a feature
+// silently until a user hit it (the multi-drive flapping incident, audit 2026-06-29). A non-root
+// agent that can't run a command it depends on is DEGRADED and must SAY so — at cutover, not days
+// later. The companion build-time test (manifest_test.go) asserts every manifest vector is covered
+// by a sudoers pattern, catching authoring gaps in CI before they ship.
+package capability
+
+import (
+	"strings"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
+)
+
+// Capability is one privileged command the agent depends on. Name is a stable id; Feature is the
+// human-readable thing that breaks if the grant is missing (used in logs + the operator alert).
+// Binary is the absolute path the runner invokes; ReprArgs is a CONCRETE argument vector that
+// matches the corresponding sudoers glob (e.g. a vmid "9201" matches `[0-9]*`, a device "/dev/sda"
+// matches `/dev/*`). Critical marks the user-facing ones — the hub alerts only when a Critical
+// capability is degraded (non-critical degradations still ride the report snapshot + agent log).
+type Capability struct {
+	Name     string
+	Feature  string
+	Binary   string
+	ReprArgs []string
+	Critical bool
+	// GatedBy names the config gate this capability sits behind ("" = always active). A healthy
+	// probe of a gated capability whose gate is OFF reports StatusInactive instead of ok — see
+	// Prober.GateActive. Broken plumbing stays degraded regardless of the gate.
+	GatedBy string
+}
+
+// GatePBSDR is the gate id for the PBS-DR tier: active when this box carries a live (enabled)
+// pbs_dr descriptor or a previously-converged marker (pbsdr.Manager.DRConfigured).
+const GatePBSDR = "pbs_dr"
+
+// Manifest is the required set, seeded from the 2026-06-29 sudoers audit (felhom-agent/REPORT.md):
+// the OK + newly-CLOSED rows. The SURFACED/DEFERRED rows are deliberately EXCLUDED — they are not
+// required capabilities: the general `pct exec <vmid> -- *` (controller-swap; arbitrary exec, an
+// open operator decision), `pct create` (golden build, maintenance, no daemon caller), `mount
+// UUID=…` (legacy/unreferenced), and the callerless `sensors -j`. Adding them here would assert
+// grants the agent neither has nor should depend on.
+//
+// Each ReprArgs is a representative instance; the probe LISTS it (`sudo -n -l`) and never runs it,
+// so even mkfs/pct-set entries are side-effect-free to probe.
+//
+// Gating (v0.86.0): every `pbsdr-*` capability is behind GatePBSDR — the whole pbsdr vector IS
+// the DR-tier apply path, so the gate rides the stable name prefix (a new pbsdr-* op is gated
+// automatically; nothing else ever is). Applied here, not in the literals, to keep the manifest
+// entries positional.
+func Manifest() []Capability {
+	out := make([]Capability, len(manifest))
+	copy(out, manifest)
+	for i := range out {
+		if strings.HasPrefix(out[i].Name, "pbsdr-") {
+			out[i].GatedBy = GatePBSDR
+		}
+	}
+	return out
+}
+
+// reprFingerprint is a shape-valid all-zero SHA-256 colon fingerprint for list-mode repr vectors
+// (matches the wrapper's fingerprint validation; never executed).
+const reprFingerprint = "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
+
+var manifest = []Capability{
+	// ---- Intermediary drive model (the multi-drive path — mostly Critical) ----
+	{"guest-init-pid", "drive-gate guest-sees check (multi-drive concurrency)", "/usr/bin/lxc-info", []string{"-n", "9201", "-p", "-H"}, true, ""},
+	{"parent-self-bind", "intermediary shared-parent self-bind", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-drives", "/mnt/felhom-drives"}, true, ""},
+	{"parent-make-shared", "intermediary shared-parent propagation", "/usr/bin/mount", []string{"--make-shared", "/mnt/felhom-drives"}, true, ""},
+	{"parent-make-private", "intermediary shared-parent peer-group isolation", "/usr/bin/mount", []string{"--make-private", "/mnt/felhom-drives"}, true, ""},
+	{"drive-bind", "drive attach (felhom-data bind under parent)", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-usb/felhom-data", "/mnt/felhom-drives/felhom-usb"}, true, ""},
+	{"drive-umount", "drive detach (fail-closed unmount)", "/usr/bin/umount", []string{"/mnt/felhom-drives/felhom-usb"}, true, ""},
+	{"drives-mkdir-parent", "stable parent dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives"}, false, ""},
+	{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
+	{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
+	{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
+	{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
+	{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
+	{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
+	{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
+
+	// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
+	{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
+	{"disk-lsblk", "disk topology read (format gate)", "/usr/bin/lsblk", []string{"-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", "/dev/sda"}, true, ""},
+	{"disk-mkfs-ext4", "guarded format (ext4)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "ext4"}, true, ""},
+	{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
+	{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
+	{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
+
+	// ---- Storage mount units (watchdog re-mount) ----
+	{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
+	{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
+	{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
+	{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
+	{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
+
+	// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
+	{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
+	{"netmount-rmdir", "removed-share mountpoint cleanup (F1)", "/usr/bin/rmdir", []string{"/mnt/felhom-drives/media"}, false, ""},
+
+	// ---- Node self-heal (CAMPAIGN-3 F12-class, appliance-gated in code) ----
+	{"selfheal-networking-start", "appliance networking recovery at boot (F12 defense in depth)", "/usr/bin/systemctl", []string{"start", "networking.service"}, false, ""},
+
+	// ---- Provisioning back-half ----
+	{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
+	{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
+	{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
+
+	// ---- Pre-start self-heal hook + guest lifecycle ----
+	{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
+	{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
+	{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
+	{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
+
+	// ---- LAN split-horizon resolver (dnsmasq) ----
+	{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
+	{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
+	{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
+	{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
+	{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
+	{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
+	{"dnsmasq-guest-ip", "guest LAN IP discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "-4", "-o", "addr", "show", "dev", "eth0"}, false, ""},
+	{"dnsmasq-guest-domain", "guest domain discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "exec", "felhom-controller", "cat", "/opt/docker/felhom-controller/controller.yaml"}, false, ""},
+
+	// ---- Guest-network watchdog (FELHOM_GUESTNET, R-54, v0.92.0) ----
+	// Deliberately NOT Critical: a host that has not yet taken the new sudoers file should show
+	// degraded (visible, diagnosable) without paging an operator for every box on rollout day —
+	// the R-50b lesson that an amber fleet is background noise. The address read is covered by
+	// dnsmasq-guest-ip above and is not duplicated here (one command, one capability row).
+	{"guestnet-route", "guest default-route probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "route", "show", "default"}, false, ""},
+	{"guestnet-ifaces", "guest interface-mode read", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/network/interfaces"}, false, ""},
+	{"guestnet-dhclient-probe", "guest DHCP-client liveness probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "pgrep", "-x", "dhclient"}, false, ""},
+	{"guestnet-heal", "guest DHCP-client restart (the 2026-07-20 heal)", "/usr/sbin/pct", []string{"exec", "9201", "--", "dhclient", "-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"}, false, ""},
+
+	// ---- Controller-swap / managed auto-update (FELHOM_CONTROLLERSWAP, v0.45.0; Critical: a
+	// silently-broken fleet auto-update is operator-alert-worthy) ----
+	{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
+	{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
+	{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
+	{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
+	{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
+
+	// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
+	// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
+	{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
+
+	// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
+	// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
+	// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
+	// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
+	// handshake read is the ONLY wg invocation (never `dump`). ----
+	{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
+	{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
+	{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
+	{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
+	{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
+	{"wg-handshake-read", "tunnel handshake-age read", "/usr/bin/wg", []string{"show", "wg-felhom", "latest-handshakes"}, true, ""},
+
+	// ---- PBS DR tier apply (FELHOM_PBSDR, slice 2). NON-critical (the selfupdate rationale):
+	// applying the tier is an occasional hub-driven provisioning op, not a steady-state serving
+	// path — a degraded grant means "can't provision/reconcile the PBS entry" (the bridge reports
+	// loudly anyway), not a serving outage. The steady-state backup path is covered by the wg +
+	// storage capabilities. List-mode representations only; never executed. ----
+	{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
+	{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
+	{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false, ""},
+	// R-39 leg (b), v0.91.0: the credential READ path. Its absence is what made the PBS verify loop
+	// permanently blind to an applied-but-401 tier, so a host missing this verb is DEGRADED in a way
+	// that matters — it cannot detect the failure this whole tier exists to survive.
+	{"pbsdr-read", "PBS DR credential read (verify-loop auth probe)", "/usr/local/sbin/felhom-pbs-apply", []string{"read", "felhom-pbs", "/etc/pve/priv/storage"}, false, ""},
+
+	// ---- Escrow ceremony (FELHOM_ESCROW, controller-driven, v0.88.0). Critical: the customer
+	// wizard's whole run path IS this one grant — a dropped line silently breaks every ceremony.
+	// GatedBy is set EXPLICITLY (the name deliberately says "escrow", not "pbsdr-": the feature is
+	// the ceremony, but it only exists behind the DR tier — no PBS key, no ceremony). ReprArgs is
+	// the SHARED argv constant (internal/escrow/ceremony.go) — the exec runner uses the same one,
+	// so runner ↔ manifest can't drift, and TestManifestCoveredBySudoers locks manifest ↔ sudoers.
+	// List-mode probe only (`sudo -n -l`), spike-confirmed side-effect-free — never a real ceremony.
+	{"escrow-ceremony", "customer recovery-code ceremony (controller-driven)", escrow.CeremonyBinary, escrow.CeremonyArgs(), true, GatePBSDR},
+
+	// ---- Agent self-update (FELHOM_SELFUPDATE, D1). NON-critical: self-update is an occasional
+	// operator-driven op, not a steady-state serving path — a degraded grant means "can't
+	// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
+	// staging-dir path + a placeholder sha (list-mode never runs it). ----
+	{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
+	{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
+	{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
+}

internal/capability/manifest_test.go

@@ -0,0 +1,296 @@
+package capability
+
+import (
+	"os"
+	"reflect"
+	"regexp"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
+)
+
+// sudoersPath is the in-repo allowlist, relative to this test file (internal/capability/).
+const sudoersPath = "../../configs/felhom-agent.sudoers"
+
+// parseSudoersEntries returns every command pattern from the Cmnd_Alias blocks, with the sudoers
+// escapes (`\,` `\:`) unescaped. It joins continuation lines and splits the alias RHS on commas
+// that are NOT backslash-escaped (escaped commas are literal arg chars, e.g. the lvs `-o` list).
+func parseSudoersEntries(t *testing.T, text string) []string {
+	t.Helper()
+	// 1. Collapse line continuations, keeping only Cmnd_Alias RHS text.
+	var rhs strings.Builder
+	lines := strings.Split(text, "\n")
+	inAlias := false
+	for _, ln := range lines {
+		trimmed := strings.TrimSpace(ln)
+		if strings.HasPrefix(trimmed, "#") {
+			continue
+		}
+		if strings.HasPrefix(trimmed, "Cmnd_Alias ") {
+			inAlias = true
+			if eq := strings.IndexByte(trimmed, '='); eq >= 0 {
+				trimmed = trimmed[eq+1:]
+			}
+		} else if !inAlias {
+			continue
+		}
+		// The final NOPASSWD line ("felhom-agent ALL=...") ends the alias region.
+		if strings.Contains(trimmed, "ALL=(") {
+			inAlias = false
+			continue
+		}
+		cont := strings.HasSuffix(trimmed, "\\")
+		rhs.WriteString(strings.TrimSuffix(trimmed, "\\"))
+		rhs.WriteString(" ")
+		if !cont {
+			// A non-continued line is the last entry of this alias. Emit a comma so it does not
+			// merge with the next alias's first entry when all RHS text is concatenated.
+			rhs.WriteString(", ")
+			inAlias = false
+		}
+	}
+	// 2. Split on unescaped commas → individual command entries.
+	raw := rhs.String()
+	var entries []string
+	var cur strings.Builder
+	for i := 0; i < len(raw); i++ {
+		c := raw[i]
+		if c == '\\' && i+1 < len(raw) {
+			cur.WriteByte(raw[i+1]) // unescape: keep the next char literally (\, → , ; \: → :)
+			i++
+			continue
+		}
+		if c == ',' {
+			entries = appendTrimmed(entries, cur.String())
+			cur.Reset()
+			continue
+		}
+		cur.WriteByte(c)
+	}
+	entries = appendTrimmed(entries, cur.String())
+	return entries
+}
+
+func appendTrimmed(entries []string, s string) []string {
+	if t := strings.Join(strings.Fields(s), " "); t != "" {
+		return append(entries, t)
+	}
+	return entries
+}
+
+// globToRegex translates a sudoers fnmatch pattern to an anchored regex. It is NOT a perfect sudo
+// emulator — it only needs to catch a removed/renamed grant (the real failure mode). `*` → `.*`,
+// `[...]` char classes pass through (valid regex), regex metachars are escaped.
+func globToRegex(pat string) *regexp.Regexp {
+	var b strings.Builder
+	b.WriteString("^")
+	for i := 0; i < len(pat); i++ {
+		c := pat[i]
+		switch {
+		case c == '*':
+			b.WriteString(".*")
+		case c == '[': // copy the char class verbatim (valid in regex too)
+			if j := strings.IndexByte(pat[i:], ']'); j > 0 {
+				b.WriteString(pat[i : i+j+1])
+				i += j
+				continue
+			}
+			b.WriteString("\\[")
+		case strings.IndexByte(`.+()|{}^$\?`, c) >= 0:
+			b.WriteByte('\\')
+			b.WriteByte(c)
+		default:
+			b.WriteByte(c)
+		}
+	}
+	b.WriteString("$")
+	return regexp.MustCompile(b.String())
+}
+
+// matchesAny reports whether cmdline matches at least one sudoers entry pattern.
+func matchesAny(cmdline string, entries []string) bool {
+	for _, e := range entries {
+		if globToRegex(e).MatchString(cmdline) {
+			return true
+		}
+	}
+	return false
+}
+
+// TestManifestCoveredBySudoers is the headline build-time gate: EVERY manifest capability's
+// representative command line must be permitted by at least one sudoers pattern. This is exactly
+// the check that would have caught the lxc-info / make-private grants being dropped at the
+// 2026-06-28 cutover — in CI, before shipping.
+func TestManifestCoveredBySudoers(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers %s: %v", sudoersPath, err)
+	}
+	entries := parseSudoersEntries(t, string(data))
+	if len(entries) < 20 {
+		t.Fatalf("parsed only %d sudoers entries — parser likely broke", len(entries))
+	}
+	for _, c := range Manifest() {
+		cmdline := strings.TrimSpace(c.Binary + " " + strings.Join(c.ReprArgs, " "))
+		if !matchesAny(cmdline, entries) {
+			t.Errorf("capability %q (%s) NOT covered by any sudoers grant:\n  %s",
+				c.Name, c.Feature, cmdline)
+		}
+	}
+}
+
+// TestRedProof_DroppedGrantFailsCheck is the companion red-proof: with the lxc-info line removed
+// from an in-memory copy of the sudoers, the coverage check for guest-init-pid MUST fail. Proves
+// the build gate actually catches the regression (a green test that can never go red is hollow).
+func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers: %v", err)
+	}
+	// Drop the lxc-info grant line.
+	var kept []string
+	for _, ln := range strings.Split(string(data), "\n") {
+		if strings.Contains(ln, "lxc-info") {
+			continue
+		}
+		kept = append(kept, ln)
+	}
+	mutated := strings.Join(kept, "\n")
+	if strings.Contains(mutated, "lxc-info") {
+		t.Fatal("setup: lxc-info line not removed")
+	}
+	entries := parseSudoersEntries(t, mutated)
+
+	var guestInit Capability
+	for _, c := range Manifest() {
+		if c.Name == "guest-init-pid" {
+			guestInit = c
+		}
+	}
+	if guestInit.Name == "" {
+		t.Fatal("manifest missing guest-init-pid")
+	}
+	cmdline := guestInit.Binary + " " + strings.Join(guestInit.ReprArgs, " ")
+	if matchesAny(cmdline, entries) {
+		t.Errorf("red-proof FAILED: guest-init-pid still matches after dropping the lxc-info grant — the build gate would NOT catch the regression")
+	}
+
+	// Sanity: the UNMUTATED file MUST cover it (so the failure above is specific to the drop).
+	full := parseSudoersEntries(t, string(data))
+	if !matchesAny(cmdline, full) {
+		t.Errorf("guest-init-pid should be covered by the real sudoers")
+	}
+}
+
+// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
+// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
+// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
+// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
+func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers: %v", err)
+	}
+	var kept []string
+	for _, ln := range strings.Split(string(data), "\n") {
+		if strings.Contains(ln, "tee /etc/felhom-controller-image") {
+			continue
+		}
+		kept = append(kept, ln)
+	}
+	mutated := strings.Join(kept, "\n")
+	entries := parseSudoersEntries(t, mutated)
+
+	var write Capability
+	for _, c := range Manifest() {
+		if c.Name == "controllerswap-write" {
+			write = c
+		}
+	}
+	if write.Name == "" {
+		t.Fatal("manifest missing controllerswap-write")
+	}
+	cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
+	if matchesAny(cmdline, entries) {
+		t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
+	}
+	if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
+		t.Errorf("controllerswap-write should be covered by the real sudoers")
+	}
+}
+
+// TestEscrowCeremonyArgvPinned locks the ceremony argv copies together (Scenario G, v0.88.0).
+// The exec runner and the manifest entry both consume escrow.CeremonyArgs() (one shared source),
+// and TestManifestCoveredBySudoers proves manifest ⊆ sudoers — so pinning the shared source to
+// the EXPECTED literal here transitively locks all three: runner == manifest == sudoers.
+// Red-proof: mutate one element of the argv in internal/escrow/ceremony.go and THIS test fails
+// (and so does the sudoers coverage); a sudoers-side mutation is caught by the existing
+// TestRedProof_* machinery.
+func TestEscrowCeremonyArgvPinned(t *testing.T) {
+	wantBinary := "/usr/local/bin/felhom-agent"
+	wantArgs := []string{"--config", "/etc/felhom-agent/agent.json", "--selftest=escrow-create", "--upload", "--output=json"}
+
+	if escrow.CeremonyBinary != wantBinary {
+		t.Errorf("escrow.CeremonyBinary = %q, want %q", escrow.CeremonyBinary, wantBinary)
+	}
+	if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
+		t.Errorf("escrow.CeremonyArgs() = %q, want %q (the sudoers line + manifest entry must stay byte-identical)", got, wantArgs)
+	}
+
+	var entry Capability
+	for _, c := range Manifest() {
+		if c.Name == "escrow-ceremony" {
+			entry = c
+		}
+	}
+	if entry.Name == "" {
+		t.Fatal("manifest missing escrow-ceremony")
+	}
+	if entry.Binary != escrow.CeremonyBinary || !reflect.DeepEqual(entry.ReprArgs, escrow.CeremonyArgs()) {
+		t.Errorf("manifest escrow-ceremony argv diverged from the shared constant: %s %q", entry.Binary, entry.ReprArgs)
+	}
+	if !entry.Critical {
+		t.Error("escrow-ceremony must be Critical (the wizard's whole run path is this one grant)")
+	}
+	if entry.GatedBy != GatePBSDR {
+		t.Errorf("escrow-ceremony GatedBy = %q, want %q (no PBS key → no ceremony; inactive, never red, on a DR-off box)", entry.GatedBy, GatePBSDR)
+	}
+	// CeremonyArgs must return a COPY — a caller mutating its slice must not poison the source.
+	mutated := escrow.CeremonyArgs()
+	mutated[0] = "--poisoned"
+	if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
+		t.Error("escrow.CeremonyArgs() shares its backing array — callers can mutate the source")
+	}
+}
+
+// TestWGCapabilityCriticality pins the exact S4 (v0.66.0) Critical set for the FELHOM_WG entries:
+// the backup path (conf install, unit enable/restart, handshake read) is operator-alert-worthy now
+// that offsite backups ride the tunnel; the one-time apt install and the deliberate disable
+// (revocation) are NOT. Red-proof: flip any one entry's Critical in manifest.go and this fails.
+func TestWGCapabilityCriticality(t *testing.T) {
+	wantCritical := map[string]bool{
+		"wg-tools-install":  false,
+		"wg-conf-install":   true,
+		"wg-enable":         true,
+		"wg-restart":        true,
+		"wg-disable":        false,
+		"wg-handshake-read": true,
+	}
+	seen := map[string]bool{}
+	for _, c := range Manifest() {
+		want, ok := wantCritical[c.Name]
+		if !ok {
+			continue
+		}
+		seen[c.Name] = true
+		if c.Critical != want {
+			t.Errorf("%s: Critical = %v, want %v", c.Name, c.Critical, want)
+		}
+	}
+	for name := range wantCritical {
+		if !seen[name] {
+			t.Errorf("manifest missing wg capability %q", name)
+		}
+	}
+}

internal/capability/probe.go

@@ -0,0 +1,122 @@
+package capability
+
+import (
+	"context"
+	"os"
+)
+
+// Status is one capability's live result — the wire shape the agent attaches to its hub report
+// (HostReport.Capabilities). The hub mirrors this struct field-for-field and keys its alert on
+// Critical+degraded. Reason is empty when ok.
+//
+// "inactive" (v0.86.0, DR-tier-by-default): a config-GATED capability whose plumbing is HEALTHY
+// (binary present, sudo granted) but whose gating feature is disabled by configuration. Distinct
+// from degraded on purpose — disabled ≠ broken; the hub renders it as a neutral chip, never red.
+// Broken plumbing (binary missing / grant denied) stays DEGRADED even when the gate is off: an
+// un-migrated box must never look deliberately disabled.
+type Status struct {
+	Name     string `json:"name"`
+	Feature  string `json:"feature"`
+	Critical bool   `json:"critical"`
+	Status   string `json:"status"` // "ok" | "degraded" | "inactive"
+	Reason   string `json:"reason,omitempty"`
+}
+
+const (
+	StatusOK       = "ok"
+	StatusDegraded = "degraded"
+	StatusInactive = "inactive"
+)
+
+// ReasonInactive is the fixed reason string for the inactive state (the hub + operator docs
+// reference it verbatim).
+const ReasonInactive = "disabled by configuration"
+
+// Runner is the minimal exec seam the probe needs (satisfied by proxmox.ExecRunner). The probe
+// runs `sudo -n -l -- <binary> <args…>` LITERALLY — a sudo POLICY LIST that never executes the
+// command — so the Runner MUST be a DIRECT runner (RunnerDirect), not the sudo-prepending one
+// (else it would double-sudo). exit 0 ⇔ the command is permitted under the NOPASSWD allowlist.
+type Runner interface {
+	Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
+}
+
+// Prober checks the manifest against the live host. Exists defaults to an os.Stat check on the
+// absolute binary path (what `command -v` would resolve for an absolute path) when nil.
+// GateActive answers "is the feature behind this gate id configured on?" for GATED capabilities
+// (Capability.GatedBy). nil, or a gate it answers true for, keeps the historical behavior; false
+// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
+type Prober struct {
+	Runner     Runner
+	Exists     func(path string) bool  // nil → os.Stat
+	GateActive func(gate string) bool // nil → every gate treated active
+}
+
+// Probe lists every manifest capability against the sudo policy and checks its binary exists,
+// mapping to ok/degraded (§8 of the spec). It NEVER executes a probed command and NEVER returns a
+// fatal error (serve-degraded): a probe failure is reported, not raised. If sudo itself is
+// unavailable for the agent (the drop-in is missing / the user has no sudo at all), it collapses
+// to ONE aggregate degraded signal instead of N identical ones.
+func (p Prober) Probe(ctx context.Context) []Status {
+	exists := p.Exists
+	if exists == nil {
+		exists = func(path string) bool { _, err := os.Stat(path); return err == nil }
+	}
+	caps := Manifest()
+
+	// Preflight: a bare `sudo -n -l` lists the user's allowed commands. For our NOPASSWD service
+	// user it exits 0; if it fails, the drop-in isn't installed (or sudo is gone) and EVERY vector
+	// would individually fail — collapse to one aggregate signal so the operator gets one alert.
+	if p.Runner != nil {
+		if _, _, err := p.Runner.Run(ctx, "sudo", "-n", "-l"); err != nil {
+			return []Status{{
+				Name:     "sudo",
+				Feature:  "the entire privileged surface (mount/format/pct/dnsmasq/lxc-info)",
+				Critical: true,
+				Status:   StatusDegraded,
+				Reason:   "sudoers drop-in not installed / sudo unavailable",
+			}}
+		}
+	}
+
+	out := make([]Status, 0, len(caps))
+	for _, c := range caps {
+		s := Status{Name: c.Name, Feature: c.Feature, Critical: c.Critical, Status: StatusOK}
+		switch {
+		case !exists(c.Binary):
+			s.Status, s.Reason = StatusDegraded, "binary not found"
+		case p.Runner != nil && !p.granted(ctx, c):
+			s.Status, s.Reason = StatusDegraded, "sudo policy denied"
+		}
+		// Config gate (v0.86.0): only a HEALTHY probe is downgraded to inactive — a degraded one
+		// stays degraded (missing binary/grant = un-migrated or mis-installed box, never "off").
+		if s.Status == StatusOK && c.GatedBy != "" && p.GateActive != nil && !p.GateActive(c.GatedBy) {
+			s.Status, s.Reason = StatusInactive, ReasonInactive
+		}
+		out = append(out, s)
+	}
+	return out
+}
+
+// granted reports whether `sudo -n -l -- <binary> <reprArgs…>` is permitted (exit 0). List-mode is
+// side-effect-free — the command is matched against the policy, never run.
+func (p Prober) granted(ctx context.Context, c Capability) bool {
+	args := append([]string{"-n", "-l", "--", c.Binary}, c.ReprArgs...)
+	_, _, err := p.Runner.Run(ctx, "sudo", args...)
+	return err == nil
+}
+
+// Summarize returns (okCount, total, degraded) for logging. degraded lists DEGRADED statuses
+// only — inactive is a deliberate, healthy state and must not land in the error log (it is
+// counted via len(statuses)-ok-len(degraded) by callers that want it).
+func Summarize(statuses []Status) (ok, total int, degraded []Status) {
+	total = len(statuses)
+	for _, s := range statuses {
+		switch s.Status {
+		case StatusOK:
+			ok++
+		case StatusDegraded:
+			degraded = append(degraded, s)
+		}
+	}
+	return ok, total, degraded
+}

internal/capability/probe_test.go

@@ -0,0 +1,202 @@
+package capability
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+)
+
+// fakeRunner returns a canned error per (command line) and records calls. deny holds binaries (or
+// the bare "sudo -n -l" preflight) that should fail; everything else exits 0.
+type fakeRunner struct {
+	preflightErr error
+	denyBinary   map[string]bool // binary path → policy-denied
+	calls        int
+	executedReal bool // set if a probed command was ever run WITHOUT -l (must never happen)
+}
+
+func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	f.calls++
+	// Preflight is `sudo -n -l` (exactly 2 args, no `--`).
+	if name == "sudo" && len(args) == 2 && args[0] == "-n" && args[1] == "-l" {
+		return nil, nil, f.preflightErr
+	}
+	// Every real probe must be a LIST: `sudo -n -l -- <binary> …`.
+	if name != "sudo" || len(args) < 4 || args[0] != "-n" || args[1] != "-l" || args[2] != "--" {
+		f.executedReal = true
+		return nil, nil, nil
+	}
+	binary := args[3]
+	if f.denyBinary[binary] {
+		return nil, nil, errors.New("sudo: a password is required")
+	}
+	return nil, nil, nil
+}
+
+func find(statuses []Status, name string) Status {
+	for _, s := range statuses {
+		if s.Name == name {
+			return s
+		}
+	}
+	return Status{}
+}
+
+// §7-A: all grants present + binaries exist → every capability ok.
+func TestProbe_AllOK(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+	ok, total, degraded := Summarize(statuses)
+	if total != len(Manifest()) {
+		t.Fatalf("total=%d want %d", total, len(Manifest()))
+	}
+	if ok != total || len(degraded) != 0 {
+		t.Fatalf("expected all ok, got %d/%d (degraded: %+v)", ok, total, degraded)
+	}
+	if r.executedReal {
+		t.Fatal("probe executed a command without -l (must be list-only)")
+	}
+}
+
+// §7-B: one grant denied → that capability degraded "sudo policy denied", others ok. Serve-degraded.
+func TestProbe_OneDenied(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{"/usr/bin/lxc-info": true}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+
+	gi := find(statuses, "guest-init-pid")
+	if gi.Status != StatusDegraded || gi.Reason != "sudo policy denied" {
+		t.Fatalf("guest-init-pid = %+v, want degraded/sudo policy denied", gi)
+	}
+	if !gi.Critical {
+		t.Fatal("guest-init-pid should be Critical")
+	}
+	// A sibling stays ok.
+	if s := find(statuses, "drive-bind"); s.Status != StatusOK {
+		t.Fatalf("drive-bind = %+v, want ok", s)
+	}
+	ok, total, _ := Summarize(statuses)
+	if ok != total-1 {
+		t.Fatalf("expected exactly one degraded, got ok=%d total=%d", ok, total)
+	}
+}
+
+// §7-D: binary missing but policy granted → degraded "binary not found".
+func TestProbe_BinaryMissing(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(path string) bool { return path != "/usr/bin/lxc-info" }}
+	statuses := p.Probe(context.Background())
+	gi := find(statuses, "guest-init-pid")
+	if gi.Status != StatusDegraded || gi.Reason != "binary not found" {
+		t.Fatalf("guest-init-pid = %+v, want degraded/binary not found", gi)
+	}
+}
+
+// §8 aggregate: sudo itself unavailable for the user → ONE aggregate degraded, not N.
+func TestProbe_SudoUnavailableAggregates(t *testing.T) {
+	r := &fakeRunner{preflightErr: errors.New("Sorry, user felhom-agent may not run sudo"), denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+	if len(statuses) != 1 {
+		t.Fatalf("expected 1 aggregate status, got %d", len(statuses))
+	}
+	s := statuses[0]
+	if s.Name != "sudo" || s.Status != StatusDegraded || !s.Critical || !strings.Contains(s.Reason, "drop-in not installed") {
+		t.Fatalf("aggregate = %+v, want critical degraded sudo-unavailable", s)
+	}
+}
+
+// Probe must never raise — even with a nil runner (e.g. a dev path) it returns statuses.
+func TestProbe_NilRunnerNoPanic(t *testing.T) {
+	p := Prober{Runner: nil, Exists: func(string) bool { return true }}
+	if got := len(p.Probe(context.Background())); got != len(Manifest()) {
+		t.Fatalf("nil-runner probe returned %d statuses, want %d", got, len(Manifest()))
+	}
+}
+
+// ── DR-tier gate (v0.86.0) ─────────────────────────────────────────────────────────────────────
+
+// Gate OFF + healthy plumbing → the gated pbsdr-* capabilities report INACTIVE (the neutral
+// "disabled by configuration" state), NOT ok and NOT degraded — Scenario B of the DR-by-default
+// spec. Ungated siblings are untouched. Red-proof partner: collapse inactive into ok (drop the
+// gate branch in Probe) → this fails while TestProbe_GateOffBinaryMissingStaysDegraded passes.
+func TestProbe_GateOffHealthyIsInactive(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{
+		Runner:     r,
+		Exists:     func(string) bool { return true },
+		GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
+	}
+	statuses := p.Probe(context.Background())
+	// v0.88.0: escrow-ceremony joins the gate EXPLICITLY (non-pbsdr name, GatedBy literal) —
+	// the ceremony only exists behind the DR tier (no PBS key, no ceremony).
+	// v0.91.0: pbsdr-read (the R-39 credential-read verb) rides the same `pbsdr-` prefix gate — a new
+	// pbsdr-* op is gated by construction, which is exactly the property this list is here to hold.
+	for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant", "pbsdr-read", "escrow-ceremony"} {
+		s := find(statuses, name)
+		if s.Status != StatusInactive || s.Reason != ReasonInactive {
+			t.Fatalf("%s = %+v, want inactive/%q", name, s, ReasonInactive)
+		}
+	}
+	// An ungated sibling stays plain ok.
+	if s := find(statuses, "drive-bind"); s.Status != StatusOK {
+		t.Fatalf("drive-bind = %+v, want ok (ungated)", s)
+	}
+	// Summarize must NOT count inactive as degraded (it is not error-log-worthy).
+	ok, total, degraded := Summarize(statuses)
+	if len(degraded) != 0 {
+		t.Fatalf("inactive leaked into degraded: %+v", degraded)
+	}
+	if ok != total-5 {
+		t.Fatalf("ok=%d total=%d, want exactly the 5 gated ones non-ok", ok, total)
+	}
+}
+
+// Gate OFF + BROKEN plumbing (binary missing) → DEGRADED stays degraded. An un-migrated
+// pre-v1.15.0 box must never masquerade as deliberately disabled ("never silently pretend").
+func TestProbe_GateOffBinaryMissingStaysDegraded(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{
+		Runner:     r,
+		Exists:     func(path string) bool { return path != "/usr/local/sbin/felhom-pbs-apply" },
+		GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
+	}
+	statuses := p.Probe(context.Background())
+	for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant"} {
+		s := find(statuses, name)
+		if s.Status != StatusDegraded || s.Reason != "binary not found" {
+			t.Fatalf("%s = %+v, want degraded/binary not found even with the gate off", name, s)
+		}
+	}
+}
+
+// Gate ON (DR configured) + healthy plumbing → plain ok, exactly the pre-v0.86.0 behavior.
+// A nil GateActive must behave the same (fails ACTIVE).
+func TestProbe_GateOnOrNilIsOK(t *testing.T) {
+	for _, gate := range []func(string) bool{nil, func(string) bool { return true }} {
+		r := &fakeRunner{denyBinary: map[string]bool{}}
+		p := Prober{Runner: r, Exists: func(string) bool { return true }, GateActive: gate}
+		statuses := p.Probe(context.Background())
+		if s := find(statuses, "pbsdr-create"); s.Status != StatusOK {
+			t.Fatalf("pbsdr-create = %+v, want ok (gate active/nil)", s)
+		}
+	}
+}
+
+// The gate covers exactly the pbsdr-* entries (name-prefix mechanism) PLUS escrow-ceremony (an
+// explicit GatedBy literal — v0.88.0: the ceremony only exists behind the DR tier, but its name
+// says what the feature is). Nothing else may be gated (a regression here would silently un-gate
+// the tier or gate an unrelated capability).
+func TestManifest_ExactlyPBSDRGated(t *testing.T) {
+	for _, c := range Manifest() {
+		wantGated := strings.HasPrefix(c.Name, "pbsdr-") || c.Name == "escrow-ceremony"
+		if gated := c.GatedBy == GatePBSDR; gated != wantGated {
+			t.Fatalf("%s: GatedBy=%q, want gated=%v", c.Name, c.GatedBy, wantGated)
+		}
+		if c.GatedBy != "" && c.GatedBy != GatePBSDR {
+			t.Fatalf("%s: unknown gate id %q", c.Name, c.GatedBy)
+		}
+	}
+}

internal/config/backup_tiers_test.go

@@ -0,0 +1,182 @@
+package config
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+)
+
+// R-82 Slice A.1 — per-target cadence + retention resolution.
+//
+// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
+// resolve to exactly one tier that behaves as it does now. The second property is that a
+// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
+// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
+// R-82 exists to fix.
+
+func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
+	// Exactly the shape live on demo-felhom today.
+	var b BackupConfig
+	raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
+	if err := json.Unmarshal([]byte(raw), &b); err != nil {
+		t.Fatal(err)
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(warnings) != 0 {
+		t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
+	}
+	if len(tiers) != 1 {
+		t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
+	}
+	got := tiers[0]
+	if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
+		t.Fatalf("legacy tier changed: %+v", got)
+	}
+}
+
+// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
+// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
+func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
+	tiers, warnings := BackupConfig{}.BackupTiers()
+	if len(warnings) != 0 || len(tiers) != 1 {
+		t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
+	}
+	if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
+		t.Fatalf("zero-config defaults changed: %+v", tiers[0])
+	}
+}
+
+// The whole point: local daily + PBS weekly, each with its OWN retention.
+func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
+	var b BackupConfig
+	raw := `{
+	  "local_backup_target":"local",
+	  "local_backup_retention":3,
+	  "backup_cadence_seconds":86400,
+	  "backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
+	}`
+	if err := json.Unmarshal([]byte(raw), &b); err != nil {
+		t.Fatal(err)
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(warnings) != 0 {
+		t.Fatalf("unexpected warnings: %v", warnings)
+	}
+	if len(tiers) != 2 {
+		t.Fatalf("want 2 tiers, got %+v", tiers)
+	}
+	if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
+		t.Fatalf("primary tier wrong: %+v", tiers[0])
+	}
+	if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
+		t.Fatalf("PBS tier wrong: %+v", tiers[1])
+	}
+	// THE knob-sharing check: the two retentions are independent values, not one shared number.
+	if tiers[0].KeepLast == tiers[1].KeepLast {
+		t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
+	}
+}
+
+// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
+// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
+func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(tiers) != 1 {
+		t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
+	}
+	if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
+		t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
+	}
+	if !strings.Contains(warnings[0], "felhom-pbs") {
+		t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
+	}
+}
+
+func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets: []BackupTargetConfig{
+			{TargetID: "", CadenceSeconds: 3600},
+			{TargetID: "local", CadenceSeconds: 3600},        // repeats the primary
+			{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
+			{TargetID: "felhom-pbs", CadenceSeconds: 99},     // duplicate
+		},
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
+		t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
+	}
+	if len(warnings) != 3 {
+		t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
+	}
+}
+
+// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
+// itself because someone forgot a field.
+func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
+	}
+	tiers, _ := b.BackupTiers()
+	if len(tiers) != 2 {
+		t.Fatalf("got %+v", tiers)
+	}
+	if tiers[1].KeepLast != 0 {
+		t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
+	}
+	// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
+	b.ExtraTargets[0].KeepLast = -5
+	tiers, _ = b.BackupTiers()
+	if tiers[1].KeepLast != 0 {
+		t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
+	}
+}
+
+// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
+func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
+	for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
+		b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
+		tiers, _ := b.BackupTiers()
+		if tiers[0].KeepLast != tc.want {
+			t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
+		}
+	}
+}
+
+// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
+// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
+// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
+// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
+// makes the next attempt collide with the guest lock vzdump still holds.
+func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
+	}
+	tiers, _ := b.BackupTiers()
+	if len(tiers) != 2 {
+		t.Fatalf("got %+v", tiers)
+	}
+	if tiers[0].WaitTimeout != 30*time.Minute {
+		t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
+	}
+	if tiers[1].WaitTimeout != 12*time.Hour {
+		t.Fatalf("an offsite tier must default to a GENEROUS wait (operator ruling: let the first backup run as long as needed) — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
+	}
+	// And it must be overridable per tier.
+	b.ExtraTargets[0].WaitTimeoutSeconds = 3600
+	tiers, _ = b.BackupTiers()
+	if tiers[1].WaitTimeout != time.Hour {
+		t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
+	}
+	// The two tiers must NOT share one bound.
+	if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
+		t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
+	}
+}

internal/config/config_test.go

@@ -0,0 +1,216 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+// TestRestoreTestPBSRestoreTimeout mirrors the BackupCadence accessor contract: positive as-is,
+// 0 → default (120m), negative → default.
+func TestRestoreTestPBSRestoreTimeout(t *testing.T) {
+	cases := []struct {
+		secs int
+		want time.Duration
+	}{
+		{0, 120 * time.Minute},
+		{-5, 120 * time.Minute},
+		{1800, 30 * time.Minute},
+		{7200, 120 * time.Minute},
+	}
+	for _, c := range cases {
+		got := BackupConfig{RestoreTestPBSRestoreTimeoutSeconds: c.secs}.RestoreTestPBSRestoreTimeout()
+		if got != c.want {
+			t.Errorf("RestoreTestPBSRestoreTimeout(secs=%d) = %v, want %v", c.secs, got, c.want)
+		}
+	}
+}
+
+func TestRedactedMasksSecret(t *testing.T) {
+	c := Default()
+	c.Proxmox.Token = "felhom-agent@pve!agent=b6547d9d-08ec-4f22-beb8-a551dc2cd69d"
+	got := c.Redacted().Proxmox.Token
+	if strings.Contains(got, "b6547d9d") {
+		t.Fatalf("secret leaked in redacted token: %q", got)
+	}
+	if !strings.HasPrefix(got, "felhom-agent@pve!agent=") {
+		t.Errorf("redacted token lost its public prefix: %q", got)
+	}
+	// The original must be untouched (Redacted returns a copy).
+	if !strings.Contains(c.Proxmox.Token, "b6547d9d") {
+		t.Errorf("Redacted mutated the original config")
+	}
+}
+
+func TestValidate(t *testing.T) {
+	c := Default()
+	c.Proxmox.Node = "demo-felhom"
+	c.Proxmox.Token = "felhom-agent@pve!agent=secret"
+	if err := c.Validate(); err != nil {
+		t.Fatalf("valid config rejected: %v", err)
+	}
+	c.Proxmox.Token = "no-bang-no-eq"
+	if err := c.Validate(); err == nil {
+		t.Errorf("malformed token accepted")
+	}
+}
+
+func TestRedactedMasksHubKey(t *testing.T) {
+	c := Default()
+	c.Hub.APIKey = "hub-secret-abcdef"
+	if got := c.Redacted().Hub.APIKey; got == "hub-secret-abcdef" || got == "" {
+		t.Fatalf("hub key not masked: %q", got)
+	}
+	if !strings.Contains(c.Hub.APIKey, "abcdef") {
+		t.Error("Redacted mutated the original hub key")
+	}
+}
+
+func TestHubConfigValidate(t *testing.T) {
+	base := HubConfig{URL: "https://hub.felhom.eu", HostID: "h1", APIKey: "k"}
+	if err := base.Validate(); err != nil {
+		t.Fatalf("valid hub config rejected: %v", err)
+	}
+	bad := []HubConfig{
+		{HostID: "h", APIKey: "k"},                              // no URL
+		{URL: "https://x", APIKey: "k"},                         // no host
+		{URL: "https://x", HostID: "h"},                         // no key
+		{URL: "http://hub.felhom.eu", HostID: "h", APIKey: "k"}, // http non-loopback
+		{URL: "ftp://x", HostID: "h", APIKey: "k"},              // bad scheme
+	}
+	for i, h := range bad {
+		if err := h.Validate(); err == nil {
+			t.Errorf("case %d: expected validation error for %+v", i, h)
+		}
+	}
+	// http is allowed for loopback (tests).
+	if err := (HubConfig{URL: "http://127.0.0.1:8443", HostID: "h", APIKey: "k"}).Validate(); err != nil {
+		t.Errorf("http loopback should be allowed: %v", err)
+	}
+}
+
+func TestHubEnvOverlayAndDefaults(t *testing.T) {
+	t.Setenv("FELHOM_AGENT_HUB_URL", "https://hub.example")
+	t.Setenv("FELHOM_AGENT_HUB_HOST_ID", "env-host")
+	t.Setenv("FELHOM_AGENT_HUB_API_KEY", "env-key")
+	t.Setenv("FELHOM_AGENT_HUB_POLL_SECONDS", "120")
+	cfg, err := Load("")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Hub.URL != "https://hub.example" || cfg.Hub.HostID != "env-host" || cfg.Hub.APIKey != "env-key" {
+		t.Errorf("hub env overlay failed: %+v", cfg.Hub)
+	}
+	if cfg.Hub.PollSeconds != 120 {
+		t.Errorf("poll seconds = %d, want 120", cfg.Hub.PollSeconds)
+	}
+	// withDefaults fills zero timeout.
+	if (HubConfig{}).WithDefaults().TimeoutSeconds != 30 {
+		t.Error("WithDefaults should set TimeoutSeconds=30")
+	}
+}
+
+func TestLoadFileThenEnvOverride(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "agent.json")
+	if err := os.WriteFile(path, []byte(`{"proxmox":{"node":"file-node","token":"u@pve!t=filesecret"}}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("FELHOM_AGENT_PROXMOX_NODE", "env-node")
+	cfg, err := Load(path)
+	if err != nil {
+		t.Fatalf("Load: %v", err)
+	}
+	if cfg.Proxmox.Node != "env-node" {
+		t.Errorf("env did not override node: %q", cfg.Proxmox.Node)
+	}
+	if cfg.Proxmox.Token != "u@pve!t=filesecret" {
+		t.Errorf("token from file lost: %q", cfg.Proxmox.Token)
+	}
+	if cfg.Proxmox.Endpoint != "https://127.0.0.1:8006" {
+		t.Errorf("default endpoint lost: %q", cfg.Proxmox.Endpoint)
+	}
+}
+
+// CAMPAIGN-3 Part 6: deployment_mode gates node self-heal, and it is FAIL-SAFE to byo — absent or any
+// unknown value is byo, ONLY the exact "appliance" unlocks the remedy.
+func TestIsAppliance_FailSafeToByo(t *testing.T) {
+	cases := []struct {
+		mode string
+		want bool
+	}{
+		{"appliance", true},
+		{"byo", false},
+		{"", false},          // absent field → byo (fail-safe)
+		{"Appliance", false}, // case-sensitive — a typo must not unlock the remedy
+		{"garbage", false},
+	}
+	for _, c := range cases {
+		cfg := &Config{DeploymentMode: c.mode}
+		if got := cfg.IsAppliance(); got != c.want {
+			t.Errorf("IsAppliance(mode=%q) = %t, want %t", c.mode, got, c.want)
+		}
+	}
+}
+
+// The env overlay can set deployment_mode (FELHOM_AGENT_DEPLOYMENT_MODE).
+func TestDeploymentModeEnvOverlay(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "agent.json")
+	if err := os.WriteFile(path, []byte(`{"proxmox":{"node":"n","token":"u@pve!t=s"},"deployment_mode":"byo"}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("FELHOM_AGENT_DEPLOYMENT_MODE", "appliance")
+	cfg, err := Load(path)
+	if err != nil {
+		t.Fatalf("Load: %v", err)
+	}
+	if !cfg.IsAppliance() {
+		t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode)
+	}
+}
+
+// R-50: the island NIC fields are all-or-nothing and the guest addr must be a CIDR. A half-set or
+// malformed island must fail at config load (a botched install) rather than silently fall back to
+// LAN-only, which would leave a guest with an island bind and no island NIC — the exact silent break
+// R-50 exists to kill. Covers LocalAPIConfig.Validate + IslandEnabled.
+func TestLocalAPIConfig_IslandValidation(t *testing.T) {
+	base := LocalAPIConfig{Enable: true, ListenAddr: "169.254.253.1:8443"}
+
+	// both empty → fine (pre-R-50 default), IslandEnabled false
+	if err := base.Validate(); err != nil {
+		t.Errorf("no island config must validate: %v", err)
+	}
+	if base.IslandEnabled() {
+		t.Errorf("IslandEnabled must be false when unset")
+	}
+	// both set, valid CIDR → fine, IslandEnabled true
+	ok := base
+	ok.IslandBridge, ok.IslandGuestAddr = "vmbr9", "169.254.253.2/30"
+	if err := ok.Validate(); err != nil {
+		t.Errorf("valid island config must validate: %v", err)
+	}
+	if !ok.IslandEnabled() {
+		t.Errorf("IslandEnabled must be true when both set")
+	}
+	// bridge only → rejected (all-or-nothing)
+	half := base
+	half.IslandBridge = "vmbr9"
+	if err := half.Validate(); err == nil {
+		t.Errorf("half-set island (bridge only) must be rejected")
+	}
+	// guest addr only → rejected
+	half2 := base
+	half2.IslandGuestAddr = "169.254.253.2/30"
+	if err := half2.Validate(); err == nil {
+		t.Errorf("half-set island (guest addr only) must be rejected")
+	}
+	// both set but guest addr is not a CIDR → rejected
+	bad := base
+	bad.IslandBridge, bad.IslandGuestAddr = "vmbr9", "169.254.253.2" // missing /30
+	if err := bad.Validate(); err == nil {
+		t.Errorf("island guest addr without a CIDR mask must be rejected")
+	}
+}

internal/desired/consumer_test.go

@@ -0,0 +1,90 @@
+package desired
+
+// S3 Group C — the raw-consumer fan-out seam: called on generation advance, NOT on no-advance,
+// and a panicking consumer is contained (the guest reconcile path must never break).
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"sync"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+type recordingConsumer struct {
+	mu    sync.Mutex
+	calls []*hub.DesiredStateResponse
+	panic bool
+}
+
+func (r *recordingConsumer) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
+	r.mu.Lock()
+	r.calls = append(r.calls, resp)
+	r.mu.Unlock()
+	if r.panic {
+		panic("consumer exploded")
+	}
+}
+
+func (r *recordingConsumer) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.calls) }
+
+type stubFetcher struct{ resp *hub.DesiredStateResponse }
+
+func (s *stubFetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
+	return s.resp, nil
+}
+
+func testResp(gen int64) *hub.DesiredStateResponse {
+	return &hub.DesiredStateResponse{
+		Generation: gen,
+		DesiredState: hub.WireDesiredState{
+			Guests:    []hub.WireDesiredGuest{},
+			Wireguard: &hub.WireWireguard{Pubkey: "PK", AssignedIP: "10.77.0.2/32"},
+		},
+	}
+}
+
+func TestSyncer_ConsumerCalledOnAdvanceOnly(t *testing.T) {
+	provider := reconcile.NewCachingProvider()
+	f := &stubFetcher{resp: testResp(2)}
+	s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	c := &recordingConsumer{}
+	s.AddConsumer(c)
+
+	// Advance → fetch → consumer called with the raw doc (wireguard block intact).
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
+	if c.count() != 1 {
+		t.Fatalf("consumer calls = %d, want 1", c.count())
+	}
+	if c.calls[0].DesiredState.Wireguard == nil || c.calls[0].DesiredState.Wireguard.Pubkey != "PK" {
+		t.Fatalf("consumer got %+v — the raw wireguard block must ride through", c.calls[0].DesiredState.Wireguard)
+	}
+
+	// No advance → no fetch → no consumer call (the negative).
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
+	if c.count() != 1 {
+		t.Errorf("consumer called without a generation advance: %d", c.count())
+	}
+}
+
+func TestSyncer_PanickingConsumerContained(t *testing.T) {
+	provider := reconcile.NewCachingProvider()
+	f := &stubFetcher{resp: testResp(1)}
+	s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	bomb := &recordingConsumer{panic: true}
+	after := &recordingConsumer{}
+	s.AddConsumer(bomb)
+	s.AddConsumer(after)
+
+	// Must not panic out; the second consumer still runs; the provider still updated.
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 1})
+	if after.count() != 1 {
+		t.Errorf("consumer after the panicking one not called: %d", after.count())
+	}
+	if provider.Generation() != 1 {
+		t.Errorf("provider generation = %d, want 1 (guest path unaffected)", provider.Generation())
+	}
+}

internal/desired/syncer.go

@@ -0,0 +1,137 @@
+// Package desired bridges the hub's "Down" channel (the control-envelope generation signal +
+// the desired-state fetch) to the reconcile engine's provider (slice 10A). It implements
+// hub.EnvelopeObserver: on each heartbeat it inspects the envelope's DesiredGeneration and, only
+// when it has ADVANCED past the cached one, fetches the full desired-state and updates the
+// engine's CachingProvider. So the heartbeat stays light; the heavy state moves on change.
+//
+// It lives in its own package because it imports BOTH hub (the wire client + types) and reconcile
+// (the domain DesiredState + CachingProvider). hub does not import it (the loop sees only the
+// hub.EnvelopeObserver seam) and reconcile does not import it — so there is no import cycle.
+package desired
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// Fetcher fetches this host's desired-state from the hub. Satisfied by *hub.Client.
+type Fetcher interface {
+	FetchDesiredState(ctx context.Context) (*hub.DesiredStateResponse, error)
+}
+
+// RawConsumer receives the FULL fetched desired-state document after each successful
+// generation-advance fetch (S3 seam — internal/wgtunnel consumes its wireguard block this way
+// without the reconcile engine learning about tunnels). Implementations must not block: do the
+// cheap store-and-nudge, never network/exec inline.
+type RawConsumer interface {
+	OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse)
+}
+
+// Syncer keeps the engine's CachingProvider in step with the hub's authoritative desired-state.
+type Syncer struct {
+	fetcher   Fetcher
+	provider  *reconcile.CachingProvider
+	consumers []RawConsumer
+	logger    *slog.Logger
+}
+
+// AddConsumer registers a raw desired-state consumer (nil-safe no-op). Not concurrency-safe —
+// call during wiring, before the hub loop starts.
+func (s *Syncer) AddConsumer(c RawConsumer) {
+	if c != nil {
+		s.consumers = append(s.consumers, c)
+	}
+}
+
+// NewSyncer builds a Syncer over the hub fetcher and the engine's provider.
+func NewSyncer(fetcher Fetcher, provider *reconcile.CachingProvider, logger *slog.Logger) *Syncer {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Syncer{fetcher: fetcher, provider: provider, logger: logger}
+}
+
+// OnEnvelope implements hub.EnvelopeObserver. It fetches + caches the desired-state ONLY when the
+// envelope's generation advances past the provider's cached generation — otherwise it is a no-op
+// (the cached state is already current). A fetch failure keeps the last-known state (the engine
+// keeps reconciling toward it) and is retried on the next advance signal.
+func (s *Syncer) OnEnvelope(ctx context.Context, env *hub.ControlEnvelope) {
+	if env == nil || s.provider == nil {
+		return
+	}
+	have := s.provider.Generation()
+	if env.DesiredGeneration <= have {
+		return // cached: the heavy desired-state moves only on a generation advance
+	}
+	s.logger.Debug("desired: generation advanced — fetching desired-state",
+		"have_generation", have, "envelope_generation", env.DesiredGeneration)
+	start := time.Now()
+	resp, err := s.fetcher.FetchDesiredState(ctx)
+	if err != nil {
+		s.logger.Warn("desired: fetch failed; keeping cached desired-state",
+			"have_generation", have, "envelope_generation", env.DesiredGeneration, "err", err)
+		return
+	}
+	s.logger.Debug("desired: fetched", "generation", resp.Generation,
+		"duration_ms", time.Since(start).Milliseconds())
+	state := mapWire(resp.DesiredState, s.logger)
+	// Cache against the FETCHED generation (not the envelope's) — robust to a generation that
+	// advanced again between the heartbeat and this fetch (we won't re-fetch the same state).
+	s.provider.Update(resp.Generation, state)
+	s.logger.Info("desired: updated from hub",
+		"generation", resp.Generation, "guests", len(state.Guests))
+	// S3: fan the raw document out to registered consumers (wgtunnel etc). A panicking consumer
+	// is contained — the guest reconcile path must never break over a tunnel add-on.
+	for _, c := range s.consumers {
+		s.notifyConsumer(ctx, c, resp)
+	}
+	if env.HasSignedOps {
+		// 10A only notes the flag; fetching + verifying + executing signed ops is slice 10B.
+		s.logger.Info("desired: hub reports pending signed ops (fetch/execute is slice 10B)")
+	}
+}
+
+// notifyConsumer delivers one raw document with panic containment.
+func (s *Syncer) notifyConsumer(ctx context.Context, c RawConsumer, resp *hub.DesiredStateResponse) {
+	defer func() {
+		if r := recover(); r != nil {
+			s.logger.Error("desired: raw consumer panicked (contained)", "panic", r)
+		}
+	}()
+	c.OnDesiredState(ctx, resp)
+}
+
+// mapWire maps the hub wire desired-state to the reconcile domain. 10A acts only on guests; the
+// forward-compat fields (restore_directive — 10D — etc.) are carried on the wire and logged, but
+// not translated into actions here.
+func mapWire(w hub.WireDesiredState, logger *slog.Logger) reconcile.DesiredState {
+	guests := make(map[int]reconcile.DesiredGuest, len(w.Guests))
+	for _, g := range w.Guests {
+		dg := reconcile.DesiredGuest{
+			VMID:         g.VMID,
+			Spec:         g.Spec,
+			Description:  g.Description,
+			Decommission: g.Decommission,
+		}
+		switch g.Run {
+		case "running":
+			dg.Run = reconcile.RunRunning
+		case "stopped":
+			dg.Run = reconcile.RunStopped
+		default:
+			dg.Run = reconcile.RunUnspecified // unknown/empty → unmanaged (planner leaves run alone)
+		}
+		guests[g.VMID] = dg
+	}
+	if w.RestoreDirective != nil {
+		// The reconcile mapping does NOT act on the directive; the DR consumer (raw-consumer seam,
+		// S5 internal/dr) surfaces it as an inspectable restore PLAN — no restore is executed here.
+		logger.Info("desired: restore_directive present (surfaced as a restore PLAN by the DR consumer; not acted on in the reconcile mapping)",
+			"mode", w.RestoreDirective.Mode)
+	}
+	return reconcile.DesiredState{Guests: guests}
+}

internal/desired/syncer_test.go

@@ -0,0 +1,117 @@
+package desired
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// fakeFetcher counts FetchDesiredState calls and returns a fixed response (or error).
+type fakeFetcher struct {
+	resp  *hub.DesiredStateResponse
+	err   error
+	calls int
+}
+
+func (f *fakeFetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
+	f.calls++
+	return f.resp, f.err
+}
+
+func env(gen int64, signed bool) *hub.ControlEnvelope {
+	return &hub.ControlEnvelope{DesiredGeneration: gen, HasSignedOps: signed}
+}
+
+// The headline caching behaviour: desired-state is fetched ONCE when the generation advances, and
+// NOT re-fetched while the generation is unchanged (the heartbeat stays light).
+func TestSyncer_FetchesOnceOnGenerationAdvance(t *testing.T) {
+	run := "running"
+	f := &fakeFetcher{resp: &hub.DesiredStateResponse{
+		Generation: 1,
+		DesiredState: hub.WireDesiredState{Guests: []hub.WireDesiredGuest{
+			{VMID: 100, Run: run},
+			{VMID: 200, Decommission: true},
+		}},
+	}}
+	p := reconcile.NewCachingProvider()
+	s := NewSyncer(f, p, quiet())
+	ctx := context.Background()
+
+	// Generation 0 (fresh host, no intent) → NO fetch.
+	s.OnEnvelope(ctx, env(0, false))
+	if f.calls != 0 {
+		t.Fatalf("fetched %d times at generation 0, want 0", f.calls)
+	}
+
+	// Generation advances to 1 → fetch exactly once, cache updated.
+	s.OnEnvelope(ctx, env(1, false))
+	if f.calls != 1 {
+		t.Fatalf("fetched %d times on advance, want 1", f.calls)
+	}
+	if p.Generation() != 1 {
+		t.Errorf("cached generation = %d, want 1", p.Generation())
+	}
+	st, _ := p.Desired(ctx)
+	if st.Guests[100].Run != reconcile.RunRunning {
+		t.Errorf("guest 100 run = %q, want running", st.Guests[100].Run)
+	}
+	if !st.Guests[200].Decommission {
+		t.Errorf("guest 200 decommission = false, want true")
+	}
+
+	// Same generation on the next heartbeats → NO re-fetch (cached).
+	s.OnEnvelope(ctx, env(1, false))
+	s.OnEnvelope(ctx, env(1, false))
+	if f.calls != 1 {
+		t.Errorf("re-fetched on an unchanged generation (calls=%d, want 1)", f.calls)
+	}
+
+	// A further advance → one more fetch.
+	f.resp = &hub.DesiredStateResponse{Generation: 2, DesiredState: hub.WireDesiredState{}}
+	s.OnEnvelope(ctx, env(2, false))
+	if f.calls != 2 || p.Generation() != 2 {
+		t.Errorf("second advance: calls=%d gen=%d, want 2/2", f.calls, p.Generation())
+	}
+}
+
+// A fetch failure keeps the last-known cache (the engine keeps reconciling toward it) and does not
+// advance the cached generation (so the next heartbeat retries).
+func TestSyncer_FetchFailureKeepsCache(t *testing.T) {
+	p := reconcile.NewCachingProvider()
+	p.Update(1, reconcile.DesiredState{Guests: map[int]reconcile.DesiredGuest{100: {VMID: 100, Run: reconcile.RunRunning}}})
+	f := &fakeFetcher{err: errors.New("hub down")}
+	s := NewSyncer(f, p, quiet())
+
+	s.OnEnvelope(context.Background(), env(5, false)) // generation jumped, but fetch fails
+	if p.Generation() != 1 {
+		t.Errorf("generation advanced to %d despite fetch failure, want kept at 1", p.Generation())
+	}
+	st, _ := p.Desired(context.Background())
+	if st.Guests[100].Run != reconcile.RunRunning {
+		t.Errorf("cache lost on fetch failure: %+v", st.Guests)
+	}
+}
+
+// The fetched generation (not the envelope's) is what gets cached — robust to a generation that
+// advanced again between the heartbeat and the fetch.
+func TestSyncer_CachesFetchedGeneration(t *testing.T) {
+	f := &fakeFetcher{resp: &hub.DesiredStateResponse{Generation: 7, DesiredState: hub.WireDesiredState{}}}
+	p := reconcile.NewCachingProvider()
+	s := NewSyncer(f, p, quiet())
+	s.OnEnvelope(context.Background(), env(5, false)) // envelope said 5, fetch returned 7
+	if p.Generation() != 7 {
+		t.Errorf("cached generation = %d, want 7 (the fetched generation)", p.Generation())
+	}
+	// A later envelope at generation 6 must NOT trigger a re-fetch (we already have 7).
+	s.OnEnvelope(context.Background(), env(6, false))
+	if f.calls != 1 {
+		t.Errorf("re-fetched at generation 6 when cache is 7 (calls=%d)", f.calls)
+	}
+}

internal/dr/plan.go

@@ -0,0 +1,134 @@
+// Package dr consumes the host-loss restore_directive (slice 10D / S5) into an inspectable restore
+// PLAN. It is DERIVE-AND-SURFACE only: the plan is logged (and exposed for the report), never
+// executed — the destructive restore is a separate, operator-present, STOP-gated step. The Consumer
+// has NO restore/destroy API by construction, so "execute nothing" is a structural guarantee.
+package dr
+
+import (
+	"context"
+	"log/slog"
+	"sync"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// RestorePlan is the derived-but-not-executed host-loss plan: per guest → restore coords + sizing;
+// per drive → durable_id → expected mount. No secrets (coordinates/identifiers/sizes only).
+type RestorePlan struct {
+	Mode   string          `json:"mode"`
+	Guests []PlannedGuest  `json:"guests"`
+	Drives []PlannedDrive  `json:"drives"`
+	PBS    *hub.DRPBSCoord `json:"pbs,omitempty"` // WHERE the offsite backups live (repo/ns/latest snapshot)
+}
+
+// PlannedGuest is one guest to restore in place, from the offsite datastore, at its original sizing.
+type PlannedGuest struct {
+	VMID          int    `json:"vmid"`
+	Archive       string `json:"archive,omitempty"` // explicit archive from the directive; "" → resolve latest at restore time
+	TargetStorage string `json:"target_storage"`    // where the restored volumes land (e.g. local-lvm)
+	Cores         int    `json:"cores"`
+	MemoryBytes   int64  `json:"memory_bytes"`
+	DiskBytes     int64  `json:"disk_bytes"`
+}
+
+// PlannedDrive is one data drive to re-attach BY DURABLE_ID (the wrong-disk guard: a match attaches,
+// a non-match is refused — the matcher, exercised in the Part-4 spike, never resolves to a near disk).
+type PlannedDrive struct {
+	DurableID     string `json:"durable_id"`
+	ExpectedMount string `json:"expected_mount"`
+	Intent        string `json:"intent"`
+}
+
+// BuildRestorePlan derives the plan from a host_loss directive + the live DR recipe. Returns
+// (nil,false) for a guest_loss/absent directive or a nil recipe (nothing to plan). PURE: reads
+// nothing, executes nothing — the whole point of this slice's safe half.
+func BuildRestorePlan(directive *hub.WireRestoreDirective, recipe *hub.DRRecipeHostHalf, restoreStorage string) (*RestorePlan, bool) {
+	if directive == nil || directive.Mode != "host_loss" || recipe == nil {
+		return nil, false
+	}
+	plan := &RestorePlan{Mode: directive.Mode, PBS: recipe.PBS}
+	for _, g := range recipe.Guests {
+		pg := PlannedGuest{
+			VMID:          g.VMID,
+			TargetStorage: restoreStorage,
+			Cores:         g.Cores,
+			MemoryBytes:   g.MemoryBytes,
+			DiskBytes:     g.DiskBytes,
+		}
+		// The directive may name an explicit archive for a specific guest (else the restore step
+		// resolves the latest snapshot from the PBS coord at execution time).
+		if directive.Archive != "" && (directive.VMID == 0 || directive.VMID == g.VMID) {
+			pg.Archive = directive.Archive
+		}
+		plan.Guests = append(plan.Guests, pg)
+	}
+	for _, d := range recipe.Drives {
+		plan.Drives = append(plan.Drives, PlannedDrive{
+			DurableID:     d.DurableID,
+			ExpectedMount: d.MountPath,
+			Intent:        d.Intent,
+		})
+	}
+	return plan, true
+}
+
+// RecipeFunc yields the current DR recipe (the agent-derived scaffolding). It is called ONLY when a
+// host_loss directive is present (a rare DR event), so an on-demand Collect is acceptable.
+type RecipeFunc func(ctx context.Context) *hub.DRRecipeHostHalf
+
+// Consumer implements desired.RawConsumer: on a host_loss restore_directive it builds + SURFACES the
+// plan (structured log + LastPlan for the report/inspection) and executes NOTHING. A guest_loss or
+// absent directive clears the plan. It holds no restore/destroy dependency — surfacing is all it can do.
+type Consumer struct {
+	recipe         RecipeFunc
+	restoreStorage string
+	logger         *slog.Logger
+
+	mu       sync.Mutex
+	lastPlan *RestorePlan
+}
+
+// NewConsumer builds the DR plan consumer. recipe may be nil (then no plan can be built — logged).
+func NewConsumer(recipe RecipeFunc, restoreStorage string, logger *slog.Logger) *Consumer {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Consumer{recipe: recipe, restoreStorage: restoreStorage, logger: logger}
+}
+
+// OnDesiredState implements desired.RawConsumer. Non-host_loss → clear + no-op.
+func (c *Consumer) OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse) {
+	if resp == nil {
+		return
+	}
+	dir := resp.DesiredState.RestoreDirective
+	if dir == nil || dir.Mode != "host_loss" {
+		c.mu.Lock()
+		c.lastPlan = nil
+		c.mu.Unlock()
+		return
+	}
+	var recipe *hub.DRRecipeHostHalf
+	if c.recipe != nil {
+		recipe = c.recipe(ctx)
+	}
+	plan, ok := BuildRestorePlan(dir, recipe, c.restoreStorage)
+	if !ok {
+		c.logger.Warn("dr: host_loss restore_directive present but no DR recipe available yet — cannot build a plan",
+			"directive_vmid", dir.VMID)
+		return
+	}
+	c.mu.Lock()
+	c.lastPlan = plan
+	c.mu.Unlock()
+	// SURFACE only — the destructive restore is a separate, operator-present step.
+	c.logger.Warn("dr: host_loss RESTORE PLAN derived (NOT executed — supervised in-place restore is a separate, gated step)",
+		"mode", plan.Mode, "guests", len(plan.Guests), "drives", len(plan.Drives), "plan", plan)
+}
+
+// LastPlan returns the most recently derived plan (nil if none / cleared). For the report + tests.
+func (c *Consumer) LastPlan() *RestorePlan {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.lastPlan
+}

internal/dr/plan_test.go

@@ -0,0 +1,92 @@
+package dr
+
+import (
+	"context"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+func sampleRecipe() *hub.DRRecipeHostHalf {
+	return &hub.DRRecipeHostHalf{
+		RecipeVersion: 1,
+		Guests:        []hub.DRGuest{{VMID: 9201, Cores: 2, MemoryBytes: 12 << 30, DiskBytes: 32 << 30}},
+		PBS:           &hub.DRPBSCoord{RepoID: "felhom-offsite", Namespace: "demo-felhom-01", LatestSnapshotID: "9201"},
+		Drives:        []hub.DRDrive{{DurableID: "uuid:abc", MountPath: "/mnt/felhom-drives/photos", Intent: "enrolled", TotalBytes: 500 << 30}},
+	}
+}
+
+// TestBuildRestorePlan_HostLoss: a host_loss directive + recipe yields per-guest {vmid, archive,
+// target, sizing} + per-drive {durable_id → mount} + the offsite PBS coord.
+func TestBuildRestorePlan_HostLoss(t *testing.T) {
+	dir := &hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201, Archive: "felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z"}
+	plan, ok := BuildRestorePlan(dir, sampleRecipe(), "local-lvm")
+	if !ok || plan == nil {
+		t.Fatal("host_loss must yield a plan")
+	}
+	if plan.Mode != "host_loss" || len(plan.Guests) != 1 || len(plan.Drives) != 1 {
+		t.Fatalf("plan shape = %+v", plan)
+	}
+	g := plan.Guests[0]
+	if g.VMID != 9201 || g.TargetStorage != "local-lvm" || g.Cores != 2 || g.DiskBytes != 32<<30 {
+		t.Errorf("planned guest = %+v", g)
+	}
+	if g.Archive != dir.Archive {
+		t.Errorf("planned guest archive = %q, want the directive's %q", g.Archive, dir.Archive)
+	}
+	d := plan.Drives[0]
+	if d.DurableID != "uuid:abc" || d.ExpectedMount != "/mnt/felhom-drives/photos" {
+		t.Errorf("planned drive (durable_id→mount) = %+v", d)
+	}
+	if plan.PBS == nil || plan.PBS.RepoID != "felhom-offsite" {
+		t.Errorf("plan must carry the offsite PBS coord, got %+v", plan.PBS)
+	}
+}
+
+// TestBuildRestorePlan_NoPlanCases is the red-proof anchor: guest_loss / absent / nil-recipe yield
+// NO plan (execute-nothing on the wrong mode). Relaxing the mode gate → the guest_loss case fails.
+func TestBuildRestorePlan_NoPlanCases(t *testing.T) {
+	if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "guest_loss", VMID: 9201}, sampleRecipe(), "local-lvm"); ok {
+		t.Error("guest_loss must NOT yield a host-loss plan")
+	}
+	if _, ok := BuildRestorePlan(nil, sampleRecipe(), "local-lvm"); ok {
+		t.Error("absent directive must NOT yield a plan")
+	}
+	if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "host_loss"}, nil, "local-lvm"); ok {
+		t.Error("nil recipe must NOT yield a plan")
+	}
+}
+
+// TestConsumer_SurfacesPlanNeverExecutes: the consumer surfaces the plan on host_loss, consults the
+// recipe only then, and clears it otherwise. It has NO restore/destroy dependency (execute-nothing
+// is structural — the type literally cannot call a restore).
+func TestConsumer_SurfacesPlanNeverExecutes(t *testing.T) {
+	recipeCalls := 0
+	c := NewConsumer(func(context.Context) *hub.DRRecipeHostHalf { recipeCalls++; return sampleRecipe() }, "local-lvm", nil)
+	ds := func(d *hub.WireRestoreDirective) *hub.DesiredStateResponse {
+		return &hub.DesiredStateResponse{DesiredState: hub.WireDesiredState{RestoreDirective: d}}
+	}
+
+	// non-host_loss → no plan, recipe NOT consulted.
+	c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "guest_loss"}))
+	if c.LastPlan() != nil {
+		t.Error("guest_loss set a plan")
+	}
+	if recipeCalls != 0 {
+		t.Errorf("recipe consulted on a non-host_loss directive (%d calls)", recipeCalls)
+	}
+	// host_loss → plan surfaced, recipe consulted once.
+	c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201}))
+	p := c.LastPlan()
+	if p == nil || len(p.Guests) != 1 || p.Guests[0].VMID != 9201 {
+		t.Fatalf("host_loss plan = %+v", p)
+	}
+	if recipeCalls != 1 {
+		t.Errorf("recipe calls = %d, want 1", recipeCalls)
+	}
+	// absent directive clears the plan.
+	c.OnDesiredState(context.Background(), ds(nil))
+	if c.LastPlan() != nil {
+		t.Error("absent directive did not clear the plan")
+	}
+}

internal/escrow/ceremony.go

@@ -0,0 +1,49 @@
+package escrow
+
+// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
+// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
+// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
+// matches that vector byte-for-byte (spike §2.2: any alteration — value, extra flag, order,
+// config path — is refused), so the argv below is the SINGLE SOURCE OF TRUTH shared by the
+// exec (localapi), the capability manifest entry, and (byte-identically) the FELHOM_ESCROW
+// sudoers line. Never build it with flag helpers and never normalize `--` to `-` — Go's flag
+// package would accept either spelling, sudoers only the literal one.
+
+// CeremonyBinary is the installed agent binary path the sudoers line pins.
+const CeremonyBinary = "/usr/local/bin/felhom-agent"
+
+// ceremonyArgv is the fixed vector. --config is pinned explicitly: `sudo -n` env_reset strips
+// FELHOM_AGENT_CONFIG, and the pin closes env-injection of an alternate config (spike probe (e)).
+var ceremonyArgv = []string{
+	"--config", "/etc/felhom-agent/agent.json",
+	"--selftest=escrow-create",
+	"--upload",
+	"--output=json",
+}
+
+// CeremonyArgs returns a fresh copy of the fixed argv (callers must not be able to mutate the
+// shared source).
+func CeremonyArgs() []string {
+	out := make([]string, len(ceremonyArgv))
+	copy(out, ceremonyArgv)
+	return out
+}
+
+// CeremonyOutput is the --output=json wire contract (version 1): the ONE JSON object json mode
+// emits on stdout — nothing else lands there; every human/info line goes to stderr. RecoveryCode
+// is the only secret field: the consumer must extract it, hand it to the one-shot claim holder,
+// and zero both the parsed struct and the raw stdout buffer. (Best-effort — Go's GC may hold
+// stale copies; the discipline still shrinks the exposure window.)
+type CeremonyOutput struct {
+	Version           int     `json:"version"`
+	RecoveryCode      string  `json:"recovery_code"`
+	KeyFingerprint    string  `json:"key_fingerprint"`
+	EntropyBits       float64 `json:"entropy_bits"`
+	BlobBytes         int     `json:"blob_bytes"`
+	IdentityBlobBytes int     `json:"identity_blob_bytes"`
+	ResticPwSealed    bool    `json:"restic_pw_sealed"`
+	Uploaded          bool    `json:"uploaded"`
+}
+
+// CeremonyOutputVersion is the current CeremonyOutput.Version value.
+const CeremonyOutputVersion = 1

internal/escrow/consume.go

@@ -0,0 +1,146 @@
+package escrow
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// Consume is the slice-10C production escrow-consumption path — the throwaway 10C spike harness
+// turned into a real, tested function. It recovers the PBS client encryption key `K` from an
+// R-wrapped escrow `blob`, GATES it on the expected key fingerprint, and installs it at `keyDest`
+// for the PBS restore path. It is the inverse of Create: Create wraps `K` under `R`; Consume
+// unwraps it back. The DR orchestration around it (re-enroll in restore mode, source the directive
+// from the hub, decide which guests to restore) is slice 10D — Consume takes its four inputs as
+// PARAMETERS so it stays standalone-testable (exactly as the spike harness was).
+//
+// The four inputs (the consumption contract — spike findings):
+//   - blob:                the opaque R-wrapped escrow blob (hub-served restore directive in 10D).
+//   - recoveryCode (R):    BY HAND from the customer — NEVER hub-sourced (zero-knowledge holds: a
+//                          hub compromise alone cannot decrypt). A secret: never logged/persisted.
+//   - expectedFingerprint: hub-served — the gate target (F-C4).
+//   - keyDest:             where the PBS restore reads the key (`--keyfile` or the default
+//                          `$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json`).
+//
+// Order (spike F-C2/F-C3/F-C4/F-C6): Unwrap → fingerprint-gate → install. On ANY failure there is
+// NO partial install (nothing at keyDest) and no key material leaks: the recovered key lives only
+// in a 0600 tempdir that is always removed; `R` and key bytes are never logged (only fingerprint
+// prefixes); the input `blob` is read-only, so a failed Consume is RETRYABLE.
+func Consume(ctx context.Context, blob []byte, recoveryCode, expectedFingerprint, keyDest string) error {
+	if len(blob) == 0 {
+		return fmt.Errorf("escrow: Consume needs a non-empty blob")
+	}
+	if recoveryCode == "" {
+		return fmt.Errorf("escrow: Consume needs the recovery code (R)")
+	}
+	if expectedFingerprint == "" {
+		// The fingerprint gate is mandatory — without it we'd install whatever a (wrong-but-valid)
+		// unwrap produced. There is no "skip the gate" path.
+		return fmt.Errorf("escrow: Consume needs the expected key fingerprint (the gate target)")
+	}
+	if keyDest == "" {
+		return fmt.Errorf("escrow: Consume needs a key destination path")
+	}
+
+	// Work in a private 0700 tempdir; RemoveAll always runs, so the recovered key never lingers.
+	work, err := os.MkdirTemp("", "felhom-consume-")
+	if err != nil {
+		return fmt.Errorf("escrow: tempdir: %w", err)
+	}
+	defer os.RemoveAll(work)
+	recovered := filepath.Join(work, "recovered.key")
+
+	// Operate on a COPY of the blob (F-C6: the input blob is read-only → a failed Consume is
+	// retryable). Unwrap re-keys this copy in place.
+	if err := os.WriteFile(recovered, blob, 0o600); err != nil {
+		return fmt.Errorf("escrow: staging blob: %w", err)
+	}
+
+	// 1. Unwrap with R. A WRONG R fails closed at the scrypt KDF (F-C3): nonzero exit, no key
+	// emitted. Surface a clear, R-free error; nothing is written to keyDest.
+	if err := Unwrap(ctx, recovered, recoveryCode); err != nil {
+		return fmt.Errorf("escrow: the recovery code did not unwrap the escrow (wrong recovery code, or a corrupt blob): %w", err)
+	}
+
+	// 2. Fingerprint gate (F-C4) — the cheap correctness check, BEFORE any install or multi-GB
+	// restore. A mismatch means a wrong escrow/datastore (or a recovery code that, against the odds,
+	// produced a different key): fail fast + loud, no install.
+	recFP, err := KeyFingerprint(ctx, recovered)
+	if err != nil {
+		return fmt.Errorf("escrow: reading recovered key fingerprint: %w", err)
+	}
+	if !fingerprintsEqual(recFP, expectedFingerprint) {
+		return fmt.Errorf("escrow: recovered key fingerprint %s != expected %s — wrong escrow/datastore (not installing)",
+			shortFP(recFP), shortFP(expectedFingerprint))
+	}
+
+	// 3. Install atomically at keyDest (0600) — the only thing Consume writes outside its tempdir.
+	// A failure here leaves nothing partial at keyDest (the temp sibling is removed).
+	if err := installKey(recovered, keyDest); err != nil {
+		return fmt.Errorf("escrow: installing recovered key: %w", err)
+	}
+	return nil
+}
+
+// installKey atomically places the recovered key at dest with 0600: write a sibling temp on the
+// SAME filesystem, fsync, then rename (atomic). The parent dir is created 0700 if missing. On any
+// error the temp is removed so dest is never left partial.
+func installKey(recovered, dest string) error {
+	if err := os.MkdirAll(filepath.Dir(dest), 0o700); err != nil {
+		return fmt.Errorf("key dir: %w", err)
+	}
+	b, err := os.ReadFile(recovered)
+	if err != nil {
+		return err
+	}
+	var rnd [8]byte
+	_, _ = rand.Read(rnd[:])
+	tmp := dest + ".tmp-" + hex.EncodeToString(rnd[:])
+	f, err := os.OpenFile(tmp, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0o600)
+	if err != nil {
+		return err
+	}
+	if _, err := f.Write(b); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := f.Sync(); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := f.Close(); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	if err := os.Rename(tmp, dest); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return nil
+}
+
+// fingerprintsEqual compares two PBS key fingerprints tolerant of formatting (case + ':'/' '
+// separators) — the hub-stored value and `key show`'s output may differ only cosmetically.
+func fingerprintsEqual(a, b string) bool {
+	return normFP(a) == normFP(b) && normFP(a) != ""
+}
+
+func normFP(s string) string {
+	return strings.ToLower(strings.NewReplacer(":", "", " ", "", "\t", "", "\n", "").Replace(strings.TrimSpace(s)))
+}
+
+// shortFP returns a log-safe truncated fingerprint prefix (a key fingerprint is an identifier, not
+// the key — but we still show only a prefix in logs/errors).
+func shortFP(s string) string {
+	s = strings.TrimSpace(s)
+	if len(s) > 23 { // ~ first 8 colon-separated octets
+		return s[:23] + "…"
+	}
+	return s
+}

internal/escrow/consume_test.go

@@ -0,0 +1,186 @@
+package escrow
+
+import (
+	"bytes"
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"testing"
+)
+
+// ---- pure-unit tests (no proxmox-backup-client) ----------------------------------------------
+
+func TestConsume_InputValidation(t *testing.T) {
+	ctx := context.Background()
+	dest := filepath.Join(t.TempDir(), "k.json")
+	cases := []struct {
+		name       string
+		blob       []byte
+		r, fp, dst string
+	}{
+		{"empty blob", nil, "R", "fp", dest},
+		{"empty R", []byte("x"), "", "fp", dest},
+		{"empty fingerprint", []byte("x"), "R", "", dest},
+		{"empty dest", []byte("x"), "R", "fp", ""},
+	}
+	for _, c := range cases {
+		if err := Consume(ctx, c.blob, c.r, c.fp, c.dst); err == nil {
+			t.Errorf("%s: expected an error", c.name)
+		}
+	}
+	// no key written on a validation failure
+	if _, err := os.Stat(dest); !os.IsNotExist(err) {
+		t.Error("a validation failure left a key behind")
+	}
+}
+
+func TestFingerprintsEqual_NormalizesFormatting(t *testing.T) {
+	if !fingerprintsEqual("01:36:E9:FE", "0136e9fe") {
+		t.Error("case/colon-insensitive compare failed")
+	}
+	if fingerprintsEqual("", "") {
+		t.Error("two empty fingerprints must NOT compare equal (no gate-bypass on empty)")
+	}
+	if fingerprintsEqual("01:36", "ff:ee") {
+		t.Error("distinct fingerprints compared equal")
+	}
+}
+
+func TestInstallKey_AtomicAnd0600(t *testing.T) {
+	dir := t.TempDir()
+	src := filepath.Join(dir, "src")
+	if err := os.WriteFile(src, []byte("keymaterial"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	dest := filepath.Join(dir, "sub", "installed.json") // parent dir must be created
+	if err := installKey(src, dest); err != nil {
+		t.Fatalf("installKey: %v", err)
+	}
+	b, _ := os.ReadFile(dest)
+	if string(b) != "keymaterial" {
+		t.Errorf("installed content = %q", b)
+	}
+	if runtime.GOOS != "windows" { // Windows does not enforce Unix file modes
+		if info, _ := os.Stat(dest); info.Mode().Perm() != 0o600 {
+			t.Errorf("installed key mode = %v, want 0600", info.Mode().Perm())
+		}
+	}
+	// no .tmp- siblings left behind
+	entries, _ := os.ReadDir(filepath.Dir(dest))
+	for _, e := range entries {
+		if filepath.Ext(e.Name()) != ".json" && len(e.Name()) > 4 && e.Name()[:4] == "inst" {
+			t.Errorf("temp sibling left behind: %s", e.Name())
+		}
+	}
+}
+
+// ---- integration tests (real proxmox-backup-client wrap/unwrap round-trip) --------------------
+
+func ensurePbc(t *testing.T) {
+	t.Helper()
+	if !pbcAvailable() {
+		t.Skip("skipping: proxmox-backup-client + linux required (runs on the demo/build host)")
+	}
+	if _, err := os.Stat(pbcBinary); err != nil {
+		if p, e := exec.LookPath("proxmox-backup-client"); e == nil {
+			pbcBinary = p
+		}
+	}
+}
+
+// makeBlob creates a throwaway kdf=none key Kt, wraps it under R, and returns (blobBytes, Kt-fp).
+func makeBlob(t *testing.T, dir, R string) ([]byte, string) {
+	t.Helper()
+	ctx := context.Background()
+	Kt := filepath.Join(dir, "Kt.json")
+	if out, err := exec.Command(pbcBinary, "key", "create", Kt, "--kdf", "none").CombinedOutput(); err != nil {
+		t.Fatalf("key create: %v: %s", err, out)
+	}
+	fp, err := KeyFingerprint(ctx, Kt)
+	if err != nil {
+		t.Fatalf("fingerprint: %v", err)
+	}
+	blobPath := filepath.Join(dir, "escrow.blob")
+	if err := Wrap(ctx, Kt, blobPath, R); err != nil {
+		t.Fatalf("Wrap: %v", err)
+	}
+	blob, err := os.ReadFile(blobPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return blob, fp
+}
+
+// VALID: Consume installs a key whose fingerprint matches the expected; blob is byte-unchanged.
+func TestConsume_ValidInstallsGatedKey(t *testing.T) {
+	ensurePbc(t)
+	ctx := context.Background()
+	dir := t.TempDir()
+	const R = "throwaway-correct-horse-battery-staple"
+	blob, wantFP := makeBlob(t, dir, R)
+	blobBefore := append([]byte(nil), blob...)
+
+	dest := filepath.Join(dir, "freshbox", "encryption-key.json")
+	if err := Consume(ctx, blob, R, wantFP, dest); err != nil {
+		t.Fatalf("Consume(valid): %v", err)
+	}
+	// key installed + fingerprint matches the expected gate target.
+	gotFP, err := KeyFingerprint(ctx, dest)
+	if err != nil {
+		t.Fatalf("fingerprint(dest): %v", err)
+	}
+	if !fingerprintsEqual(gotFP, wantFP) {
+		t.Errorf("installed key fingerprint %q != expected %q", gotFP, wantFP)
+	}
+	// 0600.
+	if info, _ := os.Stat(dest); info.Mode().Perm() != 0o600 {
+		t.Errorf("installed key mode = %v, want 0600", info.Mode().Perm())
+	}
+	// blob read-only / unchanged (retryable).
+	if !bytes.Equal(blob, blobBefore) {
+		t.Error("Consume mutated the input blob — must be read-only/retryable")
+	}
+}
+
+// WRONG R: Unwrap fails closed → clear error, NO file at dest, blob unchanged.
+func TestConsume_WrongRNoInstall(t *testing.T) {
+	ensurePbc(t)
+	ctx := context.Background()
+	dir := t.TempDir()
+	blob, wantFP := makeBlob(t, dir, "the-correct-code")
+	blobBefore := append([]byte(nil), blob...)
+
+	dest := filepath.Join(dir, "freshbox", "encryption-key.json")
+	err := Consume(ctx, blob, "DEFINITELY-the-wrong-code", wantFP, dest)
+	if err == nil {
+		t.Fatal("a wrong recovery code must fail")
+	}
+	if _, statErr := os.Stat(dest); !os.IsNotExist(statErr) {
+		t.Errorf("a wrong-R Consume left a key at dest (%v) — must install nothing", statErr)
+	}
+	if !bytes.Equal(blob, blobBefore) {
+		t.Error("wrong-R Consume mutated the blob — must be retryable")
+	}
+}
+
+// FINGERPRINT MISMATCH: right unwrap, wrong expected fingerprint → fail fast, NO install (proves
+// the gate runs BEFORE any restore).
+func TestConsume_FingerprintMismatchNoInstall(t *testing.T) {
+	ensurePbc(t)
+	ctx := context.Background()
+	dir := t.TempDir()
+	const R = "the-correct-code"
+	blob, _ := makeBlob(t, dir, R)
+
+	dest := filepath.Join(dir, "freshbox", "encryption-key.json")
+	// Right R, but an expected fingerprint for a DIFFERENT datastore/key.
+	err := Consume(ctx, blob, R, "ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff", dest)
+	if err == nil {
+		t.Fatal("a fingerprint mismatch must fail")
+	}
+	if _, statErr := os.Stat(dest); !os.IsNotExist(statErr) {
+		t.Errorf("a fingerprint-mismatch Consume left a key at dest — the gate must run before install")
+	}
+}