v0.126.0: a fetch failure is not a wrong recovery code (R-224)

A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.

The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.

escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.

A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.

Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.

29 packages ok, vet clean, agent gates OK.

.gitea/workflows/gates.yml

@@ -0,0 +1,116 @@
+# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
+# pushed or what they typed.
+#
+#   *** THIS REPORTS. IT CANNOT REFUSE. ***
+#
+# felhom repos push straight to `main` with no pull request, so there is no merge for a status
+# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
+# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
+# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
+#
+# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
+# the runner is a host-mode container with python3 and git and nothing else (see
+# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
+# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
+#
+# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
+# layer up. That is the last step, and it runs ONLY on failure.
+name: gates
+on: [push]
+
+jobs:
+  gates:
+    runs-on: felhom-gates
+    steps:
+      - name: Fetch the pushed commit and the sibling clone it needs
+        # This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
+        # door and is deliberately never copied here — so CI has to reproduce the workspace's
+        # sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
+        # needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
+        run: |
+          # Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
+          # which can move under us if two pushes race.
+          mkdir -p ws/felhom-agent
+          cd ws/felhom-agent
+          git init -q .
+          git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
+          git fetch -q --depth 1 origin "$GITHUB_SHA"
+          git checkout -q FETCH_HEAD
+          echo "checked out $(git rev-parse HEAD)"
+          cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
+          echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
+
+      - name: Run the gate entry point
+        # The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
+        # exit code IS the result: no `|| true`, no pipe that could swallow it.
+        #
+        # THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
+        # container runtime" and exists for `.githooks/pre-push`, where a push must not fail
+        # because Gitea blinked or because someone is on a train. CI is the opposite machine: it
+        # has the network, it is not in anyone's way, and it is the half that emails. The
+        # published-versions gate — the R-115 mechanism, which asks Gitea whether a released
+        # version can actually be downloaded — is network-bound and therefore runs ONLY here.
+        # Leaving `--fast` in place would have registered that gate and never run it, which is the
+        # built-but-never-wired failure this project has shipped four times.
+        env:
+          # In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
+          GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
+        run: cd ws/felhom-agent && python3 scripts/agent_gates.py
+
+      - name: Alarm on failure
+        # THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
+        # notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
+        # is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
+        # existing transactional path (Resend, the same one the hub uses), and prints the provider's
+        # accepted id so "a message left the machine" is an observable, not an assumption.
+        #
+        # Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
+        # else on purpose, and the first version of this step died on `curl: command not found`.
+        # Reaching for a bigger image to send one HTTP request would have been the wrong trade.
+        if: failure()
+        env:
+          RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
+        run: |
+          python3 - <<'PY'
+          import json, os, sys, urllib.request, urllib.error
+
+          key = os.environ.get("RESEND_API_KEY", "")
+          if not key:
+              sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
+                       "silent alarm is worse than none. Set the user-level Actions secret.")
+
+          repo = os.environ.get("GITHUB_REPOSITORY", "?")
+          sha  = os.environ.get("GITHUB_SHA", "?")
+          run  = os.environ.get("GITHUB_RUN_NUMBER", "?")
+          srv  = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
+
+          body = json.dumps({
+              "from": "Felhom CI <monitoring@felhom.eu>",
+              "to": ["admin@felhom.eu"],
+              "subject": "[felhom CI] gates FAILED in %s" % repo,
+              "text": (
+                  "The gate entry point exited non-zero.\n\n"
+                  "Repository : %s\n"
+                  "Commit     : %s\n"
+                  "Run        : %s/%s/actions/runs/%s\n\n"
+                  "The failing gate names itself in the run log.\n\n"
+                  "If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
+                  "disagree - that is a finding about the gates themselves, not about CI, and it\n"
+                  "outranks whatever the push was for.\n"
+              ) % (repo, sha, srv, repo, run),
+          }).encode()
+
+          req = urllib.request.Request(
+              "https://api.resend.com/emails", data=body, method="POST",
+              headers={"Authorization": "Bearer %s" % key,
+                       "Content-Type": "application/json",
+                       # Cloudflare fronts api.resend.com and BLOCKS the default
+                       # "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
+                       # exactly like an auth failure and is not one. Measured 2026-08-02.
+                       "User-Agent": "felhom-ci/1.0"})
+          try:
+              with urllib.request.urlopen(req, timeout=30) as r:
+                  print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
+          except urllib.error.HTTPError as e:
+              sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
+          PY

.githooks/pre-push

@@ -0,0 +1,82 @@
+#!/bin/sh
+# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
+#
+# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
+# container runtime, so a push stays a push and never pulls images or starts containers. The slow
+# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
+# the bypass becomes the habit.
+#
+# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
+# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
+#
+# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
+#   * per-clone   — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
+#                     git config core.hooksPath .githooks
+#                   Any manual entry-point run WARNS when the clone is unarmed.
+#   * skippable   — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
+#                   hatch that cannot be reached is one that gets removed the first time it is
+#                   inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
+# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
+#
+# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
+# is the repo root whether `git push` is issued from the root or from any subdirectory. The
+# explicit rev-parse below does not depend on that.
+set -u
+
+root=$(git rev-parse --show-toplevel 2>/dev/null) || {
+    echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
+    exit 1
+}
+cd "$root" || exit 1
+
+# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
+# Refuse a push from a clone outside the felhom workspace.
+#
+# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
+# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
+# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
+# fixed by writing it down again — it has to be asserted where it can bite.
+#
+# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
+# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
+#
+# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
+# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
+# failed — this hook must not brick a legitimate clone on a different host.
+#
+# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
+FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
+if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
+    ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
+    root_real=$(pwd -P) || root_real=""
+    if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
+        case "$root_real/" in
+            "$ws_real"/*) : ;;  # inside the workspace — proceed
+            *)
+                echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
+                echo "          clone:    $root_real" >&2
+                echo "          expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
+                echo "          Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
+                echo "          and state that you did in the session report." >&2
+                exit 1
+                ;;
+        esac
+    fi
+fi
+
+if ! command -v python3 >/dev/null 2>&1; then
+    echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
+    echo "          pass by default. Install python3, or push with --no-verify and say so." >&2
+    exit 1
+fi
+
+echo "pre-push [felhom-agent]: running scripts/agent_gates.py --fast ..."
+python3 "scripts/agent_gates.py" --fast
+rc=$?
+if [ "$rc" -ne 0 ]; then
+    echo "pre-push [felhom-agent]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
+    echo "          'git push --no-verify' and state that you did in the session report." >&2
+else
+    echo "pre-push [felhom-agent]: gates OK - push proceeding."
+fi
+exit $rc

.gitignore

@@ -0,0 +1,11 @@
+# build output
+/felhom-agent
+/felhom-agent.exe
+/dist/
+
+# local config that may carry a real token secret
+/agent.json
+*.local.json
+
+# go
+/vendor/

AGENT-001-FIX-NOTES.md

@@ -0,0 +1,52 @@
+# AGENT-001 fix — inline customer-confirmed wipe: anti-retarget re-resolution
+
+**Branch:** `fix/agent-001-wipe-durable-reresolve` (off `main` @ d17b5ab) — **PENDING REVIEW, NOT deployed.**
+**Status:** branch-only. No agent build, no deploy, no golden rebake, no re-provision. That is reserved for a supervised session (this fix touches the destructive storage-wipe path).
+
+## The bug
+
+`internal/localapi/disks.go` `handleDiskFormat`, the data-bearing **customer-confirmed** branch
+(`dec.Allowed`), inspected and gate-bound the device by its durable id but then called
+`s.disks.Format(ctx, req.Device, …)` on the **caller-supplied mutable `/dev` path**. The durable id
+only bound the *confirmation*; it was never re-resolved to the device just before `mkfs`. If a USB
+re-enumeration reassigned `/dev/sdbN` to a **different physical disk** between inspection and `mkfs`
+(the classify→mkfs TOCTOU), the wipe hit the wrong drive — data loss on an unintended disk.
+
+The signed-jobs path (`internal/signedjobs/wipe.go` `WipeExecutor.Execute`) already does this
+correctly: resolve durable→device, re-derive the device's durable id and require an exact match,
+re-inspect, then format the *re-resolved* device. The inline path did not.
+
+## What changed
+
+- **New `internal/localapi/wipe_reresolve.go`:**
+  - `antiRetargetResolve(durableID, resolve, derive, inspect)` — pure, injected-deps helper mirroring
+    `WipeExecutor`: refuse empty durable id; resolve→current device; re-derive and require exact
+    match (a `/dev` node now pointing at a different disk derives a different id → refuse); re-inspect
+    and require still-`DataBearing()`. Returns the **re-resolved device** to format.
+  - `(*Server).reresolveDurableForWipe` wires it with the real `storage.ResolveDurableDevice` /
+    `storage.DeviceDurableID` / `s.disks.InspectDevice`.
+- **`internal/localapi/disks.go`:** the `dec.Allowed` branch now calls `s.reresolveWipe(ctx, deviceDurable)`
+  and formats the **returned** device (never `req.Device`). On any refusal it returns `409 Conflict`
+  with the reason and does **not** call `mkfs`. The response/log now reference the re-resolved device.
+- **`internal/localapi/server.go`:** new injectable field `reresolveWipe`, defaulted in `NewServer` to
+  `s.reresolveDurableForWipe`. Production behaviour unchanged (same real functions); the seam exists so
+  the inline path is unit-testable without real `/dev/disk/by-*`.
+- **Tests:** `wipe_reresolve_test.go` covers `antiRetargetResolve` directly — happy path, empty durable,
+  no-longer-resolves, **durable-id mismatch (the core retarget case)**, no-longer-data-bearing,
+  re-inspect error. The existing integration test `TestFormat_DataBearing_UserDataConfirmed_Formats`
+  stubs `reresolveWipe` (the test helper sets it). Full `go build/vet/test ./...` green at the branch HEAD.
+
+## Why it can't break legitimate wipes
+
+`storage.DeviceDurableID` produces `byid:`/`byuuid:` ids and `storage.ResolveDurableDevice` accepts
+exactly those — they round-trip. A normal user-data drive resolves back to itself, re-derives the same
+id, and inspects data-bearing → format proceeds. Only three new refusals occur, all correct:
+durable id gone, durable-id mismatch (retarget), or no-longer-data-bearing.
+
+## Review checklist for the supervised merge+deploy+golden-rebake session
+
+1. Confirm `antiRetargetResolve` refuses on mismatch/gone/blank/empty (run `go test ./internal/localapi/ -run AntiRetarget -v`).
+2. Sanity-check `DeviceDurableID`↔`ResolveDurableDevice` scheme round-trip on the real demo USB (`felhom-usb`) so a genuine customer wipe still resolves+matches (read-only `DeviceDurableID` on the live device; do NOT wipe).
+3. Confirm the `409 Conflict` refusal surfaces sensibly in the controller UI (controller `agentapi` maps the format response — verify it doesn't treat 409 as a hard error that hides the reason).
+4. Bump agent version + CHANGELOG, build, golden-rebake, re-provision per the agent deploy runbook.
+5. Live-validate one happy customer-confirmed wipe on a demo scratch drive (re-resolve matches → mkfs) and confirm the audit log line shows the re-resolved device == the confirmed durable id.

CLAUDE.md

@@ -0,0 +1,216 @@
+# CLAUDE.md — `felhom-agent`
+
+> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
+> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
+> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
+
+## What this repo is
+
+`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
+Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
+control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
+
+- Renamed former `proxmox-controller` repo.
+- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
+  creds). Do not confuse them.
+- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
+
+## Read before writing code
+
+- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
+  in the same commit that changes a shared helper or pattern.
+- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
+- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
+  `felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
+
+## Layout (verified against the tree)
+
+```
+cmd/felhom-agent/    main + flags + --selftest modes + the daemon entry
+cmd/felhom-opsign/   offline operator signing CLI (SSHSIG)
+internal/authz/      operator signed-op verifier (SSHSIG) + durable FileNonceStore
+internal/backup/     vzdump backup runner + restore-test scheduler + report store
+internal/capability/ live sudo-policy capability probe (degradation visibility)
+internal/config/     JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
+internal/desired/    hub desired-state syncer (envelope observer)
+internal/escrow/     PBS-key escrow (zero-knowledge recovery code)
+internal/guesthook/  pre-start self-heal hookscript install
+internal/hub/        daemon: HostReport collector + Bearer client + resilient Loop
+internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
+internal/localapi/   per-guest local API: token store, disks/format, guest binds, controller swap,
+                     stale-lock recovery, pinned self-signed leaf
+internal/log/        slog setup
+internal/pbs/        PBS-API client (fingerprint-pinned) + verify maintenance loop
+internal/provision/  guest bootstrap back-half (token mint → bootstrap.json → pct bind)
+internal/proxmox/    API-first Client + fenced root-CLI Privileged + UPID WaitTask
+internal/reconcile/  reconcile engine + reversibility gate + op journal + crash recovery
+internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
+internal/storage/    storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
+```
+
+## Build / run
+
+- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
+- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
+  DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
+  host — build and run live tests locally.
+- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
+- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
+
+> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
+> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
+> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
+> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
+> elsewhere).
+
+> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
+> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
+> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
+> and undownloadable, where a documented-path reinstall would have silently downgraded them while
+> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
+> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
+> verifies by an **independent download** rather than trusting the publish step's own output.
+> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
+> than reimplementing it.
+>
+> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
+> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
+> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
+> package makes the published-versions gate correctly fail a correct release — it did, on roughly
+> every second release, and R-168 sends that failure to you by mail. The invariant the old order
+> protected is asserted directly instead: the gate now also refuses a **published version with no
+> tag**. If the push fails after a successful publish the script says so loudly and prints the
+> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
+>
+> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
+> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
+> before this, a rebuild could not reproduce the sha you were vouching. To check any published
+> version yourself:
+>
+> ```bash
+> V=0.122.0
+> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
+>     -o /tmp/felhom-agent-check ./cmd/felhom-agent
+> sha256sum /tmp/felhom-agent-check
+> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
+> ```
+>
+> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
+> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
+> ever changes, change both or one version name means two binaries again.
+
+| Step | Where | One-liner |
+|---|---|---|
+| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
+| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
+| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
+| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
+| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
+| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
+| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
+
+## Proxmox model (the load-bearing rules)
+
+- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
+  keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
+  `Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
+- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
+  the POST is **not** success; authorization can fail at task execution.
+- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
+- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
+  AND the token's ACLs — the role must be granted on **both**, or every call 403s.
+- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
+  `Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
+
+## Demo host (for live tests)
+
+Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
+available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
+`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
+`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
+is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
+historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
+**Which box is safe to break, and what may be done to each:
+`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
+nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
+fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
+demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
+
+> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
+> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
+> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
+> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
+> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
+> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
+> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
+> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
+> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
+>
+> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
+> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
+> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
+> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
+> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
+> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
+
+> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
+> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
+> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
+> (`cygpath -w` for the local scp path; CRLF hazard on config files).
+
+## Conventions
+
+### Trunk-based — no branches
+
+All shippable work commits **directly to `main`**; `main` equals what is deployed.
+- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
+- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
+- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
+
+> **In every repository where you make a change, update both files in that repo:**
+> - **`CHANGELOG.md`** — cumulative log, newest on top.
+> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
+>
+> **Never write secrets** into any committed file — reference them as "stored out-of-band".
+
+- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
+  guess** when you'd otherwise invent input/output.
+- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
+  even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
+  measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
+  checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
+- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
+- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
+  the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
+  repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
+  check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
+  this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
+  by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
+  network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
+  skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
+  — a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
+  **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
+  is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
+  WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
+  session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
+- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
+- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
+  (remote pulls) — English, keys-never-values, durations on outcomes; full rules in
+  `felhom.eu/documentation/runbooks/logging-conventions.md`.
+
+### Live validation
+
+Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
+BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
+invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
+mechanism tests where the direct call IS the mechanism are exempt.
+
+## Workflow & artifacts
+
+- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
+  CHANGELOG + REPORT.md.
+- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
+  for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
+  felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
+  decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
+  real customer data — demo scratch guests are fair game.

CONTEXT.md

@@ -0,0 +1,504 @@
+# CONTEXT — felhom-agent working state
+
+> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
+> entry = current); the end-of-task detail lives in `REPORT.md`.
+
+## R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked
+
+`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
+customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
+(`hub.Client.FetchIdentityEscrow` → `GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
+per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
+repository password plus its sha256.
+
+**Rules that must not erode:**
+- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
+  trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
+- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
+  no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
+- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
+  never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
+  log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
+- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
+  the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
+  at age's KDF, nothing written).
+- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
+  `main` → `runDaemon` → `buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
+  `Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
+  self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
+  Links 6 and 7 were two of this project's six built-but-never-wired instances.
+
+**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
+A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
+recovered password, reopens a repository, or restores a file.
+
+**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
+"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
+into the same bundle — anyone reading it would conclude the password was not there. It now names what
+THIS bundle carried and what it did not.
+
+## Current
+
+- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
+  `FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
+  storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
+  through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
+  archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
+  restore-testable and nothing said so.
+  - **The permission question is asked directly**, because unlike the listing it has a definite
+    answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
+    own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
+    composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
+    belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
+    degraded alert applies with **no hub change**.
+  - **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
+    403 — it carries the privileges INHERITED from the box-wide `/` grant
+    (`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
+    blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
+    changing that constant (`storeGrantRequiredPriv`, red-proved).
+  - **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
+    defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
+    does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
+    ok when it could not ask.
+  - **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
+    grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
+    and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
+    correctly picking the 08-02 archive (08-03 had not settled 24 h).
+  - **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
+    grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.
+
+- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
+  None touches data; all three cost attention, which every other signal depends on.
+  - **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
+    from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
+    timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
+    a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
+    twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
+    still read, and a record missing archive-or-tier is NOT reported), exposes
+    `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
+    newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
+    twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
+    "host-report gauge" in its doc comment and had **no caller** for weeks.
+  - **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
+    future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
+    tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
+  - **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
+    locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
+    made the gate correctly fail a correct release ~half the time. The old order's invariant is now
+    asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
+    a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
+  - **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
+    bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
+    `CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
+    identical. The verification command is in `CLAUDE.md`.
+
+- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
+  now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
+  `settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
+  weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
+  literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
+  tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
+  where it matters most. Red-proved at 0 runs over 5 simulated days.
+  - **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
+    keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
+    upgrade, deliberately.
+  - **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
+    (verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
+    naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
+    (default 24 h). It is NOT carried into the evaluation interval.
+  - **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
+    both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
+    evaluation interval is also its retry interval for a multi-GB restore.
+  - The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
+    able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
+    failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
+  - New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
+  - **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
+    DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
+    a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
+    evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
+    with its error in that same line.
+  - **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
+    encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
+    named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
+  - **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
+    `/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
+    host tier has never been restore-testable there, and the due-check cannot distinguish that from
+    a newborn tier.
+
+- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
+  New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
+  stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
+  `RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
+  lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
+  down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
+  invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
+  guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
+  instead of keeping a second private definition of "should be running". Retry bounded at 3
+  (1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
+  Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
+  proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
+
+- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
+  structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
+  restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
+  **(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
+  VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
+  **(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
+  PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
+  `/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
+  counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
+  `Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
+  `pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
+  (`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
+  consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
+  privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
+  binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
+  PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
+
+- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
+  self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
+  its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
+  F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
+  `sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
+  ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
+
+- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
+  `NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
+  backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
+  the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
+  deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
+  measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
+  **Size is the only tier-agnostic discriminator** — `verification` and `encrypted` are absent on
+  every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
+  either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
+  smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
+  test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
+  demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
+  observable. Deployed on both boxes. Detail: `REPORT.md`.
+  **Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
+  2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
+  accumulate. Filed as R-99 (LOW).
+
+- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
+  drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
+  `/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
+  `/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
+  **SMART-only** device from its containing filesystem (never touches backing/durable_id — the
+  removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
+  watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
+  `felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
+- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
+  `Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
+  enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
+  controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
+  ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
+  so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
+- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
+  Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
+  (`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
+  clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
+  ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
+  **The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
+  static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
+
+- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
+  felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
+  host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
+  `bind: cannot assign requested address` — the daemon exits ~1.1 s after start, systemd gave up after
+  4 retries, and a manual restart reproduces it exactly. Deployed binary is **0.90.0**. Fix needs
+  `listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest bootstrap endpoint (plus the
+  pinned leaf's SAN) → **Viktor GO**; re-pinning to another literal just re-breaks on the next lease.
+  Also re-observed each start: `pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw:
+  permission denied` (R-39-adjacent). Evidence + ranked findings:
+  `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
+
+- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
+  demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
+  v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)** —
+  self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
+  every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and
+  applies via PVE `SetConfig` — **live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
+  break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
+  after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
+  memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
+  (esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
+  trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
+  Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
+  `reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
+  (3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
+  pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
+  hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
+  agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.
+
+- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
+  Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)** —
+  closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
+  the root wrapper + re-reads instead of aborting before the grant (the demo's `felhom-offsite`
+  case). (2) **escrow config live-reload** — `/escrow/preflight`'s `pbs_storage_id` row now reads
+  the live agent.json (late-bound `CurrentPBSStorageID`) so a pbsdr-seeded id flips green with no
+  restart. (3) **agent-plane poke listener (Direction-2a)** — `internal/poke`: contentless UDP poke
+  bound to the box WG /32 (port **51822**), leading-edge debounced, fires the hub-loop out-of-band
+  trigger for an immediate desired-state cycle; enabled with `wg_tunnel.enabled`; first slice of
+  R-13. Red-proofs for all three (run-fail-revert). **ALL THREE LIVE LEGS PROVEN on the demo
+  (2026-07-17), demo now LIVE on 0.89.0:** Scenario 4 floor-driven A/B train 0.88→0.89 (operator
+  signed+enqueued the `agent_update` op — the vouch+floor alone does NOT trigger it; committed, no
+  rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → `pre-check 403 … self-granting
+  (R-22)` → `converged state=adopted` in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick
+  ~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.
+
+- **v0.88.0** (2026-07-13 eve) — **controller-driven escrow ceremony (agent half), LIVE on demo
+  host + drill VM (63/63 capabilities both).** `--output=json` machine mode (text mode
+  byte-identical; extraction into `escrowCeremony()`); the ONE fixed argv
+  (`escrow.CeremonyArgs()` — shared by the localapi exec + the `escrow-ceremony` capability
+  (Critical, pbs_dr-gated EXPLICIT) + the new `FELHOM_ESCROW` sudoers alias, three-way
+  pin-tested); localapi job endpoints (`POST /escrow/ceremony` single-flight 60 s,
+  status, ONE-SHOT claim → 410, 10-min TTL → `unclaimed_void`, `GET /escrow/preflight`).
+  R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely).
+  Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW
+  list-probe) → job ~4 s → hub blob `restic_pw_sha256` covering (repaired the spike's hash-less
+  blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard).
+  **OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got
+  direct deploys.** Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).
+
+- **v0.87.0** (2026-07-13) — **SystemDisks device-mapper walk (IA finding 2, MEDIUM):
+  legacy-boot hosts get a working drive wizard.** Operator ruling (approved 2026-07-13,
+  verbatim): *resolve device-mapper/raid parents — for the root filesystem's backing block
+  device, walk `/sys/block/<dev>/slaves` recursively down to physical disks; those, plus any
+  ESP holder when present, are system. Disks outside that set become wizard candidates (still
+  subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk
+  failure — it returns to being the error case, not the legacy-boot common case.* Implemented as
+  `physicalDisksOf`/`walkSlaves` + `HostReader.BlockSlaves` (one seam method); per-branch
+  conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test
+  `TestSystemDisks_WalkTopologies` (root-backing disk ALWAYS system — never weaken). §3 spike
+  transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda →
+  byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted →
+  torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also
+  surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4
+  502-through-CF + native-alert ban).
+
+- **v0.83.0** (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — **observability
+  pass** (pairs with controller v0.116.1 + hub v0.46.0). `applog.New` → `(logger, *Ring)`: slog
+  fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. `GET /debug/logs`
+  (local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG
+  middleware. Heartbeat log-pull: envelope `log_tail_requested` → next heartbeat ships `log_tail`
+  (128 KB, consume-once; failed push re-armed by the next envelope; `operator log pull served` INFO
+  on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome,
+  /proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount
+  unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration,
+  selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull +
+  health verdicts, desired/loop per-exchange DEBUG. Logging conventions:
+  `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: the hub-side live pull awaits
+  the operator's button click (hub UI password-gated); pre-existing lanresolver
+  permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.
+
+- **v0.77.0** (2026-07-09) — **fork-4: escrow the offsite restic repo password under R.** `IdentityBundle`
+  gains `ResticRepoPassword` (rides the existing age-under-R `WrapIdentityBundle` path — validated by the
+  custody spike `febdc56`). New `POST /escrow/stage-secret` (`withGuest`) transiently stages the
+  controller-pushed password (0600, never logged), which the `--selftest=escrow-create` ceremony
+  auto-injects into the bundle and then wipes. `AttachResticPassword`/`StagedResticPasswordPath`/
+  `WipeStagedResticPassword` added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject +
+  `DRResticCoord`). **NOT yet live-validated** — the supervised escrow ceremony is operator-run.
+
+- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
+  manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
+  full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
+  DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
+  all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
+  (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
+  `MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
+  parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
+  rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.
+
+- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
+  actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
+  COMPLETE explicit restore param set derived from the archive's embedded config (NEW
+  `Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
+  params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
+  without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
+  swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
+  seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
+  archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
+  overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
+  before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
+  while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
+  supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.
+
+- **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
+  bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
+  standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0
+  is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record
+  `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`). **Day-0 manifest vouch = operator
+  step** (agent 0.74.0 / golden 0.103.0).
+
+- **v0.74.0** (2026-07-07) — **campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom** (LIVE on
+  felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; `pct restore --pool`
+  sets it only at CREATE, so a restore-over-existing dropped 9201 from the `felhom` pool → no
+  `VM.Audit` → restore-test's *existing* `bindMountOverrides` never ran → "mp8 … only possible for
+  root". Fix: `Client.PoolAddVMID` + bring-up re-asserts membership post-restore (warn-not-fail).
+  Role/ACL + `bindMountOverrides` untouched (both correct). **Live restore-test PASSED for the first
+  time** once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore →
+  boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code.
+  OPEN: DR `bring-up -mode dr` bind-override gap (spike `SPIKE-dr-bindmount-source-2026-07-07.md`:
+  small known-constant override reusing `bindMountOverrides`; mp8/mp9 are structural constants).
+
+- **v0.73.0** (2026-07-06) — **F2 mount-role fallback CLOSED** (LIVE on felhom-pve). `roleForMountPath`
+  gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE
+  storage, so it fail-safe'd to `system` and the eject/decommission gates 403'd EVERY user-data drive
+  (campaign F2, `where=/mnt/teszt_enroll role=system`). Device-keyed classification + whole-disk containment
+  (`storage.SameWholeDisk`); Observe-error keeps the fail-safe BEFORE the fallback. Only `roleForMountPath`
+  touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart,
+  end==pre). OPEN follow-up: the `deviceRole`/`roleForMountPath` unification refactor (deferred).
+
+- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
+  `SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
+  AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
+  render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
+  `inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
+  RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
+  grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
+  felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
+  path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
+- **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
+  health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
+  Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make
+  `/run/sshd` boot-persistent AND auto-heal it every ~60s **agent-independently** (heals with the
+  agent stopped — proven live: `/run/sshd` removed → restored in 30.0s, `:22` back, no login).
+  `internal/mgmtplane` reports the additive `mgmt_plane` heartbeat stanza; hub v0.34.1 raises
+  `mgmt_plane_healed`. **NO unit declares `RuntimeDirectory=`** (the incident cause). H1 may now
+  assume `/run/sshd` is guaranteed present. Live on felhom-pve; rollback `.bak-0.70.0`.
+- **v0.70.0** (2026-07-05) — **agent self-update (operator-signed A/B slots + crash-loop
+  auto-rollback)** — TASK D1, provenance `SPIKE-agent-selfupdate-2026-07-05`. An operator-signed
+  `agent_update` op (version+sha256, sha is the only integrity root) rides the signed-jobs gate;
+  `internal/selfupdate.Executor` downloads+verifies+hands to `felhom-selfupdate-guarded apply` (root
+  re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a
+  60s dwell; a crash-looping binary is auto-reverted by `OnFailure=felhom-agent-rollback.service`
+  (first-crash trigger [SF-1]) with the tuned `[Unit]` start-limit (120s/4) as backstop. Host
+  artifacts + sudoers `FELHOM_SELFUPDATE` + `felhom-host-install.sh` day-0 install + report field
+  `selfupdate_pending`. Green tests + companions. **LIVE-VALIDATED on felhom-pve (2026-07-05): all 4
+  drills PASS** — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback
+  (0.70.2-crash → OnFailure → **~2s crash-to-recovered**, byte-identical revert, no loop), no-pending
+  guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the
+  hub `signed_jobs` queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested).
+  Box restored to canonical **v0.70.0** (host artifacts KEPT installed; scratch operator key REMOVED —
+  self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback
+  `felhom-agent.bak-0.69.0`. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update
+  auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent —
+  future: serialize the rollback oneshot). Detail: REPORT.md.
+
+- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
+  **Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
+  refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
+  wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
+  fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
+  logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
+  sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
+  no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
+  mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
+  on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
+  (`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
+  near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
+  — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
+  confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
+  OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
+  felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
+- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
+  **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
+  `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
+  `SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
+  killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
+  PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
+  offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
+  vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
+  not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
+  VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
+  OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
+  Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
+- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
+  wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
+  lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
+  loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
+  + FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
+  **namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
+  unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
+  per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
+  `demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
+  token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
+  **Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
+  (NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
+  DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
+  `backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
+  offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
+  a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
+  escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
+  `felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
+- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
+  resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
+  TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
+  (and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
+  outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
+  **Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
+  endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
+  (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
+  apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
+  true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
+  Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
+  optional future note — NOT needed for MTU correctness.
+- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
+  (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
+  the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
+  revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
+  `IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
+  gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
+  on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
+  10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
+  hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
+  on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
+  grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
+  publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
+- **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
+  findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
+  argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the
+  guide's manual D.1b update) and the golden now bakes a `felhom-controller-bootstrap.path` unit
+  (controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer
+  v1.9.1's reboot is a redundant belt, kept). **Golden 0.98.3** baked on the drill VM, clean-room
+  validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact
+  Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches
+  **agent 0.63.0 + golden 0.98.3** (the v0.63.0 vouch follow-up below is DONE). Fresh installs land
+  current and self-manage. NEW operator follow-up (SECURITY): the customer-config `git.token` has
+  Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
+- **v0.63.0** (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — **drill findings
+  B3 + B2 FIXED** (`DRILL-day0-cleanroom-2026-07-03.md`): `TokenStore.Lookup` reloads the append-only
+  store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install
+  `/controller/swap` 401 / manual restart; size short-circuit bounds the cost; behind the
+  `TokenAuthority` seam) + `guesthook.InstallSnippet` issues a fenced `mkdir -p /var/lib/vz/snippets`
+  first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained
+  exactly that one grant — **ship sudoers WITH the binary** (done on felhom-pve). Red-proofed both;
+  Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path.
+  **OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0** — until then fresh installs get
+  0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
+- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
+  pool-intersected (`staleLockController.Guests()` = `ListLXC` ∩ `Client.Pool("felhom")` members),
+  fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
+  "pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest` —
+  **deploy order on any box: rescope ACL first, then this agent.** Per
+  `SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
+- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
+- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
+- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
+  `felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`: random temp
+  staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO
+  re-checks (validated by `scripts/mkfs-guarded-harness.sh`), classifyClaim empty-lsblk fail-safe,
+  and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
+- Deployed on demo host `felhom-pve` (node `demo-felhom`), non-root `felhom-agent` service user,
+  pool-scoped token (`felhom` pool).
+
+## Open threads
+
+- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
+  server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
+- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
+  shared-box operator format gate.
+- BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
+- Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted
+  member; a live /dev re-enumeration race during a real format).

PLAN.md

@@ -0,0 +1,136 @@
+# Slice 6 Phase A — backup + restore-test orchestration (felhom-agent)
+
+## Context
+
+The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a
+signed-op gate (slice 4), but it has no guest-level **backup/restore** layer and — the point
+of this slice — no **self-restore-test**, which closes the "a backup you haven't restored
+isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the
+produced archive, a benign restore-to-a-**new** guest, and a journaled **self-restore-test**
+(restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash
+recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden
+base (§9) are **slice 7**; PBS/offsite/zero-knowledge (§8 offsite tier) are **Phase B**.
+
+Everything here is **benign** (backup, restore-to-new, scratch teardown): reuse the slice-4
+classifier/gate/journal — **no new destructive class, no new crypto**. Version → **v0.6.0-rc1**,
+stop at the checkpoint.
+
+**Locked decisions (from the task + clarifications):**
+- Backups are **crash-consistent only** (marked so in the report); app-consistency needs the
+  controller quiesce (slice 8).
+- Restore is **to a NEW guest only** (no overwrite anywhere this slice).
+- Restore-test verify = **net link-down (benign SetConfig) → boot → reaches `running`** (no
+  in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live
+  host; it is test-safety, **not** slice-7 identity reset.
+- Restore-test cadence **defaults ON at 24h**, configurable; runs only when a valid scratch
+  VMID band is set; also on-demand via the selftest harness.
+- Scratch VMID band **990000–990009** (lowest-free, excludes 9999/real guests); refuse to run
+  if unset/invalid.
+- **Local target only** this phase; `PBSSnapshot` stays a `struct{}` stub (Phase B).
+- Bulk volumes: **report the gap only** (which `backup=0` mountpoints the guest vzdump omits).
+
+## The crash-safe core (load-bearing)
+
+The restore-test is journaled as a **single** entry whose `VMID` is the scratch id and
+`Kind = "scratch_restore_test"` with a new `Scratch bool` flag. The entry is **terminal only
+after teardown** — NOT when the restore sub-task's UPID completes. `Recover` must special-case
+Scratch entries **before** the generic UPID-recheck path (else it would mark the entry
+`succeeded` because the restore task ran OK, and drop it while the guest still exists → leak).
+
+`RunRestoreTest` (pseudocode) — `defer` teardown so it runs on every path incl. failed verify:
+```
+vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999)        // refuse if band unset/invalid
+opID := "scratch-restore-"+vmid+"-"+seq
+append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted})  // BEFORE any mutation
+defer teardownScratch(opID, vmid)   // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal
+upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid)   // UPID = error-detection, NOT terminal
+for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"})  // benign
+Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout)
+res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err}
+return res    // defer tears down regardless
+```
+
+`Recover` scratch branch (added before the existing UPID logic):
+```
+if entry.Scratch {
+    guests, err := ListLXC; if err { Unresolved++; continue }              // can't decide -> leave in-flight
+    if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue }  // already gone
+    dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil)
+    if !dec.Allowed { Unresolved++; continue }                             // fail-safe (should be benign)
+    upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover
+    waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++
+}
+```
+Idempotent: crash mid-destroy → next `Recover` finds the guest gone → `ScratchClean`. Both
+teardown paths (normal + recovery) go through `gate.Authorize` (benign `ClassGuestDestroy` +
+`Provenance{AgentTaggedScratch:true}`) for the audit trail.
+
+## Files to create
+
+- **`internal/reconcile/restoretest.go`** (+ `restoretest_test.go`): `RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult` (engine method — it needs the journal/gate/queue internals), `RestoreTestResult` (reconcile-local data — must NOT return `hub.RestoreTest`, to avoid a reconcile→hub edge), `IntentForScratchDestroy(hostID, vmid)`, `pickScratchVMID`. Runs the sequence on the scratch VMID's `Queue` lane.
+- **`internal/backup/{doc,runner,store,schedule}.go`** (+ tests):
+  - `runner.go`: `BackupRunner.Backup(ctx, vmid) (hub.Backup, error)` = `Vzdump`+`WaitTask`+volid-resolve + bulk-gap from `GuestConfig.MountPoints()` `backup=0`; `LatestBackup(ctx, target) (volid, error)`; the `RestoreTestRunner` seam (satisfied by `*reconcile.Engine`); `RestoreTestResult → hub.RestoreTest` mapping.
+  - `store.go`: mutex-guarded in-memory latest-`Backup`-per-target + latest-`RestoreTest`, implementing the hub `BackupReporter`/`RestoreTestReporter` seams.
+  - `schedule.go`: cadence goroutine (default 24h; 0=disabled) → `LatestBackup` → `engine.RunRestoreTest` → write `Store`. No-ops cleanly when no backup exists yet.
+  - Imports `reconcile`+`hub`+`proxmox` (acyclic; hub imports neither).
+- **`configs/`**: example agent config with the `backup` block.
+
+## Files to modify
+
+- **`internal/proxmox/mutate.go`**: `DestroyLXC(ctx, vmid)` → `DELETE /nodes/{node}/lxc/{vmid}` with `purge=1&destroy-unreferenced-disks=1` via `dataString` (async→UPID); add `Notes string` to `VzdumpOptions` → `notes-template` param (probe `notes-template` vs `notes` on the demo PVE 9.2.2 first, read-only).
+- **`internal/proxmox/query.go`**: `LatestBackupVolID(ctx, store, vmid) (string, error)` — `StorageContent` filtered `Content=="backup" && VMID==vmid`, max `CTime`.
+- **`internal/reconcile/state.go`**: add `RestoreLXC`, `DestroyLXC`, `GuestStatus` to the `GuestAPI` interface (`*proxmox.Client` already satisfies all three).
+- **`internal/reconcile/journal.go`**: add `Scratch bool \`json:"scratch,omitempty"\``; carry it (and `Kind`) through `terminal()`.
+- **`internal/reconcile/recover.go`**: the scratch branch above; add `ScratchClean`/`ScratchDestroyed` to `RecoverResult`.
+- **`internal/reconcile/recover_test.go`** + **`engine_test.go`**: `fakeAPI` gains `RestoreLXC`/`DestroyLXC`/`GuestStatus` recorders; new `TestRecover_LeakedScratchDestroyed` (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), `…AlreadyGone` (ScratchClean), `…ListUnreadable` (Unresolved).
+- **`internal/hub/report.go`**: fill `Backup` + `RestoreTest` structs (see wire shapes); `PBSSnapshot` stays `struct{}`.
+- **`internal/hub/collect.go`**: add `BackupReporter { Backups(ctx) []Backup }` + `RestoreTestReporter { RestoreTests(ctx) []RestoreTest }` consumer seams (mirror `StorageObserver`); `collectBackups`/`collectRestoreTests` degrade nil/err → non-nil empty.
+- **`internal/hub/contract_test.go`** + **`internal/hub/testdata/host-report.golden.json`**: populated `backups[0]`/`restore_tests[0]`; assert their key sets (bidirectional, slice-5 pattern).
+- **`internal/config/config.go`**: `BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage}` + `RestoreTestCadence()` accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. `Config.Validate` stays proxmox-only.
+- **`cmd/felhom-agent/main.go`**: wire `backup.Store` into `NewCollector`; add the cadence goroutine alongside `engine.Run`/`loop.Run`/`watchdog.Run`; add `backup` + `restore-test` to `selftestFlag.Set` + the switch + a `-archive` flag; `runSelftestBackup`/`runSelftestRestoreTest` reuse the `runSelftestStorage` wiring (`NewGate(nil, hostID, SlogAudit{}, logger)` + journal + engine).
+- **felhom.eu/hub**: `hub/internal/api/handler.go` — add `hostBackup`/`hostRestoreTest` mirror structs to `hostReportPayload`, parse, persist via existing `report_json` (no new DB columns — slice-5 precedent), and **log a FAILED restore-test prominently** (`[WARN]`, the loudest DR signal). `hub/internal/api/testdata/host-report.golden.json` byte-identical with the agent golden; `host_test.go` adds `TestHostBackup_GoldenContract`/`TestHostRestoreTest_GoldenContract` (bidirectional key-set, slice-5 pattern).
+- **`CHANGELOG.md`** (prepend v0.6.0-rc1), **`REPORT.md`** (overwrite), `CLAUDE.md` current-state line.
+
+## Proposed wire shapes (draft — must land byte-identical in both repos)
+
+```go
+type Backup struct {
+    TargetID         string   `json:"target_id"`          // backup storage name
+    VMID             int      `json:"vmid"`
+    Archive          string   `json:"archive"`            // produced volid
+    Mode             string   `json:"mode"`               // snapshot|stop
+    CrashConsistent  bool     `json:"crash_consistent"`   // always true this slice
+    SizeBytes        int64    `json:"size_bytes"`
+    Success          bool     `json:"success"`
+    Error            string   `json:"error,omitempty"`
+    StartedAt        string   `json:"started_at"`         // RFC3339
+    DurationSeconds  float64  `json:"duration_seconds"`
+    UncoveredVolumes []string `json:"uncovered_volumes"`  // backup=0 mountpoints (bulk gap)
+}
+type RestoreTest struct {
+    SourceArchive   string  `json:"source_archive"`
+    SourceTier      string  `json:"source_tier"`     // "local" (pbs = Phase B)
+    ScratchVMID     int     `json:"scratch_vmid"`
+    Pass            bool    `json:"pass"`
+    Verified        string  `json:"verified"`        // "boot+running"
+    Error           string  `json:"error,omitempty"`
+    TestedAt        string  `json:"tested_at"`       // RFC3339
+    DurationSeconds float64 `json:"duration_seconds"`
+}
+type PBSSnapshot struct{} // Phase B stub
+```
+
+## Verification
+
+- `go test ./...` (local, Windows) + `go test -race ./...` on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded.
+- New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign `ClassCreate` passes the gate; restore-test end-to-end against fakes incl. **(a) teardown-on-failed-verify** and **(b) journal-recovery cleanup** (extend the slice-4 recover test: in-flight Scratch → `Recover` destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests.
+- Build the linux binary on 192.168.0.180, relay to `felhom-pve` (see memory `demo-felhom-live-agent`: `MSYS_NO_PATHCONV=1`, config at `/root/.config/felhom-agent/agent.json`, mode=direct, served-cert pin `BA:7C:99…`).
+- **Live on the demo** (the checkpoint validation): `--selftest=backup -vmid <small stopped guest>` to a local target (e.g. `felhom-usb` or `local`, content=backup) → print the `Backup` record; then `--selftest=restore-test -archive <volid>` → restore into a 990000-band scratch guest, net-link-down, boot, verify `running`, teardown → print the `RestoreTest` record. Confirm no leaked scratch guest remains (`pct list`), and that a simulated mid-test crash + restart triggers `Recover` teardown.
+
+## Push & checkpoint
+
+Push **v0.6.0-rc1** to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps
+runbook if needed), update CHANGELOG/REPORT, then **stop and await validation** (restore-test
+teardown + recovery, the benign classifications, reporting) + the live demo restore-test.
+**Phase B (next): PBS** — datastore on the USB, zero-knowledge key custody, restore-from-PBS,
+PBS integrity-verify as the lighter frequent check.

README.md

@@ -0,0 +1,350 @@
+# felhom-agent
+
+The **host agent** for the Felhom platform — the operator-tier component that runs on each
+Proxmox host and owns *all* Proxmox interaction (provision/restore guests, host storage,
+backups, host+tunnel monitoring, hub control loop, per-guest local API). Design:
+[`felhom.eu/documentation/architecture/03-host-agent.md`](https://gitea.dooplex.hu/admin/felhom.eu/raw/branch/main/documentation/architecture/03-host-agent.md).
+
+> **Status — slice 1 of N.** This repo currently contains the project scaffold and the
+> **`internal/proxmox`** interaction layer (the typed library every other module will call to
+> talk to Proxmox), plus a runnable read-only `--selftest`. **No** reconcile loop, hub client,
+> signing, or storage/backup orchestration yet — those are later slices.
+
+Module: `gitea.dooplex.hu/admin/felhom-agent` · binary: `felhom-agent` · Go 1.24.
+
+## Layout
+
+```
+cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
+internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
+internal/config/       # JSON config + env overrides (secrets never logged)
+internal/log/          # slog setup
+internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
+internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
+internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
+internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
+internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
+configs/agent.example.json
+```
+
+## The `capability` package — privileged self-check (v0.44.0)
+
+The non-root agent depends on a fixed set of `sudo -n` grants (`configs/felhom-agent.sudoers`). When
+the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke
+unnoticed until a user hit them. `internal/capability` makes that loud:
+
+- **`Manifest()` is the source of truth** for the required `(binary, representative-arg)` vectors,
+  seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When
+  you add a privileged `runner.Run` call that needs a NEW grant, add it to BOTH the sudoers file AND
+  the manifest — `manifest_test.go` asserts **manifest ⊆ sudoers** at build time (with a red-proof
+  that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
+- **`Prober.Probe`** lists each vector with `sudo -n -l` (a policy LIST — never executes, safe for
+  mkfs/pct) + an `os.Stat` existence check → an `ok`/`degraded` snapshot. It runs at startup (logged
+  loud) and on every collect, riding the hub report as `HostReport.Capabilities`; the hub
+  (`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
+  — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
+
+**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
+(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
+**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
+`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
+grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
+target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
+dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
+
+## The `storage` package — observe + watchdog (slice 5)
+
+Read-only this slice (no hub desired-state until slice 10):
+
+- **Observer** builds the host-report's `storage_targets` from `ListStorage`/`NodeStorage`
+  joined with non-privileged host reads (`/proc/mounts`, `/dev/disk/by-uuid`,
+  `/sys/.../rotational`). It reports each target's `durable_id` (the DR-load-bearing
+  re-attach key: fs-UUID for usb/local-dir, `server:export` for nfs/cifs,
+  `repo+fingerprint` for pbs, `vg/pool` for lvmthin), state, usage, a rotational
+  **class hint** (never authoritative — class is hub-owned), and the lvmthin **thin-pool
+  data fill** (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
+- **Watchdog** is the third daemon goroutine: a fast poll (seconds) over the *known*
+  target set that detects an `attached↔disconnected` transition and fires a **debounced,
+  out-of-band** host-report so the hub learns of a USB drop in seconds rather than at the
+  ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in
+  Phase B). The `HostReader` seam keeps it root-free and unit-testable with no real devices.
+
+The reported `StorageTarget` shape is a cross-repo contract duplicated in `felhom.eu/hub`;
+`internal/hub/testdata/host-report.golden.json` is byte-identical with the hub's copy and a
+bidirectional key-set test guards drift.
+
+### DR recipe — the storage/guest/PBS half (v0.38.0)
+
+The host-report carries an additive `dr_recipe` section (`internal/hub/dr_recipe.go`) — the agent half
+of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`). It is the non-secret
+**re-provision scaffolding** that complements escrow (keys) + PBS/restic (bytes): `guests[]` sizing,
+`drives[]` (user-data drives by `durable_id` → role → mount → intent), `pve_storage[]` (the
+`storage.cfg` defs), and `pbs` coordinates. Built by the pure `BuildDRRecipeHostHalf` from facts the
+report already collects — **no new reads**. **Boundary:** every field is an identifier/intent/size/
+coordinate — never a key/password/token/hash/`ENC:`; the PBS key + restic password stay in escrow.
+`recipe_version=1`, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with
+the controller's app half into one customer recipe.
+
+### The privileged `HostOps` surface (slice 5 Phase B)
+
+The write side — the one place the agent steps outside its Proxmox API token into OS-root —
+is isolated behind the `HostOps` seam (`hostops.go`): production `SudoHostOps` shells out via
+a narrow **sudoers allowlist** (`configs/felhom-agent.sudoers`) with **fixed argument vectors
+and no shell**; tests use a fake (no real root in the suite).
+
+- **Persistent mounts** are **systemd `.mount` units keyed by fs-UUID**
+  (`What=/dev/disk/by-uuid/<UUID>`, enabled so they survive reboot) — not raw fstab or a
+  transient `mount`. Benign re-mount is idempotent; **detach** (stop+disable) is destructive
+  and routes through the gate.
+- **Host-reboot remount re-resolution (v0.37.0):** at startup (before binding drives into the
+  guest) and on the periodic tick, `ReassertEnrolledMounts` re-asserts every enrolled `.mount`
+  unit that isn't currently mounted: it re-resolves the drive's `uuid:<fs-uuid>` durable id to
+  its **current** `/dev` node by re-scanning `/dev/disk/by-uuid` (`ResolveStorageDevice`, never a
+  cached node) and re-runs the idempotent `enable --now`. This re-enables a unit a prior detach
+  left `disabled` AND tolerates kernel re-enumeration moving a drive's letter (`/dev/sdb`→`sdc`)
+  — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
+- **Every argument is validated before any command is constructed** (`validate.go`): UUIDs
+  against a strict hex regex, mount paths confined + traversal-checked, SMART devices
+  whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in
+  `validate_test.go` proves a hostile UUID / path / device is refused with **zero** exec.
+- **SMART** (`smart.go`) fills `StorageTarget.smart` via `smartctl -a -j` — SATA *and* NVMe
+  attribute sets, degrading to `UNKNOWN` for devices that expose no SMART (e.g. a USB bridge).
+  **`lvs`** fills the lvmthin thin-pool **metadata** fill (metadata exhaustion corrupts a pool
+  like data exhaustion).
+- The **watchdog** gains a benign **re-mount response**: when a known mount-backed target's
+  device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed
+  through the gate as benign. The **disk-grow executor** (`pct resize`, **grow-only**) lands
+  in `internal/reconcile` as a benign action; **destructive storage ops** (detach/wipe/
+  data-losing-resize) construct a `ClassStorageWipe`/`ClassDecommission` intent bound to the
+  storage **target identity** and go through the slice-4 gate (built + tested, inert live).
+
+### `--selftest=storage` (live storage harness)
+
+Runs standalone on the Proxmox host (no hub needed):
+- bare: an **observe pass** printing the full `StorageTarget` table incl. the SMART summary
+  and thin-pool data+metadata fill.
+- `-watch <dur>` (e.g. `--selftest=storage -watch 3m`): runs the watchdog verbose for the
+  window with the **re-mount response live**, so an operator can physically cycle a drive and
+  watch detect → report → re-mount in the logs.
+
+## The `proxmox` package — model
+
+Two backends, one fixed routing policy (the fence is structural — `Client` never shells out,
+`Privileged` never makes an HTTP call; asserted in `routing_test.go`):
+
+| | Backend | Used for |
+|---|---|---|
+| **API (default)** | `proxmox.Client` | everything the scoped **FelhomAgent** token can do |
+| **root-CLI (fenced)** | `proxmox.Privileged` | the **three** proven OS-root exceptions only |
+
+Grounded entirely in the spike findings (`felhom.eu/documentation/proxmox-platform.md`,
+`tests/phase{0,1-2,3}-findings.md`). Every mutating API op is **async**: it returns a UPID and
+the caller `WaitTask`s until the task stops, then asserts `exitstatus == "OK"` — authorization
+can surface at task execution, not the HTTP POST (phase1-2 §1.3).
+
+### Public surface
+
+`Client` (API):
+
+- Read: `Version`, `Nodes`, `NodeStatus`, `ListLXC`, `GuestStatus`, `GuestConfig`,
+  `ListStorage`, `NodeStorage`, `StorageContent`.
+- Async mutating (return UPID): `RestoreLXC` (primary create path), `Vzdump`, `Snapshot`,
+  `Rollback`, `DeleteSnapshot`, `SetConfig`, `Start`, `Stop`.
+- Tasks: `WaitTask`, `TaskStatusOnce`, `TaskLogTail`.
+- Errors: `*APIError` (parses the offending privilege from a 403), `*TaskError` (parses it from
+  a failed task `exitstatus`).
+
+`Privileged` (fenced root-CLI) — each method documents *why it can't be the API*:
+
+- `CreateGoldenLXC` — `pct create` with `keyctl=1` (root@pam-only; the only root-fenced create —
+  the per-customer path provisions by **restore**, which preserves keyctl).
+- `MountUSBByUUID` — host mount-by-UUID (not a Proxmox API op).
+- `SMART`, `Sensors` — hardware reads (not API-exposed).
+
+### API-vs-root routing table
+
+See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary: the entire guest
+lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
+`keyctl` create, host mounts, and SMART/sensors (phase3 §B3).
+
+### Controller swap (agentic controller update, Phase 1 — v0.42.0)
+
+The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
+replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
+pre-pulls the target image then calls the agent:
+
+- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
+  (`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
+  write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll
+  the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
+  guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
+  - **(v0.47.0) Verify hardening (F1):** the health poll reads `{{.RestartCount}}` (a 4th inspect
+    field) — `running && RestartCount>0` is **not** healthy (a process that already crash-restarted
+    isn't stably up), and a **no-healthcheck** image must report ok on 3 **consecutive** polls (a
+    stability dwell) before acceptance; a real `healthy` result is trusted immediately. This closes the
+    hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass
+    the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the
+    `docker inspect -f *` grant (the `*` spans the extended template) are unchanged.
+- `GET /controller/swap/status` → `{state, current, previous, target, error}`.
+
+The agent is **external** to the controller container, so it survives the controller being killed
+mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
+seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.
+
+## Agent self-update (operator-signed, A/B slots, crash-loop auto-rollback — v0.70.0, TASK D1)
+
+The agent updates ITSELF the same way it swaps the controller: **the thing that performs rollback is
+never the thing being updated.** For the agent that means systemd + an ~80-line root shell wrapper
+(`configs/felhom-selfupdate-guarded`) that changes almost never; the Go binary is what flips.
+
+**Trust model.** An update is an **operator-signed `agent_update` op** delivered through the existing
+signed-jobs pipeline (same LOCKED authz gate as `storage_wipe`/`decommission`). The signed params pin
+the exact **version + sha256**, so the pinned sha is the *only* integrity root — **neither a
+compromised hub (dumb transport) nor a compromised Gitea (dumb storage) can substitute a binary.**
+The operator signs offline with `felhom-opsign -op agent_update -agent-version <v> -sha256 <hex>`.
+
+**The flow** (`internal/selfupdate/` = the Go half; the wrapper = the root half):
+
+1. The control loop sees a pending signed op → the gate verifies it (pinned-key SSHSIG → namespace →
+   allow-list → crypto → host → time → **durable nonce-burn**) → the `agent_update` executor runs.
+2. Executor downloads the binary for the signed version from the config'd artifact host
+   (`selfupdate.url_template`, `{version}` interpolated) to `/var/lib/felhom-agent/selfupdate/`,
+   verifies its sha256 against the **signed** value (mismatch → refuse, remove, agent untouched),
+   and hands it to `sudo -n felhom-selfupdate-guarded apply <staged> <sha>`. The job is completed on
+   the hub **after verify+download, before apply** (the nonce is already burned — a queued job would
+   only re-fetch and no-op on the spent nonce; a failed/rolled-back update is visible via the report).
+3. The wrapper (as root) **re-verifies** the sha, confines the staged path to the staging dir, asserts
+   same-filesystem (the atomic-rename guarantee), snapshots the current binary to `.prev`, atomically
+   `mv`s the new binary into place, writes a `pending.json` marker, `reset-failed`s, and schedules a
+   **detached** restart (`systemd-run --on-active=2s … systemctl restart felhom-agent`, so the caller
+   survives to log the handoff).
+4. The **new** binary boots; after it has run cleanly for a dwell (`selfupdate.dwell_seconds`, default
+   60) *and* core init is done, `internal/selfupdate.Manager` calls the wrapper's `commit` (clears the
+   marker; `.prev` retained as a manual net). A pending marker naming a *different* version than the
+   running binary is **not** committed — loud WARN, marker left so the report shows why (a human
+   decides).
+5. **Crash-loop auto-rollback (the safety property).** If the new binary crashes, systemd's
+   `OnFailure=felhom-agent-rollback.service` (the `felhom-agent-limits.conf` drop-in) runs the
+   wrapper's `rollback`: pending marker present → restore `.prev` byte-identical → clear marker →
+   restart → the old binary is back **within seconds of the first crash**. On systemd 257 `OnFailure=`
+   fires on *every* crash, so rollback triggers at the first one; the marker-guard makes every later
+   fire (and any crash with no update in flight) a harmless no-op. The tuned start-limit
+   (`[Unit] StartLimitIntervalSec=120 + StartLimitBurst=4`) is the terminal **backstop** (e.g. an
+   environmental crash loop of the known-good binary → terminal `failed` ≈20s → the hub's
+   `host_staleness` dead-man's-switch alerts the operator).
+
+**Design provenance:** every systemd behaviour above is empirically validated in
+`felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md` (the SF-findings). The host
+report carries `selfupdate_pending` (+ version) so a runs-but-never-commits binary is visible even
+though it never crashes. v1 scope: no hub-floor auto-update, no auto-retry of a failed update, no
+pending-timeout auto-rollback (a stuck-but-alive binary is caught by `host_staleness`).
+
+### TLS trust
+
+The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in
+config: `ca_file` (PEM, full verify), `fingerprint` (SHA-256 of the host leaf cert — pinned
+exact-cert match; the `/nodes` API returns each node's `ssl_fingerprint` to pin), or the
+explicitly-named `insecure_skip_verify` (off by default; selftest-against-127.0.0.1 only).
+
+## Provisioning the token (out-of-band, operator side)
+
+The agent only **consumes** a privilege-separated API token; role setup is a provisioning step.
+The role must be granted on **both the user AND the token** for the same path, or the
+intersection is empty and every call 403s (phase1-2 §1.2):
+
+```bash
+pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
+  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
+  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
+  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
+pveum user add felhom-agent@pve
+pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
+pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
+pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent
+```
+
+(`VM.Config.CPUMemory` is **not** a real privilege; `SDN.Use` **is** required for bridge use.)
+
+## Run
+
+```bash
+go build ./...
+# read-only health check against the host:
+./felhom-agent --config configs/agent.example.json --selftest
+# or via env (keeps the secret off disk):
+FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
+FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
+FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
+FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
+  ./felhom-agent --selftest
+```
+
+`--selftest` (read-only) loads config, builds the API client, and runs the read queries (version,
+nodes, node status, guests, storage), printing a short health report. It mutates nothing and says
+so cleanly if the token/endpoint isn't configured.
+
+`--selftest=task --vmid N` (explicitly gated) exercises `WaitTask` on a **reversible** op
+(snapshot → rollback → delete-snapshot) against guest `N`. Default `--selftest` never mutates.
+
+`--selftest=bring-up|provision` accept an optional operator **CPU/RAM cap**: `-cores N` and
+`-memory M` (MiB). Both default to `0` = keep the golden's baked sizes. When set, the cap is written
+into the SAME pre-start config PUT as the identity reset (via `BringUpSpec.Cores/MemoryMB`), so the
+guest never boots uncapped — useful when the appliance shares a host with other guests.
+
+**Pool-scoped restore (v0.53.0):** every restore (provision bring-up, DR, and restore-test) allocates
+the guest INTO the `felhom` PVE pool (`reconcile.DefaultPool`; `RestoreLXCOptions.Pool` → `pct restore
+--pool`). This is what lets the agent token be scoped to `/pool/felhom` + `/storage/<targets>` instead
+of `/` (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under
+that scoped token (`VM.Allocate`+`Pool.Allocate` at `/pool/felhom`). Layout + validation:
+`felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`.
+
+**Drive discovery + tracking (v0.55.0, Impl-2a):** `GET /disks/candidates` lists host disks the Impl-1
+filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced
+from the intent registry + Felhom `.mount` units (`RegistryKnownTargets`), NOT PVE storages — so a drive
+enrolled with no PVE dir-storage is still health-tracked (`Observe()` stays for real PVE storages). The
+controller wizard consuming this is Impl-2b.
+
+**Format safety (v0.54.0, Impl-1; hardened v0.61.0, audit D1/D2/D3):** `Format` (mkfs) is gated by a
+mandatory **unclaimed-disk guard** (`internal/storage/claim.go`) — it refuses any device not provably
+free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read
+error, and on an empty/target-absent lsblk topology — D2), independent of `DataBearing`. Below the
+agent, mkfs runs ONLY through `configs/felhom-mkfs-guarded.sh` (the sole mkfs the sudoers permits),
+which re-checks the catastrophic cases as root: system disk, LVM PV (absolute-path pvs), foreign
+mount, read-only device, and any LVM/ZFS/mdraid/LUKS/swap member signature (D1 — validated by
+`scripts/mkfs-guarded-harness.sh`, a loop-device + recorder harness). The blank-format local-API path
+binds to the device's durable id and anti-retarget re-resolves before mkfs, same as the confirmed
+wipe (D3, AGENT-001's benign-branch twin). The pool-scoped token does NOT touch mkfs (sudo op) — the
+filter + wrapper are the guard. See `SPIKE-drive-enrollment-2026-07-01.md` and
+`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`.
+
+## Process model
+
+Native Go binary, systemd service, **non-root** `felhom-agent` service user holding the scoped token,
+with a **narrow sudoers allowlist** for the fenced host-root ops. `privileged.mode: "sudo"` matches
+this; `"direct"` is for dev/CI where the agent is already root.
+
+The canonical artifacts (BUNDLE slice):
+- **`configs/felhom-agent.service`** — the canonical unit (`User=felhom-agent`,
+  `ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`). It deliberately sets
+  **no** `NoNewPrivileges` and **no** mount-namespacing hardening (`ProtectHome`/`PrivateTmp`/…): the
+  first would block the `sudo` the agent needs, the second would put the agent in a private mount
+  namespace so its `mount --bind` drive enrollments wouldn't propagate into the running guest. The
+  security boundary is the sudoers allowlist, not systemd sandboxing.
+- **`configs/felhom-agent.sudoers`** → `/etc/sudoers.d/felhom-agent` (0440, `visudo -cf`-validated).
+- **`scripts/publish-agent.sh`** publishes the binary to Gitea as a generic package
+  (`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`), printing the sha256 the operator
+  records in the hub artifact manifest.
+
+**Install is automated.** The host-bootstrap script (`felhom.eu/scripts/felhom-host-install.sh`) fetches
+the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user +
+binary + sudoers + unit + config — no manual agent install step.
+
+## Test
+
+```bash
+go vet ./... && go test ./...
+```
+
+Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, `WaitTask`
+(running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response
+decoding against the captured live shapes, and the API-vs-root routing fence.

REPORT-storage-split-spike.md

@@ -0,0 +1,90 @@
+# Findings — OS/Docker-data storage-split spike (Part B)
+
+**Date:** 2026-06-13 · **Host:** demo-felhom (PVE 9.2.2) · **Throwaway:** unprivileged LXC 9300 (created, exercised, destroyed) · **Builds nothing** — this report gates the provisioning spec.
+
+**Goal:** prove the mechanics of splitting the controller guest's **OS rootfs** from its **Docker/data** onto separate `local-lvm` volumes, and the live-migration path, before writing the provisioning spec. Target shape to validate: a small OS rootfs (~32 GB default) + a separate large Docker-data volume (~256 GB), sizes configurable at install.
+
+**Method:** all risky steps were run on a throwaway unprivileged LXC (9300) replicating guest 9201's config (`unprivileged: 1`, `features nesting=1,keyctl=1`, rootfs on `local-lvm`), never on 9201's live data. Docker 29.5.3 — **identical to production 9201** (verified: both `overlayfs` driver + `io.containerd.snapshotter.v1`).
+
+---
+
+## Verdict summary
+
+| Item | Verdict |
+|---|---|
+| B1 — second volume + Docker `data-root` on unprivileged LXC | ✅ **Works.** overlayfs on the ext4 `local-lvm` mountpoint; no idmap/ownership problem; survives reboot. |
+| B2 — live move-then-verify migration | ✅ **Works**, copy-not-move is the safety net. Caveat: rsync must exist + its exit code must be checked. |
+| B3 — vzdump/PBS coverage of the new mountpoint | ⚠️ **NOT by default — contradicts the spec's premise.** Extra CT mountpoints are **excluded** from vzdump unless `backup=1` is set **and the CT restarted**. |
+| B4 — live resize vs fresh-install carving | ✅ Live grow is online/non-disruptive (`pct resize`). Fresh-install carving = installer `hdsize`/`maxroot`/`maxvz` (host-level, documented below). |
+| B5 — provisioning seams | ✅ Mapped: golden rootfs size + `BringUpSpec.RootfsGrowGB`/`Mounts`; `bringup.go:313` is the exact spot that today omits `backup=1`. |
+
+---
+
+## B1 — Second volume + Docker `data-root` on an unprivileged LXC ✅
+
+- Added a second `local-lvm` mountpoint: `pct set 9300 -mp0 local-lvm:16,mp=/mnt/dockerdata`. PVE created `vm-9300-disk-1`, formatted it **ext4**, mounted it; in-container it is `root:root 0755` — **no idmap/ownership fix needed** (PVE maps the unprivileged offset for a fresh volume automatically).
+- Pointed Docker at it via `/etc/docker/daemon.json` `{"data-root":"/mnt/dockerdata"}` → after `systemctl restart docker`, `docker info` reports `Docker Root Dir: /mnt/dockerdata`, `Storage Driver: overlayfs`.
+- **overlay works on the mountpoint's filesystem** (ext4): pulled an image + created a named volume + wrote to both the volume and the container's overlay layer. Image/volume bytes landed on `/mnt/dockerdata` (228K → 64M) while the rootfs `/var/lib/docker` stayed flat (232K).
+- **Survives reboot:** after `pct reboot`, `data-root` is still `/mnt/dockerdata`, the volume's marker file persists, the image is present.
+
+**Gotcha (Docker 29 / containerd-snapshotter):** the daemon already auto-starts at boot, so `systemctl start docker` is a no-op and won't pick up a freshly-written `daemon.json` — use `systemctl restart docker`. Also note the driver is the new **`overlayfs`** (containerd-snapshotter), not legacy `overlay2` — relevant to B2.
+
+## B2 — Live migration sequence (move-then-verify) ✅
+
+Proven on the throwaway with seeded **running** apps (postgres with a known row `migrate-survive` in a named volume + nginx). The safe sequence:
+
+1. `docker stop <apps>` 2. `systemctl stop docker docker.socket` 3. **copy, not move:** `rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/dockerdata/` 4. write `data-root` → `/mnt/dockerdata` in `daemon.json` 5. `systemctl start docker` 6. **verify** images + `docker volume ls` + start apps + query the DB row 7. **only then** reclaim the old location (`rm -rf /var/lib/docker/*`).
+
+Result: post-switch the `pgdata` volume was present, both apps booted, and the DB row was intact. Reclaiming the old path while apps ran on the new `data-root` did not disturb them — the new `data-root` is fully self-sufficient.
+
+**Two hard lessons (both nearly caused a silent false-positive):**
+- **`rsync` was not installed** on the fresh guest; the first attempt's `rsync … | grep …` swallowed the `127` exit (pipe returns grep's status, `set -e` doesn't fire), so the migration "completed" having copied **nothing**. The migration script MUST install rsync and **check rsync's own exit code**, not a piped one.
+- With **containerd-snapshotter** the image content store sits under `data-root/containerd…`; switching `data-root` to an empty volume made images *appear* present (they resolve from the snapshotter's own addressing) while volumes/containers were empty — a misleading partial state. The reliable approach is to **rsync the entire `data-root` with docker stopped** (captures the containerd store + volumes + containers) and verify functional integrity (volume + DB row + boot), or treat images as re-pullable and migrate only the volumes (the irreplaceable data). The copy-not-move discipline meant the source was always intact and recoverable — I reverted `data-root` and the original apps + DB row came straight back.
+
+## B3 — Backup coverage after the split ⚠️ (the load-bearing finding)
+
+**The spec's premise is wrong.** It assumed the new mountpoint "should be [captured], as a CT mountpoint unless `backup=0`." The opposite is true on PVE 9.2.2:
+
+```
+# mp0 with NO backup flag:
+INFO: excluding volume mount point mp0 ('/mnt/dockerdata') from backup (disabled)
+```
+
+Additional LXC mountpoints are **excluded from vzdump by default**. They are included **only** when `backup=1` is set on the mountpoint. And setting it on a running CT is a **pending change** (`[pve:pending]`) — vzdump kept excluding mp0 until the **CT was restarted**, after which:
+
+```
+# mp0 with backup=1, post-restart:
+INFO: including mount point mp0 ('/mnt/dockerdata') in backup
+```
+
+Confirmed end-to-end: the resulting archive contains `./mnt/dockerdata/volumes/pgdata/_data/…` — the DB data is in PBS/vzdump only once `backup=1` is active.
+
+**Implication:** if the OS/data split lands without `backup=1` on the docker-data mountpoint, **every named-volume database silently falls out of the PBS whole-guest snapshot** — exactly the class of silent-default trap the spike exists to catch (cf. the PBS `ignore-verified` default). This is the single must-fix for the provisioning spec.
+
+Honest note (as the spec asked): because **both** volumes stay in PBS once `backup=1` is set, the backup **size does not shrink** — the win of the split is independent sizing/growth and a smaller, faster-to-restore OS rootfs, **not** a smaller backup. (Excluding images from PBS would shrink it but would drop DB coverage — out of scope; not done.)
+
+## B4 — Resize paths: live vs fresh install ✅
+
+- **Live (within the existing 349 GB `local-lvm`), non-disruptive:** `pct resize 9300 rootfs +2G` and `pct resize 9300 mp0 +4G` both grew the volume **and** the in-guest ext4 online (7.8→9.8 GB rootfs, 16→20 GB docker-data) with the CT running and the DB still queryable — no reboot, no downtime. Adding a brand-new mountpoint (`pct set -mpN`) is also non-disruptive to the rootfs, but the **mount itself only appears after a CT restart** on an unprivileged guest (same pending-activation behaviour seen with `backup=1` and with drive binds — consistent with the agent's existing "activate at next boot" note). Grow-only is safe; shrinking LVM-thin volumes is not supported online and was not attempted. The host root (`local`) was **not** repartitioned on the live system.
+- **Fresh install (host-level carving — this is where "configurable at install" lives):** the Proxmox ISO installer's *Advanced LVM options* expose `hdsize` (total of the disk PVE uses), `maxroot` (cap on the host `/` = `pve/root`), `maxvz` (cap on the `data` thin pool = `local-lvm`; set to 0 / leave headroom to keep space unallocated), `minfree` (reserved free space in the VG), and `swapsize`. These decide how the SSD is split between a small **host** root and a large **`local-lvm` thin pool**. Note the distinction the spec's target shape blurs: those installer knobs size the **host's** local-vs-thinpool; the **guest's** 32 GB-rootfs + 256 GB-docker-data split is sized at *provision* time via `pct` (rootfs size + an additive mountpoint), carved out of the thin pool — not by the installer. On the demo, the thin pool already has ~350 GB free, so per-guest carving needs no host repartition.
+
+## B5 — Provisioning integration points ✅
+
+Where the guest is sized today, and where the split + configurable sizes slot in:
+
+- **Golden rootfs size — `configs/build-golden.sh:40`:** `--rootfs "${ROOTFS_STORAGE}:8"` — the OS rootfs is **hardcoded at 8 GB**. The golden also `docker pull`s the controller + app images into `/var/lib/docker` **on the rootfs** (lines ~71–92), so the baked-image set is bounded by this size. `ROOTFS_STORAGE` is the 3rd script arg (default `local-lvm`). The golden runs the same docker-ce as production (→ 29.5.3 overlayfs/containerd-snapshotter).
+- **Provision bring-up — `internal/reconcile/bringup.go`:** `BringUpSpec` already carries the right seams: `RestoreStorage` (rootfs target storage), `RootfsGrowGB` (grow-only rootfs resize, applied as its own `ResizeLXC` call at line ~205), and `Mounts []GuestMount{Storage,SizeGB,MountPoint}` (additive `mpN`, attached in `buildConfigParams` at line ~313). So a docker-data volume is a natural `GuestMount`, and the OS rootfs can be grown per-customer via `RootfsGrowGB`.
+- **🔴 Exact code spot for the B3 fix — `bringup.go:313`:** today it builds `fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)` — **no `backup=` flag**, so any additive mount (including a future docker-data volume) is created `backup=0` = **excluded from PBS**. The docker-data mount MUST be attached with `,backup=1` (and the provision flow must account for the restart-to-activate behaviour). `GuestMount` should gain a `Backup bool` (or always-on for the data mount).
+- **Where the size config comes from:** currently caller-provided — the `--selftest=provision`/bring-up path and `cfg.Backup.RestoreStorage`. The in-code marker (`bringup.go:49-50`) states slice 10 wires the **hub storage manifest** into `Mounts`; that is the right home for per-customer sizes, consistent with bootstrap-v2 (the controller already pulls customer-scoped config from the hub). So "configurable at install" = a per-customer field in the hub desired-state → `RootfsGrowGB` + `GuestMount.SizeGB`.
+
+---
+
+## Open questions for the provisioning spec to resolve
+
+1. **Where does `data-root` get redirected for a *fresh* guest?** The golden bakes images into `/var/lib/docker` on the rootfs. Mounting an empty data volume **over** `/var/lib/docker` hides the baked images. Options: (a) bake `daemon.json` `data-root=/mnt/dockerdata` into the golden and attach the (empty) data mount before first Docker start, re-pulling images on first boot; (b) bake the split into the golden itself (data volume present at golden-build time, images baked onto it); (c) first-boot migration step. Pick one; (b) keeps the baked-image fast-start, (a) is simplest but loses it.
+2. **`backup=1` is mandatory on the docker-data mount** (B3). Encode it at `bringup.go:313` and decide whether the OS rootfs alone (no app data) is worth a separate lighter backup cadence.
+3. **Restart-to-activate** (B3/B4): attaching a mount or flipping `backup=1` on a running unprivileged guest is pending until reboot. Provisioning attaches mounts *before* first start, so this is a non-issue at provision time, but any *post-hoc* migration of an existing customer guest must schedule a reboot.
+4. **Default sizes & source:** confirm 32 GB rootfs / 256 GB docker-data defaults, and wire them through the hub storage manifest (per-customer), sized to the actual SSD via the installer carving facts in B4.
+5. **Backup size does not shrink** (B3): set expectations — the split is for independent sizing/growth and a smaller OS-rootfs restore, not a smaller PBS footprint.
+
+**Nothing was shipped.** Throwaway LXC 9300 and its test archives were destroyed after the spike.

REPORT.md

@@ -0,0 +1,53 @@
+# REPORT — felhom-agent v0.126.0: a fetch failure is not a wrong recovery code (R-224)
+
+**Scope: this repo's half of R-224.** The controller half ships as felhom-controller v0.202.0.
+
+## Why the agent changed at all
+
+The task that commissioned this work scoped `felhom-agent` as **untouched**. It could not be. Its
+Scenario A (a hub outage must not blame the customer's code) and Scenario C (a genuine mistype must be
+told to re-check the ten words) are **mutually unsatisfiable** while this agent answers both with one
+HTTP 400 and one sentence. No value available to the controller separates them. The task's own §5
+anticipates this — *"if the step is not recoverable from the value, make it so, and say what that
+cost"* — and §4.3 says the source outranks the register's recorded shape. **The cost is this version,
+a publish, and a `MinAgent` coupling on the controller side.**
+
+## What changed
+
+| File | Change |
+|---|---|
+| `internal/escrow/recover.go` | new `ErrBundleFetch` sentinel; the fetch leg joins it with `%w: %w` so the cause survives for the operator log |
+| `internal/localapi/escrow_recover.go` | new `case errors.Is(err, escrow.ErrBundleFetch)` → **502** with its own words; the `default` now carries only the wrong-code case and drops the "or" |
+| `internal/escrow/recover_test.go` | three new tests; the pre-existing `FetchErrorIsDistinct` re-pointed from a string to the sentinel, with the reason it failed to protect |
+| `internal/localapi/escrow_recover_class_test.go` | new — the consequence-level test: four situations, four statuses |
+
+**Four statuses:** `502` fetch failed (the code was **not used**) · `400` fetched and refused ·
+`404` no bundle · `409` bundle predates the field.
+
+## The finding this turned up
+
+**A green test named the defect and did not prevent it.** `TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct`
+has asserted since v0.125.0 that *"the operator must not be sent to re-read their recovery code because
+the hub was unreachable"*. It passed throughout, because it checked this package's error **string** one
+layer below the local-api `default` that did the merging — and a string is not something a caller can
+branch on. **Mechanism asserted, consequence unpinned**; the project's own rule names this exact case.
+It is also a comment-vs-code entry: `recover.go`'s header said the errors were *"DISTINCT on purpose"*
+and named **three** situations while a fourth was silently folded into one of them.
+
+## Green gate
+
+`go build ./...` clean · `go vet ./...` clean · `go test ./...` → **29 packages ok** ·
+`python3 scripts/agent_gates.py --fast` → all gates OK.
+
+**Red-proofs, each demonstrated failing then restored:**
+
+| Mutation | Result |
+|---|---|
+| remove the `%w: %w` join (pre-R-224 wrap) | `FetchFailureIsClassifiedAsFetch` **FAILS** |
+| delete the `ErrBundleFetch` handler case | fetch answers `400 "the recovery code did not open the sealed bundle"` — **the exact defect**, and both status tests **FAIL** |
+
+## Not changed
+
+No Proxmox surface, no privileged path, no report/hub contract, no config schema. The route's
+success path, its scoping and its R-handling discipline (`R = ""` on both paths, never logged, never
+persisted) are untouched.

REUSE.md

@@ -0,0 +1,201 @@
+# REUSE.md — felhom-agent
+
+> Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid.
+> Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper.
+> Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.
+
+## 1. Canonical helpers (MUST reuse — do not reinvent)
+
+### Allowlisted exec / privileged surface (sudoers)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Runner` / `ExecRunner.Run`, `RunStdin` | internal/proxmox/privileged.go | `Run(ctx, name, args...) (stdout, stderr []byte, err)` | ALL host command exec (direct or `sudo -n` prefix) | Arg vectors only, never a shell string; `capBuf` caps output at 1 MiB |
+| `Privileged` (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors) | internal/proxmox/privileged.go | methods on `*Privileged` | the 3 fenced root-CLI exceptions ONLY | Do NOT add methods — fence is structural (`routing_test.go` asserts it) |
+| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
+| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
+| `stageTemp` | internal/localapi/intermediary.go | `stageTemp(pattern, content) (path, err)` | random-named temp before a root `install` (audit B1) | Fixed /tmp names are a TOCTOU — sudoers globs expect `/tmp/felhom-*-*.ext` |
+| `guesthook.InstallSnippet` / `Register` | internal/guesthook/install.go | `InstallSnippet(ctx, runner) error` | pre-start self-heal hook install (C1 net) | Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues `mkdir -p /var/lib/vz/snippets` FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv) |
+
+### Disk / format safety (role gates, durable IDs, format guards)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `SudoHostOps.Format` | internal/storage/hostops.go | `Format(ctx, device, fstype) error` | THE only mkfs path | Guards, in order: `ValidateBlockDevice` + `ValidateFSType` → mandatory `deviceUnclaimed` (claim.go) → exec `felhom-mkfs-guarded` (sudoers allowlists ONLY the wrapper, not raw mkfs) |
+| `SudoHostOps.InspectDevice` + `DeviceProbe.DataBearing` | internal/storage/hostops.go | `InspectDevice(ctx, device) (DeviceProbe, error)` | data-bearing verdict from the AGENT's own read | Fail-safe: `Probed=false` ⇒ DataBearing=true; blkid output is evidence, lsblk is read-success authority |
+| `classifyClaim` / `SudoHostOps.deviceUnclaimed` | internal/storage/claim.go | `classifyClaim(claimFacts) (unclaimed bool, reason string)` | "is this disk provably free to format" | Pure function of `gatherClaimFacts`; ANY read error/ambiguity/empty-lsblk ⇒ CLAIMED (audit D2); Felhom's own `/mnt/felhom-drives` mounts are not a foreign claim |
+| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
+| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
+| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
+| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
+| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
+| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
+| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
+| `storage.ResolveStorageDevice` | internal/storage/durable_device.go | `ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)` | re-resolve enrolled STORAGE drives (remount) | `uuid:` scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node |
+| `deriveDurableID` | internal/storage/durableid.go | `deriveDurableID(typ, s, backingDevice, uuid) string` | storage-target durable id (DR re-attach key) | Deterministic per type; `uuid:` for usb/local-dir; PBS id includes `#<fingerprint>` |
+| `SystemDisks` / `isSystemBacked` / `RoleForStorage` / `RoleForRawDevice` | internal/storage/role.go | `RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole` | protection-tier classification | Fails safe to `system` (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied |
+| `ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName`, `UnitNameForMount` | internal/storage/validate.go | `Validate*(v) error` | EVERY arg that reaches a root shell-out | The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); `systemdEscapePath` computed in-process |
+| `ValidateNetworkMountSpec` | internal/storage/netmount.go | `ValidateNetworkMountSpec(spec) error` | NAS mount input boundary | Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under `NetworkMountRoot` |
+
+### Mount lifecycle (host + guest binds)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `SudoHostOps.EnsureMount` | internal/storage/hostops.go | `EnsureMount(ctx, MountSpec) error` | persistent by-UUID systemd .mount | Validate→render→stage→`install`→`enable --now`; idempotent |
+| `SudoHostOps.Unmount` | internal/storage/hostops.go | `Unmount(ctx, where) error` | detach a mount unit | DESTRUCTIVE — caller MUST have gated it; does not self-authorize |
+| `SudoHostOps.ReassertEnrolledMounts` | internal/storage/hostops.go | `ReassertEnrolledMounts(ctx)` | reboot remount (re-resolve by UUID) | Re-asserts unless mounted AND enabled (`shouldReassertMount`); skips absent UUIDs |
+| `GuestBinder.AttachDrive` / `DetachDrive` | internal/localapi/intermediary.go | `AttachDrive(ctx, vmid, where) (guestPath, err)` | live drive hot-swap under `/mnt/felhom-drives` | Normalizes to EXACTLY ONE bind via `countHostMounts` (converges double-binds); force re-bind when guest can't see it |
+| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
+| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
+| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
+| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
+| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
+| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
+| `ClassifyNetVerifyFailure` | internal/storage/netverify.go | `ClassifyNetVerifyFailure(journalTail, tcpReachable) (code, hint)` | NAS verify failure categories | String-based BY DESIGN (every mount failure is rc=32); substrings verbatim from SPIKE-nas-verify Q4; `nfs_export` merges not-found/not-permitted (NFSv4 identical) |
+
+### Durable stores (atomic state)
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
+| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
+| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
+| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
+| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
+| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
+
+### Local-API plumbing
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Server.withGuest` | internal/localapi/server.go | `withGuest(fn(w, r, vmid)) http.HandlerFunc` | bearer auth + self-scoping for EVERY route | Token→vmid is the authority; explicit `?vmid=` only a consistency check (403 on mismatch) |
+| `Server.scopedFromBody` | internal/localapi/server.go | `scopedFromBody(w, bodyVMID, tokenVMID, path) bool` | POST-body vmid self-scope check | Call right after decode; false = already 403'd |
+| `decodeBody` / `writeOK` / `writeErr` / `writeStatus` | internal/localapi/server.go | `writeStatus(w, code, ok, data, errMsg)` | ALL local-API JSON I/O | Do not hand-roll response envelopes |
+| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
+| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
+| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
+| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
+
+### Proxmox client / hub / PBS / provisioning
+
+| Symbol | File | Short signature | Use for | Gotchas |
+|---|---|---|---|---|
+| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
+| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
+| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+| `Client.PoolAddVMID` | internal/proxmox/mutate.go | `PoolAddVMID(ctx, pool, vmid) error` | re-assert pool membership after a restore-over-existing (campaign-2 R2) | SYNC (no UPID, don't WaitTask); PVE `PUT /pools` is additive (merge, not replace) — `delete=1` removes; idempotent (already-member swallowed); needs `Pool.Allocate` at `/pool/<pool>`. `pct restore --pool` sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore |
+| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
+| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
+| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
+| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 60–3600 s |
+| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
+| `buildBringUpConfig` island NIC | internal/reconcile/bringup.go | (pure) `BringUpSpec{IslandBridge,IslandGuestAddr}` → `params["net1"]` | R-50 island control plane | When BOTH island fields are set (from `cfg.LocalAPI`), attaches a static `net1=name=eth1,bridge=<vmbr9>,ip=<.2/30>` (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in `LocalAPIConfig.Validate`. The guestnet healer is eth0-only (`parseMode` is dev-scoped) so it never touches the static island NIC |
+| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
+| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
+| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
+| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
+| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
+| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
+| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
+| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
+| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
+| `wgtunnel.Manager.TunnelConvergence` / `reconcile.Engine.LastResult` | internal/wgtunnel/manager.go, internal/reconcile/engine.go | `TunnelConvergence() (desired, operational bool)`; `LastResult() (Result, bool)` | cached convergence snapshots for the fast-tick | TunnelConvergence is refreshed at the END of every `Apply` (its own cadence) so the fast-tick never execs `wg`/`systemctl`; LastResult is mutex-recorded per `reconcileOnce`, ok=false until the first pass |
+
+## 2. Canonical patterns (copy structure from THE named file)
+
+| Pattern | Canonical file | Key traits |
+|---|---|---|
+| Validate-then-exec privileged op | internal/storage/hostops.go (`EnsureMount`) | validate EVERY arg (validate.go) → render → stage in agent dir → root `install` → `systemctl`; refuse before any command exists |
+| Fail-safe pure classifier over gathered facts | internal/storage/claim.go (`classifyClaim` over `claimFacts`) | pure function ⇒ fixture-testable; any error/ambiguity refuses; gather separated from verdict |
+| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
+| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
+| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
+| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
+| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
+| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
+| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
+| Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) |
+| Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash |
+| Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted |
+| Root-file install via random temp | internal/localapi/intermediary.go (`installSharedParentUnit`) | `stageTemp` (os.CreateTemp) → sudoers-globbed `install -m` → pinned destination |
+| Detached destructive job + restart recovery | internal/localapi/formatjob.go | persist `running` → run off baseCtx → record outcome; recovery re-resolves durable id, never a path |
+| Signed-op verify pipeline | internal/authz/verifier.go (`Verify`) | armor→namespace→key-material allowlist→crypto over RAW bytes→blob→target→window→nonce LAST |
+| Resilient daemon loop | internal/hub/loop.go (`Loop.Run`) | ctx-cancel returns nil; errors logged and loop continues |
+| Guarded-below-the-agent wrapper | configs/felhom-mkfs-guarded.sh | root re-checks catastrophic cases (system disk/LVM PV/foreign mount/RO/member FS) even against an agent bug |
+
+## 3. Dangerous lookalikes — do NOT reuse
+
+| Trap | Why it bites | Use instead |
+|---|---|---|
+| Acting on the caller's `req.Device` (or any remembered /dev path) after inspection | classify→mkfs TOCTOU (AGENT-001): /dev re-enumeration retargets the node to a different physical disk | `Server.reresolveWipe`/`reresolveBlank` → format the RE-RESOLVED device |
+| Exec'ing raw `mkfs.*` (incl. `Binaries.MkfsExt4/MkfsXfs`) | sudoers no longer allowlists raw mkfs; bypasses the claim filter + wrapper re-checks | `SudoHostOps.Format` (→ `felhom-mkfs-guarded`) |
+| `DiskInfo.DurableID` (`uuid:`) as a wipe-confirmation id | wipe gate resolves `byid:`/`byuuid:` — a `uuid:` id is a `binding_mismatch` (F20-BUG2) | `DiskInfo.WipeDurableID` / `storage.DeviceDurableID` |
+| `ResolveDurableDevice` for enrolled-storage remounts (or vice versa) | schemes differ: wipe = `byid:`/`byuuid:`, storage = `uuid:` — each refuses the other | `ResolveStorageDevice` for mounts; `ResolveDurableDevice` for wipes |
+| `authz.MemoryNonceStore` on a real host | replay protection dies on restart | `authz.FileNonceStore` |
+| Adding methods to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers Cmnd_Alias + validate.go-style checks |
+| Calling `Client.DestroyLXC`/`Vzdump`/`SetConfig` outside the gate/queue/journal | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
+| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can brick guest boot (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
+| `isHostMountpoint` to reconcile bind state | boolean can't converge stacked double-binds (the /mnt doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
+| Acting on a raw `ListLXC` list as if it were "guests the agent owns" | audit A1 (pre-v0.62.0 the stale-lock reaper did exactly this — contained only by the pool-scoped token) | ownership must be PROVEN: intersect with `Client.Pool` membership like `staleLockController.Guests()` (v0.62.0), fail-safe on read failure |
+
+## 4. Seams & interfaces (testing + cross-repo)
+
+| Interface | Defined in | Implemented by | Fakes/tests at |
+|---|---|---|---|
+| `proxmox.Runner` | internal/proxmox/privileged.go | `*ExecRunner` (direct/sudo) | `mockRunner` internal/proxmox/mock_test.go; runner fakes in storage tests |
+| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
+| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
+| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
+| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
+| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
+| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
+| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
+| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
+| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
+| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
+| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
+| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
+| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
+| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
+| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
+| `hub.AddressEnumerator` (v0.119.0) | internal/hub/hostaddr.go | **defaults to the REAL `systemInterfaces`** when `Collector.addrEnum` is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record) | internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from `ip -o addr show` on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. `filterHostAddresses` keeps GLOBAL UNICAST only: one predicate that drops loopback, `fe80::/10`, and `169.254/16` — the last being the R-50 island literal, identical on every box and actively misleading if surfaced |
+| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
+| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
+| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go |
+| `applog.Ring` (always-DEBUG capture ring) + fan-out `applog.New → (logger, ring)` | internal/log/log.go | wired in cmd/felhom-agent/main.go → `localapi.Options.LogRing` + `Loop.SetLogTailSource(ring.Lines)` | internal/log/log_test.go; localapi/debuglogs_test.go — v0.83.0; the byte-capped `Lines` is the heartbeat tail source |
+| `pbsdr.StorageReader` / `SecretConsumer` / `Manager.probeFP` (func seam) | internal/pbsdr/manager.go | `*proxmox.Client`; `*hub.Client`; `pbs.ProbeFingerprint` | `fakeStorage`/`fakeConsumer`/`fakeRunner` internal/pbsdr/manager_test.go (argv+stdin recorder) |
+| `capability.Runner` | internal/capability/probe.go | `*proxmox.ExecRunner` (RunnerDirect) | `fakeRunner` internal/capability/probe_test.go |
+| Cross-repo: local API ↔ controller | internal/localapi/server.go routes; contract seeded by internal/provision/doc.go (`bootstrap.json`: endpoint + leaf fingerprint + token) | felhom-controller's agentapi client | pin = served leaf cert (memory gotcha) |
+| Cross-repo: agent ↔ hub | internal/hub/report.go (`HostReport`), `ControlEnvelope`; POST `/api/v1/host-report` | hub mirrors structs field-for-field | new event/report fields need hub-side ingest changes |
+| Cross-repo: shipped host artifacts | configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, configs/felhom-pbs-apply, shared-parent script/unit (inline in internal/localapi/intermediary.go) | deployed WITH the binary | sudoers globs must match `stageTemp` patterns + staging dirs exactly |
+| Operator signing | internal/authz (OpBlob, SSHSIG) | cmd/felhom-opsign (offline CLI) | blob/verify tests in internal/authz |
+
+## 5. Extension points (where new features plug in)
+
+- **Local-API route**: add to `Server.Handler` (internal/localapi/server.go) wrapped in `s.withGuest`; new deps go into `Options` as OPTIONAL fields degrading to "not configured".
+- **New signed-job verb**: implement `signedjobs.Executor` (return `ErrNoExecutor` for foreign ops) and append to the `signedjobs.ExecutorChain` in cmd/felhom-agent/main.go; add the op class + role scoping in internal/reconcile (classify.go, gate.go `roleAuthorizes`).
+- **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
+- **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
+- **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo).
+- **DR-recipe section (host-half)**: add the field to `DRRecipeHostHalf` (internal/hub/dr_recipe.go) **AND** to
+  the hub's `hostHalfShape` + `AssembledRecipe` (felhom.eu `hub/internal/store/dr_recipe.go`). Those two
+  hub structs are **ALLOW-LISTS**: a section only the agent knows about is stored intact and silently
+  dropped before any operator sees it — that is R-122, which cost `offsite_restic` its entire existence.
+  Then update BOTH copies of `testdata/host-report.golden.json` (byte-identical, cross-repo) and extend
+  `TestAssembleDRRecipe_CarriesEveryEmittedSection`. **A recipe field that cannot be resolved records an
+  explicit unknown with a reason — never a default, an empty string, or a placeholder** (`DRState*` /
+  `DRReason*`); a recipe read during a rebuild must not present a guess as a fact.
+- **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go.
+- **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go.
+- **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output).
+
+## 6. Known duplication (observed — NOT fixed)
+
+- Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go).
+- Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map).
+- **SMART device resolution (v0.95.0):** `smartDeviceFor` (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (`dmWholeDisk` in internal/storage/smartdev.go, via `/sys/block/<dm>/slaves`, `sysBlockRoot` test seam). `storage.SmartReader.SMARTForBacking` is the shared read the localapi `/disks` union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-`local` SMART device comes from `containingMountDevice` (SMART-only; never feeds backing/durable_id).
+- Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
+- `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go).
+- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go). **In localapi they were unified in v0.117.0**: `isHostMountpoint` and `countHostMounts` are now one-liners over `hostMountEntries`, the single parser that also yields devno/fstype/super-options for `bindLiveness`.
+- Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 1–3 (internal/signedjobs/wipe.go) across packages.
+- `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins.

RUNBOOK-provisioning-storage.md

@@ -0,0 +1,129 @@
+# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
+
+How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
+separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
+(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
++ 50 GB user-data**, rest of the thin pool free for other guests.
+
+## The design (locked)
+
+Three thin volumes are baked into the golden and grown at provision:
+
+| mp slot | Mount | Holds | Golden default | Provision grow knob |
+|---------|-------|-------|----------------|---------------------|
+| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
+| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
+| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
+
+- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
+  if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
+  unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
+- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
+  (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
+  partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
+  protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
+  baked log rotation), not by placement.
+- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
+  decommissions — it is part of the golden's baked layout. The controller auto-discovers
+  `<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
+  mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
+  the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
+  the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
+  use).
+- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
+  default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
+  data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
+
+## Golden build (per controller-image baseline)
+
+`configs/build-golden.sh` builds the template guest with the split baked in:
+
+```
+# defaults shown; override via env
+OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
+  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
+```
+
+- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
+  + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+  + `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
+- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
+  grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
+  mount; the golden does NOT pre-create it.
+- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
+  classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
+  AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
+  or mp1** (the B3 trap).
+- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
+  image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
+  the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
+  (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
+  overlay + volumes) under data-root = the data volume, which is what the split + the controller's
+  `statfs("/")` prevention guard both require.
+
+## Provision (per customer guest)
+
+The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
+baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
+are never shadowed:
+
+```
+felhom-agent --selftest=provision \
+  -archive <golden-volid> -vmid <VMID> \
+  -customer-id <id> -hub-password <retrieval-passphrase> \
+  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
+  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
+  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
+  -sysdata-grow <N>  # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
+  -sysdata-mount mp1 # the golden's user-data slot (default mp1)
+```
+
+In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
+— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
+controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
+the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
+mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
+USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
+
+**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to
+**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back
+on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) —
+a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the
+drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind.
+(Guests provisioned before v0.41.0 need a one-time `pct set <vmid> -onboot 1`.)
+
+**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
+home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
+the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
+
+### Default sizing guidance
+- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
+  (`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
+  other guests.
+- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
+  for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
+  written.
+
+## Fresh PVE host install — thin-pool carving (host-level, do this at install)
+
+This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
+carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:
+
+- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
+- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
+- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
+  rest, OR cap it to reserve unallocated VG space.
+- `minfree` — reserved free space in the VG (thin-pool safety).
+- `swapsize` — host swap.
+
+**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
+time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
+~350 GB free).
+
+## In-place migration of an EXISTING guest (NOT used by this slice)
+
+This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
+used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
+upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
+rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
+the new mount and a CT restart to activate it are required (spike B3/B4).

cmd/felhom-agent/archivetier_test.go

@@ -0,0 +1,23 @@
+package main
+
+import "testing"
+
+// R-82 live regression (2026-07-26): the restore-test derived its tier from the CONFIGURED default
+// target instead of the archive's own storage. Restoring a `felhom-pbs:` archive on a box whose
+// primary target is "local" was classified "local" → the 10-minute local wait instead of the
+// generous PBS one → the wait expired mid-restore at 600s against a 14.46 GB WAN restore, teardown
+// fired at a still-restoring guest, and the scratch leaked.
+func TestArchiveStorageID(t *testing.T) {
+	cases := []struct{ in, want string }{
+		{"felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", "felhom-pbs"},
+		{"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst", "local"},
+		{"", ""},
+		{"no-prefix", ""},
+		{":leading-colon", ""}, // i>0 guard: a leading colon is not a storage id
+	}
+	for _, c := range cases {
+		if got := archiveStorageID(c.in); got != c.want {
+			t.Fatalf("archiveStorageID(%q) = %q, want %q", c.in, got, c.want)
+		}
+	}
+}

cmd/felhom-agent/escrow_contract_test.go

@@ -0,0 +1,36 @@
+package main
+
+import (
+	"encoding/json"
+	"reflect"
+	"sort"
+	"testing"
+)
+
+// TestEscrowUploadContract pins the agent→hub escrow wire shape. It MUST match the hub's ingest
+// struct (felhom-hub api.escrowUploadRequest). Cross-repo, no shared module — this is the agent
+// half of the contract guard; the hub has the mirror in its own test.
+func TestEscrowUploadContract(t *testing.T) {
+	b, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t", ResticPwSHA256: "h"})
+	var m map[string]any
+	if err := json.Unmarshal(b, &m); err != nil {
+		t.Fatal(err)
+	}
+	got := make([]string, 0, len(m))
+	for k := range m {
+		got = append(got, k)
+	}
+	sort.Strings(got)
+	want := []string{"blob_b64", "created_at", "key_fingerprint", "posture", "restic_pw_sha256"}
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("escrow wire contract drift: got %v want %v (must match the hub ingest struct)", got, want)
+	}
+	// SLICE 3: no staged password folded in → the hash field is OMITTED on the wire (the hub stores NULL →
+	// the controller never matches → stays pending; correct — the blob doesn't cover the key).
+	b2, _ := json.Marshal(escrowUploadRequest{BlobB64: "x", KeyFingerprint: "y", Posture: "z", CreatedAt: "t"})
+	var m2 map[string]any
+	_ = json.Unmarshal(b2, &m2)
+	if _, present := m2["restic_pw_sha256"]; present {
+		t.Fatal("restic_pw_sha256 must be omitted when no staged password was sealed")
+	}
+}

cmd/felhom-agent/escrow_recover_wiring_test.go

@@ -0,0 +1,188 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"strings"
+	"testing"
+)
+
+// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
+// grepping for a string.
+//
+// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
+// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
+// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
+// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
+// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
+// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
+// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
+
+func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
+	t.Helper()
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
+	if err != nil {
+		t.Fatalf("parsing main.go: %v", err)
+	}
+	return fset, f
+}
+
+// callsWithin returns the set of function names called (directly, by identifier or selector) inside
+// the named top-level function.
+func callsWithin(f *ast.File, fnName string) map[string]bool {
+	out := map[string]bool{}
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			ce, ok := n.(*ast.CallExpr)
+			if !ok {
+				return true
+			}
+			switch fn := ce.Fun.(type) {
+			case *ast.Ident:
+				out[fn.Name] = true
+			case *ast.SelectorExpr:
+				if x, ok := fn.X.(*ast.Ident); ok {
+					out[x.Name+"."+fn.Sel.Name] = true
+				}
+				out[fn.Sel.Name] = true
+			}
+			return true
+		})
+	}
+	return out
+}
+
+// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
+func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
+	_, f := parseMain(t)
+
+	// 1. main() reaches runDaemon.
+	if !callsWithin(f, "main")["runDaemon"] {
+		t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
+	}
+	// 2. runDaemon reaches buildLocalAPIServer.
+	if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
+		t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
+	}
+
+	// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
+	//    an escrow.OffsiteKeyRecoverer is constructed there.
+	var optionsHasField, recovererConstructed bool
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			cl, ok := n.(*ast.CompositeLit)
+			if !ok {
+				return true
+			}
+			sel, ok := cl.Type.(*ast.SelectorExpr)
+			if !ok {
+				return true
+			}
+			pkg, _ := sel.X.(*ast.Ident)
+			if pkg == nil {
+				return true
+			}
+			switch pkg.Name + "." + sel.Sel.Name {
+			case "localapi.Options":
+				for _, el := range cl.Elts {
+					kv, ok := el.(*ast.KeyValueExpr)
+					if !ok {
+						continue
+					}
+					if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
+						optionsHasField = true
+					}
+				}
+			case "escrow.OffsiteKeyRecoverer":
+				recovererConstructed = true
+			}
+			return true
+		})
+	}
+	if !recovererConstructed {
+		t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
+			"production assembly point (the built-but-never-wired shape, seventh instance)")
+	}
+	if !optionsHasField {
+		t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
+			"exists and the route would answer 503 forever")
+	}
+}
+
+// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
+// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
+// key is used.
+func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
+	fset, f := parseMain(t)
+	var fetchUsesHubClient bool
+	for _, d := range f.Decls {
+		fd, ok := d.(*ast.FuncDecl)
+		if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
+			continue
+		}
+		ast.Inspect(fd.Body, func(n ast.Node) bool {
+			ce, ok := n.(*ast.CallExpr)
+			if !ok {
+				return true
+			}
+			sel, ok := ce.Fun.(*ast.SelectorExpr)
+			if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
+				return true
+			}
+			if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
+				fetchUsesHubClient = true
+			} else {
+				t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
+					fset.Position(ce.Pos()))
+			}
+			return true
+		})
+	}
+	if !fetchUsesHubClient {
+		t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
+			"not wired, or it uses a client whose credentials are not this host's")
+	}
+}
+
+// The route itself must be registered on the local API. A handler with no route is the same defect
+// one layer down, and it has shipped here before.
+func TestRecoverRouteIsRegistered(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parsing localapi/server.go: %v", err)
+	}
+	var registered bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		ce, ok := n.(*ast.CallExpr)
+		if !ok || len(ce.Args) < 2 {
+			return true
+		}
+		sel, ok := ce.Fun.(*ast.SelectorExpr)
+		if !ok || sel.Sel.Name != "HandleFunc" {
+			return true
+		}
+		lit, ok := ce.Args[0].(*ast.BasicLit)
+		if !ok {
+			return true
+		}
+		if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
+			registered = true
+		}
+		return true
+	})
+	if !registered {
+		t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
+			"exists and nothing can reach it")
+	}
+}

cmd/felhom-agent/guesthook_bulletproof_test.go

@@ -0,0 +1,60 @@
+package main
+
+import (
+	"context"
+	"testing"
+	"time"
+)
+
+// F10/rc255 (CAMPAIGN-3): a guest-hook phase body that PANICS must never crash the process — the hook
+// must return cleanly so the guest start proceeds (a nonzero exit blocks the start). runHookPhase
+// recovers the panic and returns.
+func TestRunHookPhase_PanicRecovered(t *testing.T) {
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		runHookPhase("9201", "pre-start", 5*time.Second, func(context.Context) {
+			panic("simulated heal panic (e.g. a future Heal bug)")
+		})
+	}()
+	select {
+	case <-done:
+		// returned cleanly — the guest start would proceed
+	case <-time.After(3 * time.Second):
+		t.Fatal("runHookPhase did not return after a panicking body (would have crashed the hook)")
+	}
+}
+
+// A phase body that overruns the timeout must be abandoned — the hook returns rather than hanging the
+// PVE start task. (The body's context is cancelled; the hook does not wait for the body to notice.)
+func TestRunHookPhase_TimeoutReturns(t *testing.T) {
+	bodyCtxCancelled := make(chan struct{}, 1)
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		runHookPhase("9201", "post-start", 20*time.Millisecond, func(ctx context.Context) {
+			<-ctx.Done() // simulate a body that respects cancellation eventually
+			bodyCtxCancelled <- struct{}{}
+		})
+	}()
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("runHookPhase did not return after the timeout (would hang the guest start)")
+	}
+	select {
+	case <-bodyCtxCancelled:
+		// the body's context was cancelled at the deadline — the intended signal
+	case <-time.After(time.Second):
+		t.Fatal("the phase body's context was not cancelled at the timeout")
+	}
+}
+
+// A body that errors (returns normally, no panic) is fine — the hook returns cleanly.
+func TestRunHookPhase_NormalBodyReturns(t *testing.T) {
+	ran := false
+	runHookPhase("9201", "pre-start", time.Second, func(context.Context) { ran = true })
+	if !ran {
+		t.Fatal("the phase body must run")
+	}
+}

cmd/felhom-agent/guesthook_wiring_test.go

@@ -0,0 +1,33 @@
+package main
+
+import (
+	"context"
+	"testing"
+)
+
+// The hook WIRING red-proof target: `guest-hook <vmid> post-start` must invoke the network
+// reassert with the vmid; pre-start and unknown phases must NOT. (Companion red-proof: remove the
+// PhasePostStart case from runGuestHook → the invoked assertion fails.)
+func TestRunGuestHook_PostStartInvokesNetworkReassert(t *testing.T) {
+	orig := postStartNetworkReassertFn
+	t.Cleanup(func() { postStartNetworkReassertFn = orig })
+
+	var gotVMIDs []string
+	postStartNetworkReassertFn = func(_ context.Context, vmid string) {
+		gotVMIDs = append(gotVMIDs, vmid)
+	}
+
+	runGuestHook([]string{"9201", "post-start"})
+	if len(gotVMIDs) != 1 || gotVMIDs[0] != "9201" {
+		t.Fatalf("post-start must invoke the network reassert with vmid 9201, got %v", gotVMIDs)
+	}
+
+	// pre-start must not touch the network reassert (it is the placeholder-heal phase; the heal
+	// no-ops on a nonexistent config path and never blocks).
+	runGuestHook([]string{"9201", "pre-start"})
+	// unknown phases are ignored entirely.
+	runGuestHook([]string{"9201", "pre-stop"})
+	if len(gotVMIDs) != 1 {
+		t.Fatalf("only post-start may invoke the network reassert, got %v", gotVMIDs)
+	}
+}

cmd/felhom-agent/guestnet_wiring_test.go

@@ -0,0 +1,100 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"testing"
+)
+
+// R-54 §9 rule 6 — the seam-discipline test, and the one this repo has the most reason to write:
+// v0.91.0 shipped the PBS auth-probe seam with `main.go` never calling `SetAuthSink`, every unit
+// test green because they all injected the seam directly. The guestnet watchdog has the identical
+// shape (a component + a reporter seam + a goroutine), so its wiring is asserted here rather than
+// trusted.
+//
+// This walks the AST rather than grepping the source: a commented-out call still satisfies a
+// substring match (found while red-proofing the controller's twin of this test), and a comment is
+// not a caller.
+func TestMainWiresGuestNetWatchdog(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+
+	var constructed, reporterWired, started bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.CallExpr:
+			switch fn := node.Fun.(type) {
+			case *ast.SelectorExpr:
+				switch fn.Sel.Name {
+				case "New":
+					// guestnet.New(...)
+					if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "guestnet" {
+						constructed = true
+					}
+				case "SetGuestNetReporter":
+					reporterWired = true
+				}
+			}
+		case *ast.GoStmt:
+			if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
+				if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
+					started = true
+				}
+			}
+		}
+		return true
+	})
+
+	if !constructed {
+		t.Error("main.go never calls guestnet.New — the watchdog does not exist at runtime")
+	}
+	if !reporterWired {
+		t.Error("main.go never calls collector.SetGuestNetReporter — the guest_net stanza would " +
+			"never reach the hub (the exact v0.91.0 inert-seam defect)")
+	}
+	if !started {
+		t.Error("main.go never starts the watchdog with `go gnWatchdog.Watch(ctx)` — it would be " +
+			"constructed, reported on, and never probe anything")
+	}
+}
+
+// The watchdog must NOT join the errc fan-out: a guest-network watchdog that can terminate the
+// agent turns a customer's DHCP problem into an operator-plane outage. If it is ever changed to
+// `errc <- ...`, the drain bound at the bottom of main() also has to change — this catches the
+// first half of that mistake.
+func TestGuestNetWatchdogIsNotInTheErrcFanout(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+	bad := false
+	ast.Inspect(f, func(n ast.Node) bool {
+		send, ok := n.(*ast.SendStmt)
+		if !ok {
+			return true
+		}
+		if ch, ok := send.Chan.(*ast.Ident); !ok || ch.Name != "errc" {
+			return true
+		}
+		call, ok := send.Value.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" {
+			if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" {
+				bad = true
+			}
+		}
+		return true
+	})
+	if bad {
+		t.Fatal("the guestnet watchdog was added to the errc fan-out — a watchdog over customer " +
+			"guests must never be able to bring the agent down (and the drain bound in main() " +
+			"would now be off by one, hanging shutdown)")
+	}
+}

cmd/felhom-agent/restoretest_due_wiring_test.go

@@ -0,0 +1,156 @@
+package main
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"testing"
+)
+
+// R-86 Scenario I — the seam-discipline test for the due-check.
+//
+// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
+// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
+// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
+// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
+// tested, never called — and this repo has shipped it four times.
+//
+// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
+// match, and a comment is not a caller.
+func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var settlePicker, oldPicker, settleWired, evalInterval bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.SelectorExpr:
+			// runner.PickSettledRestoreCandidateOn passed as a value (not called).
+			switch node.Sel.Name {
+			case "PickSettledRestoreCandidateOn":
+				settlePicker = true
+			case "PickRestoreCandidateOn":
+				oldPicker = true
+			}
+		case *ast.KeyValueExpr:
+			key, ok := node.Key.(*ast.Ident)
+			if !ok {
+				return true
+			}
+			if key.Name == "Settle" {
+				settleWired = true
+			}
+		case *ast.CallExpr:
+			if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
+				evalInterval = true
+			}
+		}
+		return true
+	})
+
+	if !settlePicker {
+		t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
+			"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
+	}
+	if oldPicker {
+		t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
+			"two pickers means the one under test is not the one running")
+	}
+	if !settleWired {
+		t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
+			"and every freshly-landed archive would be an immediate candidate")
+	}
+	if !evalInterval {
+		t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
+			"the retired cadence knob")
+	}
+}
+
+// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
+// RUN, and the shared heavy-operation gate is still handed to the scheduler.
+func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var inFlightWired, specIsAFunc bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		kv, ok := n.(*ast.KeyValueExpr)
+		if !ok {
+			return true
+		}
+		key, ok := kv.Key.(*ast.Ident)
+		if !ok {
+			return true
+		}
+		switch key.Name {
+		case "InFlight":
+			inFlightWired = true
+		case "Spec":
+			// A FuncLit means it is evaluated per run; anything else is a frozen value.
+			if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
+				specIsAFunc = true
+			}
+		}
+		return true
+	})
+
+	if !inFlightWired {
+		t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
+			"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
+	}
+	if !specIsAFunc {
+		t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
+			"(the tier and its timeout evaluated once at daemon start, forever)")
+	}
+}
+
+func parseMainForWiring(t *testing.T) *ast.File {
+	t.Helper()
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+	return f
+}
+
+// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
+//
+// This test exists because the method it feeds is the project's own cautionary tale:
+// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
+// was written and **had no caller at all** — a seam built, documented and never connected, found
+// only when a live restore-test's PASS reached no host-report. The fix must not become the next
+// instance, so the wiring is asserted rather than trusted.
+//
+// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
+// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
+func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var wired, feedsState bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		sel, ok := call.Fun.(*ast.SelectorExpr)
+		if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
+			return true
+		}
+		wired = true
+		// ...and it must be fed the PERSISTED state, not the in-memory store.
+		if len(call.Args) == 1 {
+			if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
+				feedsState = true
+			}
+		}
+		return true
+	})
+
+	if !wired {
+		t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
+			"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
+	}
+	if wired && !feedsState {
+		t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
+			"that does NOT survive a restart, so wiring it here would fix nothing")
+	}
+}

cmd/felhom-agent/seam_witnesses.go

@@ -0,0 +1,28 @@
+package main
+
+import (
+	"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
+)
+
+// COMPILE-TIME WITNESSES for OPTIONAL interfaces that are satisfied by a RUNTIME type assertion.
+//
+// WHY THIS FILE EXISTS. `localapi.BackupArchiveLister` is asserted at server.go's `newestArchiveOn`
+// via `tier.Service.(BackupArchiveLister)`. A failed assertion does not error — it degrades to
+// `archiveAbsent`, i.e. the pre-R-84 "ask the in-memory record only" behaviour. That degrade is
+// SILENT and it is behaviour-relevant: it is exactly the R-84 bug (a cold store after a restart
+// reading as "no backup ever") coming back, with nothing in any log to say so.
+//
+// The precedent is not hypothetical. During R-88 Part 2 the controller's `quiesceBackend` stopped
+// satisfying `quiesce.TieredBackend` when a signature changed, and `go build` AND `go vet` both
+// passed — because the interface is only ever asserted at runtime. Every box would have degraded to
+// the single-tier path, losing R-82's multi-tier backups, with no error anywhere. It was caught by
+// accident.
+//
+// A witness costs one line and converts that class of failure from a silent production degrade into
+// a compile error.
+//
+// THIS DOES NOT MAKE THE INTERFACE REQUIRED. The optionality is deliberate — it is what lets a
+// BackupService without a lister still work. The witness pins the IMPLEMENTATION (this concrete type
+// really does satisfy it), not the CONTRACT.
+var _ localapi.BackupArchiveLister = (*backup.BackupRunner)(nil)

cmd/felhom-agent/storegrant_test.go

@@ -0,0 +1,417 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"go/ast"
+	"io"
+	"log/slog"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
+)
+
+// R-185 — a tier the box cannot READ must say so.
+//
+// THE OBSERVATION (demo-felhom, 2026-08-03, reproduced at the start of this session): root lists
+// three archives on `felhom-backup`; the agent's own token gets `{"data":[]}` from the same
+// endpoint; and `local`, which has the grant, lists through that same token. The token is the
+// variable, not the storage.
+//
+// The defect is NOT the missing grant — that is one command. It is that an empty content listing is
+// what a FORBIDDEN tier and a NEWBORN tier both return, so the box could not tell them apart and
+// said nothing. These tests pin the distinction.
+
+// permAnswer is the shape /access/permissions really returns, taken from the live measurement:
+// an UNGRANTED path answers with the privileges inherited from the box-wide grant — NOT empty, and
+// NOT a 403.
+var (
+	permGranted   = map[string]int{"Datastore.Allocate": 1, "Datastore.AllocateSpace": 1}
+	permUngranted = map[string]int{"Sys.Audit": 1, "SDN.Use": 1, "Datastore.Audit": 1}
+)
+
+// probeWith calls the PRODUCTION decision with a permissions answer. **Naming the seam:** everything
+// below is true up to `storeGrantVerdict`; that the live call feeds it the real API answer is what
+// Part 0's measurement established and what the live run on the box demonstrates. An earlier draft
+// of this file re-implemented the branch here — it passed, and would have kept passing while
+// production diverged, which is the hollow shape this project keeps catching in its own tests.
+func probeWith(privs map[string]int, targetID string, critical bool) capability.Status {
+	return storeGrantVerdict(targetID, critical, privs, nil)
+}
+
+// ── SCENARIO A — a forbidden storage is REPORTED, not passed over ────────────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-03): delete the store-grant probes from `probeAll` in
+// main.go — i.e. restore `append(capProber.Probe(ctx), poolReadStatus(ctx, px))` — and
+// TestMainWiresTheStoreGrantProbe fails with "main.go never calls storeGrantStatuses". That is
+// today's behaviour on the live box: complete silence about a tier it cannot read.
+func TestStoreGrant_ForbiddenStorageIsDegradedAndNamed(t *testing.T) {
+	s := probeWith(permUngranted, "felhom-backup", true)
+
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("a storage the agent may not read must be DEGRADED, not %q — silence is the defect", s.Status)
+	}
+	if !s.Critical {
+		t.Fatal("it must be CRITICAL: the hub alerts only on critical, so a non-critical entry is the same silence with extra steps")
+	}
+	if !strings.Contains(s.Reason, "felhom-backup") {
+		t.Fatalf("the reason must NAME the storage — 'a grant is missing' costs a diagnosis at 07:00; got %q", s.Reason)
+	}
+	if !strings.Contains(s.Reason, "FelhomAgentStore") {
+		t.Fatalf("the reason must name the ROLE to grant, so the fix is in the alert; got %q", s.Reason)
+	}
+}
+
+// THE TRAP THE LIVE MEASUREMENT CAUGHT, pinned so it cannot be re-introduced: the ungranted answer
+// is not empty and not a 403 — it carries the INHERITED box-wide privileges. A probe that asked
+// "did the path come back?" or "does it have Datastore.Audit?" would report the blinded storage
+// healthy.
+func TestStoreGrant_InheritedPrivilegesAreNotAGrant(t *testing.T) {
+	if len(permUngranted) == 0 {
+		t.Fatal("fixture wrong: the ungranted answer is NOT empty — that is the whole trap")
+	}
+	if permUngranted["Datastore.Audit"] != 1 {
+		t.Fatal("fixture wrong: the ungranted path DOES carry Datastore.Audit, inherited box-wide")
+	}
+	if s := probeWith(permUngranted, "felhom-backup", true); s.Status != capability.StatusDegraded {
+		t.Fatalf("checking for the wrong privilege reports a blinded storage healthy; got %q", s.Status)
+	}
+	// ...and the privilege actually checked is the one whose absence was measured to blind listing.
+	if storeGrantRequiredPriv != "Datastore.AllocateSpace" {
+		t.Fatalf("the probed privilege changed to %q — re-measure before trusting it", storeGrantRequiredPriv)
+	}
+}
+
+// ── SCENARIO B — a newborn tier is still silent ──────────────────────────────────────────────
+//
+// A storage the agent IS allowed to read but which simply holds no archives yet is HEALTHY. The
+// probe must not look at content at all, or every freshly provisioned box alarms and the signal dies.
+//
+// COMPANION RED-PROOF (observed): make the probe degrade on an empty content listing instead of on
+// the permission — a granted-but-empty storage then reports degraded, i.e. every newborn box alarms.
+func TestStoreGrant_GrantedButEmptyIsHealthy(t *testing.T) {
+	s := probeWith(permGranted, "felhom-pbs", true)
+	if s.Status != capability.StatusOK {
+		t.Fatalf("a readable tier is healthy whether or not it holds archives yet; got %q (%s)", s.Status, s.Reason)
+	}
+	if s.Reason != "" {
+		t.Fatalf("a healthy probe carries no reason; got %q", s.Reason)
+	}
+}
+
+// ── SCENARIO C — the two states are distinguishable at a glance ──────────────────────────────
+func TestStoreGrant_ForbiddenAndNewbornAreDistinguishable(t *testing.T) {
+	forbidden := probeWith(permUngranted, "felhom-backup", true)
+	newborn := probeWith(permGranted, "felhom-pbs", true)
+
+	if forbidden.Status == newborn.Status {
+		t.Fatalf("the two states must differ — today both read as 'no settled archive yet'; got %q for both", forbidden.Status)
+	}
+	if forbidden.Name == newborn.Name {
+		t.Fatalf("each tier needs its own capability id, or one tier's fault hides another's; got %q twice", forbidden.Name)
+	}
+}
+
+// §8.3, weighed once and pinned: a box with NO dedicated target ("local" — host-install's own
+// DEGRADED fallback) must not turn an ordinary configuration into an operator page. It is still
+// probed and still reported; only the paging differs.
+func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
+	if storeGrantCritical("local") {
+		t.Fatal("a box whose backup target is the 'local' fallback must not page the operator about " +
+			"an ordinary, documented configuration")
+	}
+	for _, dedicated := range []string{"felhom-backup", "felhom-pbs", "some-nvme"} {
+		if !storeGrantCritical(dedicated) {
+			t.Fatalf("a DEDICATED target that cannot be read is user-facing and must be critical; %q was not", dedicated)
+		}
+	}
+	// The fallback is still reported — silence for it would be the original defect, scoped smaller.
+	if s := probeWith(permUngranted, "local", storeGrantCritical("local")); s.Status != capability.StatusDegraded {
+		t.Fatalf("the fallback target must still report degraded when unreadable; got %q", s.Status)
+	}
+}
+
+// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
+// probe, because it looks like coverage.
+func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
+	s := storeGrantStatus(context.Background(), nil, "felhom-backup", true, nil)
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
+	}
+	if s.Reason == "" {
+		t.Fatal("it must say why it could not ask")
+	}
+}
+
+// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
+//
+// This project's "built but never wired" count reached six last week. The fix for a SILENCE must not
+// itself be silent. AST, not grep: a commented-out call still contains the string.
+func TestMainWiresTheStoreGrantProbe(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var wired bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" {
+			wired = true
+		}
+		return true
+	})
+	if !wired {
+		t.Error("main.go never calls storeGrantStatuses — the probe would exist and report to nobody, " +
+			"which is precisely the silence R-185 is about")
+	}
+}
+
+// ── R-190 — the grant repairs itself, and the repair is VISIBLE ──────────────────────────────
+//
+// R-190 is a storage grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24,
+// with a host reinstall, logged `pveum` activity and cluster-log entries all ruled out. The cause is
+// open; the resilience is not conditional on it.
+//
+// The half that matters is the RECORD. R-190's own words: the probe sees the state, nothing sees the
+// transition. A self-repair that leaves only "ok" behind destroys the only evidence a loss happened,
+// so a recurring loss becomes undetectable forever — strictly worse than the fault it fixes.
+
+// fakeRepairRunner records wrapper invocations and can be made to fail.
+type fakeRepairRunner struct {
+	calls [][]string
+	fail  bool
+}
+
+func (f *fakeRepairRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	f.calls = append(f.calls, append([]string{name}, args...))
+	if f.fail {
+		return nil, []byte("pveum: refused"), errors.New("exit status 2")
+	}
+	return nil, nil, nil
+}
+
+func newRepairer(f *fakeRepairRunner) *storeGrantRepairer {
+	return &storeGrantRepairer{run: f.Run, log: slog.New(slog.NewTextHandler(io.Discard, nil))}
+}
+
+// ── SCENARIO F — the repair is BOUNDED ───────────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-04): make mayAttempt always return true (drop the
+// storeGrantRepairMinInterval check) →
+//
+//	--- FAIL: TestGrantRepair_IsBounded
+//	    storegrant_test.go: a repair must not run on every cycle; 5 cycles produced 5 attempt(s)
+//
+// which is a re-grant every report cycle, forever, against a fault an ACL cannot fix. Restored.
+func TestGrantRepair_IsBounded(t *testing.T) {
+	f := &fakeRepairRunner{}
+	r := newRepairer(f)
+	// Jittered, so the series never lands exactly on the interval boundary — a perfectly regular
+	// series is how a threshold test passes its own mutation, which has happened here before.
+	base := time.Date(2026, 8, 4, 9, 17, 43, 0, time.UTC)
+	offsets := []time.Duration{0, 13*time.Minute + 7*time.Second, 27*time.Minute + 51*time.Second,
+		41*time.Minute + 19*time.Second, 55*time.Minute + 3*time.Second}
+	attempts := 0
+	for _, off := range offsets {
+		if r.mayAttempt("felhom-backup", base.Add(off)) {
+			attempts++
+		}
+	}
+	if attempts != 1 {
+		t.Fatalf("a repair must not run on every cycle; %d cycles produced %d attempt(s) within %s",
+			len(offsets), attempts, storeGrantRepairMinInterval)
+	}
+	// ...and once the interval has genuinely passed, it may try again — a bound is not a ban.
+	if !r.mayAttempt("felhom-backup", base.Add(storeGrantRepairMinInterval+2*time.Minute+11*time.Second)) {
+		t.Fatal("after the interval a repair must be allowed again — otherwise one failure disables the repair forever")
+	}
+	// A DIFFERENT tier is not throttled by this one's attempt.
+	if !r.mayAttempt("felhom-pbs", base.Add(time.Minute)) {
+		t.Fatal("the bound must be per tier — one tier's attempt must not suppress another's")
+	}
+}
+
+// A nil repairer (or one with no runner) never attempts, and never panics.
+func TestGrantRepair_NilIsSafe(t *testing.T) {
+	var r *storeGrantRepairer
+	if r.mayAttempt("felhom-backup", time.Now()) {
+		t.Fatal("a nil repairer must never claim an attempt")
+	}
+	if (&storeGrantRepairer{}).mayAttempt("felhom-backup", time.Now()) {
+		t.Fatal("a repairer with no runner must never claim an attempt")
+	}
+}
+
+// The repair calls the EXISTING wrapper verb, with the storage id — no new privileged surface.
+func TestGrantRepair_CallsTheExistingWrapperVerb(t *testing.T) {
+	f := &fakeRepairRunner{}
+	r := newRepairer(f)
+	if err := r.repair(context.Background(), "felhom-backup"); err != nil {
+		t.Fatalf("repair should succeed with a healthy runner: %v", err)
+	}
+	if len(f.calls) != 1 {
+		t.Fatalf("exactly one wrapper invocation expected; got %d", len(f.calls))
+	}
+	got := f.calls[0]
+	want := []string{"/usr/local/sbin/felhom-backup-target-apply", "grant", "felhom-backup"}
+	if len(got) != len(want) {
+		t.Fatalf("wrapper argv = %v, want %v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("wrapper argv = %v, want %v — the sudoers vector is `grant *`; anything else is a policy change", got, want)
+		}
+	}
+}
+
+// A repair that FAILS must surface the failure, not swallow it (Scenario E's precondition).
+func TestGrantRepair_FailureIsReturned(t *testing.T) {
+	f := &fakeRepairRunner{fail: true}
+	if err := newRepairer(f).repair(context.Background(), "felhom-backup"); err == nil {
+		t.Fatal("a failed wrapper run must return its error — a repair that cannot run must never read as done")
+	}
+}
+
+// ── SCENARIO D (the half that matters) — the REPAIR MUST BE VISIBLE ──────────────────────────
+//
+// A repair that leaves only "ok" behind is worse than the fault: the tier works, and the fact that a
+// permission vanished is gone with it. R-190 exists because nothing saw the transition.
+//
+// The channel is the hub's EXISTING ok→degraded→ok edge (§8.5) — nothing new was built. That only
+// works if the agent deliberately reports ONE degraded cycle after repairing, and if the explanation
+// rides the field the hub actually puts in the operator's e-mail. The hub's message is built from the
+// capability NAME and FEATURE (`internal/monitor/host_capability.go` emitTransition) — **not** from
+// Reason — so the Feature must carry it.
+//
+// COMPANION RED-PROOF (observed 2026-08-04): after a successful repair, report ok instead —
+//
+//	s.Status = capability.StatusOK; s.Feature unchanged
+//
+// →	--- FAIL: TestGrantRepair_ARepairedGrantIsReportedAsATransition
+//
+//	storegrant_test.go: a self-repair must still report DEGRADED for one cycle so the hub raises
+//	its edge; got "ok" — the loss would be invisible
+//
+// i.e. exactly the silence R-190 is about. Restored.
+func TestGrantRepair_ARepairedGrantIsReportedAsATransition(t *testing.T) {
+	// THE PRODUCTION verdict, not a copy of it. An earlier draft of this test built the Status
+	// itself and asserted its own construction — it would have passed while production reported ok,
+	// which is precisely the silence being guarded against.
+	if pre := probeWith(permUngranted, "felhom-backup", true); pre.Status != capability.StatusDegraded {
+		t.Fatalf("precondition: a missing grant is degraded; got %q", pre.Status)
+	}
+	s := storeGrantRepairedVerdict("felhom-backup", true)
+
+	if s.Status != capability.StatusDegraded {
+		t.Fatalf("a self-repair must still report DEGRADED for one cycle so the hub raises its edge; "+
+			"got %q — the loss would be invisible", s.Status)
+	}
+	// The hub e-mails the FEATURE text. If the explanation is not there, the operator is told a
+	// capability was degraded and never learns it repaired itself or that anything vanished.
+	for _, want := range []string{"MISSING", "RESTORED", "felhom-backup", "R-190"} {
+		if !strings.Contains(s.Feature, want) {
+			t.Fatalf("the Feature text is what the hub puts in the operator's e-mail; it must contain %q. Got: %s", want, s.Feature)
+		}
+	}
+	if !s.Critical {
+		t.Fatal("the transition must be CRITICAL or the hub does not alert on it at all")
+	}
+}
+
+// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
+//
+// The wrapper's `grant` verb is itself a "built but never wired" example: it exists, is
+// sudoers-permitted for any id, and had only ever been called at storage CREATION. The repair must
+// not become the seventh instance. AST, not grep — a commented-out call still contains the string.
+func TestMainWiresTheGrantRepair(t *testing.T) {
+	f := parseMainForWiring(t)
+
+	var built, passed bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		switch node := n.(type) {
+		case *ast.CompositeLit:
+			if id, ok := node.Type.(*ast.Ident); ok && id.Name == "storeGrantRepairer" {
+				built = true
+			}
+		case *ast.CallExpr:
+			if id, ok := node.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" && len(node.Args) == 4 {
+				if a, ok := node.Args[3].(*ast.Ident); ok && a.Name == "grantRepairer" {
+					passed = true
+				}
+			}
+		}
+		return true
+	})
+	if !built {
+		t.Error("main.go never constructs a storeGrantRepairer — nothing would ever repair a lost grant")
+	}
+	if !passed {
+		t.Error("storeGrantStatuses is not passed the repairer — the probe would detect the loss and " +
+			"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
+	}
+}
+
+// The transition must survive a probe that is NOT the one feeding the hub.
+//
+// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
+// production while every unit test passed: `probeAll` is called independently by the self-check LOG
+// and by the collector building a host-report. The repairing call was the log's; the report three
+// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
+// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
+//
+// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
+//
+//	--- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
+//	    storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
+//	    the host-report would carry ok and the operator would never learn the grant vanished
+//
+// Restored.
+func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
+	r := newRepairer(&fakeRepairRunner{})
+	// Jittered, never landing on the window boundary.
+	repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
+	r.noteRepaired("felhom-backup", repairedAt)
+
+	// The DECISION a later probe makes — the production function, not the helper it calls. An
+	// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
+	// the latch's USE left the helper untouched.
+	healthy := probeWith(permGranted, "felhom-backup", true)
+	if healthy.Status != capability.StatusOK {
+		t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
+	}
+	got := storeGrantHealthyVerdict("felhom-backup", true,
+		healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
+	if got.Status != capability.StatusDegraded {
+		t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
+			"would carry ok and the operator would never learn the grant vanished", got.Status)
+	}
+	if !strings.Contains(got.Feature, "RESTORED") {
+		t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
+	}
+	// Outside the window it reports plain ok again.
+	late := storeGrantHealthyVerdict("felhom-backup", true,
+		healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
+	if late.Status != capability.StatusOK {
+		t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
+			"would be its own false alarm", late.Status)
+	}
+	if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
+		t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
+	}
+	// ...and it clears on its own rather than latching a box degraded forever.
+	if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
+		t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
+	}
+	// It is per tier.
+	if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
+		t.Fatal("one tier's repair must not latch another tier's status")
+	}
+	// The window MUST exceed the report interval — the property, asserted rather than assumed.
+	if storeGrantRepairReportWindow <= 15*time.Minute {
+		t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
+			"be missed entirely", storeGrantRepairReportWindow)
+	}
+}

cmd/felhom-opsign/main.go

@@ -0,0 +1,222 @@
+// Command felhom-opsign is the OPERATOR's offline signing CLI for destructive ops (slice 10B,
+// decision (a): offline operator key + signing CLI, hardware-key-ready).
+//
+// It constructs the canonical OpBlob bytes by REUSING internal/authz.CanonicalBlob — the exact
+// production path the agent's verifier authenticates over — so signer and verifier can never drift.
+// It signs that canonical message with the operator's key via `ssh-keygen -Y sign`, which makes it
+// hardware-ready: an `sk-`/YubiKey key works through ssh-keygen unchanged. The output is a signed-op
+// envelope { op_blob_b64, sig_armored } to hand to the hub's jobs queue (optionally uploaded with
+// --upload). This CLI touches ONLY the operator's signing key — never the hub's or agent's keys.
+//
+// Example — sign a data-bearing wipe (closing the 8C pending_signature gap):
+//
+//	felhom-opsign -op storage_wipe -host demo-felhom-01 \
+//	   -durable-id byid:wwn-0x5000c500abcd1234 -fstype ext4 \
+//	   -key-id ops-key-1 -key ~/.ssh/felhom_op_ed25519 -ttl 30m
+package main
+
+import (
+	"bytes"
+	"crypto/rand"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/authz"
+)
+
+func main() {
+	if err := run(); err != nil {
+		fmt.Fprintln(os.Stderr, "felhom-opsign:", err)
+		os.Exit(1)
+	}
+}
+
+func run() error {
+	var (
+		op        = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update")
+		host      = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
+		guest     = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
+		keyID     = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
+		paramsRaw = flag.String("params", "", "op params as JSON (overrides -durable-id/-fstype)")
+		durableID = flag.String("durable-id", "", "storage_wipe: the DURABLE device id (byid:…|byuuid:…); decommission: the drive's STORAGE durable-id (e.g. uuid:<fs-uuid>)")
+		fstype    = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe")
+		agentVer  = flag.String("agent-version", "", "for agent_update: the target agent version (e.g. 0.70.1)")
+		sha256Hex = flag.String("sha256", "", "for agent_update: the pinned lowercase-hex sha256 of the target binary")
+		keyFile   = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign")
+		ttl       = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)")
+		nonce     = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)")
+		uploadURL = flag.String("upload", "", "optional hub base URL to POST the signed op to the jobs queue")
+		hubKey    = flag.String("hub-key", "", "hub bearer key for --upload (operator's hub key — NOT a signing key)")
+		out       = flag.String("o", "", "write the envelope JSON to this file (default: stdout)")
+	)
+	flag.Parse()
+
+	if *op == "" || *host == "" || *keyID == "" || *keyFile == "" {
+		return fmt.Errorf("-op, -host, -key-id and -key are required")
+	}
+
+	// Params: explicit JSON, or built from the convenience flags.
+	params := strings.TrimSpace(*paramsRaw)
+	if params == "" {
+		switch *op {
+		case "storage_wipe":
+			if *durableID == "" {
+				return fmt.Errorf("storage_wipe needs -durable-id (byid:…|byuuid:…) — a path-only binding is refused by the agent")
+			}
+			pj, _ := json.Marshal(map[string]string{"durable_id": *durableID, "fstype": *fstype})
+			params = string(pj)
+		case "decommission":
+			// Decommission binds to the drive's STORAGE durable-id (the watchdog's key, e.g.
+			// "uuid:<fs-uuid>"), NOT the device-level byid:/byuuid: scheme. The agent records this
+			// id into the intent map, so it must match what the storage observer reports.
+			if *durableID == "" {
+				return fmt.Errorf("decommission needs -durable-id (the drive's storage durable-id, e.g. uuid:<fs-uuid>)")
+			}
+			pj, _ := json.Marshal(map[string]string{"durable_id": *durableID})
+			params = string(pj)
+		case "agent_update":
+			// The agent downloads the binary for -agent-version and verifies it against -sha256.
+			// The sha is the ONLY integrity root, so both are mandatory and the sha is strict-validated.
+			if *agentVer == "" || *sha256Hex == "" {
+				return fmt.Errorf("agent_update needs -agent-version and -sha256 (the pinned binary hash)")
+			}
+			if !isHex64(*sha256Hex) {
+				return fmt.Errorf("agent_update -sha256 must be 64 lowercase hex chars (got %d)", len(*sha256Hex))
+			}
+			pj, _ := json.Marshal(map[string]string{"version": *agentVer, "sha256": *sha256Hex})
+			params = string(pj)
+		default:
+			params = "{}"
+		}
+	}
+
+	n := *nonce
+	if n == "" {
+		var b [16]byte
+		if _, err := rand.Read(b[:]); err != nil {
+			return fmt.Errorf("generating nonce: %w", err)
+		}
+		n = hex.EncodeToString(b[:])
+	}
+
+	now := time.Now().UTC()
+	issued := now
+	expires := now.Add(*ttl)
+
+	// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+	blob, err := authz.CanonicalBlob(*op, *host, *guest, *keyID, n, params, issued, expires)
+	if err != nil {
+		return err
+	}
+
+	sigArmored, err := signWithSSHKeygen(blob, *keyFile)
+	if err != nil {
+		return err
+	}
+
+	env := map[string]string{
+		"op_blob_b64": base64.StdEncoding.EncodeToString(blob),
+		"sig_armored": sigArmored,
+	}
+	envJSON, _ := json.Marshal(env)
+
+	// Emit the envelope (stdout or file). Also print the human summary to stderr (never the key).
+	fmt.Fprintf(os.Stderr, "signed: op=%s host=%s guest=%q key_id=%s nonce=%s expires=%s\n",
+		*op, *host, *guest, *keyID, n, expires.Format(time.RFC3339))
+	if *out != "" {
+		if err := os.WriteFile(*out, envJSON, 0o600); err != nil {
+			return err
+		}
+		fmt.Fprintln(os.Stderr, "wrote envelope to", *out)
+	} else {
+		fmt.Println(string(envJSON))
+	}
+
+	if *uploadURL != "" {
+		if *hubKey == "" {
+			return fmt.Errorf("--upload needs --hub-key (the operator's hub bearer key)")
+		}
+		if err := upload(*uploadURL, *hubKey, *host, envJSON); err != nil {
+			return fmt.Errorf("upload to hub: %w", err)
+		}
+		fmt.Fprintln(os.Stderr, "uploaded signed op to the hub jobs queue")
+	}
+	return nil
+}
+
+// isHex64 reports whether s is exactly 64 lowercase hex chars (a sha256 hex digest).
+func isHex64(s string) bool {
+	if len(s) != 64 {
+		return false
+	}
+	for _, c := range s {
+		if (c < '0' || c > '9') && (c < 'a' || c > 'f') {
+			return false
+		}
+	}
+	return true
+}
+
+// signWithSSHKeygen signs `message` with `ssh-keygen -Y sign -n <namespace>`, the hardware-ready
+// path (sk-/YubiKey keys work unchanged). It writes the message to a temp file, runs ssh-keygen,
+// and reads the armored SSHSIG it produces. The namespace is the agent's FIXED domain separator.
+func signWithSSHKeygen(message []byte, keyFile string) (string, error) {
+	dir, err := os.MkdirTemp("", "felhom-opsign-")
+	if err != nil {
+		return "", err
+	}
+	defer os.RemoveAll(dir)
+	msgPath := filepath.Join(dir, "op.blob")
+	if err := os.WriteFile(msgPath, message, 0o600); err != nil {
+		return "", err
+	}
+	cmd := exec.Command("ssh-keygen", "-Y", "sign", "-n", authz.Namespace, "-f", keyFile, msgPath)
+	var stderr bytes.Buffer
+	cmd.Stderr = &stderr
+	if err := cmd.Run(); err != nil {
+		return "", fmt.Errorf("ssh-keygen -Y sign: %w: %s", err, strings.TrimSpace(stderr.String()))
+	}
+	sig, err := os.ReadFile(msgPath + ".sig")
+	if err != nil {
+		return "", fmt.Errorf("reading produced signature: %w", err)
+	}
+	return string(sig), nil
+}
+
+// upload POSTs the signed-op envelope to the hub's jobs queue (POST /api/v1/admin/hosts/{id}/jobs).
+// The queued blob is base64(envelope JSON); the hub stores it opaquely (it cannot forge or open it).
+func upload(baseURL, hubKey, hostID string, envJSON []byte) error {
+	body, _ := json.Marshal(map[string]string{
+		"blob_b64": base64.StdEncoding.EncodeToString(envJSON),
+	})
+	// 10A's enqueue lives under /admin/hosts/{id}/jobs (operator/global key). The queued blob is
+	// base64(envelope JSON); the hub stores it opaquely.
+	url := strings.TrimRight(baseURL, "/") + "/api/v1/admin/hosts/" + hostID + "/jobs"
+	req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Authorization", "Bearer "+hubKey)
+	req.Header.Set("Content-Type", "application/json")
+	hc := &http.Client{Timeout: 15 * time.Second}
+	resp, err := hc.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
+	}
+	return nil
+}

cmd/felhom-opsign/main_test.go

@@ -0,0 +1,24 @@
+package main
+
+import "testing"
+
+// TASK D1 Group D — agent_update opsign param validation. isHex64 is the sha gate the CLI applies
+// before it will build an agent_update envelope (the agent re-validates too, but a bad sha should
+// never even be signed).
+func TestIsHex64(t *testing.T) {
+	good := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" // 64 'a'
+	if !isHex64(good) {
+		t.Errorf("isHex64(%q) = false, want true", good)
+	}
+	for name, bad := range map[string]string{
+		"too short":     "abcdef",
+		"too long":      good + "a",
+		"uppercase hex": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
+		"non-hex char":  "g" + good[1:],
+		"empty":         "",
+	} {
+		if isHex64(bad) {
+			t.Errorf("%s: isHex64(%q) = true, want false", name, bad)
+		}
+	}
+}

configs/agent.example.json

@@ -0,0 +1,63 @@
+{
+  "proxmox": {
+    "endpoint": "https://127.0.0.1:8006",
+    "node": "demo-felhom",
+    "token": "felhom-agent@pve!agent=REPLACE_WITH_SECRET",
+    "tls": {
+      "ca_file": "",
+      "fingerprint": "BA:7C:99:7D:45:D0:67:91:E2:F2:72:74:6E:D6:9F:83:51:D1:61:E5:C3:BD:F6:A0:B8:0B:E3:D8:DB:89:5B:CF",
+      "insecure_skip_verify": false
+    }
+  },
+  "privileged": {
+    "mode": "sudo",
+    "sudo_path": "sudo"
+  },
+  "authz": {
+    "nonce_store_path": "/var/lib/felhom-agent/nonces.log",
+    "signers": [
+      {
+        "key_id": "felhom-op-1",
+        "role": "operational",
+        "public_key": "ssh-ed25519 AAAA... felhom-op-1"
+      },
+      {
+        "key_id": "felhom-recovery-1",
+        "role": "recovery",
+        "public_key": "ssh-ed25519 AAAA... felhom-recovery-1"
+      }
+    ]
+  },
+  "hub": {
+    "url": "https://hub.felhom.eu",
+    "host_id": "demo-host-01",
+    "api_key": "REPLACE_WITH_PER_HOST_HUB_KEY",
+    "poll_seconds": 900,
+    "timeout_seconds": 30,
+    "ca_file": ""
+  },
+  "backup": {
+    "local_backup_target": "local",
+    "restore_storage": "local-lvm",
+    "restore_test_cadence_seconds": 0,
+    "scratch_vmid_min": 990000,
+    "scratch_vmid_max": 990009,
+    "pbs_verify_cadence_seconds": 0,
+    "pbs_secret_dir": "/etc/pve/priv/storage",
+    "backup_cadence_seconds": 0
+  },
+  "local_api": {
+    "enable": true,
+    "listen_addr": "169.254.253.1:8443",
+    "cert_file": "/var/lib/felhom-agent/local-api.crt",
+    "key_file": "/var/lib/felhom-agent/local-api.key",
+    "token_store": "/var/lib/felhom-agent/local-tokens.log",
+    "island_bridge": "vmbr9",
+    "island_guest_addr": "169.254.253.2/30"
+  },
+  "lan_resolver": {
+    "enable": true,
+    "host_ip": "192.168.0.162"
+  },
+  "log_level": "info"
+}

configs/build-golden.sh

@@ -0,0 +1,447 @@
+#!/usr/bin/env bash
+# build-golden.sh — build the Felhom golden base LXC archive (slice 7).
+#
+# Produces a minimal Debian + Docker, unprivileged, nesting=1,keyctl=1, overlayfs LXC, baked
+# identity-clean, and archives it for a token-restore by the bring-up reconcile job
+# (internal/reconcile/bringup.go). Run as root@pam on a Proxmox host (the keyctl=1 feature flag
+# is root-only — phase3 #1; this is the ONE root step, off the per-customer path).
+#
+# Grounded by documentation/tests/slice7-bringup-spike-findings.md (commit 3342993):
+#   - F3: removing the SSH host keys does NOT auto-regenerate them on Debian (pct restore runs no
+#     keygen hook), so a baked, Condition-gated first-boot unit regenerates them — keeping the
+#     agent's front half host-side-only. The gate (ConditionPathExists=!…) makes it fire on a
+#     provision (golden, keys absent) and no-op on a DR restore (customer backup, keys present),
+#     symmetric with machine-id.
+#   - machine-id: truncated; systemd regenerates it on first boot for free (no unit needed).
+#
+# Slice 8A — the golden now also BAKES the in-guest controller (decision: image baked at golden
+# build on the trusted host, so NO registry credential ever enters a customer guest at deploy) and
+# a controller-bootstrap unit that, on boot, deploys the baked image from the agent-populated
+# config mount (/etc/felhom-bootstrap/bootstrap.json) — no docker login/pull at deploy. Refreshing
+# the golden bumps the controller baseline; controller self-update covers in-between drift.
+#
+# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE
+#        CONTROLLER_IMAGE is REQUIRED (no default) — pass the released controller tag explicitly,
+#        e.g. gitea.dooplex.hu/admin/felhom-controller:0.98.3.
+# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
+# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
+#
+# OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
+#
+# The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
+# NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
+#
+#     /var/lib/felhom/docker     --bind-->  /var/lib/docker    (Docker's data-root)
+#     /var/lib/felhom/sys_drive  --bind-->  /mnt/sys_drive     (the controller's system_data_path)
+#
+# WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
+# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
+# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
+# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
+#
+# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
+# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
+# a different documented guarantee:
+#   * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
+#     stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
+#     destroys every local recovery unit on the box;
+#   * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
+#     container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
+#     /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
+#     felhom-data-namespace mounts" would become false.
+# The neutral mount breaks neither, for one extra path and one extra fstab line.
+#
+# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
+# agent-recoverable if the data volume fills (the controller's prevention layer, and since
+# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
+# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
+# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
+# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
+set -euo pipefail
+
+# Script provenance — logged into every bake transcript next to the baked controller tag, so an
+# archive can always be traced to the script that produced it. Bump on any behavior change.
+GOLDEN_SCRIPT_VERSION="3.0.0"
+
+VMID="${1:-9100}"
+TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
+ROOTFS_STORAGE="${3:-local-lvm}"
+ARCHIVE_STORAGE="${4:-local}"
+BRIDGE="${5:-vmbr0}"
+# CONTROLLER_IMAGE is MANDATORY — no default. The hand-bumped default rotted twice (0.43.0 →
+# 0.85.1 → stale again): each time, a fresh provision would have booted a pre-floor controller
+# needing a manual install-day update (drill finding B5). A required argument cannot rot, and
+# auto-resolving "latest" could bake a tag the hub manifest never vouched — so the caller states
+# the released controller tag explicitly at every rebuild.
+CONTROLLER_IMAGE="${6:-}"
+if [ -z "$CONTROLLER_IMAGE" ]; then
+  echo "[golden] FATAL: CONTROLLER_IMAGE (argument 6) is required — pass the released controller tag explicitly." >&2
+  echo "Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE" >&2
+  echo "  e.g.: build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3" >&2
+  exit 1
+fi
+REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
+# OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
+#
+# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
+# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
+# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
+# per-customer target (bringup.go DataVolGrowGB).
+OS_SIZE_GB="${OS_SIZE_GB:-32}"
+# 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
+# did before the merge. It is deliberately NOT a per-customer size: provision grows it.
+GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
+# The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
+GOLDEN_VOLUME_MP="/var/lib/felhom"
+
+echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
+echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
+# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
+# whole point of R-165 is that it stops existing rather than being made bigger.
+pct create "$VMID" "$TEMPLATE" \
+  --hostname felhom-golden --unprivileged 1 \
+  --features nesting=1,keyctl=1 \
+  --rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
+  --mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
+  --net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
+
+echo "[golden] starting + installing Docker (official repo, trixie channel) …"
+pct start "$VMID"
+# wait for DHCP/DNS
+for i in $(seq 1 30); do
+  if pct exec "$VMID" -- getent hosts download.docker.com >/dev/null 2>&1; then break; fi
+  sleep 1
+done
+pct exec "$VMID" -- bash -c '
+  set -e
+  export DEBIAN_FRONTEND=noninteractive
+  apt-get update -qq
+  apt-get install -y -qq ca-certificates curl >/dev/null
+  install -m0755 -d /etc/apt/keyrings
+  curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
+  echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable" \
+    > /etc/apt/sources.list.d/docker.list
+  apt-get update -qq
+  apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
+'
+echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
+# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
+# /var/lib/containerd — which is NOT /var/lib/docker, so it would stay on the OS rootfs and the split
+# would only move named volumes, defeating the whole point (validated: images landed on the rootfs).
+# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
+# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
+# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
+# DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
+# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
+# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
+# still needs no override because the path is unchanged. Log caps kill the most common runaway.
+pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
+{
+  "features": { "containerd-snapshotter": false },
+  "log-driver": "json-file",
+  "log-opts": { "max-size": "10m", "max-file": "3" }
+}
+JSON'
+echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
+# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
+# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
+# round would hide those files under the bind and silently ship a golden whose baked images are on the
+# rootfs — the exact failure class the assertions below exist to catch.
+#
+# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
+# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
+# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
+pct exec "$VMID" -- bash -c "
+  set -e
+  systemctl stop docker docker.socket containerd 2>/dev/null || true
+  mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
+  if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
+    cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
+    rm -rf /var/lib/docker/*
+  fi
+  mkdir -p /var/lib/docker /mnt/sys_drive
+  printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker'    >> /etc/fstab
+  printf '%s /mnt/sys_drive none bind 0 0\n'  '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
+  systemctl daemon-reload
+  mount /var/lib/docker
+  mount /mnt/sys_drive
+  systemctl start containerd
+"
+
+echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
+# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
+# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
+pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm hello-world >/dev/null && echo "  docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"'
+# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
+# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
+pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
+# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
+# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
+# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
+# its baked images there.
+pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo "  /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
+# ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
+# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
+# (clearing that warning is the whole point of the volume).
+pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo "  /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
+# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
+# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
+# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
+# there, and a shared pool neither `df` can see coming.
+pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo "  both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
+
+echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
+# docker login is used ONCE here on the trusted build host, then logged out before archiving so
+# the credential is NEVER baked into the golden. The IMAGE is what gets baked (in Docker storage).
+if [ -n "${REGISTRY_USER:-}" ] && [ -n "${REGISTRY_TOKEN:-}" ]; then
+  pct exec "$VMID" -- bash -c "systemctl start docker; sleep 1; echo '$REGISTRY_TOKEN' | docker login '$REGISTRY_HOST' -u '$REGISTRY_USER' --password-stdin >/dev/null"
+fi
+pct exec "$VMID" -- bash -c "docker pull '$CONTROLLER_IMAGE'"
+# Record the baked image ref for the bootstrap unit (so the unit needs no login/pull).
+pct exec "$VMID" -- bash -c "printf '%s\n' '$CONTROLLER_IMAGE' > /etc/felhom-controller-image"
+
+# Bake EVERY controller-managed infra image so enabling an infra stack on a fresh box is near-instant
+# instead of a multi-minute silent registry pull.
+#
+# THE LIST COMES FROM THE CONTROLLER WE JUST PULLED, not from a copy here. This script used to carry
+# a hand-maintained array of three tags with a comment telling the reader to keep it in sync with the
+# controller's internal/infra constants. It drifted the moment a fourth stack was added: felhom-samba
+# was never added here, so the golden baked 3 of 4 and turning on Megosztás pulled from the registry
+# with zero UI feedback — observed live, twice. Asking the binary (`--print-infra-images`, backed by
+# infra.Images()) makes golden-vs-controller drift structurally impossible: the golden bakes exactly
+# what THIS controller version will request.
+echo "[golden] asking the controller which infra images it manages …"
+INFRA_LIST="$(pct exec "$VMID" -- bash -c "docker run --rm --entrypoint /usr/local/bin/felhom-controller '$CONTROLLER_IMAGE' --print-infra-images 2>/dev/null" || true)"
+mapfile -t INFRA_IMAGES < <(printf '%s\n' "$INFRA_LIST" | grep -E '^[a-z0-9._/-]+:[A-Za-z0-9._-]+$' || true)
+if [ "${#INFRA_IMAGES[@]}" -eq 0 ]; then
+  # Controllers older than v0.147.0 have no --print-infra-images. Fall back to the historical list so
+  # an old-controller bake still works — but say so loudly, because the fallback is exactly the
+  # drift-prone thing this change removed, and it CANNOT know about stacks added after it was written.
+  echo "[golden] WARN: '$CONTROLLER_IMAGE' does not support --print-infra-images (pre-0.147.0)."
+  echo "[golden] WARN: falling back to the historical 3-image list. felhom-samba will NOT be baked,"
+  echo "[golden] WARN: so enabling Megosztás on this golden will pull at runtime. Bake >=0.147.0."
+  INFRA_IMAGES=(
+    "traefik:v3.6.7"
+    "cloudflare/cloudflared:2026.6.0"
+    "gtstef/filebrowser:1.3.3-stable"
+  )
+fi
+echo "[golden] baking infra images (${#INFRA_IMAGES[@]}): ${INFRA_IMAGES[*]} …"
+for img in "${INFRA_IMAGES[@]}"; do
+  # Hard gate: fail loudly BEFORE pulling if a pinned tag doesn't resolve (a bad pin otherwise fails
+  # mid-bake with a confusing error).
+  pct exec "$VMID" -- bash -c "docker manifest inspect '$img' >/dev/null 2>&1" \
+    || { echo "[golden] FATAL: pinned infra image does not resolve: $img"; exit 1; }
+  pct exec "$VMID" -- bash -c "docker pull '$img'"
+done
+
+# Log out LAST — not right after the controller pull. felhom-samba lives on the same private registry
+# as the controller, so the infra loop above needs the credential; logging out first made that pull
+# 401. The credential is still never archived: the logout + config.json removal happen before the
+# guest is stopped and templated.
+pct exec "$VMID" -- bash -c "docker logout '$REGISTRY_HOST' >/dev/null 2>&1 || true; rm -f /root/.docker/config.json"
+pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
+  || { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
+
+echo "[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …"
+pct push "$VMID" /dev/stdin /usr/local/sbin/felhom-controller-bootstrap.sh --perms 700 <<'BOOTSH'
+#!/bin/bash
+# felhom controller-bootstrap (slice 8A): the host agent's back-half populated the read-only
+# config mount /etc/felhom-bootstrap; this golden-baked oneshot deploys the BAKED controller image
+# with that config. NO docker login / NO docker pull — the image is already in this golden's Docker
+# storage (and self-update handles version drift). Host-side only; the agent never enters the guest.
+set -euo pipefail
+CFG=/etc/felhom-bootstrap/bootstrap.json
+[ -r "$CFG" ] || { echo "[ctrl-bootstrap] no $CFG — not provisioned, nothing to do"; exit 0; }
+IMAGE=$(cat /etc/felhom-controller-image 2>/dev/null || true)
+[ -n "$IMAGE" ] || { echo "[ctrl-bootstrap] FATAL: /etc/felhom-controller-image missing"; exit 1; }
+# Per-guest container hostname (slice base-infra/3A): derive from the bootstrap's customer.id so the
+# controller's os.Hostname() (its hub-reported hostname) is the customer id, not the Docker container
+# ID. Portable, dependency-free parse (NO jq in the golden) — bootstrap.json has exactly one "id" key
+# (customer.id). Falls back to no --hostname if the parse yields nothing (fail-safe).
+CUSTOMER_ID=$(sed -n 's/.*"id"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$CFG" | head -1)
+# SECURITY: $CUSTOMER_ID is interpolated into `docker run` — reject anything that isn't a DNS-safe
+# label so a malformed/hostile customer.id can't smuggle extra docker flags (e.g. --privileged, -v).
+# Then pass it via a quoted array (never word-split). Empty/invalid → no --hostname (fail-safe).
+case "$CUSTOMER_ID" in
+  ""|*[!a-zA-Z0-9._-]*|-*) CUSTOMER_ID="" ;;
+esac
+HOSTNAME_ARGS=()
+[ -n "$CUSTOMER_ID" ] && HOSTNAME_ARGS=(--hostname "$CUSTOMER_ID")
+echo "[ctrl-bootstrap] deploying $IMAGE from $CFG (hostname=${CUSTOMER_ID:-<unset>})"
+docker rm -f felhom-controller >/dev/null 2>&1 || true
+# Section-G fix (base-infra slice): the controller writes app/infra compose stacks under
+# /opt/docker/stacks INSIDE the container, but `docker compose up` is executed by the GUEST daemon
+# (shared socket), which resolves every relative bind source on the GUEST filesystem. Without a
+# SAME-PATH host bind for /opt/docker/stacks, the daemon can't see those bind sources and silently
+# creates empty dirs → every bind-mounted stack (base infra AND customer apps) breaks. A named volume
+# would NOT fix this (it resolves to /var/lib/docker/volumes/...). Pre-create the dir, then same-path bind.
+mkdir -p /opt/docker/stacks
+# slice 10 P2: the controller must SEE enrolled external data drives, which the agent binds into the
+# guest at /mnt/<name>. Make /mnt a SHARED mount so those binds (and later host-remount self-heal)
+# propagate into the controller container, then bind /mnt :rslave (host->container propagation ONLY —
+# the container can't mutate the guest mount tree). This is the ONE bind the 8C de-privileging left
+# out; scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the
+# customer's other on-drive data. rbind preserves existing submounts; re-running re-shares (safe).
+mkdir -p /mnt
+mountpoint -q /mnt || mount --rbind /mnt /mnt
+mount --make-rshared /mnt
+# Otherwise still DE-PRIVILEGED: disk EXECUTION (scan/format/mount) stays the agent's — NO --privileged,
+# no /dev, no /etc/fstab. Bootstrap config (ro), data volume, stacks dir (same-path), the /mnt :rslave
+# view, and the docker socket. The controller reaches the agent's local API for disk management.
+docker run -d --name felhom-controller --restart unless-stopped "${HOSTNAME_ARGS[@]}" \
+  -e FELHOM_BOOTSTRAP_PATH=/etc/felhom-bootstrap/bootstrap.json \
+  -v /etc/felhom-bootstrap:/etc/felhom-bootstrap:ro \
+  -v felhom-controller-data:/opt/docker/felhom-controller \
+  -v /opt/docker/stacks:/opt/docker/stacks \
+  -v /mnt:/mnt:rslave \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  "$IMAGE"
+echo "[ctrl-bootstrap] controller started"
+BOOTSH
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-controller-bootstrap.service <<UNIT
+[Unit]
+Description=Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)
+After=docker.service network-online.target
+Wants=docker.service network-online.target
+ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/felhom-controller-bootstrap.sh
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-controller-bootstrap.service'
+
+# B1 (DRILL-day0-cleanroom-2026-07-03 R6/B1): the service's ConditionPathExists is evaluated only
+# when the service is STARTED — i.e. at boot via multi-user.target — but the agent back-half
+# hot-plugs the bootstrap mount into the ALREADY-RUNNING guest, so on a provision the boot-time
+# start races the mount and loses on slow hardware. This path unit watches for bootstrap.json and
+# starts the service when it APPEARS — covering the provision hot-plug without a reboot. The boot
+# case is still served by the enabled service itself; RemainAfterExit=yes on the service means the
+# path unit cannot re-trigger it in a loop once it has run.
+echo "[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …"
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-controller-bootstrap.path <<UNIT
+[Unit]
+Description=Watch for the agent-populated bootstrap config; start the controller bootstrap when it appears
+
+[Path]
+PathExists=/etc/felhom-bootstrap/bootstrap.json
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-controller-bootstrap.path'
+
+echo "[golden] baking the first-boot SSH host-key regeneration unit (F3) …"
+pct exec "$VMID" -- bash -c 'cat > /etc/systemd/system/felhom-regen-hostkeys.service <<UNIT
+[Unit]
+Description=Regenerate SSH host keys on first boot if absent
+ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key
+DefaultDependencies=no
+After=local-fs.target
+Before=ssh.service sshd.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/ssh-keygen -A
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl enable felhom-regen-hostkeys.service'
+
+echo "[golden] identity-clean + minimize …"
+pct exec "$VMID" -- bash -c '
+  systemctl stop docker containerd 2>/dev/null || true
+  apt-get clean; rm -rf /var/lib/apt/lists/*
+  rm -f /etc/ssh/ssh_host_*            # regenerated on first boot by the baked unit (F3)
+  truncate -s 0 /etc/machine-id        # systemd regenerates on first boot (free)
+  rm -f /var/lib/dbus/machine-id; ln -sf /etc/machine-id /var/lib/dbus/machine-id
+  rm -rf /var/log/*; : > /root/.bash_history
+  rm -f /etc/hostname                  # set per-guest at provision (host-side token config)
+'
+
+echo "[golden] stop + archive …"
+pct stop "$VMID"
+# --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
+# INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
+# exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
+# Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
+# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
+vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
+if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
+  exit 1
+fi
+# ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
+# archive". After the merge there is no mp1 — so the same failure now looks like the volume being
+# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
+# inclusion line names the volume's actual mount path.
+if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
+  grep -iE "mount point" /tmp/golden-vzdump.log || true
+  exit 1
+fi
+# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
+# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
+if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
+  echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
+  exit 1
+fi
+grep -q "including mount point mp0" /tmp/golden-vzdump.log \
+  || echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
+
+VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
+echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
+
+#-------------------------------------------------------------------------------
+# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
+#-------------------------------------------------------------------------------
+# The host-install script fetches the golden from Gitea
+# (/api/packages/admin/generic/felhom-golden/<golden-version>/golden.tar.zst) and verifies its sha256
+# against the hub-vouched artifact manifest before importing it. <golden-version> = the BAKED
+# controller version (so the golden's published version tracks what it ships). Publishing is OPT-IN:
+# only runs when GITEA_USER + GITEA_TOKEN (or REGISTRY_USER/REGISTRY_TOKEN) are set. The local-golden
+# auto-discovery in the host-install script stays as a fallback either way.
+GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
+GITEA_OWNER="${GITEA_OWNER:-admin}"
+PUB_USER="${GITEA_USER:-${REGISTRY_USER:-}}"
+PUB_TOKEN="${GITEA_TOKEN:-${REGISTRY_TOKEN:-}}"
+# Golden version = the baked controller tag's version (strip the image path + any leading 'v').
+GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"; GOLDEN_VERSION="${GOLDEN_VERSION#v}"
+
+if [ -z "$VOLID" ]; then
+  echo "[golden] WARN: could not resolve the archive volid — skipping Gitea publish."
+elif [ -z "$PUB_USER" ] || [ -z "$PUB_TOKEN" ]; then
+  echo "[golden] Gitea publish SKIPPED (set GITEA_USER+GITEA_TOKEN or REGISTRY_USER+REGISTRY_TOKEN to enable)."
+  echo "[golden]   would publish version=$GOLDEN_VERSION from volid $VOLID"
+else
+  # Resolve the archive's on-disk path (pvesm path turns a volid into a filesystem path).
+  ARCHIVE_PATH="$(pvesm path "$VOLID" 2>/dev/null || true)"
+  if [ -z "$ARCHIVE_PATH" ] || [ ! -f "$ARCHIVE_PATH" ]; then
+    echo "[golden] WARN: cannot resolve archive path for $VOLID — skipping publish."
+  else
+    GOLDEN_SHA256="$(sha256sum "$ARCHIVE_PATH" | awk '{print $1}')"
+    PUB_URL="${GITEA_BASE}/api/packages/${GITEA_OWNER}/generic/felhom-golden/${GOLDEN_VERSION}/golden.tar.zst"
+    echo "[golden] publishing golden ($(wc -c < "$ARCHIVE_PATH") bytes, sha256 ${GOLDEN_SHA256:0:16}…) → $PUB_URL"
+    # Delete-then-PUT so re-publishing the same version overwrites cleanly (idempotent).
+    dcode="$(curl -fsS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X DELETE "$PUB_URL" 2>/dev/null || true)"
+    echo "[golden]   pre-delete existing: HTTP ${dcode} (404/204 expected)"
+    ucode="$(curl -sS -o /dev/null -w '%{http_code}' -u "${PUB_USER}:${PUB_TOKEN}" -X PUT --upload-file "$ARCHIVE_PATH" "$PUB_URL")"
+    if [ "$ucode" = "201" ] || [ "$ucode" = "200" ]; then
+      echo "[golden]   upload OK (HTTP $ucode)"
+      echo "GOLDEN_VERSION=${GOLDEN_VERSION}"
+      echo "GOLDEN_SHA256=${GOLDEN_SHA256}"
+      echo "[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden ${GOLDEN_VERSION} / ${GOLDEN_SHA256}"
+    else
+      echo "[golden] WARN: golden upload failed (HTTP $ucode) — the local archive is still usable via auto-discovery."
+    fi
+  fi
+fi
+
+echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"

configs/felhom-agent-limits.conf

@@ -0,0 +1,19 @@
+# felhom-agent-limits.conf — start-limit + rollback-trigger drop-in for felhom-agent.service
+# (TASK D1). Install as /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf
+# and `systemctl daemon-reload`.
+#
+# Values are the SPIKE-agent-selfupdate-2026-07-05 tuned set, verbatim [SF-2]: with the unit's
+# Restart=on-failure + RestartSec=5s and systemd 257's compiled defaults (10s/5), a crash-looping
+# binary NEVER trips the start limit and loops forever (35 starts/180s measured). 120s/4 gives a
+# terminal `failed` ≈20s after the first crash.
+#
+# PLACEMENT TRAP [SF-3]: these keys MUST be in [Unit]. On systemd 257 a [Service] placement is
+# HALF-APPLIED — StartLimitBurst is accepted while StartLimitIntervalSec is silently ignored
+# (journal warning only). Never split them; never put them in [Service].
+#
+# OnFailure fires on EVERY crash on systemd 257 [SF-1] — see the comment block in
+# felhom-agent-rollback.service for why that is safe (pending-marker guard).
+[Unit]
+StartLimitIntervalSec=120
+StartLimitBurst=4
+OnFailure=felhom-agent-rollback.service

configs/felhom-agent-rollback.service

@@ -0,0 +1,21 @@
+# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping agent
+# self-update (TASK D1; SPIKE-agent-selfupdate-2026-07-05).
+#
+# Install as /etc/systemd/system/felhom-agent-rollback.service. It is referenced by the
+# felhom-agent-limits.conf drop-in's OnFailure= line.
+#
+# THE PER-CRASH-FIRING REALITY [SF-1]: on systemd 257, OnFailure= fires on EVERY crash of the main
+# unit — even while it is merely `activating (auto-restart)`, long before (and regardless of) the
+# terminal start-limit `failed` state. During one crash incident this unit therefore runs MANY
+# times. That is safe BY DESIGN: the wrapper's rollback verb is pending-marker-guarded — the first
+# fire after a bad update reverts the binary and clears the marker; every later fire (and any fire
+# with no update in flight at all) is a logged no-op that touches nothing. Consequence: a bad
+# update is rolled back at the FIRST crash (~seconds), not after the start-limit burst — the tuned
+# start-limit in the drop-in is the terminal BACKSTOP (e.g. an environmental crash loop of the
+# known-good binary), not the rollback trigger.
+[Unit]
+Description=Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/felhom-selfupdate-guarded rollback

configs/felhom-agent.service

@@ -0,0 +1,50 @@
+# felhom-agent.service — canonical systemd unit for the Felhom host agent.
+#
+# Install as /etc/systemd/system/felhom-agent.service (the host-bootstrap script installs this from
+# the Gitea-published binary; previously this unit was hand-made on each host). The agent runs as the
+# NON-root `felhom-agent` service user (the documented production model — README "Process model";
+# `privileged.mode: "sudo"`) and shells the few host-root ops out via `sudo -n` against the fixed-arg
+# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the
+# sudoers `felhom-agent ALL=(root) NOPASSWD: …` MUST name the SAME user.
+#
+# Paths MUST match what the sudoers / host-install script expect:
+#   binary  /usr/local/bin/felhom-agent
+#   config  /etc/felhom-agent/agent.json   (0600 felhom-agent:felhom-agent — secrets live here)
+#   state   /var/lib/felhom-agent          (nonces, local-api cert/key/tokens, staged units, guests)
+#
+# === DELIBERATELY NO SANDBOXING — read before adding any hardening directive ===
+#
+# 1. NoNewPrivileges is NOT set. It is INCOMPATIBLE with the agent's privilege model: it blocks the
+#    setuid `sudo` the agent relies on for EVERY host-root op (mount, format, pct, dnsmasq …), so the
+#    agent would silently lose all privileged capability. The narrow surface comes from the sudoers
+#    fixed-arg allowlist + the agent's in-process fine validation (internal/storage/validate.go), NOT
+#    from NoNewPrivileges. Do not add it.
+#
+# 2. NO mount-namespacing hardening (ProtectHome, ProtectSystem, PrivateTmp, ReadOnlyPaths,
+#    ProtectControlGroups, …). Any of these give the unit a PRIVATE mount namespace — and the agent's
+#    intermediary-mount drive model does `mount --make-shared /mnt/felhom-drives` + `mount --bind` and
+#    relies on those propagating into the RUNNING customer guest. In a private namespace the binds
+#    would be invisible to the host/guest and every external-drive enrollment would silently break.
+#    The agent MUST share the host mount namespace. The security boundary is the sudoers allowlist.
+
+[Unit]
+Description=Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)
+Documentation=https://gitea.dooplex.hu/admin/felhom-agent
+After=network-online.target pve-cluster.service pveproxy.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=felhom-agent
+Group=felhom-agent
+ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
+Restart=on-failure
+RestartSec=5s
+
+# State dir: created 0750 felhom-agent:felhom-agent on start if absent (local-api cert/key/tokens,
+# nonces, staged .mount units, per-guest bootstrap dirs all live here).
+StateDirectory=felhom-agent
+StateDirectoryMode=0750
+
+[Install]
+WantedBy=multi-user.target

configs/felhom-agent.sudoers

@@ -0,0 +1,308 @@
+# felhom-agent sudoers allowlist — the NARROW host-root surface (slice 5 Phase B, doc 03 §3/§7).
+#
+# Install as a drop-in: /etc/sudoers.d/felhom-agent (mode 0440, root:root), validated with
+# `visudo -cf`. The agent runs as the non-root `felhom-agent` service user and shells out via
+# `sudo -n` with FIXED argument vectors (no shell). The fine-grained validation is done IN
+# the agent BEFORE exec (internal/storage/validate.go): UUIDs against a strict hex regex,
+# mount paths confined+traversal-checked, SMART devices whitelisted to raw disks, LVM names
+# charset-checked. These sudoers wildcards are the COARSE allowlist; the agent is the fine
+# gate, so a wildcard can never be abused by a value the agent didn't already validate.
+#
+# Binary paths MUST match the agent config (privileged.systemctl/install/smartctl/lvs). Adjust
+# for your distro (Debian/PVE shown). A missing/declined entry degrades the agent with a
+# warning (SMART→UNKNOWN, mount→logged error), it does not crash.
+
+Cmnd_Alias FELHOM_MOUNT = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.mount, \
+  /usr/bin/systemctl daemon-reload, \
+  /usr/bin/systemctl enable --now -- *.mount, \
+  /usr/bin/systemctl disable -- *.mount, \
+  /usr/bin/systemctl stop -- *.mount
+
+Cmnd_Alias FELHOM_DISK = \
+  /usr/sbin/smartctl -a -j /dev/sd[a-z]*, \
+  /usr/sbin/smartctl -a -j /dev/nvme[0-9]*n[0-9]*, \
+  /usr/sbin/smartctl -a -j /dev/vd[a-z]*, \
+  /usr/sbin/smartctl -a -j /dev/hd[a-z]*, \
+  /usr/sbin/lvs --reportformat json --units b -o lv_name\,data_percent\,metadata_percent -- *, \
+  /usr/sbin/pvs --reportformat json --noheadings -o pv_name, \
+  /usr/sbin/zpool status -P
+
+# Provisioning back-half (slice 8A, doc 03 §6): populate a guest's bootstrap config mount
+# host-side (internal/provision). These are host-root ops the API token cannot do — a bind mount
+# is root@pam-only, and the chown maps the 0600 bootstrap.json to the unprivileged-LXC guest-root
+# (uid/gid 100000, spike gotcha 1). The host dir is AGENT-OWNED state under /var/lib/felhom-agent/
+# (the wildcard only ever names a path the agent itself created), and the bootstrap file the agent
+# writes there is the only thing these touch. ':' is escaped per sudoers grammar.
+Cmnd_Alias FELHOM_PROVISION = \
+  /usr/bin/chown -R 100000\:100000 /var/lib/felhom-agent/guests/*, \
+  /usr/sbin/pct set [0-9]* -mp[0-9]* /var/lib/felhom-agent/guests/*, \
+  /usr/sbin/pct set [0-9]* -onboot 1
+
+# Disk inspection + format (slice 8C + Impl-1). blkid/lsblk read the device's data-bearing evidence
+# (the agent decides data-bearing-ness from THIS, never the caller's claim). Format goes ONLY through
+# felhom-mkfs-guarded (Impl-1 Part B): raw mkfs.* is NO LONGER allowlisted, so even a bad agent cannot
+# mkfs the OS disk — the wrapper re-checks the catastrophic cases (system disk / LVM PV / foreign mount)
+# as root and refuses, and the agent's unclaimed-disk filter (claim.go) is the primary guard above it.
+Cmnd_Alias FELHOM_FORMAT = \
+  /usr/sbin/blkid -p -o export /dev/*, \
+  /usr/bin/lsblk -J -o NAME\,FSTYPE\,PTTYPE\,MOUNTPOINT /dev/*, \
+  /usr/local/sbin/felhom-mkfs-guarded /dev/* *
+
+# LAN split-horizon resolver (internal/lanresolver): the agent manages a host-side dnsmasq that
+# answers *.<customer-domain> with each guest's live LAN IP. install only ever writes felhom-*.conf
+# drop-ins (from agent-written /tmp temp files); the two `pct exec` reads are FIXED command vectors
+# (the guest's eth0 IPv4 + the controller's pulled controller.yaml for the domain) — NOT a general
+# `pct exec`. systemctl is scoped to the dnsmasq unit only. The agent never edits /etc/resolv.conf.
+Cmnd_Alias FELHOM_DNSMASQ = \
+  /usr/bin/apt-get install -y -q dnsmasq, \
+  /usr/bin/install -m 0644 /tmp/felhom-resolver-*.conf /etc/dnsmasq.d/felhom-*.conf, \
+  /usr/bin/systemctl enable --now dnsmasq, \
+  /usr/bin/systemctl reload dnsmasq, \
+  /usr/bin/systemctl restart dnsmasq, \
+  /usr/bin/rm -f /etc/dnsmasq.d/felhom-*.conf, \
+  /usr/sbin/pct exec [0-9]* -- ip -4 -o addr show dev eth0, \
+  /usr/sbin/pct exec [0-9]* -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
+
+# Guest mountpoint lifecycle (intermediary-mount re-architecture + C1 net). The pre-start self-heal hook
+# wrapper is installed once into the PVE snippets dir (from an agent-written /tmp file) and registered
+# per-guest; decommission/eject DELETE the dead mountpoint slot so a missing bind source can't brick the
+# guest at next boot (the B3 C1 fix). The agent fine-validates the vmid (numeric) + slot (mp[0-9]+) and
+# the snippet path is fixed — the wildcards are the coarse allowlist. The install SOURCE is a
+# random-named agent temp (os.CreateTemp, audit B1 — a fixed /tmp name was a local TOCTOU), hence the
+# glob; the DESTINATION stays pinned. The `mkdir -p` creates the snippets dir on a FRESH box —
+# `install` won't create parents, so without it the hook install failed silently on Day-0 boxes
+# (B2, DRILL-day0-cleanroom-2026-07-03; fixed agent v0.63.0).
+Cmnd_Alias FELHOM_GUESTHOOK = \
+  /usr/bin/mkdir -p /var/lib/vz/snippets, \
+  /usr/bin/install -m 0755 -- /tmp/felhom-guest-hook-*.sh /var/lib/vz/snippets/felhom-guest-hook.sh, \
+  /usr/sbin/pct set [0-9]* --hookscript local\:snippets/felhom-guest-hook.sh, \
+  /usr/sbin/pct set [0-9]* --delete mp[0-9]*, \
+  /usr/sbin/pct reboot [0-9]*
+
+# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
+# /mnt/felhom-drives (self-bind + make-shared + a boot-persistence systemd unit) and binds/unbinds each
+# drive's felhom-data namespace UNDERNEATH it so the change propagates into the running guest live (no
+# pct, no reboot). The agent fine-validates the drive name + confines paths before any exec; the trailing
+# `*` (matching the comma-laden mp spec) mirrors the existing FELHOM_PROVISION pattern.
+# `lxc-info -n <vmid> -p -H` resolves the guest init PID for the GuestSeesMount / bound_under_parent check
+# (a READ — the drive-gate's "is the drive live in the guest?" signal); WITHOUT it the non-root agent gets
+# an empty PID and reports every drive absent (multi-drive flapping, audit 2026-06-29). `make-private`
+# isolates the parent's peer group on FIRST setup only (EnsureSharedParent guards on mountpoint, so it
+# never re-churns a live parent); without it the parent stays in root's group and submounts double.
+Cmnd_Alias FELHOM_INTERMEDIARY = \
+  /usr/bin/mkdir -p /mnt/felhom-drives, \
+  /usr/bin/mkdir -p /mnt/felhom-drives/*, \
+  /usr/bin/mkdir -p /mnt/*/felhom-data, \
+  /usr/bin/chown 100000\:100000 /mnt/*/felhom-data, \
+  /usr/bin/mount --bind /mnt/felhom-drives /mnt/felhom-drives, \
+  /usr/bin/mount --make-shared /mnt/felhom-drives, \
+  /usr/bin/mount --make-private /mnt/felhom-drives, \
+  /usr/bin/mount --bind /mnt/*/felhom-data /mnt/felhom-drives/*, \
+  /usr/bin/umount /mnt/felhom-drives/*, \
+  /usr/bin/install -m 0755 -- /tmp/felhom-shared-parent-*.sh /usr/local/sbin/felhom-shared-parent.sh, \
+  /usr/bin/install -m 0644 -- /tmp/felhom-shared-parent-*.service /etc/systemd/system/felhom-shared-parent.service, \
+  /usr/bin/systemctl enable felhom-shared-parent.service, \
+  /usr/bin/lxc-info -n [0-9]* -p -H, \
+  /usr/sbin/pct set [0-9]* -mp8 /mnt/felhom-drives*
+
+# Controller-swap / managed auto-update (Option A, non-root). The agent owns the in-guest controller
+# image SWAP (it survives the controller being killed mid-swap): read the baked image ref, check the
+# pre-pulled target is present, rewrite /etc/felhom-controller-image, restart the bootstrap unit,
+# health-check, roll back on failure. Each grant is bounded — NO general `pct exec` and NO `bash -c`:
+#   cat <fixed file>            — read the current image ref (read-only)
+#   docker image inspect *      — is the pre-pulled target present? (read-only)
+#   docker inspect -f *         — container running/health/image (read-only; `*` spans the -f template
+#                                 + container across spaces, spike-confirmed)
+#   systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
+#   tee <FIXED image file>      — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
+#                                 the agent strict-validates the ref (controllerImageRe) before the write.
+# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
+Cmnd_Alias FELHOM_CONTROLLERSWAP = \
+  /usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image, \
+  /usr/sbin/pct exec [0-9]* -- docker image inspect *, \
+  /usr/sbin/pct exec [0-9]* -- docker inspect -f *, \
+  /usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service, \
+  /usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image
+
+# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
+# agent clears the STALE lock at startup (only when no vzdump is in-flight). `pct unlock` is the one op
+# with no API equivalent (snapshot-delete + start go through the API token); the agent fine-validates the
+# vmid (numeric) before exec — the `[0-9]*` is the coarse allowlist.
+Cmnd_Alias FELHOM_STALELOCK = \
+  /usr/sbin/pct unlock [0-9]*
+
+# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
+# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
+# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
+# out of reach (403 VM.Allocate) holding its disks until a human removes it.
+#
+# TWO API-SIDE FIXES WERE TRIED AND BOTH REFUTED LIVE on 2026-07-28, which is why this grant exists:
+#   1. Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
+#      VM.Allocate on the VM being added — pool membership cannot bootstrap its own authority.
+#   2. Grant FelhomAgentGuest per-path at /vms/990000..990009. Durable for exactly one use per slot:
+#      PVE's own destroy path calls `AccessControl::remove_vm_access($vmid)` (LXC.pm:906), which DELETES
+#      every ACL at /vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it
+#      authorises, so after ten teardowns the band is ungranted and the defect returns.
+#
+# WHY THIS IS THE TIGHTEST AVAILABLE FENCE, not a widening: sudo matches the vmid LITERALLY, so
+# `99000[0-9]` is exactly the ten-slot scratch band the restore-test picks from — nothing else. There is
+# no `[0-9]*` coarse allowlist here on purpose: unlike `pct unlock`, this op DESTROYS, so the band must
+# be in the policy and not merely validated in the agent. Even a compromised agent asking for
+# `pct destroy 9201` is refused by sudo itself. Unlike an ACL, a sudoers rule is not consumed by use.
+# The agent re-checks the band in code before exec (defence in depth); this is the outer fence.
+Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
+  /usr/sbin/pct destroy 99000[0-9] --purge
+
+# Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share
+# HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates
+# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
+# A NAS is NOT a drive — no durable-id, no SMART, no wipe; these grants only install/enable/remove the
+# unit pair. The agent fine-validates every value (share name, server, export, uid/gid, creds path) before
+# any unit is rendered (internal/storage/netmount.go ValidateNetworkMountSpec); the trailing globs are the
+# COARSE allowlist. The `.mount` install/enable/disable/stop reuse FELHOM_MOUNT; this alias adds the
+# `.automount` variants + the unit-file removal. The unit FILE name is the systemd-escaped mountpoint,
+# which always begins `mnt-felhom` (the mountpoint is /mnt/felhom-drives/<name>), so the rm glob is scoped
+# to felhom mount units only. mkdir of the mountpoint reuses FELHOM_INTERMEDIARY's /mnt/felhom-drives/*.
+# CAMPAIGN-3 additions (loud, per the no-widening rule):
+#  - `systemctl reset-failed -- mnt-felhom*`: F10 (CRITICAL) — a NAS automount that hit
+#    mount-start-limit-hit during an outage was re-armable by NO platform path; the reassert now
+#    reset-failed's the stuck unit before `enable --now` (which the start-limit otherwise refuses),
+#    and RemoveNetworkMount clears failed-state residue (F2). Scoped to felhom mount units (the unit
+#    name is the systemd-escaped mountpoint, always beginning `mnt-felhom`). reset-failed only clears
+#    a unit's failed latch — it cannot start/stop/alter anything.
+#  - `rmdir /mnt/felhom-drives/*`: F1 — remove the now-empty mountpoint dir a removed share leaves
+#    behind (the campaign accumulated 10 stub-shaped leftovers). rmdir ONLY (never rm -rf): it refuses
+#    a non-empty dir, so unexpected data is preserved, not destroyed — a fail-safe grant.
+Cmnd_Alias FELHOM_NETMOUNT = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/units/* /etc/systemd/system/*.automount, \
+  /usr/bin/systemctl enable --now -- *.automount, \
+  /usr/bin/systemctl disable -- *.automount, \
+  /usr/bin/systemctl stop -- *.automount, \
+  /usr/bin/systemctl reset-failed -- mnt-felhom*, \
+  /usr/bin/rmdir /mnt/felhom-drives/*, \
+  /usr/bin/rm -f /etc/systemd/system/mnt-felhom*
+
+# Offsite WG tunnel (S3, doc 06 §3.3). The agent manages wg-quick@wg-felhom as an agent-managed
+# host service (the dnsmasq/lanresolver shape): conf staged in the agent-owned StateDir (never
+# /tmp), installed 0600 to the FIXED destination, unit enable/restart/disable. The ONLY wg read
+# is `latest-handshakes` — `wg show <if> dump` is FORBIDDEN everywhere (its interface line
+# carries the PRIVATE KEY; the S1 session-log incident). Both install paths are FIXED (no glob):
+# the agent has exactly one tunnel conf to manage.
+Cmnd_Alias FELHOM_WG = \
+  /usr/bin/apt-get install -y -q wireguard-tools, \
+  /usr/bin/install -o root -g root -m 0600 -- /var/lib/felhom-agent/wg/wg-felhom.conf /etc/wireguard/wg-felhom.conf, \
+  /usr/bin/systemctl enable --now wg-quick@wg-felhom, \
+  /usr/bin/systemctl restart wg-quick@wg-felhom, \
+  /usr/bin/systemctl disable --now wg-quick@wg-felhom, \
+  /usr/bin/wg show wg-felhom latest-handshakes
+
+# Agent self-update (TASK D1, SPIKE-agent-selfupdate-2026-07-05). The agent downloads the
+# operator-SIGNED binary (sha256 pinned in the signed op — neither hub nor Gitea compromise can
+# substitute it), verifies the sha in-process, then hands off to the guarded wrapper, which
+# RE-verifies the sha as root, confines the staged path to /var/lib/felhom-agent/selfupdate/,
+# performs the A/B flip (atomic same-fs rename, .prev retained) and schedules a detached restart.
+# The apply args are a COARSE glob (spike S4b: sudoers fnmatch makes a [a-f0-9]* sha pattern
+# first-char-only anyway) — the wrapper's own sha re-verify + path confinement is the real gate.
+# `rollback` is normally run by felhom-agent-rollback.service (root, OnFailure=), not via sudo;
+# granting it here keeps the verb probe-able (capability self-check) and operator-invokable.
+Cmnd_Alias FELHOM_SELFUPDATE = \
+  /usr/local/sbin/felhom-selfupdate-guarded apply /var/lib/felhom-agent/selfupdate/* *, \
+  /usr/local/sbin/felhom-selfupdate-guarded commit, \
+  /usr/local/sbin/felhom-selfupdate-guarded rollback
+
+# Dedicated OOB sshd (TASK H1). The agent manages felhom-sshd like wg-felhom/dnsmasq: it RENDERS the
+# config (Port from its claim) + the operator's authorized_keys, validates with `sshd -t`, and reloads
+# (never restart-on-change [SF-2]). Both install SOURCES are the agent-owned staged files under
+# StateDir; both DESTINATIONS are FIXED. `sshd -t/-T` are the validate/discover reads. The
+# systemctl verbs are SCOPED to felhom-sshd only. reset-failed precedes a deliberate restart [SF-5].
+# NOTHING here can touch the stock sshd, :22, or /etc/ssh.
+Cmnd_Alias FELHOM_SSHD = \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/sshd_config /etc/felhom-sshd/sshd_config, \
+  /usr/bin/install -o root -g root -m 0644 -- /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op /etc/felhom-sshd/authorized_keys/felhom-op, \
+  /usr/sbin/sshd -t -f /var/lib/felhom-agent/felhom-sshd/sshd_config, \
+  /usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config, \
+  /usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config, \
+  /usr/bin/systemctl enable --now felhom-sshd, \
+  /usr/bin/systemctl reload felhom-sshd, \
+  /usr/bin/systemctl restart felhom-sshd, \
+  /usr/bin/systemctl reset-failed felhom-sshd, \
+  /usr/bin/wg show wg-felhom latest-handshakes
+
+# PBS DR tier apply (slice 2, SPIKE-pbs-tier-provisioning-2026-07-10 §2b). Storage-entry
+# lifecycle is /storage-ROOT-gated in the PVE API (spike Probe 1: create/modify/delete all check
+# Datastore.Allocate on /storage), so the agent token cannot do it — this wrapper is the pinned
+# vector. THE SET-ONLY LAW: the wrapper contains NO deletion path (entry deletion destroys the
+# client encryption key = un-decryptable backups); verbs are create/reconcile/grant only. The
+# token secret rides the wrapper's STDIN — sudo logs argv, so it must never appear here. The
+# agent fine-validates every field (charset + descriptor equality) before exec; these globs are
+# the coarse allowlist.
+#
+# `read` (R-39 leg b, agent v0.91.0) is the ONE added verb. It prints a token secret to stdout and
+# performs no mutation. It exists because the agent writes that file through this wrapper but could
+# never read it back (/etc/pve/priv is 0700 root:www-data), leaving its PBS verify loop permanently
+# blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory
+# and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to
+# STDOUT, never argv — sudo logs argv.
+# E-2a: the backup-target storage shim. Creating a PVE storage needs Datastore.Allocate at /storage
+# and the grant needs Permissions.Modify -- the agent holds NEITHER by design (blast-radius
+# containment; Permissions.Modify would let it rewrite its own authority). Both live behind this
+# fixed-vocabulary root shim instead, exactly like the mkfs and pbs-apply wrappers. The wrapper has
+# NO storage-removal path, enforces is_mountpoint 1, and refuses a target on the root device.
+Cmnd_Alias FELHOM_BACKUPTARGET = \
+  /usr/local/sbin/felhom-backup-target-apply create *, \
+  /usr/local/sbin/felhom-backup-target-apply grant *
+
+Cmnd_Alias FELHOM_PBSDR = \
+  /usr/local/sbin/felhom-pbs-apply create *, \
+  /usr/local/sbin/felhom-pbs-apply reconcile *, \
+  /usr/local/sbin/felhom-pbs-apply grant *, \
+  /usr/local/sbin/felhom-pbs-apply read *
+
+# OOB nft belt (TASK H1). The STATIC table `inet felhom_oob` is installed once by host-install; the
+# agent mutates ONLY its two SETS — @operator_ips (the operator /32) + @ssh_port (the claimed port).
+# SET ELEMENTS ONLY [trap 4]: NO `nft add rule`, NO `nft -f`, NO `flush ruleset/table` — a rule grant
+# would let the agent firewall anything. The agent fine-validates every element (netip / int range)
+# before exec; the trailing wildcards are the coarse allowlist (values only).
+Cmnd_Alias FELHOM_OOB = \
+  /usr/sbin/nft list set inet felhom_oob operator_ips, \
+  /usr/sbin/nft list set inet felhom_oob ssh_port, \
+  /usr/sbin/nft flush set inet felhom_oob operator_ips, \
+  /usr/sbin/nft flush set inet felhom_oob ssh_port, \
+  /usr/sbin/nft add element inet felhom_oob operator_ips *, \
+  /usr/sbin/nft add element inet felhom_oob ssh_port *
+
+# Escrow ceremony (controller-driven, TASK 2026-07-13; mechanics validated by
+# SPIKE-controller-escrow-2026-07-13). ONE fixed argv — sudoers matches the argument vector
+# byte-for-byte (spike §2.2): any alteration (value, extra flag, order, config path) is refused.
+# --config pinned: env_reset strips FELHOM_AGENT_CONFIG and the pin closes alternate-config
+# injection. R rides the subprocess stdout pipe only; sudo logs argv = secrets-free. The argv
+# MUST stay byte-identical to escrow.CeremonyArgs() (internal/escrow/ceremony.go) — the
+# capability manifest entry + TestManifestCoveredBySudoers lock the three copies together.
+Cmnd_Alias FELHOM_ESCROW = \
+  /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload --output=json
+
+# Node self-heal (CAMPAIGN-3 Part 6, F12-class defense in depth). The ONE fixed unit the appliance
+# watchdog may (re)start when a boot leaves networking down — the exact command the morning recovery
+# ran by hand after the F12 host loss. FIXED unit, no glob: this grant alone cannot harm — starting
+# networking.service is precisely what the boot should have done. The remedy is ALSO code-gated on
+# deployment_mode="appliance" (the Manager refuses to invoke it on a byo host); the sudoers grant is
+# the coarse floor, the mode gate is the fine one.
+Cmnd_Alias FELHOM_SELFHEAL = \
+  /usr/bin/systemctl start networking.service
+
+# Guest-network watchdog (internal/guestnet, R-54). The guest's DHCP client is unsupervised — when it
+# died on 2026-07-20 the box lost its address ~80 minutes later and went off the internet for 1h15m
+# (INCIDENT-guest-dhclient-killed-2026-07-20). Four FIXED read vectors plus ONE fixed heal vector; the
+# heal is the incident's own restored invocation, byte for byte. This is NOT a general `pct exec`: every
+# argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
+# the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
+# and the same command must not be granted twice under two names.
+Cmnd_Alias FELHOM_GUESTNET = \
+  /usr/sbin/pct exec [0-9]* -- ip route show default, \
+  /usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
+  /usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
+  /usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
+
+felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN

configs/felhom-backup-target-apply

@@ -0,0 +1,111 @@
+#!/bin/bash
+#===============================================================================
+# felhom-backup-target-apply — the ONLY path the felhom-agent sudoers permits for creating the
+# whole-guest backup TARGET storage and granting the agent access to it (E-2a).
+#
+# WHY A WRAPPER AT ALL. Creating a PVE storage needs `Datastore.Allocate` at `/storage`, and the ACL
+# grant needs `Permissions.Modify`. The agent holds NEITHER by design — its token is scoped per
+# storage path for blast-radius containment, and `Permissions.Modify` would let it rewrite its own
+# authority. Widening the PVE role to make the move possible would trade the entire containment model
+# for one feature. So the privileged half lives here: a minimal, auditable root shim with a fixed
+# vocabulary, exactly like felhom-mkfs-guarded and felhom-pbs-apply.
+#
+# THE NO-DELETE LAW (inherited from felhom-pbs-apply, same reasoning class). This wrapper contains NO
+# storage-removal path of any kind. `pvesm remove` on a dir storage does not delete the archives, but
+# it DOES silently orphan a configured backup tier, and a "cleanup" verb here would be reachable by
+# any bug in the agent. Retiring a target is a deliberate operator op, not this tool. Grep-assertable;
+# do not add one.
+#
+# THE TWO LAWS E-1 PAID FOR ON LIVE HARDWARE, both enforced here rather than trusted to the caller:
+#
+#   F-1  the storage path must BE the drive's own mountpoint. A subdirectory fails the agent's
+#        exactMount check, so the target reports `disconnected` FOREVER and its durable id degrades
+#        off the filesystem UUID. Enforced: `mountpoint -q` must pass on the exact path given.
+#
+#   F-2  --is_mountpoint 1 is not optional. Without it, an unplugged or late-mounting drive leaves a
+#        bare directory on the ROOT filesystem and vzdump writes the whole-guest backup onto the
+#        system drive — the exact device the whole change exists to escape — while PVE reports the
+#        storage `active` and advertises the root filesystem's free space. Proven live: the unguarded
+#        form had already created dump/ on pve-root. Hardcoded below; not a caller-supplied flag.
+#
+# Ops (all non-secret; nothing here touches a credential, so nothing arrives on stdin):
+#   create <id> <mountpoint>
+#           Create a `dir` storage with content=backup at <mountpoint>, is_mountpoint 1.
+#           IDEMPOTENT: an existing entry with the SAME path is accepted (re-run safe, and the
+#           installer re-run path depends on it). An existing entry with a DIFFERENT path is REFUSED
+#           — silently repointing a live backup target is the failure this whole arc closes.
+#   grant  <id>
+#           The dual grant: FelhomAgentStore on /storage/<id> to the agent user AND token (privsep
+#           intersection — a token's rights are the intersection, so granting one is granting neither).
+#           Without it every backup 403s on first run (E-1 finding F-3, found by the first real backup).
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-backup-target-apply: REFUSED: $*" >&2; exit 1; }
+
+op="${1:-}"; id="${2:-}"
+[[ -n "$op" && -n "$id" ]] || die "usage: felhom-backup-target-apply <create|grant> <storage-id> [mountpoint]"
+
+# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
+[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
+
+STORECFG=/etc/pve/storage.cfg
+
+# current_path_of <id> — the configured `path` of dir storage <id>, or "" when absent/not-a-dir.
+current_path_of() {
+  awk -v want="dir: $1" '
+    $0 == want { found=1; next }
+    found && /^[a-z]+: / { exit }
+    found && $1 == "path" { print $2; exit }
+  ' "$STORECFG" 2>/dev/null || true
+}
+
+case "$op" in
+create)
+  [[ $# -eq 3 ]] || die "create takes <id> <mountpoint>"
+  mp="$3"
+  # Absolute, normalized, no traversal, no shell metacharacters. The value reaches pvesm and the
+  # filesystem, so it is validated here rather than assumed well-formed.
+  [[ "$mp" = /* ]]            || die "mountpoint must be absolute ($mp)"
+  [[ "$mp" != *".."* ]]       || die "mountpoint must not contain .. ($mp)"
+  [[ "$mp" =~ ^[A-Za-z0-9/_.-]+$ ]] || die "mountpoint has unexpected characters ($mp)"
+  [[ "$mp" != "/" ]]          || die "refusing / as a backup target"
+
+  # F-1 + F-2, checked as one: the path must BE a mountpoint right now. A bare directory here is
+  # precisely the silent-retarget shape, and is_mountpoint would make PVE refuse it later anyway —
+  # better to refuse now, with a reason, than to create a storage that can never activate.
+  mountpoint -q "$mp" || die "$mp is not a mountpoint — the backup target must be the drive's OWN mountpoint (F-1), and an unmounted path would silently retarget onto the system drive (F-2)"
+
+  # Never the system disk: a target on the root filesystem is not drive-loss protection, it is the
+  # thing we are escaping. The root device and the candidate's device are compared, not their paths.
+  root_dev="$(findmnt -no SOURCE / 2>/dev/null || true)"
+  mp_dev="$(findmnt -no SOURCE "$mp" 2>/dev/null || true)"
+  [[ -n "$mp_dev" ]] || die "could not resolve the backing device of $mp"
+  [[ "$mp_dev" != "$root_dev" ]] || die "$mp is backed by the ROOT device ($root_dev) — a backup target there protects against corruption only, never drive loss"
+
+  existing="$(current_path_of "$id")"
+  if [[ -n "$existing" ]]; then
+    if [[ "$existing" == "$mp" ]]; then
+      echo "felhom-backup-target-apply: storage $id already exists at $mp — nothing to do (idempotent)" >&2
+      exit 0
+    fi
+    die "storage $id already exists at $existing — refusing to repoint it at $mp (a live backup target is never silently moved)"
+  fi
+
+  # is_mountpoint 1 is HARDCODED (F-2). content=backup only: this storage exists for vzdump archives
+  # and must never become a place guests are allocated on.
+  pvesm add dir "$id" --path "$mp" --content backup --is_mountpoint 1 >&2
+  echo "felhom-backup-target-apply: created dir storage $id at $mp (content=backup, is_mountpoint 1)" >&2
+  ;;
+grant)
+  [[ $# -eq 2 ]] || die "grant takes only <id>"
+  # BOTH, always. A privsep token's rights are the intersection of the user's and the token's ACLs,
+  # so granting one of the two grants nothing usable.
+  pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
+  pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
+  echo "felhom-backup-target-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
+  ;;
+*)
+  die "unknown op ($op)"
+  ;;
+esac

configs/felhom-localapi-firewall.example

@@ -0,0 +1,49 @@
+# felhom-agent local API — host firewall narrowing (doc 03 §6; R-50 island update 2026-07-25)
+#
+# Defense-in-depth for the per-guest local API (the controller→agent channel). The PER-GUEST BEARER
+# TOKEN + the served-leaf pin are the authorization gate; a firewall rule is only an ADDITIONAL layer
+# limiting who can even open the port.
+#
+# ===  R-50 ISLAND INSTALL (the default on a fresh appliance)  =================================
+# The agent binds local_api.listen_addr on the HOST-INTERNAL island bridge — 169.254.253.1:8443 on
+# vmbr9, a bridge with NO physical port (bridge-ports none). That bind is the security win:
+#   * Nothing listens on the LAN IP at all, so no LAN host (or off-site attacker on the LAN) can
+#     reach the local API — the LAN:8443 surface is CLOSED by the bind, not by a rule.
+#   * vmbr9 has no uplink, so 169.254.253.1:8443 is reachable ONLY from the one guest wired to the
+#     /30 (169.254.253.2) — the controller. The portless bridge is the isolation.
+# So on an island install NO firewall rule is required for exposure; the topology provides it. If you
+# want belt-and-suspenders, restrict the port to the island bridge (it changes nothing, since nothing
+# off-bridge can route to a portless bridge anyway):
+#
+#   nft add rule inet filter input iifname != "vmbr9" ip daddr 169.254.253.1 tcp dport 8443 drop
+#
+# Verify: from the guest, a TLS connect to 169.254.253.1:8443 succeeds; there is no LAN listener to
+# probe (`ss -lnt 'sport = :8443'` shows only the island IP).
+#
+# ===  LEGACY LAN BIND (byo, --no-island, or an explicit --bridge-ip)  =========================
+# When the agent still binds a LAN bridge IP (e.g. 192.168.0.162:8443), the port is exposed to the
+# whole LAN and the subnet-narrowing rule below is worth applying. Replace the bridge IP, port, and
+# the guest bridge subnet with this host's values.
+#
+# Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept:
+#
+#   nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
+#       ip saddr != 192.168.0.0/24 drop
+#   nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
+#       ip saddr 192.168.0.0/24 accept
+#
+# Option B — iptables:
+#
+#   iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT
+#   iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP
+#
+# Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled.
+# Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw):
+#
+#   [RULES]
+#   IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog
+#   IN DROP   -dport 8443 -proto tcp -log nolog
+#
+# Apply at HOST SETUP — a host-level packet-filter change, intentionally OUTSIDE the agent's
+# 3-exception privileged fence (the agent never mutates the host firewall at runtime). The token +
+# leaf-pin still gate the request regardless of which bind is in force.

configs/felhom-mgmt-watchdog.service

@@ -0,0 +1,19 @@
+# felhom-mgmt-watchdog.service (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.service.
+#
+# LAYER 2 (auto-heal without login): a DUMB oneshot that runs /usr/local/sbin/felhom-mgmt-watchdog,
+# triggered every ~60s by felhom-mgmt-watchdog.timer. It recreates a missing /run/sshd and clears a
+# start-limited stock sshd — so a management-plane lockout self-corrects in ≤1 tick with NOBODY logged
+# in and WITHOUT the felhom-agent binary running (the agent only OBSERVES/reports; the heal must not
+# depend on it — SPIKE-felhom-sshd-2026-07-05 finding #9 / TASK G1 trap 1).
+#
+# CRITICAL: this unit MUST NOT declare `RuntimeDirectory=` — that directive (with value `sshd`) is the
+# incident it exists to fix. It writes only to /run (dir + heal marker) as root; no state dir needed.
+[Unit]
+Description=Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)
+# Ordering only — the timer drives cadence; no hard dependency so it runs even in degraded boots.
+After=systemd-tmpfiles-setup.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/felhom-mgmt-watchdog
+# Deliberately NO RuntimeDirectory=, NO PrivateTmp, NO sandboxing that would hide /run/sshd from it.

configs/felhom-mgmt-watchdog.sh

@@ -0,0 +1,68 @@
+#!/bin/sh
+# felhom-mgmt-watchdog — the DUMB, agent-INDEPENDENT management-plane healer (TASK G1).
+#
+# WHY THIS EXISTS: /run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by
+# every sshd on the host. If it goes missing, sshd's listener stays up but every new session RESETS
+# right after SSH2_MSG_KEXINIT — a management-plane lockout on an otherwise-healthy box
+# (SPIKE-felhom-sshd-2026-07-05 §8, caused live by a second unit's `RuntimeDirectory=sshd` cleanup).
+# The tmpfiles.d entry (felhom-privsep.tmpfiles) PREVENTS the known cause; this script AUTO-HEALS any
+# novel cause with ZERO login and ZERO dependency on the felhom-agent binary (so it self-corrects even
+# when the agent is down — the whole point). It is run by felhom-mgmt-watchdog.timer every ~60s.
+#
+# CONTRACT (all four are load-bearing):
+#   1. STAT-FIRST / IDEMPOTENT — a HEALTHY /run/sshd is NEVER touched (no mkdir/chmod/chown call), so
+#      steady state is zero mutation and the dir's mtime is preserved. Only a MISSING or
+#      wrong-mode/owner dir is corrected.
+#   2. reset-failed the STOCK sshd ONLY when it is in the `failed` state (a start-limit lockout) —
+#      never otherwise, and NEVER `restart` it (mkdir alone restores it; PID stays, spike-proven).
+#   3. HEAL MARKER — writes an RFC3339 UTC timestamp to $MARKER only on a real heal, so the agent can
+#      REPORT the condition to the hub (a recurring clobber must surface BEFORE it becomes a lockout).
+#   4. NO `RuntimeDirectory=` anywhere in this feature (that IS the incident cause) — the unit that
+#      runs this script must not declare one either.
+#
+# Touches host /run + the stock sshd unit ONLY — no guests (pool-scoping is moot here). Fail-safe:
+# any single step failing is logged and does not abort the others (`|| true`); the next tick retries.
+
+set -u
+
+PRIVSEP=/run/sshd
+MARKER=/run/felhom-mgmt-watchdog.healed
+SSHD_UNIT=ssh.service   # Debian/PVE stock sshd unit name (sshd.service is an alias)
+healed=0
+
+log() { logger -t felhom-mgmt-watchdog "$*" 2>/dev/null || true; }
+
+# 1. Privsep dir — stat-first: create only when missing; correct mode/owner only when wrong.
+if [ ! -d "$PRIVSEP" ]; then
+	if mkdir -p "$PRIVSEP" && chown root:root "$PRIVSEP" && chmod 0755 "$PRIVSEP"; then
+		healed=1
+		log "recreated missing privsep dir $PRIVSEP (0755 root:root)"
+	else
+		log "ERROR: failed to recreate $PRIVSEP"
+	fi
+else
+	mode=$(stat -c %a "$PRIVSEP" 2>/dev/null || echo "")
+	owner=$(stat -c %U:%G "$PRIVSEP" 2>/dev/null || echo "")
+	if [ "$mode" != "755" ]; then
+		chmod 0755 "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP mode ($mode -> 755)"
+	fi
+	if [ "$owner" != "root:root" ]; then
+		chown root:root "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP owner ($owner -> root:root)"
+	fi
+fi
+
+# 2. Stock sshd — clear a start-limit lockout ONLY when the unit is genuinely `failed`. Never restart
+#    it (unnecessary — a recreated privsep dir is picked up by the next fork; the listener never died).
+if systemctl is-failed --quiet "$SSHD_UNIT" 2>/dev/null; then
+	if systemctl reset-failed "$SSHD_UNIT" 2>/dev/null; then
+		healed=1
+		log "reset-failed $SSHD_UNIT (was in the failed state)"
+	fi
+fi
+
+# 3. Heal marker — only on a real heal, so the agent reports the condition (hub raises a warning).
+if [ "$healed" = "1" ]; then
+	date -u +%Y-%m-%dT%H:%M:%SZ > "$MARKER" 2>/dev/null || true
+fi
+
+exit 0

configs/felhom-mgmt-watchdog.timer

@@ -0,0 +1,18 @@
+# felhom-mgmt-watchdog.timer (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.timer.
+#
+# Drives felhom-mgmt-watchdog.service on a ~60s cadence (the auto-heal tick). OnBootSec fires shortly
+# after boot (belt-and-suspenders with the tmpfiles layer); OnUnitActiveSec=60s gives the ≤1-tick
+# heal budget the G1 acceptance drill measures. Persistent=true runs a missed tick immediately after a
+# resume/late boot. The oneshot is idempotent, so a fast cadence never churns a healthy host.
+[Unit]
+Description=Felhom management-plane watchdog timer (~60s privsep-dir auto-heal tick)
+
+[Timer]
+OnBootSec=30s
+OnUnitActiveSec=60s
+AccuracySec=5s
+Persistent=true
+Unit=felhom-mgmt-watchdog.service
+
+[Install]
+WantedBy=timers.target

configs/felhom-mkfs-guarded.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+#===============================================================================
+# felhom-mkfs-guarded — the ONLY mkfs path the felhom-agent sudoers permits (Impl-1 Part B,
+# SPIKE-drive-enrollment-2026-07-01 §SQ3). Defense-in-depth BELOW the agent: even a buggy or
+# compromised agent cannot mkfs a catastrophic target through this — it re-checks, as root, the
+# cheap catastrophic cases (OS/system disk, LVM physical volume, a foreign mount, a read-only
+# device, and any LVM/ZFS/mdraid/LUKS/swap member signature — audit D1) and refuses.
+#
+# The agent's full unclaimed-disk filter (internal/storage/claim.go) is the PRIMARY guard; this
+# wrapper is a deliberately minimal, auditable second gate. It is NOT the place for the full filter.
+#
+# Usage:  felhom-mkfs-guarded <device> <fstype:ext4|xfs>
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-mkfs-guarded: REFUSED: $*" >&2; exit 1; }
+
+dev="${1:-}"; fstype="${2:-}"
+[[ -n "$dev" && -n "$fstype" ]] || die "usage: felhom-mkfs-guarded <device> <fstype>"
+
+# The device must be a REAL block-device node under /dev — no symlink (e.g. /dev/disk/by-*), no traversal.
+[[ "$dev" == /dev/* ]]  || die "device must be under /dev ($dev)"
+[[ "$dev" != *..* ]]    || die "path traversal ($dev)"
+[[ -b "$dev" ]]         || die "not a block device ($dev)"
+[[ ! -L "$dev" ]]       || die "device must be a real node, not a symlink ($dev)"
+
+# Whole-disk of the target (a partition's parent, else the disk itself).
+pk="$(lsblk -ndo PKNAME "$dev" 2>/dev/null || true)"
+whole="$dev"; [[ -n "$pk" ]] && whole="/dev/$pk"
+
+# 1) OS/system disk — does the target's whole-disk back /, /boot or /boot/efi?
+while read -r src mnt _rest; do
+  case "$mnt" in
+    /|/boot|/boot/efi)
+      spk="$(lsblk -ndo PKNAME "$src" 2>/dev/null || true)"
+      swhole="$src"; [[ -n "$spk" ]] && swhole="/dev/$spk"
+      [[ "$swhole" == "$whole" || "$src" == "$dev" || "$src" == "$whole" ]] && die "system/OS disk ($dev backs $mnt)"
+      ;;
+  esac
+done < /proc/mounts
+
+# 2) LVM physical volume anywhere on the target disk or its partitions. pvs is resolved by ABSOLUTE
+#    path (audit D1: `command -v pvs` silently skipped this check when pvs wasn't on the caller's
+#    PATH); if neither candidate exists, check 5's LVM2_member FSTYPE loop still catches a PV
+#    independently — pvs-absent never silently drops LVM detection.
+pvsbin=""
+for c in /usr/sbin/pvs /sbin/pvs; do
+  [[ -x "$c" ]] && { pvsbin="$c"; break; }
+done
+if [[ -n "$pvsbin" ]]; then
+  while read -r pv; do
+    pv="${pv//[[:space:]]/}"; [[ -z "$pv" ]] && continue
+    pvpk="$(lsblk -ndo PKNAME "$pv" 2>/dev/null || true)"
+    pvwhole="$pv"; [[ -n "$pvpk" ]] && pvwhole="/dev/$pvpk"
+    [[ "$pvwhole" == "$whole" ]] && die "device holds an LVM physical volume ($pv)"
+  done < <("$pvsbin" --noheadings -o pv_name 2>/dev/null || true)
+fi
+
+# 3) mounted OUTSIDE Felhom's own drive area = a live foreign filesystem → catastrophic. Mounts under
+#    /mnt/felhom-drives are our own drives (the agent detaches before a re-init) → allowed.
+while read -r mp; do
+  [[ -z "$mp" ]] && continue
+  case "$mp" in
+    /mnt/felhom-drives|/mnt/felhom-drives/*) : ;;
+    *) die "device (or a partition) is mounted at $mp ($dev)" ;;
+  esac
+done < <(lsblk -nro MOUNTPOINT "$whole" 2>/dev/null || true)
+
+# 4) read-only device (audit D1): a device the kernel marks RO is never a formattable data disk.
+wbase="${whole#/dev/}"
+rof="/sys/block/$wbase/ro"
+if [[ -r "$rof" ]]; then
+  ro="$(cat "$rof" 2>/dev/null || true)"
+  [[ "$ro" == "1" ]] && die "read-only device ($whole)"
+fi
+
+# 5) member/active FSTYPEs anywhere on the target disk or its partitions (audit D1). Mirrors
+#    claim.go memberFSTypes exactly: a member of LVM/ZFS/mdraid/LUKS or active-swap signature is
+#    always a claim, never a plain formattable data disk. This also independently catches an LVM PV
+#    when pvs is not installed (check 2's belt-and-suspenders).
+while read -r fst; do
+  [[ -z "$fst" ]] && continue
+  case "$fst" in
+    LVM2_member|zfs_member|linux_raid_member|crypto_LUKS|swap)
+      die "device holds a $fst signature ($whole)" ;;
+  esac
+done < <(lsblk -nro FSTYPE "$whole" 2>/dev/null || true)
+
+# Passed the catastrophic checks → format. exec so the mkfs exit status is the wrapper's.
+case "$fstype" in
+  ext4) exec /usr/sbin/mkfs.ext4 -F "$dev" ;;
+  xfs)  exec /usr/sbin/mkfs.xfs  -f "$dev" ;;
+  *)    die "unsupported fstype ($fstype)" ;;
+esac

configs/felhom-oob-nft.service

@@ -0,0 +1,19 @@
+# felhom-oob-nft.service (TASK H1) — install as /etc/systemd/system/felhom-oob-nft.service.
+#
+# Loads the STATIC felhom_oob belt table on boot (the agent then fills its sets each tick). Oneshot +
+# RemainAfterExit so `systemctl status` reflects "loaded". Independent of pve-firewall/nftables.service
+# (the table is a self-contained `inet felhom_oob`, additive — it never flushes the ruleset).
+[Unit]
+Description=Felhom OOB belt — load the static felhom_oob nft table (H1)
+After=network-pre.target
+Before=network.target felhom-agent.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/sbin/nft -f /etc/felhom-oob.nft
+# On stop, remove the table (best-effort) so an uninstall/disable leaves nft clean.
+ExecStop=-/usr/sbin/nft delete table inet felhom_oob
+
+[Install]
+WantedBy=multi-user.target

configs/felhom-oob.nft

@@ -0,0 +1,37 @@
+#!/usr/sbin/nft -f
+# felhom OOB belt — STATIC table (TASK H1). Installed by host-install; loaded on boot by
+# felhom-oob-nft.service. The agent mutates ONLY the two SETS (@operator_ips, @ssh_port) via the
+# narrow FELHOM_OOB sudoers — it NEVER touches the rules [trap 4], so the enforcement semantics are
+# fixed here and cannot be changed by the agent.
+#
+# Enforcement (scoped to iifname "wg-felhom" — the offsite tunnel — so :22 and every other interface
+# are UNTOUCHED [SF-3/OF-5]):
+#   - ESTABLISHED,RELATED replies always pass (covers the PBS path + reply traffic; PMTU-safe).
+#   - felhom-sshd's port is reachable from the operator /32 (in @operator_ips) over the tunnel ONLY.
+#   - any other tunnel source to that port is DROPPED at the host (defense-in-depth vs the endpoint).
+#   - the port is unreachable OFF the tunnel entirely.
+# Empty sets (before the agent's first sync) = the port rules match nothing = no enforcement yet
+# (felhom-sshd is not started until the agent renders its config either). priority -5 sits just above
+# the default; policy accept so this table only ever ADDS drops for the felhom-sshd port.
+#
+# Idempotent load: create-then-delete-then-define (a re-run/boot yields a clean table with EMPTY sets;
+# the agent refills them within one tick).
+
+table inet felhom_oob
+delete table inet felhom_oob
+
+table inet felhom_oob {
+	set operator_ips {
+		type ipv4_addr
+	}
+	set ssh_port {
+		type inet_service
+	}
+	chain input {
+		type filter hook input priority -5; policy accept;
+		ct state established,related accept
+		iifname "wg-felhom" tcp dport @ssh_port ip saddr @operator_ips accept
+		iifname "wg-felhom" tcp dport @ssh_port drop
+		tcp dport @ssh_port iifname != "wg-felhom" drop
+	}
+}

configs/felhom-op.sudoers

@@ -0,0 +1,23 @@
+# felhom-op operator sudoers (TASK H1) — install as /etc/sudoers.d/felhom-op (0440 root:root).
+#
+# The DEFAULT operator identity (login via felhom-sshd only; key in /etc/felhom-sshd/authorized_keys/
+# felhom-op, outside ~/.ssh, so the customer's sshd never honours it [SF-3]). felhom-op is a normal
+# login user with NO ambient privilege — it escalates ONLY through these explicit repair verbs, so a
+# leaked operator key's blast radius is this verb set, not the box (locked decision §6). Deep repairs
+# that genuinely need root (pct, deeper systemctl) remain available via `root` as a tunnel-gated
+# break-glass account (PermitRootLogin prohibit-password on felhom-sshd), but the day-to-day identity
+# is felhom-op.
+Cmnd_Alias FELHOM_OP_REPAIR = \
+  /usr/bin/mkdir -p /run/sshd, \
+  /usr/bin/systemctl reset-failed wg-quick@wg-felhom, \
+  /usr/bin/systemctl restart wg-quick@wg-felhom, \
+  /usr/bin/systemctl start felhom-agent, \
+  /usr/bin/systemctl restart felhom-agent, \
+  /usr/bin/systemctl reset-failed felhom-sshd, \
+  /usr/bin/systemctl restart felhom-sshd, \
+  /usr/sbin/pct list, \
+  /usr/sbin/pct start [0-9]*, \
+  /usr/sbin/pct stop [0-9]*, \
+  /usr/sbin/pct unlock [0-9]*
+
+felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR

configs/felhom-pbs-apply

@@ -0,0 +1,159 @@
+#!/bin/bash
+#===============================================================================
+# felhom-pbs-apply — the ONLY storage-entry path the felhom-agent sudoers permits for the PBS DR
+# tier (slice 2; SPIKE-pbs-tier-provisioning-2026-07-10 §2b). The guarded-mkfs shape: the agent's
+# in-process validation is the PRIMARY gate (descriptor-field charset + equality checks BEFORE
+# exec); this wrapper is the minimal, auditable second gate as root.
+#
+# THE SET-ONLY LAW (spike §4, data-loss class): `pvesm` entry deletion DESTROYS the client
+# encryption key file (<id>.enc = K) — un-decryptable backups. This wrapper therefore contains
+# NO deletion path of any kind, and a re-apply is `pvesm set`-only. Grep-assertable; do not add
+# a "cleanup" verb here, ever. Deprovision is a deliberate future operator op, not this tool.
+#
+# SECRET DISCIPLINE (spike §2b): sudo logs its full argv to auth.log → the PBS token secret
+# arrives on STDIN, never as an argument to this wrapper. Inside, it is passed to pvesm's
+# --password (root-local, transient ps exposure — the accepted spike posture); it is never
+# echoed, never written anywhere except by pvesm itself (the 0600 .pw store).
+#
+# Ops (non-secret args on argv):
+#   create    <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>
+#             secret on stdin (required). Creates the pbs entry with --encryption-key autogen
+#             (K born at /etc/pve/priv/storage/<id>.enc), then places .pw/.enc copies in
+#             <secret-dir> when it differs (the §4b WARN-fix dir; escrow-create's PBSEncKeyPath
+#             must find K there — spike §4 flag).
+#   reconcile <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>
+#             secret on stdin (optional; empty = no credential change). `pvesm set` ONLY:
+#             server/fingerprint (+ --password when a secret is fed, e.g. after a hub re-issue).
+#             NOTE datastore is deliberately NOT settable, and namespace/token-id are accepted
+#             for validation parity but NOT applied — tenancy identity is adopt-only (the
+#             demo's live entry must never be repointed at a different namespace).
+#   read      <id> <secret-dir>
+#             R-39 leg (b): print the storage's token secret to STDOUT and nothing else.
+#             The non-root agent WRITES this file through this wrapper but could never read it
+#             back (/etc/pve/priv is 0700 root:www-data and there is no read verb), so its
+#             15-minute PBS verify loop was permanently blind to the one failure it exists to
+#             catch — an `applied` tier authenticating 401. This verb is that missing read.
+#             It is deliberately the narrowest thing that works: no network, no mutation, no
+#             logging of the value, one file, prefix-asserted under the given secret dir.
+#
+#   grant     <id>
+#             The Part-0-evidenced dual-grant: FelhomAgentStore on /storage/<id> to the agent
+#             user AND token (privsep intersection). Datastore.Audit reads ride the base role.
+#===============================================================================
+set -euo pipefail
+
+die() { echo "felhom-pbs-apply: REFUSED: $*" >&2; exit 1; }
+
+op="${1:-}"; id="${2:-}"
+[[ -n "$op" && -n "$id" ]] || die "usage: felhom-pbs-apply <create|reconcile|grant|read> <storage-id> ..."
+
+# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
+[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
+
+STORECFG=/etc/pve/storage.cfg
+PRIVDIR=/etc/pve/priv/storage
+
+entry_exists() { grep -Eq "^pbs: ${id}\$" "$STORECFG"; }
+
+val_server()    { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9.-]{0,62}$ ]] || die "bad server ($1)"; }
+val_datastore() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad datastore ($1)"; }
+val_ns()        { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$ ]] || die "bad namespace ($1)"; }
+val_tok()       { [[ "$1" =~ ^[A-Za-z0-9_.-]+@[A-Za-z0-9]+![A-Za-z0-9_.-]+$ ]] || die "bad token id ($1)"; }
+val_fp()        { [[ "$1" =~ ^([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}$ ]] || die "bad fingerprint"; }
+val_sdir() {
+  case "$1" in
+    /etc/pve/priv/storage) : ;;
+    /var/lib/felhom-agent/*) [[ "$1" != *..* ]] || die "secret-dir traversal" ;;
+    *) die "secret-dir must be $PRIVDIR or under /var/lib/felhom-agent ($1)" ;;
+  esac
+}
+
+read_secret() { # → SECRET (may be empty when optional)
+  local s
+  s=$(head -c 256 || true)
+  s="${s%$'\n'}"; s="${s%$'\r'}"
+  printf '%s' "$s"
+}
+
+place_copies() { # secret-dir — the §4b WARN-fix placement (non-root agent can't read /etc/pve/priv)
+  local sdir="$1"
+  [[ "$sdir" == "$PRIVDIR" ]] && return 0
+  install -d -o felhom-agent -g felhom-agent -m 0700 "$sdir"
+  [[ -f "$PRIVDIR/$id.pw" ]] && install -o felhom-agent -g felhom-agent -m 0600 "$PRIVDIR/$id.pw" "$sdir/$id.pw"
+  # K's copy: escrow-create stats PBSEncKeyPath(<secret-dir>/<id>.enc) — the spike §4 flag.
+  [[ -f "$PRIVDIR/$id.enc" ]] && install -o root -g felhom-agent -m 0640 "$PRIVDIR/$id.enc" "$sdir/$id.enc"
+  return 0
+}
+
+case "$op" in
+create)
+  [[ $# -eq 8 ]] || die "create needs 7 args: <id> <server> <datastore> <namespace> <token-id> <fingerprint> <secret-dir>"
+  server="$3"; datastore="$4"; ns="$5"; tok="$6"; fp="$7"; sdir="$8"
+  val_server "$server"; val_datastore "$datastore"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
+  entry_exists && die "entry $id already exists (reconcile is the re-apply path — set-only law)"
+  SECRET=$(read_secret)
+  [[ -n "$SECRET" ]] || die "create requires the token secret on stdin"
+  pvesm add pbs "$id" \
+    --server "$server" --datastore "$datastore" --namespace "$ns" \
+    --username "$tok" --password "$SECRET" --fingerprint "$fp" \
+    --content backup --encryption-key autogen >&2
+  [[ -f "$PRIVDIR/$id.enc" ]] || die "pvesm add succeeded but K ($PRIVDIR/$id.enc) was not born"
+  place_copies "$sdir"
+  echo "felhom-pbs-apply: created $id (K born; encryption-key autogen)" >&2
+  ;;
+reconcile)
+  [[ $# -eq 7 ]] || die "reconcile needs 6 args: <id> <server> <namespace> <token-id> <fingerprint> <secret-dir>"
+  server="$3"; ns="$4"; tok="$5"; fp="$6"; sdir="$7"
+  val_server "$server"; val_ns "$ns"; val_tok "$tok"; val_fp "$fp"; val_sdir "$sdir"
+  entry_exists || die "entry $id does not exist (create is the fresh path)"
+  SECRET=$(read_secret)
+  # R-39 (2026-07-18): NEVER pass --server to `pvesm set`. PVE treats `server` as a CREATE-ONLY
+  # parameter and rejects the ENTIRE call — "can't change value of fixed parameter 'server'" —
+  # even when the value is byte-identical to the stored one. That made every reconcile exit 255,
+  # so each hub-re-issued one-time secret was consumed-then-burned and the tier stayed pinned to
+  # a revoked credential (401 forever). Proven live on the N100 demo host: `pvesm set <id>
+  # --server <same> --fingerprint <same>` -> rejected; the same call without --server -> rc 0.
+  # The server address is immutable by construction (relocating a PBS endpoint needs a fresh
+  # create), so there is nothing here to reconcile. Guarded by
+  # TestReconcileNeverPassesServerToPvesmSet.
+  args=(--fingerprint "$fp")
+  [[ -n "$SECRET" ]] && args+=(--password "$SECRET")
+  pvesm set "$id" "${args[@]}" >&2
+  place_copies "$sdir"
+  echo "felhom-pbs-apply: reconciled $id (set-only; tenancy identity untouched)" >&2
+  ;;
+read)
+  # R-39(b): the missing read path. Prints the secret to STDOUT, nothing else — no stderr note (it
+  # would be the only verb whose success line could be confused with the value), no mutation.
+  #
+  # Traversal is refused three times over, because this is the one verb that EXFILTRATES a file and
+  # its argv is attacker-shaped if the agent is ever compromised:
+  #   1. `id` already matched ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ at the top — it cannot start with a dot
+  #      and cannot contain a slash, so "../../etc/shadow" never reaches here;
+  #   2. val_sdir pins the directory to PRIVDIR or under /var/lib/felhom-agent, rejecting "..";
+  #   3. the RESOLVED path is prefix-asserted under that directory below, so even a future change to
+  #      either grammar cannot walk out.
+  [[ $# -eq 3 ]] || die "read needs 2 args: <id> <secret-dir>"
+  sdir="$3"
+  val_sdir "$sdir"
+  target="$sdir/$id.pw"
+  # Belt: resolve and re-check the prefix (guards a symlinked <id>.pw pointing outside the dir).
+  resolved=$(readlink -f -- "$target" 2>/dev/null || true)
+  [[ -n "$resolved" ]] || die "secret file not found ($target)"
+  case "$resolved" in
+    "$sdir"/*) : ;;
+    *) die "resolved secret path escapes the secret dir" ;;
+  esac
+  [[ -f "$resolved" ]] || die "secret file not found ($target)"
+  cat -- "$resolved"
+  ;;
+grant)
+  [[ $# -eq 2 ]] || die "grant takes only <id>"
+  pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
+  pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
+  echo "felhom-pbs-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
+  ;;
+*)
+  die "unknown op ($op)"
+  ;;
+esac

configs/felhom-privsep.tmpfiles

@@ -0,0 +1,15 @@
+# felhom privsep-dir persistence (TASK G1) — install as /etc/tmpfiles.d/felhom-privsep.conf.
+#
+# LAYER 1 (prevent): make OpenSSH's shared privilege-separation directory /run/sshd boot-persistent
+# and owned by NO systemd unit's lifecycle. This closes the exact incident cause from
+# SPIKE-felhom-sshd-2026-07-05 §8: a second sshd unit declaring `RuntimeDirectory=sshd` had systemd
+# REMOVE the shared /run/sshd on that unit's stop/failure, taking the stock sshd on :22 down with it
+# (sessions reset right after SSH2_MSG_KEXINIT). A tmpfiles.d entry recreates the dir at every boot
+# independently of any unit, so no unit's RuntimeDirectory cleanup can be the sole owner.
+#
+# systemd-tmpfiles is idempotent: `systemd-tmpfiles --create` re-run is a no-op on an existing,
+# correct dir (it only creates/fixes, never churns). Complemented at runtime by the
+# felhom-mgmt-watchdog timer (layer 2), which re-heals a dir removed AFTER boot.
+#
+# Type d = create the directory if absent (leaves an existing one, only fixing mode/owner).
+d /run/sshd 0755 root root -

configs/felhom-selfupdate-guarded

@@ -0,0 +1,139 @@
+#!/bin/sh
+# felhom-selfupdate-guarded — the ROOT half of the agent's A/B self-update (TASK D1).
+#
+# Install as /usr/local/sbin/felhom-selfupdate-guarded (0755 root:root). The non-root agent invokes
+# `apply`/`commit` via `sudo -n` (FELHOM_SELFUPDATE alias); `rollback` is ALSO the ExecStart of
+# felhom-agent-rollback.service — the OnFailure= target that auto-reverts a crash-looping update.
+#
+# Design provenance: SPIKE-agent-selfupdate-2026-07-05 (its SF-findings are cited inline). The core
+# principle: the thing that performs rollback is never the thing being updated — this wrapper +
+# systemd change almost never; the agent binary is what flips.
+#
+# Trust model: the agent verifies the download against the OPERATOR-SIGNED sha before staging; this
+# wrapper RE-verifies the same sha as root (defense in depth — the sudoers arg glob is coarse, the
+# sha check here is the real gate). Path confinement: apply only ever reads from the agent's own
+# staging dir and only ever writes the fixed live path + its siblings. NO env-overridable paths —
+# path-fixedness IS the security property (a test-mode override would be a root escalation hole).
+#
+# Verbs:
+#   apply <staged> <sha256>  stage-verify → .prev → atomic flip → pending marker → detached restart
+#   commit                   clear the pending marker (idempotent; .prev retained as a manual net)
+#   rollback                 pending-guarded revert to .prev + restart (no pending → exit 0 no-op)
+set -u
+
+BIN=/usr/local/bin/felhom-agent
+PREV=$BIN.prev
+STAGING=/var/lib/felhom-agent/selfupdate
+PENDING=$STAGING/pending.json
+UNIT=felhom-agent.service
+
+# Every refusal/decision goes to stderr AND the journal (strict rule 10).
+log() { echo "felhom-selfupdate-guarded: $*" >&2; logger -t felhom-selfupdate-guarded -- "$*" 2>/dev/null || true; }
+
+case "${1:-}" in
+
+apply)
+    staged=${2:-}; want=${3:-}
+    # [SF-7] entry sweep: a kill between staging-copy and mv leaves an orphaned temp — harmless,
+    # but sweep it so temps can never accumulate.
+    rm -f "$BIN".new.*
+
+    if [ -z "$staged" ] || [ -z "$want" ]; then
+        log "refusing apply: usage: apply <staged> <sha256>"
+        exit 2
+    fi
+    # Root-side path confinement: the staged binary MUST live in the agent's staging dir.
+    case "$staged" in
+        "$STAGING"/*) ;;
+        *) log "refusing apply: staged path outside $STAGING: $staged"; exit 1 ;;
+    esac
+    case "$staged" in
+        *..*) log "refusing apply: staged path contains '..'"; exit 1 ;;
+    esac
+    [ -f "$staged" ] || { log "refusing apply: staged file missing: $staged"; exit 1; }
+    # The sha must be 64 lowercase hex chars — anything else is refused before any hashing.
+    case "$want" in
+        *[!0-9a-f]*) log "refusing apply: sha256 is not lowercase hex"; exit 1 ;;
+    esac
+    [ "${#want}" -eq 64 ] || { log "refusing apply: sha256 must be 64 hex chars (got ${#want})"; exit 1; }
+
+    # [SF-7] sha-verify FIRST — before .prev, before any mutation (spike S3a companion ordering).
+    got=$(sha256sum "$staged" | awk '{print $1}')
+    if [ "$got" != "$want" ]; then
+        log "refusing apply: sha mismatch (got=$got want=$want)"
+        exit 1
+    fi
+
+    # Same-fs assert (§8): the atomic-rename guarantee only holds within one filesystem.
+    if [ "$(stat -c %d "$staged")" != "$(stat -c %d /usr/local/bin)" ]; then
+        log "refusing apply: staging and /usr/local/bin are on different filesystems — atomic rename impossible"
+        exit 1
+    fi
+
+    old_ver=$("$BIN" --version 2>/dev/null | awk '{print $2}')
+    [ -n "$old_ver" ] || old_ver=unknown
+    # The staged filename is felhom-agent-<version> (executor contract) — version without executing.
+    new_ver=$(basename "$staged"); new_ver=${new_ver#felhom-agent-}
+
+    cp -p "$BIN" "$PREV" || { log "apply failed: cannot snapshot current binary to .prev"; exit 1; }
+    # Stage a root-owned 0755 copy next to the live path, then ATOMIC same-fs rename.
+    if ! cp "$staged" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
+        rm -f "$BIN.new.$$"; log "apply failed: staging copy"; exit 1
+    fi
+    mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "apply failed: atomic rename"; exit 1; }
+
+    # Pending marker: written AFTER the flip — its existence means "an update is awaiting commit",
+    # which is exactly the rollback unit's trigger condition.
+    printf '{"old_version":"%s","new_version":"%s","sha256":"%s","applied_at":"%s"}\n' \
+        "$old_ver" "$new_ver" "$want" "$(date -Is)" > "$PENDING" \
+        || { log "apply failed: cannot write pending marker"; exit 1; }
+
+    # [SF-4/5] deliberate restarts consume start-limit budget — clear the counter first.
+    systemctl reset-failed "$UNIT" 2>/dev/null || true
+    # [SF-6] the spike's S2b winner, verbatim: detached transient timer OUTSIDE the agent's cgroup,
+    # so the sudo/agent caller survives to log the handoff and the restart cannot be torn down
+    # by its own requester dying.
+    systemd-run --on-active=2s --timer-property=AccuracySec=100ms systemctl restart "$UNIT" \
+        || { log "apply: flip done but detached restart scheduling FAILED — restart $UNIT manually"; exit 1; }
+    log "applied $new_ver (prev $old_ver, sha $want); detached restart scheduled"
+    ;;
+
+commit)
+    if [ ! -f "$PENDING" ]; then
+        log "commit: no pending — no-op"
+        exit 0
+    fi
+    # .prev is deliberately RETAINED (spike S3d) — a manual safety net until the next apply.
+    rm -f "$PENDING" || { log "commit failed: cannot remove pending marker"; exit 1; }
+    log "committed (pending cleared, .prev retained)"
+    ;;
+
+rollback)
+    # [SF-1] On systemd 257 OnFailure= fires on EVERY crash, so this verb runs MANY times per
+    # incident — the pending-guard makes every fire after the first a harmless no-op, and makes a
+    # crash with NO update in flight touch nothing at all (spike S1d/S3e).
+    if [ ! -f "$PENDING" ]; then
+        log "rollback: no pending update — no-op"
+        exit 0
+    fi
+    [ -f "$PREV" ] || { log "rollback FAILED: pending exists but no .prev binary"; exit 1; }
+    rm -f "$BIN".new.*
+    if ! cp "$PREV" "$BIN.new.$$" || ! chmod 0755 "$BIN.new.$$" || ! chown root:root "$BIN.new.$$"; then
+        rm -f "$BIN.new.$$"; log "rollback FAILED: staging copy"; exit 1
+    fi
+    mv "$BIN.new.$$" "$BIN" || { rm -f "$BIN.new.$$"; log "rollback FAILED: atomic rename"; exit 1; }
+    # Clear pending BEFORE the restart: once the binary is reverted, later OnFailure fires must
+    # no-op (the guard above) instead of re-copying .prev forever.
+    rm -f "$PENDING"
+    # [SF-4/5] the crash burst has been eating the start-limit budget — reset before starting.
+    systemctl reset-failed "$UNIT" 2>/dev/null || true
+    # Direct restart is correct HERE: this caller is the rollback oneshot, OUTSIDE the agent cgroup.
+    systemctl restart "$UNIT" || { log "rollback: binary reverted but restart FAILED"; exit 1; }
+    log "rolled back to previous binary and restarted $UNIT"
+    ;;
+
+*)
+    log "usage: felhom-selfupdate-guarded apply <staged> <sha256> | commit | rollback"
+    exit 2
+    ;;
+esac

configs/felhom-sshd.service

@@ -0,0 +1,28 @@
+# felhom-sshd.service (TASK H1) — install as /etc/systemd/system/felhom-sshd.service.
+#
+# The dedicated OOB sshd instance: a SECOND sshd on a claimed non-22 port, COEXISTING with the
+# customer's/stock sshd on :22 (never touched). Config is AGENT-RENDERED at /etc/felhom-sshd/sshd_config
+# (Port from the agent's claim); the agent reloads on change (never restart-on-change [SF-2]).
+#
+# CRITICAL [SF-1]: this unit MUST NOT declare `RuntimeDirectory=` — that directive (value `sshd`)
+# removed the SHARED /run/sshd and broke the stock sshd live (SPIKE-felhom-sshd §8). /run/sshd is
+# guaranteed by G1 (tmpfiles + watchdog); the ExecStartPre mkdir is a harmless extra guarantee. The
+# host-install RuntimeDirectory guard will REJECT this unit if the directive is ever added.
+[Unit]
+Description=Felhom OOB sshd (dedicated instance, agent-managed; H1)
+After=network.target
+
+[Service]
+Type=notify
+ExecStartPre=-/usr/bin/mkdir -p /run/sshd
+ExecStartPre=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
+ExecStart=/usr/sbin/sshd -D -f /etc/felhom-sshd/sshd_config
+# Config-change path: validate then HUP — a bad reload is REFUSED and the running daemon SURVIVES
+# [SF-2]. The agent always writes→sshd -t→reload; it never `restart`s on a config change.
+ExecReload=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
+ExecReload=/bin/kill -HUP $MAINPID
+KillMode=process
+Restart=on-failure
+
+[Install]
+WantedBy=multi-user.target

docs/architecture/01-topology-and-trust.md

@@ -1,224 +0,0 @@
-# Felhom Controller Architecture — Part 1: Topology & Trust
-
-**Status:** draft (decisions from the topology/trust design sessions).
-**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
-records *Felhom's decisions*, not Proxmox behaviour.
-
----
-
-## 1. Model at a glance
-
-Three components. **Control is always box-initiated** — the hub never connects *into* a
-customer box.
-
-```
-        operator side                     customer box (per Proxmox host)
-   ┌───────────────────┐         ┌───────────────────────────────────────────┐
-   │       HUB         │         │  Proxmox host                              │
-   │ (dooplex.hu, k3s) │         │   ┌──────────────┐                         │
-   │  - report sink    │◀──poll──┤   │  HOST AGENT  │  operator-tier          │
-   │  - signed jobs    │  signed │   │  (Proxmox    │  • all Proxmox ops      │
-   │  - dashboard      │  jobs   │   │   token)     │  • provision / restore  │
-   │  - customer record│         │   └──────┬───────┘  • storage mgmt         │
-   │  - PBS namespace  │         │          │ local constrained API           │
-   └─────────▲─────────┘         │   ┌──────▼───────────────────────────────┐ │
-             │                   │   │  customer LXC (one per customer)      │ │
-             │  direct, app-     │   │   ┌──────────────┐   Docker:          │ │
-             └───────────────────┼───┤   │ IN-GUEST     │   [app] [app] ...  │ │
-                domain reports   │   │   │ CONTROLLER   │   (Docker containers)│
-                                 │   │   │ (Docker-only)│                    │ │
-                                 │   │   └──────────────┘                    │ │
-                                 │   └───────────────────────────────────────┘ │
-                                 └───────────────────────────────────────────┘
-   PBS (offsite) ◀── outbound, client-side-encrypted backups ── customer box
-   end-users / customer ◀── Cloudflare Tunnel ── apps + controller UI
-```
-
----
-
-## 2. The customer node
-
-- One **Proxmox host** per box (PVE 9.2, Debian 13, LVM-thin).
-- **Default workload topology:** one **customer LXC**, Docker inside it, each app a Docker
-  container/stack. Apps are isolated at the Docker layer (separate containers, networks,
-  volumes, cgroup limits); they share one LXC/kernel/Docker daemon.
-- **Escape hatch:** promote an individual app to its own guest (LXC or VM) only for a
-  specific reason — a non-Linux/Windows app, a genuinely untrusted or exposed app needing
-  hard isolation, or a resource hog needing guarantees.
-- **Multi-tenant:** one customer per host is the home default; multiple customer LXCs on
-  one host (a company environment) is **not precluded** — the agent manages a *set* of
-  guests. The only multi-tenant-specific work deferred to "if it becomes real" is resource
-  fairness (per-guest disk/RAM/CPU quotas).
-
----
-
-## 3. Components & responsibilities
-
-| | **Hub** | **Host agent** | **In-guest controller** |
-|---|---|---|---|
-| Runs on | dooplex.hu (k3s) | the Proxmox host | the customer LXC |
-| Tier | operator backend | operator (high-privilege) | customer-facing (app) |
-| Holds | customer records, signed-job source, PBS namespaces, escrowed keys | the **only** Proxmox API token; per-host operator identity | **no Proxmox creds**; its own hub API key + a local-API token to the agent |
-| Does | reporting sink, dashboard, job queue, source of durable truth | all Proxmox ops (provision, restore, snapshot, backup, storage mgmt, LXC lifecycle); polls hub for signed jobs; exposes a constrained local API to the controller; **per-guest authorization gate** | Docker/app lifecycle, catalog deploy, customer UI, app-level (data-layer) backup; reports app-domain to the hub directly |
-| Never does | initiate a connection *into* a box | — | touch the Proxmox API directly |
-
-**Key separation:** the controller manages Docker; the agent manages Proxmox. The controller's
-only path to guest-level operations (snapshot-before-deploy, "grow my RAM") is a constrained
-**local API call to the agent**, which the agent authorizes (scoped to that controller's own
-guest) and executes with its operator-tier token. This consolidates all Proxmox access and
-all per-guest authorization in one auditable place and leaves the guest with zero Proxmox
-credentials.
-
----
-
-## 4. Control plane — box-initiated
-
-- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
-  Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
-  box exposes no control endpoint at all.
-- The agent and the controller **poll** the hub; the hub never initiates inbound.
-- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
-  before executing, so a compromised hub database alone cannot forge commands.
-- All operator-initiated actions are recorded in a **customer-visible audit log**.
-
----
-
-## 5. Trust boundaries
-
-| Boundary | What crosses | Mechanism | Blast radius if breached |
-|---|---|---|---|
-| end-user ↔ apps | app traffic | Cloudflare Tunnel → Traefik (Host routing) | that app |
-| customer ↔ controller UI | management UI | Cloudflare Tunnel; UI auth (bcrypt) | the customer's own box |
-| controller ↔ agent | snapshot/resize/backup requests | local constrained RPC; agent authorizes per-guest | the controller's own guest only |
-| agent ↔ hub | reports + signed jobs | outbound poll; signed jobs | one box; signed jobs limit forgery |
-| controller ↔ hub | app-domain reports/jobs (incl. geo desired-state) | outbound, own API key | app-domain of one customer |
-| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) |
-| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
-| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
-
----
-
-## 6. Enrollment & identity
-
-- **Physical presence at provisioning** (on-site install, or pre-imaged-and-delivered).
-  This removes any zero-touch remote-enrollment problem.
-- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
-  config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
-  case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
-  everything downstream runs on durable credentials, so retries don't need it.
-- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
-  operator-tier Proxmox token), then provisions the customer LXC from the golden template and
-  deploys the controller into it — injecting the controller's hub API key and its local-API
-  token. The controller is the agent's product, never the other way around.
-- The **hub customer record is the durable source of truth**, and it survives box loss:
-  identity, domain, **Cloudflare tunnel token**, **PBS namespace**, **storage manifest**, a
-  **mirrored app inventory** (bottom-up reality, not operator-declared intent — apps themselves
-  restore from the PBS guest snapshot, never re-deployed from this record; see `05` §1/§9), and the
-  **escrowed (zero-knowledge) backup key**. This is what makes hardware replacement possible.
-
----
-
-## 7. Networking
-
-- **Cloudflare Tunnel** provides inbound access to apps and the controller UI (the CGNAT
-  solution). Tunnel token lives in the hub record → **reused on new hardware during DR**, so
-  DNS/routing stay intact through an outage.
-- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
-  endpoint exists in the chosen model.
-- **Tunnel placement: host** (resolved, Part 3 §3/§5). `cloudflared` runs on the Proxmox host
-  as its own **agent-managed systemd service** — not inside the guest — so the data path
-  survives control-plane death by construction. Geo-restriction WAF is **hub-enforced** (the
-  hub holds the CF API token; the controller only reports geo desired-state).
-
----
-
-## 8. Storage & backup
-
-**Tiers** (escalating failure scope):
-
-| Layer | Mechanism | Survives | Note |
-|---|---|---|---|
-| Snapshot | LVM-thin snapshot (transient) | *logical* loss only | whole-LXC rollback; **not a backup** |
-| Local — second storage | vzdump to `dir`/`nfs`/`cifs` | primary-disk failure (USB) / box death (NAS) | first *real* backup tier |
-| Offsite — PBS | dedup'd, incremental, encrypted | site loss | the DR substrate; paid tier |
-
-- **Storage manifest** (hub-held, agent-reconciled): per target → type, durable identity
-  (UUID / `server:/export` / repo+fingerprint), **class** (fast/slow + rough IOPS, set once
-  at attach), role, encrypted credentials, schedule/retention. The agent creates the Proxmox
-  storages, continuously checks presence/reachability, and reports per-target status (a
-  disconnected target → actionable notification).
-- **App data placement is per-volume, not per-app:** `.felhom.yml` classifies each volume
-  **hot** (DB/config/cache → fast storage, enforced) vs **bulk** (media/files → may be slow).
-  A photo app's DB stays on SSD while its blobs go to the USB.
-- **Backup scoping:** hot data (LXC rootfs) rides the guest `vzdump` → tiers + PBS. Bulk data
-  on external mount points is **excluded** from the guest vzdump (per-mount `backup` flag) and
-  gets its own per-volume policy (file-level to a tier, slower cadence — or explicitly *not*
-  backed up for re-downloadable content, with the customer informed).
-- **Tiers double as the DR restore-source priority:** restore from the fastest *surviving*
-  source (local if still attachable, PBS on true site loss).
-- **Key custody (zero-knowledge default):** three tiers the customer chooses —
-  *customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
-  the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
-  code** the operator can't open; DR needs the customer's code. Access-notification is an
-  audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
-  keyfile passphrase.)
-
----
-
-## 9. Disaster recovery
-
-- **Guest-loss (host + agent alive):** the agent restores the guest from the fastest
-  surviving tier, **resets identity** (MAC/hostname — see `proxmox-platform.md`), boots it,
-  controller returns. Validated mechanics: Phase 2.
-- **Host / hardware-loss (agent gone):** re-provision (§6) in **restore mode** — the hub,
-  knowing the customer has PBS backups, hands the freshly-enrolled agent the existing identity
-  + PBS namespace + a restore directive instead of a clean-provision directive. The agent
-  restores from PBS; the controller returns on the same domain (tunnel reused from the hub
-  record). DR = provisioning + a restore mode, not a separate mechanism.
-- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
-  post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
-
----
-
-## 10. How this embodies the product values
-
-- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it.
-- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise
-  alone can't forge commands.
-- **Customer-visible audit log** — every operator action is visible to the customer.
-- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
-  support, new deployments); the customer's data and deployed apps remain recoverable by the
-  customer (recovery code), with nothing locked behind the operator.
-
----
-
-## 11. Open sub-decisions (carried into later parts)
-
-- **RTO/RPO targets** → drive the backup + offsite-replication schedule (§8).
-- Offboarding / decommission (scenario 6) — not yet designed; must honour "never hold data
-  hostage" in credential revocation + data hand-off.
-- Multi-tenant resource fairness — deferred until multi-tenant is real (§2).
-
----
-
-## Appendix — relationship to the spike
-
-- **Phase 0** → §2: LXC-default for the workload; overhead numbers.
-- **Phase 1** → §3/§5: validated the privilege boundary (create/allocate is operator-tier).
-  The guest-side scoped-backup-token it proved possible is **not** used — we chose the
-  agent-mediated path — but it confirmed restore = operator-tier, which shapes the agent.
-- **Phase 2** → §8/§9: backup→restore round-trip; identity reset on restore.
-
----
-
-## Changelog — design-review + Phase-3 fold-in (2026-06-08)
-
-- §5 trust boundaries: **added `hub ↔ Cloudflare API`** row (hub holds the CF token, enforces
-  geo→WAF); controller↔hub row notes it carries geo desired-state (S4).
-- §7 networking: **tunnel placement resolved → host** (agent-managed systemd service); geo is
-  hub-enforced (S4/S5).
-- §11 open items: removed the now-resolved **tunnel placement** and **self-update flow** entries
-  (S5; self-update designed in 03 §11).
-- §6 durable record: **"declarative app inventory" → "mirrored app inventory"** — aligns the wording
-  with the locked two-driver model (`05` §1: apps are bottom-up mirror, never operator-declared;
-  `05` §9: apps restore from the PBS guest snapshot, not re-deployed from this record).

docs/architecture/02-controller-module-map.md

@@ -1,374 +0,0 @@
-# Felhom Controller Architecture — Part 2: Controller Module Map
-
-**Status:** audit (keep / port / delete / modify / add), grounded in the v0.33 source.
-**Subject:** the v0.33 controller in `felhom-controller/controller/` (110 `.go` files,
-~40 K LOC) audited against [01-topology-and-trust.md](01-topology-and-trust.md) and
-[../proxmox-platform.md](../proxmox-platform.md).
-
-> This is a **planning map, not the port.** No controller code was changed. Source
-> citations use `controller/internal/...:line` (a different repo, so links are not
-> clickable). Classifications reflect the **target model**: the in-guest controller is
-> **Docker-only and holds no Proxmox credentials**; everything host/disk/Proxmox moves to
-> a new **host agent** (out of scope here); the controller reaches the agent through a
-> constrained **local API**.
-
-## Classification scheme
-**KEEP** (host-agnostic, ~unchanged) · **PORT** (survives, needs rework) ·
-**DELETE (→agent)** (responsibility moves to the host agent) ·
-**DELETE (obsolete)** (no longer needed) · **MODIFY** (stays, materially changes) ·
-**NEW** (no v0.33 equivalent).
-Risk tags: **clean** · **needs-rework** · **hazard** (entangles a delete-target with a keep/port target).
-
----
-
-## 0. Executive summary
-
-- The **app domain is largely intact and portable**: stack lifecycle (`stacks/`), catalog
-  git-sync (`sync/`), app-to-app integrations (`integrations/`), `.fab` export/import
-  (`appexport/`), the scheduler, crypto, asset sync, the hub report/notify *channels*, and
-  most of the web UI **KEEP/PORT cleanly**.
-- The **disk/storage/host half deletes wholesale to the agent**: all of `storage/`,
-  `monitor/watchdog.go`, the restic/cross-drive/disk-layout/drive-mount parts of `backup/`,
-  `report/infra_backup*`+`infra_pull`, and the host-physical parts of `system/`.
-- The **setup wizard (`setup/`) is obsolete** — the agent provisions the controller.
-- **The single biggest hazard is `backup/`**: the keep side (DB dumps, Docker-volume
-  archive, per-app restore — needed by `appexport/` and the backup UI) and the delete side
-  (restic, cross-drive, drive-mount) are **interleaved inside the same files**
-  (`backup.go`, `restore.go`, `paths.go`), not cleanly file-separated. Extracting the
-  app-data-backup subset into a clean retained package is the critical refactor.
-- **Intent-vs-reality corrections** (vs the task's provisional split): `monitor/pinger.go`
-  is already **dead** (legacy Healthchecks.io, "deprecated… now handled by Hub" per
-  `main.go`) → DELETE(obsolete), not keep. `backup.go`/`restore.go`/`paths.go` do **not**
-  split on file boundaries — they split *within* the file. `settings/` is **not** pure app
-  domain — it stores disk/disconnect/decommission state. `system/` is genuinely
-  mixed-per-function, not per-file.
-
----
-
-## 1. v0.33 module inventory (package → purpose, key deps)
-
-| Package | Purpose | Key internal deps |
-|---|---|---|
-| `cmd/controller/main.go` | Entry point; wires all subsystems; 6 adapters break import cycles; branches into setup mode | imports **every** package |
-| `api/` | REST API (`router.go`) + geo endpoints (`geo.go`) | stacks, backup, metrics, notify, selfupdate, sync, system, assets, integrations, cloudflare, config, settings |
-| `appexport/` | `.fab` app export/import (config+DB+volumes, AES-256-CTR+scrypt) | **backup** (DB dump), (provider iface → stacks) |
-| `assets/` | Download/cache app assets from Hub API | — (HTTP only) |
-| `backup/` | DB dumps, Docker-volume archive, **restic**, **cross-drive rsync**, per-app restore, **drive mount**, disk-layout, infra-backup metadata | config, monitor, settings, system, util |
-| `cloudflare/` | Geo-restriction via Cloudflare WAF (zone/waf/geosync/countries) — **enforcement → hub** (S4) | settings |
-| `config/` | `controller.yaml` schema + load | — |
-| `crypto/` | AES-256-GCM for app.yaml secrets | — |
-| `integrations/` | App-to-app (OnlyOffice→FileBrowser/Nextcloud) via docker exec / config patch | stacks, crypto, settings |
-| `metrics/` | SQLite time-series: system + container metrics, log scan | system |
-| `monitor/` | App health (`healthcheck`,`pinger`) + **storage/USB watchdog** | config, notify, settings, system |
-| `notify/` | Hub event push (direct, own API key) | settings |
-| `recovery/` | Generate `recovery-info.txt` (DR guide) | — |
-| `report/` | Build+push hub report; **infra-backup payload**; **recovery pull** | backup, config, metrics, monitor, scheduler, settings, stacks, system |
-| `scheduler/` | Cron/interval jobs, Budapest TZ | — |
-| `selftest/` | Startup checks (docker/dirs/catalog/hub/**restic repos**/mountpoint) | backup, config, settings, system |
-| `selfupdate/` | Self-update: pull image, edit compose, `up -d` | config |
-| `settings/` | `settings.json` persistent state: **storage paths/disconnect/decommission**, cross-drive cfg, notif prefs, geo, integration state, DB-validation cache | — |
-| `setup/` | **First-run wizard** (scan drives, hub-restore, manual config) | backup, config, report, settings, web |
-| `stacks/` | Docker Compose lifecycle, deploy + memory validation, metadata (`.felhom.yml`), HDD-data delete | config, crypto, system |
-| `storage/` | **Physical disk** scan/format/attach/mount/migrate/fstab/safety | backup, settings, util |
-| `sync/` | Catalog git-sync (pull templates) | config |
-| `system/` | Resource info: mem/cpu/load (guest) + **temp/disk-model/USB/mount topology (host)** | — |
-| `util/` | String helper | — |
-| `web/` | Hungarian dashboard: pages, auth, deploy, backup UI, **storage/disk UI**, DR restore UI, export UI, debug | appexport, backup, config, crypto, integrations, monitor, notify, scheduler, selfupdate, settings, stacks, storage, system |
-
----
-
-## 2. Classification table (per package/file)
-
-### `cmd/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `cmd/controller/main.go` | **MODIFY** | Wiring stays, but drop the setup-mode branch, the storage/watchdog/drive-migrator/restic/cross-drive/infra-backup wiring, and add the **agent local-API client**. 6 adapters shrink. | hazard |
-
-### `api/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `api/router.go` | **PORT/MODIFY** | Keep stacks/deploy/integrations/metrics/sync/assets/selfupdate routes; **remove `/api/storage/*` (disk)**; backup routes become **agent-coordinated guest-backup** requests; `config/apply` (hub-pushes-yaml) changes since the **agent** now injects config at provision. | needs-rework |
-| `api/geo.go` | **PORT/MODIFY** | Keep the customer-facing geo **preference** endpoints (set/get global + per-app); **drop the Cloudflare-sync trigger** — enforcement → hub (S4). The controller reports geo desired-state up instead of calling the CF API. | needs-rework |
-
-### `appexport/` — KEEP/PORT (Docker-volume + DB level, no disk ops)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `crypto.go` | **KEEP** | Self-contained AES-256-CTR+HMAC+scrypt for `.fab`. | clean |
-| `manifest.go`, `provider.go` | **KEEP** | Bundle metadata; provider interface (impl in main). | clean |
-| `export.go` | **PORT** | Docker-volume `tar`, DB dump via `backup.DumpOne`, config copy. Depends on the **retained** app-data-backup subset of `backup/`; HDD-mount enumeration reworked to **per-volume placement**. | needs-rework |
-| `restore.go` | **PORT** | `docker volume create`/`tar xf`, DB import, compose up. Same per-volume rework. | needs-rework |
-| `estimate.go` | **PORT** | `du`/`df` on mounts → per-volume sizing. | clean |
-
-### `assets/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `syncer.go` | **KEEP** | Hub API download + checksum cache; already a direct hub channel. | clean |
-
-### `backup/` — THE SPLIT (delete side interleaved with keep side; see §3)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `dbdump.go` | **KEEP** | Pure `docker exec pg_dump`/`mariadb-dump` — app/DB data layer; the retained per-app backup. | clean |
-| `appdata.go` | **PORT** | App-data discovery (stacks/volumes/DB containers, `du`). "HDD mount" concept → per-volume. | needs-rework |
-| `backup.go` (1478 L) | **MODIFY (split)** | Mixes **keep** (`RunDBDumps`, `DumpAppVolumes(Safe)`, app restore) with **delete→agent** (`RunBackup`/`backupDrive`/restic snapshot/prune/check on per-drive repos). Must be torn in two. | hazard |
-| `restore.go` (442 L) | **MODIFY (split)** | `RestoreApp` restic path → agent; Docker-volume + Tier-2 rsync restore (app layer) → keep. | hazard |
-| `restore_app_linux.go`/`_other.go` | **PORT** | Per-app restore: compose pull/up, rsync app data, DB-dump restore. App layer; depends on backup location that changes. | needs-rework |
-| `paths.go` | **MODIFY (split)** | `AppDBDumpPath`/`AppVolumeDumpPath` keep; `Primary/SecondaryResticRepoPath`, `InfraBackupDir` → agent. | needs-rework |
-| `restic.go` | **DELETE (→agent)** | restic repos on drives = infra backup tier; agent does vzdump/PBS. | hazard |
-| `crossdrive.go` | **DELETE (→agent)** | Tier-2 cross-drive rsync to secondary storage = storage-tier (agent + storage manifest). | hazard |
-| `restore_drives_linux.go`/`_other.go` | **DELETE (→agent)** | `lsblk`/`blkid`/`mount`/fstab — pure host disk. | hazard |
-| `disk_layout.go` | **DELETE (→agent)** | Disk topology for DR → agent. | clean |
-| `local_infra.go` | **DELETE (→agent)** | Per-drive infra-backup metadata → agent. | clean |
-| `restore_scan.go` | **DELETE (→agent)** | Scans drives to build a DR restore plan = agent-tier DR. | needs-rework |
-
-### `cloudflare/` — DELETE (→hub): CF-API enforcement moves to the hub (S4)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
-
-### `config/`, `crypto/`, `util/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `config/config.go` | **MODIFY** | Drop `BackupConfig` (restic/retention), storage-drive keys, and `InfrastructureConfig.cf_api_token` (→hub, S4); keep customer/paths/web/git/stacks/monitoring/hub/assets/system; **add agent local-API endpoint+token**. | needs-rework |
-| `crypto/crypto.go` | **KEEP** | App.yaml secret encryption. | clean |
-| `util/strings.go` | **KEEP** | Trivial helper. | clean |
-
-### `integrations/` — all KEEP (pure app-domain)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `integrations.go`,`lifecycle.go`,`manager.go`,`onlyoffice_filebrowser.go`,`onlyoffice_nextcloud.go` | **KEEP** | App-to-app via `docker exec` / compose-config patch; no host ops. | clean |
-
-### `metrics/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `store.go`,`logscanner.go`,`telemetry.go`,`types.go` | **KEEP** | SQLite store, `docker logs` scan, container telemetry — app-domain. | clean |
-| `collector.go` | **PORT** | Container metrics (`docker stats`) keep; host metrics via `system.GetInfo` (temp, physical disk) become **agent-provided or dropped**. | needs-rework |
-| `sysinfo.go`/`sysinfo_other.go` | **MODIFY** | Reads `/host/etc`, `/proc/cpuinfo`, uptime — host static info; in-guest some is meaningful, hardware identity via agent. | needs-rework |
-
-### `monitor/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `healthcheck.go` | **PORT (split)** | Keep guest health (mem/cpu/docker/protected-containers); host health (temp, **physical disk**, storage-path mount status) becomes **agent-fed**. | needs-rework |
-| `pinger.go` | **DELETE (obsolete)** | Legacy Healthchecks.io; `main.go` itself marks it "deprecated… now handled by Hub". *(Corrects the task's KEEP/PORT guess.)* | clean |
-| `watchdog.go` (902 L) | **DELETE (→agent)** | Storage/USB disconnect monitoring: `umount -l`, `mount -T /host-fstab`, UUID probing, restic-lock cleanup — pure host storage. | hazard |
-
-### `notify/`, `recovery/`, `scheduler/`, `selftest/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `notify/notifier.go` | **KEEP/MODIFY** | Direct hub event channel (own API key) — keep; prune infra event types that move to the agent (`storage_disconnected`, `crossdrive_*`, `disaster_recovery_*`). | clean |
-| `recovery/info.go` | **DELETE (obsolete)** | Generates a DR text guide (OS install, docker-setup.sh, hub restore UI); DR is now agent+hub provisioning. | clean |
-| `scheduler/scheduler.go` | **KEEP** | Generic cron/interval, Budapest TZ. | clean |
-| `selftest/selftest.go` | **PORT** | Keep docker/dirs/catalog/hub checks; drop restic-repo + system-data **mountpoint** checks (→agent). | needs-rework |
-
-### `report/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `pusher.go` | **KEEP** | Direct hub push (`/api/v1/report`, Bearer). | clean |
-| `telemetry.go` | **KEEP** | Per-app telemetry section. | clean |
-| `builder.go` (326 L) | **MODIFY** | Keep containers/telemetry/stacks/geo/app-health; drop/relocate host system info, physical storage, **restic backup status incl. restic password**. | hazard |
-| `types.go` | **MODIFY** | Schema: drop infra fields (`restic password`, physical storage), keep app-domain. | needs-rework |
-| `infra_backup.go`/`_linux.go`/`_other.go` | **DELETE (→agent)** | Builds infra-backup payload (disk layout, restic/enc passwords) for hub. | hazard |
-| `infra_pull.go` | **DELETE (→agent)** | Pulls recovery config + infra backup from hub (setup-wizard DR). | needs-rework |
-
-### `selfupdate/` — controller is agent-managed (doc 03 §11)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
-| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
-| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
-
-### `settings/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `settings/settings.go` (1101 L) | **MODIFY (split)** | Keep notif prefs, integration state, geo, DB-validation cache, cross-drive *intent*. The **storage-path registry** (`StoragePath` with `Disconnected`/`DisconnectedAt`/`StoppedStacks`/decommission) is disk-management state → reshape to **per-volume placement** fed by the agent's storage manifest; disconnect/decommission/migrate state leaves. (UUID is *not* a persisted field — runtime-derived from fstab.) | hazard |
-
-### `setup/` — all DELETE (obsolete); the agent provisions the controller
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `handlers.go`,`setup.go`,`csrf.go`,`network.go` | **DELETE (obsolete)** | First-run wizard (hub-restore, manual config, LAN-IP detection). | needs-rework |
-| `scanner.go` | **DELETE (→agent)** | Drive scan (`lsblk`+temp mounts) for backup discovery — host op; its capability informs the agent. | clean |
-
-### `stacks/` — core app domain (KEEP/PORT)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `manager.go` (1074 L) | **KEEP/PORT** | Docker Compose orchestration, scan/state/start/stop/logs — the heart. Minor port. | clean |
-| `deploy.go` | **PORT** | Memory validation (`system.GetMemoryMB` — **guest** mem, fine in LXC), secret gen, encrypted app.yaml. **Add snapshot-before-deploy → agent** hook. | needs-rework |
-| `healthprobe.go` | **KEEP** | TCP/HTTP app probes. | clean |
-| `metadata.go` | **PORT** | `.felhom.yml` parse. **Add per-volume hot/bulk classification** (doc 01 §8). | needs-rework |
-| `delete.go` | **PORT** | Stack delete + HDD-data `os.RemoveAll` on bind mounts → per-volume cleanup. | needs-rework |
-
-### `storage/` — entire package DELETE (→agent)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `scan*`,`format*`,`attach*`,`migrate*`,`migrate_drive*`,`safety*` | **DELETE (→agent)** | Physical disk: `lsblk`/`sfdisk`/`wipefs`/`mkfs.ext4`/`partprobe`/`mount`/`umount`/fstab/`blkid`/drive-rsync. The agent owns all of this (doc 01 §3, §8). | hazard |
-
-### `sync/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `sync/sync.go` | **KEEP** | Catalog git-sync (clone/fetch/reset, copy compose+`.felhom.yml`, never overwrite app.yaml). | clean |
-
-### `system/` — split per-function (not per-file)
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `cpu_linux.go`/`cpu_other.go` | **KEEP** | `/proc/stat` works inside an LXC. | clean |
-| `info.go`/`info_other.go` | **KEEP** | Structs/stubs. | clean |
-| `info_linux.go` | **MODIFY (split)** | Keep mem (`/proc/meminfo`)/load/statfs (guest); **temp via `/host/sys`, hwmon → agent**. | needs-rework |
-| `mounts_linux.go`/`mounts_other.go` | **DELETE (→agent)** mostly | Mount-point detection, USB, disk model, fstab, probe — host/disk. Guest-meaningful `statfs` disk-usage is the only keep-candidate → fold into the kept `info`. | hazard |
-
-### `web/` — split by UI surface
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `auth.go`,`csrf.go`,`logbuffer.go`,`embed.go`,`templates.go` | **KEEP** | Session/CSRF, log ring buffer, embeds/logo. | clean |
-| `funcmap.go` | **KEEP/PORT** | Template helpers; a few backup/state labels track the backup rework. | clean |
-| `server.go` (559 L) | **MODIFY** | Routing/wiring; remove storage/DR-restore/watchdog wiring; keep app/deploy/backup/settings/export/debug. | needs-rework |
-| `handlers.go` (1883 L) | **PORT/MODIFY** | Core pages keep; the embedded **storage-path management** (add/remove/label/schedulable, storage bars, FileBrowser mount sync) → per-volume / agent-fed. | hazard |
-| `handler_export.go` | **KEEP/PORT** | `.fab` UI. | clean |
-| `handler_debug.go` (823 L) | **PORT** | Drop storage-simulate/infra-push/DR debug; keep the rest. | needs-rework |
-| `alerts.go` | **PORT/MODIFY** | Storage-disconnect alert now sourced from **agent** status; backup/update alerts keep. | needs-rework |
-| `handler_restore.go` | **DELETE (→agent) / MODIFY** | DR restore-mode UI; DR is agent-tier — replace with an agent-status view or remove. | needs-rework |
-| `storage_handlers.go` (1600 L) | **DELETE (→agent)** | Format/attach/mount/disconnect/migrate-drive/decommission disk UI. Any survivor is a **thin client calling the agent API** (e.g. per-volume placement requests). | hazard |
-| `templates/` (HTML, non-Go) | **PORT** | Remove disk-wizard + DR pages; keep app/deploy/backup/settings pages. | needs-rework |
-
-### `scripts/`
-| File | Class | Reason | Risk |
-|---|---|---|---|
-| `scripts/hashpass.go` | **KEEP** | Standalone bcrypt helper. | clean |
-
----
-
-## 3. Coupling hazards (delete-targets depended on by keep/port)
-
-1. **`backup/` is half-deleted but split *inside files*, not across them.** `backup.go`
-   contains both `RunDBDumps`/`DumpAppVolumesSafe`/app-restore (keep) and
-   `RunBackup`/`backupDrive` + restic (delete→agent); `restore.go` and `paths.go` are
-   likewise mixed. **Keep/port consumers reach into this same package:**
-   - `appexport/export.go:295` → `backup.DiscoverDatabases`/`DumpOne` (DB dump is app-layer — must survive)
-   - `report/builder.go:buildBackupReport` → backup status (MODIFY)
-   - `web/handlers.go` (backups page, `buildAppBackupRows`), `web/funcmap.go`, `web/alerts.go`, `web/handler_restore.go`, `web/handler_debug.go`
-   - `selftest/selftest.go:217` → `checkResticRepos` (restic path — delete)
-   - `main.go` scheduler chain `RunFullBackup` (DB→volume→restic→infra-push) interleaves both sides.
-   **Action:** extract the app-data-backup subset (DB dump, volume archive, per-app
-   restore) into a clean retained package *before* deleting the restic/cross-drive code,
-   or every keep consumer breaks.
-
-2. **`backup/crossdrive.go` (delete→agent) is wired as `crossDriveRunner` into**
-   `main.go`, `api/router.go`, `web/server.go`, and surfaced by `report/builder.go` and the
-   backups page. Removing it requires reworking the backup UI/report to the agent's
-   guest-backup status.
-
-3. **`storage/` (delete→agent) depended on by keep/port UI:** `web/storage_handlers.go`
-   (delete) and `web/server.go`/`web/handlers.go` (port) — the latter renders storage
-   labels/bars and runs **FileBrowser mount sync** off the storage-path registry.
-   `storage/migrate*.go` also imports `backup` (also being split). Untangle the per-volume
-   placement UI from the disk-management UI.
-
-4. **`monitor/watchdog.go` (delete→agent) depended on by** `web/alerts.go` (port),
-   `web/server.go`, `web/handler_debug.go`, `main.go`. The disconnect **alert** must instead
-   consume agent-reported storage status.
-
-5. **`system/` mixed-per-function, consumed by both sides.** Keep consumers —
-   `stacks/deploy.go` (`GetMemoryMB`, guest), `metrics/collector.go` (container) — must not
-   drag in the host-disk/temp/USB code that goes to the agent (`mounts_linux.go`,
-   `info_linux.go` temp). Also consumed by `report/builder.go` (MODIFY), `monitor/healthcheck.go`
-   (PORT), `selftest`, `crossdrive` (delete). **Split `system/` cleanly into guest-info vs
-   host-info first.**
-
-6. **`settings/StoragePath` carries disk state into an app-domain store.** Disk fields
-   (`Disconnected`,`DisconnectedAt`,`StoppedStacks`, decommission — UUID is *not* persisted, it's runtime-derived from fstab via `system.ParseFstabUUID`/`watchdog.go`) are written by
-   `watchdog.go`/`storage_handlers.go`/`crossdrive.go` (all delete) but the same struct is
-   read by `stacks`/`web` for labels and **placement** (keep). Reshape `StoragePath` to a
-   placement record fed by the agent manifest.
-
-7. **`report/builder.go` imports almost everything** (backup, monitor, scheduler, stacks,
-   system, metrics, settings, config). Its MODIFY must land *after* the backup and system
-   splits, or it pulls deleted code along.
-
-8. **`backup/paths.go` shared both ways** — `appexport` + `selftest` + the kept DB-dump
-   flow use the app-dump path helpers; the same file holds the restic/secondary helpers
-   that leave.
-
-9. **DR/provisioning chain is cross-cut:** `setup/` (obsolete) → `report/infra_pull` +
-   `recovery/info` + `backup.MountDrivesFromLayout` + `backup.ReadLocalInfraBackup`. All
-   obsolete/→agent, but `main.go`'s setup branch and `web/handler_restore.go` reference
-   them; remove together.
-
----
-
-## 4. Moves to the host agent (consolidated — feeds the future agent design)
-
-> Reporting only; **not** designing the agent here.
-
-- **All physical-disk management** — `storage/` in full: scan/classify, format
-  (`wipefs`/`sfdisk`/`mkfs.ext4`/`partprobe`), attach (raw mount + bind + fstab), per-app
-  and full-drive migration (rsync), safety checks (system-disk detection).
-- **Storage/USB watchdog** — `monitor/watchdog.go`: disconnect/reconnect detection,
-  `umount -l`, `mount -T /host-fstab`, UUID-by-id probing, safe-disconnect, restic-lock
-  cleanup.
-- **Infra/disk backup tier** — `backup/restic.go`, `crossdrive.go`,
-  `restore_drives_*`, `disk_layout.go`, `local_infra.go`, `restore_scan.go`, plus the
-  restic-snapshot half of `backup.go`, the restic-restore half of `restore.go`, and the
-  restic/secondary path helpers in `paths.go`. (Maps to the agent's `vzdump`→tiers→PBS in
-  doc 01 §8.)
-- **Infra-backup payload + recovery pull** — `report/infra_backup*`, `report/infra_pull`.
-- **Host-physical telemetry** — `system/mounts_linux.go` (mount topology, USB, disk
-  model), the temp/hwmon parts of `system/info_linux.go`, and the host-hardware parts of
-  `metrics/sysinfo.go`.
-- **Drive scanning for provisioning/DR** — `setup/scanner.go`.
-- **Self-restore-test execution** — the agent performs the restore-to-scratch-guest; the
-  controller only orchestrates/validates (see §5).
-
----
-
-## 5. New components to build (no v0.33 equivalent)
-
-1. **Agent local-API client** — the controller's only path to guest-level Proxmox
-   operations (doc 01 §3, §5): `snapshot-before-deploy` + rollback, "grow my RAM", request
-   guest backup/restore, read the storage manifest / mount placement, query per-target
-   storage status. Replaces the deleted direct host/disk code with constrained RPC. The
-   controller holds **no Proxmox creds** — only a local-API token.
-2. **Per-volume storage placement** (doc 01 §8) — `.felhom.yml` `hot`/`bulk` volume
-   classification (extend `stacks/metadata.go`), enforcement at deploy (extend
-   `stacks/deploy.go`), and a placement record in `settings`. Replaces the per-app
-   HDD-path + cross-drive model. A `bulk` volume must be realized as a `backup=0` mount point,
-   **never** a rootfs Docker named volume (validated recipe: `phase3-findings.md` B2 / doc 03 §7).
-3. **Self-restore-test status display** (read-only) — the **agent owns orchestration** (it
-   holds the PBS key and creates the scratch guest — operator-tier, doc 03 §8); the controller
-   only surfaces `GET /restore-test/status` in its UI. (Round-trip validated: Phase 2,
-   [../proxmox-platform.md](../proxmox-platform.md) §4.)
-4. **Snapshot-before-deploy/rollback flow** in the deploy path — wraps the existing
-   compose deploy with agent snapshot → health check → agent rollback-on-failure
-   (doc 01 §9). New behaviour on top of `stacks/deploy.go` + `stacks/healthprobe.go`.
-5. **Agent-provisioning bootstrap receiver** — the controller accepts its injected hub API
-   key + local-API token from the agent at provision time (doc 01 §6), replacing the
-   deleted `setup/` wizard.
-
----
-
-## 6. Open / blocked items
-
-- **Geo — resolved (S4):** CF-API **enforcement moves to the hub** (it holds the CF token and
-  reconciles geo → WAF); the controller keeps the geo **preference UI/data** and reports
-  desired-state up. Tunnel placement is settled (host, agent-managed, doc 03 §3/§5). The
-  `cloudflare/` package + `api/geo.go`'s CF-sync are DELETE-from-controller → hub.
-- **Self-update — resolved (doc 03 §11):** the controller is agent-managed; its self-update
-  path is removed.
-- **`settings`/`stacks` per-volume reshape** — depends on the storage-manifest contract
-  between hub ↔ agent ↔ controller (doc 01 §8), not yet specified.
-- **Backup UI/report surface** — depends on the agent's guest-backup status API shape
-  (what the controller can see about vzdump/PBS state) — undefined.
-- **Notification event taxonomy** — which infra events (`storage_disconnected`,
-  `crossdrive_*`, `disaster_recovery_*`) the **agent** emits vs the controller, once those
-  responsibilities move.
-
----
-
-## Changelog — design-review + Phase-3 fold-in (2026-06-08)
-
-- **M1:** removed `UUID` from the `settings.StoragePath` field lists (§ settings, hazard #6) —
-  it is runtime-derived from fstab, not persisted.
-- **S4 (geo):** `cloudflare/` reclassified **PORT(blocked) → DELETE(→hub)** (CF-API enforcement
-  moves to the hub); `api/geo.go` → **PORT/MODIFY** (keep geo *preference* endpoints, drop the
-  CF-sync trigger); `config/config.go` also drops `cf_api_token`. §6 + §1 updated.
-- **S5:** cloudflare/geo no longer "blocked on tunnel placement" (resolved).
-- **S6:** §5(3) self-restore-test → **status-display only**; the agent owns orchestration.
-- **Self-update resolved (03 §11):** `updater.go` → **DELETE(→agent)**, `state.go` →
-  DELETE(obsolete), `version.go` KEEP; §6 + §5(2) updated (bulk = `backup=0` mountpoint recipe).

docs/architecture/03-host-agent.md

@@ -1,299 +0,0 @@
-# Architecture Part 3 — The Host Agent
-
-> Status: design draft (decision content). To be grounded by Claude Code against
-> `docs/proxmox-platform.md` and `docs/architecture/02-controller-module-map.md`,
-> then placed at `docs/architecture/03-host-agent.md`.
->
-> Builds on Part 1 (`01-topology-and-trust.md`) and Part 2 (`02-controller-module-map.md`).
-> Where this doc and the locked decisions disagree, the locked decisions win and this
-> draft is wrong — flag it.
-
-## 1. Purpose & scope
-
-The **host agent** is the operator-tier component that runs on each Proxmox host and
-owns *all* Proxmox interaction. It is the trusted host actor: it provisions and restores
-guests, manages host storage, orchestrates backups and restore-tests, watches the host
-and the tunnel, talks to the hub, and exposes a narrow local API to the in-guest
-controllers it deploys.
-
-It is the privileged tier. The controller deliberately holds **no** Proxmox credentials
-(Part 1) — the privilege the controller shed by losing `storage/` did not disappear, it
-**moved here**. That makes the agent's hardening and blast-radius discipline the most
-security-sensitive part of the platform.
-
-The agent manages a **set** of guests on its host (usually one customer = one guest, but
-the multi-tenant/company case is not precluded — the agent's data model is per-host,
-N-guests, never "the guest").
-
-## 2. Responsibilities (and explicit non-responsibilities)
-
-Owns:
-
-1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
-2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
-3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
-4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub.
-5. **Provisioning** — provision a guest **by restoring the golden base image** (§9), deploy the controller into it, hand it its bootstrap config; also **build and refresh the golden base image** itself.
-6. **Hub control loop** — poll for desired state + signed jobs, reconcile, execute, report, heartbeat.
-7. **Local API** — the per-guest authorization gate the controller calls.
-8. **Self-update** — update itself (carefully — it is a host service) and update the controllers it owns.
-
-Explicitly does **not**:
-
-- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
-- Hold or proxy customer application data.
-- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
-- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.
-
-## 3. Process model & host integration
-
-- **Native Go binary, systemd service** on the host: boot-start, `Restart=always`, systemd watchdog (kill+restart on hang), journald logging, resource limits.
-- **Root-minimized (boundary settled — Phase 3 B3).** The agent runs as a **non-root** service user with the scoped `FelhomAgent` token for all API-covered work + a **narrow `sudoers` allowlist** for true host ops. Per Phase 3 (B3) the boundary is settled: the entire per-customer guest lifecycle — provision (by restore, §9), config, start/stop, snapshot, backup, **restore**, destroy — is token-covered. Genuine OS-root is confined to: (1) building/refreshing the **golden base image** (`keyctl` create is `root@pam`-only — one-time at enrollment + a maintenance cadence, §9); (2) **host mounts** (USB mount-by-UUID, systemd mount units / fstab); (3) **SMART / hardware sensors**. Root therefore never sits on the per-customer path. See `proxmox-platform.md` §3.6 for the role + boundary table.
-- **`cloudflared` is a separate systemd service**, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process.
-
-## 4. Control model — reconcile + signed destructive ops
-
-Two channels, split by **reversibility**, not by transport.
-
-**(a) Desired-state reconciliation — steady state.**
-The hub holds desired state for the host: which guests should exist (and at what spec),
-the storage manifest, backup/retention policies, controller image versions. The agent
-runs a reconcile loop converging actual Proxmox state → desired: idempotent, self-healing,
-and tolerant of missed polls (drift is corrected on the next loop). Provisioning retries,
-re-attach of a flapping USB target, redeploy of a crashed controller — all fall out of
-reconciliation for free.
-
-**(b) Signed one-shot jobs — operator actions.**
-Restore-now, decommission, force-backup, break-glass-enable. Discrete, run-once
-(idempotency key), written to the customer-visible audit log, and **outside** the reconcile
-loop — they are point-in-time and often destructive, and a reconciler must never re-run a
-restore because it "sees drift." A one-shot job names a **target** ("restore guest X from
-snapshot S"), not a procedure; the agent owns the *how*.
-
-**The reversibility gate (security-critical).**
-"Signed jobs resist hub compromise" only holds if the agent also distrusts hub-supplied
-*desired state* for destructive changes. The gate is by **provenance + data-bearing-ness, not
-by verb**:
-
-- **The reconciler MAY act without an operator signature** when: (a) creating/starting/restarting; (b) destroying resources it created earlier **within the same journaled transaction** (compensating rollback, §10); (c) destroying resources it **tagged ephemeral/scratch** (e.g. restore-test scratch guests, §8). The ephemeral/scratch tag is **agent-internal provenance and is never accepted from the hub** — else a compromised hub could relabel a data-bearing guest as scratch to walk the gate.
-- **An operator signature is always required** to destroy/overwrite any resource holding the only/primary copy of customer data — live-guest destroy, storage detach/wipe, restore-overwrite, decommission — *regardless of whether it arrives as a job or as a desired-state delta*. A compromised hub cannot forge them because the signing key is **not held by the hub** (it lives with the operator / a separate signing path; the hub only queues opaque signed blobs).
-- **Healing a crashed controller is non-destructive by construction:** it is reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. (v0.33 precedent: `watchdog.go` restarts stopped stacks, it never destroys the guest.)
-
-Signed payloads carry a **nonce + expiry** (anti-replay: a captured "restore" job cannot be
-re-injected later) and a target binding (host + guest id) so a signature can't be retargeted.
-Notification-on-destructive-op is an **audit signal, never the guard** — a compromised hub
-could both issue and suppress the notice, which is exactly why the *signature* (not the
-notification) is the control.
-
-## 5. Hub ↔ agent protocol (host domain)
-
-**Box-initiated poll.** The hub never connects inbound. Each poll cycle exchanges:
-
-- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health, agent + controller versions, audit-log tail.
-- **Down:** the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes).
-
-**Dead-man's-switch (essential, not optional).** In a box-initiated model the heartbeat
-*is* the liveness signal — a box that stops checking in is otherwise invisible. The hub
-alerts the operator when an agent misses its expected check-in window. This is the worst
-failure mode for a managed service, so it gets first-class treatment hub-side.
-
-**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
-(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit,
-**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
-the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
-logged operation; it auto-expires.
-
-## 6. Agent ↔ controller local API
-
-The controller (in its LXC) reaches the agent (on the host) over the local bridge.
-
-- **Transport:** HTTPS to the host's bridge IP on a fixed port.
-- **Auth:** a per-guest local token, minted by the agent when it deploys the controller and written into the guest's bootstrap config. The agent maps token → guest and **authorizes per guest**: a controller can only act on *its own* guest. This is the agent acting as the per-guest authorization gate from Part 1.
-- **Surface (minimal, all scoped to the caller's own guest):**
-  - `GET /storage` — mounts available to this guest and their **class** (fast/slow), so the controller can place hot vs bulk volumes per `.felhom.yml`. (The agent owns the actual mounts; the controller just binds to the paths it's given.)
-  - `POST /snapshot` — snapshot *this* guest (the snapshot-before-deploy primitive).
-  - `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
-  - `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
-  - `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
-  - `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
-
-Note what is *absent*: nothing here lets a controller touch another guest, the host, storage
-attachment, or restore-overwrite. Destructive/cross-guest power stays operator-signed (§4).
-
-A controller can only `POST /rollback` (or snapshot/backup) **its own** guest — the agent maps
-token → guest and authorizes per guest, so a compromised controller's blast radius is
-**self-scoped and bounded** to its own guest.
-
-## 7. Storage manifest & reconciliation
-
-The manifest is the load-bearing contract. It absorbs the **persisted** disk-state fields that
-`settings.StoragePath` carries today **and adds** `durable_id`/UUID — today the controller
-re-derives the UUID from fstab each boot (Part 2 / Phase-3), so persisting it is an
-improvement. Held in the hub, reconciled by the agent.
-
-Per target:
-
-| field | meaning |
-|---|---|
-| `type` | `local-dir` / `usb` / `nfs` / `cifs` / `pbs` |
-| `durable_id` | UUID (USB), `server:export` (NFS/CIFS), `repo+fingerprint` (PBS) — survives box loss |
-| `class` | `fast` or `slow`, set **once at attach**, with an IOPS marker; no runtime speed-test |
-| `role` | `primary` / `vzdump-target` / `pbs-offsite` / `bulk-data` |
-| `creds` | encrypted (NFS/CIFS/PBS); USB has none |
-| `policy` | schedule + retention for this target |
-| `state` | `attached` / `disconnected` / `decommissioned` |
-
-Reconciliation: ensure each `attached` target is mounted (USB-by-UUID via the sudoers
-allowlist), each Proxmox storage entry matches, and `disconnected` targets are surfaced to
-the hub (the storage watchdog — detect a USB drop in seconds, not at the next health cycle).
-
-**Placement is per-volume, not per-app.** Hot volumes (DB/config) → a `fast` target,
-**enforced**; bulk volumes (media) → may live on `slow`, declared in `.felhom.yml`.
-
-A `bulk` volume **MUST** be realized as a `backup=0` **volume mount point** (or an external
-bind mount) — **never** a Docker named volume in rootfs, which `vzdump` always captures
-(verified, `phase3-findings.md` B2). Proven recipe: attach
-`-mpN <storage>:<size>,mp=/mnt/bulk,backup=0`, then
-`docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk <vol>` (or a
-compose bind). The per-volume placement component (Part 2 §5(2)) enforces this at deploy. The
-**DR consequence** of excluding bulk is covered in §8.
-
-**Field re-homing (from `settings.StoragePath`, Part 2):** `Label` → manifest (canonical);
-`IsDefault`/`Schedulable` → manifest `policy`; `MigratedTo` + decommission → manifest `state`;
-`StoppedStacks` → the **controller's `settings`** (app-domain: which apps to restart on
-reconnect, not a host concern).
-
-## 8. Backup/restore orchestration
-
-Tiers double as backup *and* restore-source priority (fastest surviving source first),
-per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a backup) →
-**local second storage** (vzdump to dir/NFS/CIFS) → **PBS offsite** (the DR substrate).
-
-- **Quiescing (controller-driven for app-consistency):** an LXC has no fsfreeze
-  (`proxmox-platform.md` §4.2), so app-consistency is the controller's job: it learns a backup
-  is due (`GET /backup/due`, §6, or via its hub channel) → **quiesces** the app stack →
-  `POST /backup` → polls `GET /backup/status` → unquiesces. **An agent-initiated vzdump is
-  crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5).
-  Every Proxmox op is async → the agent polls `task exitstatus`, never trusts the POST return.
-- **Bulk volumes have no DR coverage from the guest vzdump** — they are excluded (§7). Every
-  `bulk` volume needs an explicit own-backup decision: its own backup target per the manifest
-  `policy`, **or deliberately none** when the data is re-downloadable (customer informed). On
-  host-loss, un-backed-up bulk is gone; a **bind-mounted** bulk volume re-attaches only on the
-  *same* host, so cross-host DR needs the separate backup. A deliberate per-volume choice,
-  never a silent loss.
-- **Key custody (PBS):** the **live** PBS key sits on the box so the agent can both back up
-  *and* run restore-tests. The hub holds only the **recovery-code-wrapped escrow** copy it
-  cannot open (zero-knowledge default). So: the box can restore-test; the operator cannot
-  read the data; the customer's offsite recovery code is the irreducible residual.
-- **Self-restore-test:** the closing of the "tested restore is the critical gap" theme. The
-  agent periodically restores a backup into a **throwaway scratch guest**, boots it, runs
-  health checks, reports pass/fail, and tears it down. Zero-knowledge backups can *only* be
-  restore-tested by the box (the operator lacks the key) — so this lives in the agent by
-  necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often
-  as the lighter check.
-
-## 9. Provisioning & DR flows
-
-**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
-the `keyctl=1` feature flag, which Proxmox permits only for `root@pam` (Phase 3, B3) — not the
-scoped token. But a token-authorized **restore preserves `keyctl`** (Phase 3, B3), so the agent
-provisions **by restoring a golden base image**, never by `pct create` on the per-customer path:
-
-- A **golden base archive** — minimal Debian + Docker, `nesting=1,keyctl=1`, overlayfs — is
-  built once as `root@pam` **at enrollment** (when the agent legitimately holds root to mint its
-  Proxmox token) and refreshed on a maintenance cadence. This is the one place `keyctl`/root
-  provisioning lives — off the per-customer path.
-- To provision guest G: restore the golden archive → new VMID (token-covered: `VM.Allocate` +
-  `Datastore.AllocateSpace`; `keyctl` preserved) → reset identity (MAC/hostname) → size the guest
-  (CPU/mem config + `pct resize` rootfs, token-covered) → attach storage mounts per the manifest
-  → deploy the controller → hand it bootstrap config. A mid-flight failure is journaled and
-  compensating-rolled-back (destroy the just-restored guest — allowed without a signature per §4,
-  same-transaction provenance).
-
-**Unified bring-up primitive.** Provisioning and DR-restore share the same token-covered front
-half — *restore an archive → reset identity* — and differ only in the archive and the back half:
-provisioning restores the **golden base** then deploys a fresh controller; DR-restore restores
-the **customer's backup** (already containing controller + data), brings it up, and reattaches
-external storage. One code path, exercised by every restore-test (§8).
-
-**Guest loss.** Agent restores G from the fastest surviving tier and resets identity
-(MAC/hostname) so the restored guest rejoins cleanly — this *is* the unified restore primitive
-above (customer-backup archive, DR back half).
-
-**Host/hardware loss.** Re-enroll the new host in **restore mode**; the hub — the durable
-source of truth that survives box death — hands the new agent the existing identity, PBS
-namespace, tunnel token, storage manifest, and a restore directive. Tunnel is reused from
-the hub record, so DNS stays intact.
-
-## 10. Concurrency, crash-safety, idempotency
-
-- **Per-guest serialization.** Reconcile, one-shot jobs, and local-API calls all feed a
-  work queue that serializes mutations **per guest** (Proxmox dislikes concurrent conflicting
-  ops on the same guest). Independent guests proceed in parallel.
-- **Operation journaling.** Multi-step async ops (provision, restore, controller-update, agent
-  self-update) are journaled with their in-flight Proxmox task ids. On agent restart, the
-  journal is replayed: resume-or-rollback, so a crash mid-restore never leaves a corrupt or
-  half-built guest.
-- **Idempotency keys** on one-shot jobs (run-once across retries and restarts).
-
-## 11. Self-update
-
-- **Agent (the hard case — a host service, no snapshot-rollback).** **A/B layout:** download →
-  verify signature → stage as the inactive slot → flip a `current → good|new` symlink → restart.
-  **Revert authority lives outside the swapped binary** — `Restart=always` alone just
-  crash-loops a bad binary — so a **separate health-gate** (a systemd oneshot `ExecStartPost`
-  probe, or a tiny supervisor unit) flips `current` back to last-good and restarts on a failed
-  health window. The new version is **committed as "good" only after a clean health window**.
-  Triggered by a hub signed job within the update window; manual always allowed. Journaled (§10).
-- **Controller (the easy case — it's a guest).** The agent owns the controller's lifecycle,
-  so the **agent updates the controller**: snapshot-before-update (free rollback, because the
-  controller *is* a snapshottable guest) → pull new image → redeploy → health-check → rollback
-  on failure. This resolves the Part-2 `selfupdate/` open: the controller is **agent-managed**,
-  not self-updating; the controller's old self-update path is removed.
-
-## 12. Secrets at rest on the host
-
-The agent holds, root-only on the host fs: the scoped Proxmox token, the hub API key, the
-operator's **public** verify key (for §4 signatures — public, low-risk), the Cloudflare
-tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**. The privilege
-and the secret footprint that left the controller now concentrate here — which is the whole
-argument for §3's root-minimization and a small, auditable agent.
-
-## 13. Open items / what this unblocks
-
-Resolved here: tunnel placement (host, agent-managed, own systemd service), the
-reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update
-ownership, the local-API surface, the storage-manifest schema, **provision-by-restore**, and
-the **root-vs-API boundary** (Phase 3, B3).
-
-Still open:
-
-- Multi-tenant **resource fairness** on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass.
-- Operator-side **signing tooling** — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub").
-- Hub-side **desired-state editing UX** and the host-domain report schema details — belong to the hub architecture doc.
-- **Golden base image** refresh cadence + fleet versioning — who triggers a rebuild, how the per-host image version is tracked (operational detail, not blocking; §9).
-
-This doc hands the implementation three contracts it was waiting on:
-
-1. the **local-API surface** (§6) → the controller's NEW local-API client, snapshot-before-deploy, and self-restore-test wiring (Part 2);
-2. the **storage-manifest schema** (§7) → the `settings.StoragePath` reshape and per-volume hot/bulk placement (Part 2);
-3. the **backup contract** (§7–8) → the destination for the app-data-backup package extracted in the Part-2 refactor.
-
----
-
-## Changelog — design-review + Phase-3 fold-in (2026-06-08)
-
-- **NEW provision-by-restore** (§9): the agent provisions by **restoring a golden base image**
-  (token-covered, preserves `keyctl`), never `pct create` on the per-customer path; one unified
-  restore primitive shared with DR. §2 responsibility + §3 boundary updated.
-- **B3** (§2/§3): replaced "Phase-1 minimal role" with the validated **`FelhomAgent`** operator
-  role; root-vs-API boundary **settled** (root only for golden-image build, host mounts, SMART).
-- **B1** (§4): reversibility gate rewritten as **provenance + data-bearing** (scratch tag is
-  agent-internal, never hub-supplied; crashed-controller heal is non-destructive in-place).
-- **B2** (§7/§8): validated bulk-as-`backup=0`-mountpoint recipe + the **bulk-DR consequence**
-  (excluded bulk needs its own backup decision).
-- **S1** (§6/§8): `GET /backup/due` added; controller-driven quiescing; agent vzdump is
-  crash-consistent only. **S2** (§10/§11): A/B self-update with external revert authority;
-  controller-update + agent self-update journaled. **S3** (§7): `StoragePath` field re-homing.
-  **S4:** geo non-responsibility added (§2). **M2** (§7): manifest "absorbs + adds durable_id".
-  **§6:** rollback is self-scoped/bounded. **§13:** golden-image refresh cadence added as open.

docs/architecture/04-control-plane-authorization.md

@@ -1,154 +0,0 @@
-# Architecture Part 4 — Control-plane authorization (operator signing)
-
-> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
-> To be reviewed by Claude Code against that spike + `03` §4, then placed at
-> `docs/architecture/04-control-plane-authorization.md`.
->
-> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
-> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
-
-## 1. Purpose & scope
-
-`03` §4 gates **destructive/irreversible** operations behind an operator signature the hub cannot
-forge. That gate is only real if signing is real. This doc defines the signing mechanism: the
-primitive, the keys, rotation, the three components' roles, and the operator workflow. The
-*policy* (what needs a signature) lives in `03` §4; this is the *how*.
-
-**Recap of what needs a signature** (from `03` §4, by reversibility, not by verb): destroying or
-overwriting any resource holding the only/primary copy of customer data — live-guest destroy,
-storage detach/wipe, restore-overwrite, decommission — **regardless of whether it arrives as a job
-or a desired-state delta**. Benign convergence (deploy a guest, attach storage, restore to a *new*
-guest, bump a version) runs on normal hub auth, unsigned. Most recovery is therefore unsigned;
-signed ops are rare and deliberate.
-
-## 2. Primitive — SSH signatures (SSHSIG)
-
-Confirmed by Phase 4: destructive ops carry an **SSH signature** (`ssh-keygen -Y sign`, the armored
-`SSHSIG` format), verified by the agent in Go (`golang.org/x/crypto/ssh`) — `pem.Decode` →
-`ssh.Unmarshal` → `ssh.ParsePublicKey` → `pub.Verify`. ~40 lines of framing, no hand-rolled crypto.
-
-**Why SSHSIG and not raw Ed25519 / minisign:** SSHSIG verification dispatches on the key type
-embedded in the signature, so the **same verifier accepts a software key (`ssh-ed25519`) today and
-a FIDO2 hardware key (`sk-ssh-ed25519@openssh.com`) later** — which is exactly the hardware-ready
-foundation we want (§7). A raw-Ed25519 verifier cannot consume an sk signature (flags+counter,
-different signed-data), so it would force a verifier change on every box at hardware-adoption time.
-SSHSIG buys key-type-agnosticism for a one-file framing cost (Phase 4 §5–6).
-
-### 2.1 The signed object — canonical op blob
-The signature covers an op blob (Phase 4 §2):
-
-```
-{ op, target:{host_id, guest_id}, params, nonce, issued_at, expires_at, key_id }
-```
-
-- **Canonical form is a *signer-side* requirement** — JSON, keys sorted at every level, no
-  insignificant whitespace, UTF-8 — so the blob is deterministic and human-auditable. The
-  **verifier trusts the exact bytes it receives** (it verifies the signature over the raw bytes and
-  parses those same bytes for fields), so there is no canonicalization-mismatch risk on the verify
-  side. The canonical form is the shared contract between the operator CLI and the agent (both Go).
-- `nonce` ≥128-bit random; `issued_at`/`expires_at` a short window (minutes); `key_id` identifies
-  the signing key (rotation/audit).
-
-### 2.2 Domain separation — the namespace
-The SSHSIG **namespace** `felhom-op-v1` is a **fixed constant in the verifier**, never
-caller-supplied. A signature minted for any other namespace must not verify (proven). This stops a
-signature made for one purpose being reused for another.
-
-### 2.3 Verify pipeline (order is load-bearing)
-`namespace → allow-list → crypto verify → target binding → time window → nonce`. The **nonce is
-recorded last**, only after everything else passes, so an invalid signature can never consume a
-nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
-- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
-  A cannot be replayed at box B);
-- **time window** — `now ∈ [issued_at, expires_at]`;
-- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
-  and expiry-pruned; a non-persistent store reopens the replay window after a restart).
-
-The Phase-4 reference verifier (`VerifySignedOp`) is the seed of the agent's implementation.
-
-## 3. The keys — two-key model, software now
-
-Both software (SSH-format) keys today; both are also valid FIDO2-resident keys later with no box
-change (§7).
-
-- **Operational signing key** — the "master stamp" for destructive ops. A **dedicated** key (NOT
-  the operator's daily SSH login key), passphrase-protected, on the operator workstation. Used only
-  for destructive ops — rare, so its exposure is low.
-- **Cold recovery key** — generated once, kept **offline** (password manager / a USB held back /
-  printed). Never used for ordinary ops; its sole power is to authorize rotating the operational key
-  if that key is lost or compromised.
-
-Both **public** keys are pinned onto the agent at enrollment (the allowed-signers set). The
-operational key is authorized for ops; the recovery key is authorized **only** for key-rotation
-instructions.
-
-**Allowed-signers is a set** → single signer today; **quorum (N-of-M) for the highest-blast ops is
-just set sizing + a threshold policy**, addable later without a redesign (Phase 4 §8). Out of scope
-now.
-
-## 4. Rotation & compromise recovery
-
-The agents pin the operator public keys. The danger: rotation must **not** flow as plain hub config,
-or a compromised hub re-pins its own key and forges everything. So **every re-pin is itself a signed
-op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
-
-- **Planned rotation:** the *current* operational key signs a "new operational public key = X" op;
-  the agent accepts it because it's signed by the trusted current key (key-signs-key).
-- **Operational key lost/compromised:** the **cold recovery key** signs the re-pin; the agent accepts
-  it because the recovery key is pinned and authorized for rotation. The compromised key is removed
-  from the allowed set in the same signed op.
-- **Both keys gone:** on-site physical re-enrollment (last resort — re-establishes the trust root the
-  way initial enrollment did).
-
-## 5. Component roles
-
-- **Operator tooling (the workstation).** A signing CLI behind a thin **`Signer` interface**
-  (`Sign(blob) → signature`). The backend today is a **file key**; a **FIDO2/PIV** backend drops in
-  later (§7) with no change to the blob format, the hub, or the agent. Holds the operational private
-  key (passphrase-protected); can reach the cold recovery key when rotation is needed.
-- **Hub.** Queues the **opaque** signed blobs and surfaces pending destructive ops + their signature
-  status in the operator UI. Holds **no** private key and cannot sign — a compromised hub can only
-  queue blobs the agent rejects. (Matches `03` §4 / box-initiated poll.)
-- **Agent (each box).** Pins the allowed-signers set (operational + recovery) at enrollment; runs the
-  verify pipeline (§2.3) on any destructive op before executing; writes every signed op to the
-  customer-visible **audit log**. Notification-on-destructive-op is an audit signal, never the guard
-  (a compromised hub could issue *and* suppress notice — the signature is the control).
-- **Enrollment.** Pins the initial operational + recovery public keys onto the agent during the
-  physical-presence provisioning step (the trust root is established on-site, not via the hub).
-
-## 6. Operator workflow
-
-- **Routine work** (deploy, monitor, attach storage, restore to a *new* guest): no signing, zero
-  overhead.
-- **A destructive op** (rare): the operator runs the signing CLI on their workstation — which builds
-  the canonical blob, signs it (passphrase, or later a hardware touch), and posts it to the hub
-  queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
-  desk. **Never** a site visit.
-
-## 7. Hardware readiness (Viktor's "build the foundation now")
-
-Software `ssh-ed25519` now; a FIDO2 `sk-ssh-ed25519@openssh.com` key later is a **no-op on the
-boxes** — proven end-to-end against the OpenSSH spec in Phase 4 §5 (the unchanged verifier accepts a
-spec-faithful sk signature). At hardware adoption the operator generates an sk-key, points the
-`Signer` backend at it, and updates the allowed-signers entry; nothing on the boxes changes.
-
-Two honest notes:
-- **Confirm with a real device at adoption.** §5 was validated to spec, not against live hardware —
-  a 5-minute real-key round-trip should confirm it (no surprise expected; signer/library/device all
-  follow the same spec).
-- **Optional future hardening:** require the FIDO2 **user-presence (touch) flag**. The verifier is
-  crypto-only today (correct for software keys); enforcing the flag is a small later option once
-  hardware is in use.
-
-## 8. Open items
-- **Quorum policy** (N-of-M per op-class, e.g. two signatures for decommission) — deferred; the
-  allowed-signers-set foundation supports it.
-- **Signing-key passphrase UX** on the workstation (ssh-agent / askpass) — minor operator-tooling
-  detail.
-- **Hub-side pending-op UI** (showing ops awaiting signature + audit) — belongs to the hub doc.
-
-## 9. What this unblocks
-Closes the `03` §4 "undesigned signing path." Hands the implementation: the **canonical blob spec**
-(§2.1) + the **`VerifySignedOp` reference** (Phase 4 §7) for the agent's verify path, the
-**`Signer` interface** for the operator CLI, and the **allowed-signers pinning** step for enrollment.
-The hub's signed-job queue + pending-op UI carry into the hub architecture doc.

docs/architecture/05-hub-architecture.md

@@ -1,223 +0,0 @@
-# Architecture Part 5 — The Hub
-
-> Status: design draft (decision content). To be validated by Claude Code against the **actual
-> felhom-hub source** (`felhom.eu` repo, `hub/`) + Parts 01–04, then placed at
-> `docs/architecture/05-hub-architecture.md`.
->
-> The hub is **not** greenfield — it's a mature service (felhom-hub v0.6.3, Go + SQLite on k3s,
-> `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus the new
-> data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
-
-## 1. Source-of-truth model — two drivers, two directions
-
-The single most important framing, and the one that governs everything below: the hub is **not** a
-monolithic source of truth. State flows in two directions with opposite drivers.
-
-- **Operator-driven *intent* — hub authors, agent reconciles (top-down).** Which guests should
-  exist and their spec, storage *policy* (a target's role/class/backup schedule), controller +
-  golden-image versions, identity, tunnel. The operator sets these in the hub; the agent converges
-  toward them. Here the hub *is* the source of truth.
-- **Box/customer-driven *reality* — box authors, pushes up, hub mirrors (bottom-up).** Which USB
-  drive is *physically* attached (and its `durable_id`), what apps are deployed and where, the
-  customer's controller configs/settings, host/guest health, latest PBS snapshot pointers. The
-  customer or the physical world drives these; the box reports them; the hub stays an up-to-date
-  **mirror** but is **never** the driver.
-
-They meet at a **handshake**, not a tug-of-war. Storage is the clearest case: the customer plugs in
-a drive → the agent *detects* it and reports `durable_id X attached` (reality) → the operator
-assigns `role=bulk, class=slow, backup=weekly` (policy, intent) → the agent reconciles that policy
-*onto the detected drive*. **Apps never enter the reconcile loop** — app deployment is the
-controller's domain (customer- or operator-driven, inside the guest); the hub only mirrors the
-resulting inventory. **Reconciliation applies to infrastructure; the app/customer layer is mirrored.**
-
-## 2. Data model (Part 1 decision (b): customer-anchored)
-
-A customer's deployment is one **Host** (its agent) plus one-or-more **Guests** (its controllers).
-1 customer = 1 host + N guests; the shared-host multi-tenant case is deferred (not precluded — the
-`hosts` table is the seam it would use).
-
-- **`customer_configs`** (existing) — the Customer anchor: identity, domain, email,
-  `retrieval_password`, status, config_json. Unchanged role.
-- **`hosts`** (new) — `host_id PK, customer_id, api_key` (the agent's hub key), `agent_version`,
-  desired-state intent (storage manifest + policies + golden-image version, as JSON), a per-host
-  **`desired_generation`** counter, the slim DR record (§9), timestamps.
-- **`guests`** (new) — `guest_id PK, customer_id, host_id, api_key` (the controller's hub key),
-  `display_name, controller_version`, per-guest **`desired_spec_json`** (CPU/mem/disk, versions),
-  timestamps.
-
-**Per-reporter keys:** today's per-customer `customer_configs.api_key` becomes per-reporter —
-`hosts.api_key` (agent) and `guests.api_key` (controller). The hub resolves a presented Bearer key →
-host or guest → customer; `customer_configs.api_key` goes unused once auth resolves via the new keys.
-**Clean cutover:** no dual-model support; the demo re-enrolls fresh into `host + guests`.
-
-## 3. Report ingest — two domains
-
-The single controller report splits. The de-privileged controller no longer sees host disks/storage/
-backup, so its report **slims** (it loses System/Storage/Backup, keeps app-domain).
-
-- **`POST /api/v1/host-report`** (new, agent) → **`host_reports`**: host CPU/RAM/disk, per-guest
-  up/down + spec, storage-target status (attached drives + `durable_id` + reachability), last backup
-  + restore-test per target, latest PBS snapshot pointers, `cloudflared` health, agent + controller
-  versions. Denormalized columns for the dashboard; full `report_json`. Index `(host_id, received_at
-  DESC)` + `(customer_id, received_at DESC)`.
-- **`POST /api/v1/report`** (existing, slimmed controller) → the renamed **`guest_reports`**: it
-  gains `guest_id` + `host_id`; its `cpu/memory` denorm now means *guest-level*; `backup_last_snapshot`
-  goes quiet (backup status lives in `host_reports`). App telemetry / log issues stay.
-
-These two streams are the bottom-up mirror of §1 — they keep the hub current without a separate push.
-
-## 4. Liveness / dead-man's-switch
-
-Evolves the existing staleness checker (60s **cadence**, 30m/1h **thresholds** — OK <30m, down at
-2× = >1h; today: controller-report recency → `node_stale`/`down`/`recovered`):
-
-- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
-  liveness signal; a silent agent = the box is gone (the critical alert).
-- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
-  than waiting for a guest report to go stale.
-- **Guest-report recency = secondary** app-level signal.
-
-**Backup-deadline checker:** today it is *event-based* — it scans for `backup_completed`/`backup_failed`
-events since local midnight and alerts if none. Two changes: (1) **mechanism** — move it to a field
-check on `host_reports`' last-backup-per-target (cleaner now that backup state arrives in the host
-report); (2) **emitter** — the de-privileged controller no longer runs backups, so the **agent** is the
-source of the last-backup status (Part 3 §8). Without re-homing the source, the deadline check would go
-silent after the controller stops backing up.
-
-## 5. Desired-state serving
-
-The operator's **intent** (§1 top-down) lives as JSON on `hosts`/`guests` (storage manifest +
-policies + golden version on the host; per-guest spec + versions on the guest) with a per-host
-`desired_generation`. The agent pulls its host's desired state on poll (with the generation, so it
-reconciles only on change and reports which generation it has converged to).
-
-- **Benign convergence** (create a guest, attach storage per policy, bump a version, adjust a
-  non-destructive policy) → the agent reconciles freely.
-- **Destructive convergence** (guest removal = destroy, storage detach/wipe, data-losing resize) →
-  the agent requires a **matching signed op** (§6) before executing that delta; absent/invalid → it
-  refuses and reports `pending_signature`.
-
-**Geo is *not* in the agent's desired state** — it's customer→hub→Cloudflare (§7); the agent never
-touches WAF.
-
-## 6. Authorization — signed-op queue + editing flow
-
-Implements Part 4's gate on the hub side. The hub holds **no signing key**.
-
-- **`signed_ops`** (new): `op_id, customer_id, host_id, target_guest, op_type, op_blob (canonical
-  JSON), signature (armored SSHSIG), status (pending_signature → signed → delivered → executed /
-  failed / expired / rejected), nonce, issued_at, expires_at, executed_at, result`.
-- **Editing flow:** the operator edits a customer's desired state, reusing the existing config-form +
-  diff UX. Note the **transport inverts**: today's "Push" is a hub→box *inbound* POST (forbidden by the
-  box-initiated model); here "publish" means **write to desired state, delivered on the next agent/
-  controller poll**. The form and diff carry over; the push transport does not. The hub diffs vs current
-  and **classifies each delta** (B1 rule):
-  - **benign** → published straight to desired state;
-  - **destructive** → the hub generates the canonical op blob and routes it through signing.
-- **Signing hand-off (Part 4 option (b)):** a local operator CLI (`felhom-sign --pending`) fetches
-  the pending blob from the hub, signs it on the workstation with the dedicated key, and posts the
-  signature back into `signed_ops`. The hub never sees the key.
-- The agent polls `signed_ops` for its host alongside desired state, verifies (Part 4 pipeline),
-  executes, and reports status → the hub logs to the existing **`events`** audit trail.
-- **Classification lives in both places, with different jobs:** the hub classifies at *edit time*
-  for UX (prompt to sign); the **agent's classification is the authoritative guard** (a compromised
-  hub could skip the prompt, but the agent still enforces the signature).
-- A **pending-ops view** per customer shows the lifecycle (awaiting signature → awaiting agent →
-  executed).
-
-## 7. Geo enforcement (Part-2 S4)
-
-The hub already holds the CF API token and already has a remove-all path
-(`internal/web/configs.go` `handleGeoDisable` → `cloudflare.RemoveGeoRules`). **But the token is
-dual-purpose today** — DNS-01/ACME *and* WAF/geo — and `configgen.Generate` deep-merges it (via
-`config_json`) into the generated `controller.yaml`, so it currently ships **down to the box**. Two
-things follow:
-
-- **ACME assumption (must be stated, not skipped):** in the Cloudflare-Tunnel-default model the edge
-  terminates TLS, so the box needs no public certificate and the **DNS-01/ACME use of the token goes
-  away**. Granting that, the token comes fully off the box and lives hub-only. (If any box still does
-  DNS-01, the token cannot fully come off — so this assumption is load-bearing.)
-- **`configgen` must stop emitting `cf_api_token`** into `controller.yaml` (drop it from the merge /
-  relocate it to a hub-only field).
-
-The delta: the **customer sets geo in the controller UI → the controller reports the geo desired-state
-up → the hub reconciles it into the Cloudflare WAF** (rather than the box calling the CF API). The hub
-keeps the remove-all override for self-lockout. The controller no longer calls the CF API.
-
-## 8. Enrollment (evolution of the existing retrieval-password/config-gen flow)
-
-Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase) returns a deep-merged
-`controller.yaml`. New:
-
-- Enrollment mints the **agent identity first** (the agent then provisions controllers), pins the
-  **operator signing public keys** (Part 4 — operational + cold recovery) onto the agent, and the
-  agent mints each controller's bootstrap (its hub guest key + local-API token).
-- A **restore-mode** re-enrollment (§9) hands an existing identity to a fresh agent.
-
-The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the
-agent-first + key-pinning + restore-mode steps.
-
-## 9. DR model
-
-The headline: the **old heavy infra-backup push retires** — not because the hub authors everything
-(§1 says it doesn't), but because (a) the box-driven mirror already arrives via the §3 report streams,
-and (b) the actual app **data + configs live inside the PBS guest snapshot**. So a separate
-config+secrets+restic-password infra-backup blob is redundant.
-
-What remains:
-- the **report streams** keep the hub's mirror current (storage layout + `durable_id`s, app inventory,
-  snapshot pointers) — but this mirror is **convenience, not the DR source of record** (reports are
-  pruned by age);
-- the agent **escrows the recovery-code-wrapped PBS key** to the hub (the one artifact only the box
-  can produce — zero-knowledge: the hub stores it, cannot open it);
-- a **slim DR record** on the `hosts` row (PBS namespace + repo fingerprint + the wrapped escrow key).
-  These last two are *box-reported* columns on an otherwise operator-intent row — labelled as such so
-  the §1 two-driver split stays legible per column.
-
-Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
-hit it) **and** `infra_backups` (the deprecated legacy mirror). The slim DR record folds onto `hosts`
-instead. The **controller's infra-backup push is removed** (it's de-privileged).
-
-**Recovery (host loss):** the new agent re-enrolls in **restore mode**; the hub hands it the durable
-record — and DR reads from the **durable sources, not the prunable report mirror**: operator intent
-(desired-state on `hosts`/`guests` — identity, tunnel token, storage manifest), the slim DR record
-(PBS namespace + repo fingerprint), the **wrapped escrow key**, and **PBS's own snapshot enumeration**
-(the agent lists snapshots once it has the namespace + unwrapped key). Guest inventory + app data come
-from **inside the PBS guest snapshots**, not from a retained `host_report`, so recovery doesn't degrade
-when the last report has aged out. The **customer provides their recovery code at the agent**, which
-unwraps the PBS key locally (never sent to the hub); the agent restores guests from PBS, resets
-identity, reuses the tunnel. The customer recovery code is the irreducible residual (the premium
-operator-managed custody tier avoids it, at the cost of the operator holding the key). The old
-controller-targeted `GET /recovery/{id}` is replaced by this agent restore-mode flow.
-
-## 10. What persists from today (unchanged or lightly adapted)
-
-The Customer record (`customer_configs`); config generation/retrieval (`configgen`); the two-tier
-notification system (operator English / customer Hungarian, Resend, cooldowns); `events` + audit;
-`app_telemetry` / `app_log_issues`; customer lifecycle actions (block/unblock, trigger-update,
-delete); the asset manager; and the dashboard — adapted to render the **host + guests** view per
-customer instead of a single controller.
-
-## 11. Schema deltas (grounded in store.go's idempotent style; clean cutover)
-
-- **NEW:** `hosts`, `guests`, `host_reports`, `signed_ops`.
-- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
-  migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
-  guest-level; `backup_last_snapshot` goes quiet.
-- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
-  slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
-- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
-  record replaces them on `hosts`.
-- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
-  `app_telemetry`, `app_log_issues`.
-- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth
-  rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup
-  push; any that carry over should scope to the resolved host/guest → customer under §2.
-
-## 12. Open items
-- Operator signing-key operational mechanics (Part 4 §8) — the hub-side pending-op UI is here; the
-  key custody/rotation tooling is Part 4's.
-- Multi-tenant resource fairness (deferred shared-host case).
-- Hub-side desired-state **editing UX** specifics (form/diff wiring) — to be grounded against
-  `hub/internal/web/configs.go` at implementation.
-- Golden-image refresh cadence / fleet versioning (carried from Part 3 §13).

docs/architecture/_design-review.md

@@ -1,260 +0,0 @@
-# Critical design review — Proxmox re-platform doc set
-
-> ✅ **RESOLVED (2026-06-08).** All findings folded into 01/02/03 + `proxmox-platform.md`
-> (Phase-3 spike run for B2/B3 → `tests/phase3-findings.md`). **Folded:** B1 (03 §4), B2
-> (03 §7/§8 + platform §4.7), B3 (03 §2/§3 + platform §3.6), S1 (03 §6/§8), S2 (03 §10/§11),
-> S3 (03 §7), S4 (01 §5/§7 + 02 + 03 §2), S5 (01 §7/§11 + 02 §6), S6 (02 §5), M1 (02 §3),
-> M2 (03 §7), M3 (03 §10), §6-residual (03 §6). Plus the two Phase-3 design updates:
-> provision-by-restore (03 §9) and the settled root-vs-API boundary (03 §3). **Deferred/none:**
-> no finding was deferred; the pre-existing open items (operator signing-key mechanics,
-> multi-tenant fairness, hub-side desired-state UX, golden-image refresh cadence) remain
-> flagged in 03 §13. This artifact can be deleted once confirmed.
-
-Working artifact. Review pass over `01-topology-and-trust.md`, `02-controller-module-map.md`,
-`03-host-agent.md`, `proxmox-platform.md`, and the Phase 0 / Phase 1-2 findings, grounded
-against the v0.33 source (`felhom-controller/controller/`). Every finding cites a
-file+line or a doc section. Severity: **blocking** / **should-fix** / **minor**.
-
-Two findings are self-corrections of my own earlier work (`02` and `proxmox-platform.md`) —
-flagged as such.
-
----
-
-## Ranked summary
-
-| # | Severity | Finding | Where |
-|---|---|---|---|
-| B1 | **blocking** | Reversibility gate contradicts the self-heal reconcile loop — crashed-guest healing can require a signature-gated destroy → reconcile stalls | `03` §4 vs §4(a) |
-| B2 | **blocking** | vzdump bulk-exclusion only works for **volume** mount points; Docker **named volumes live in the LXC rootfs and ARE captured** → naive placement silently backs up the 1 TB media drive. Unvalidated by spike. | `03` §7 vs `proxmox-platform.md` §4.3 + pct manpage |
-| B3 | **blocking** | Agent's Proxmox role is called "the minimal role from Phase 1" — but that role is the *narrow self-backup* role that Phase 1 proved is **denied** create/allocate/restore. The agent's operator-tier role is undefined. | `03` §2/§3 vs `phase1-2` §1.3-1.4, `01` appendix |
-| S1 | should-fix | Quiescing for agent/hub-scheduled backups has **no agent→controller channel** — the local API is controller→agent only | `03` §6, §8 |
-| S2 | should-fix | Agent self-update revert authority unspecified — if the new binary won't boot, nothing outside it can flip back | `03` §11 |
-| S3 | should-fix | Storage manifest drops fields `settings.StoragePath` carries today (Label, Schedulable/default, StoppedStacks, MigratedTo) with no re-homing stated | `03` §7 vs `settings.go:90-103` |
-| S4 | should-fix | Geo-restriction WAF ownership + Cloudflare **API token** placement unspecified after tunnel placement was locked; zone-wide token in a guest is a blast-radius concern | `03` (absent), `01` §3, `config.go` InfrastructureConfig |
-| S5 | should-fix | Cross-doc staleness: `01` §11 still lists tunnel placement OPEN; `02` §6 lists geo "blocked on tunnel placement" — both resolved by `03` §13 | `01` §11, `02` §6 vs `03` §13 |
-| S6 | should-fix (self-correct) | `02` put self-restore-test **orchestration** in the controller; `03` correctly makes it agent-owned (controller only reads status) | `02` §5(3) vs `03` §6/§8 |
-| M1 | minor (self-correct) | `02` §3 lists `UUID` as a `settings.StoragePath` field — it isn't; UUID is derived from fstab at runtime | `02` §3 vs `settings.go:91-103` |
-| M2 | minor | `03` §7 says the manifest "absorbs the disk-state fields StoragePath carries today" incl. UUID — UUID isn't persisted today, so the manifest *adds* it (an improvement, not absorption) | `03` §7 |
-| M3 | minor | controller-update is not in `03` §10's journaled-ops list, though it's a multi-step async op | `03` §10 vs §11 |
-
-**Values check: clean.** No DR/key-custody/offboarding path leaves a customer locked out.
-Zero-knowledge DR (`03` §8, `01` §8) correctly makes the customer recovery code the
-irreducible residual; the operator cannot read data and the box can still restore-test.
-No hostage path found.
-
-**Locked premises:** reviewed for soundness/consistency only; not relitigated.
-
----
-
-## Blocking findings
-
-### B1 — The reversibility gate stalls the self-healing reconcile loop
-**Where:** `03` §4(a) vs the gate in §4.
-**What:** §4(a) lists "redeploy of a crashed controller" as benign convergence that "falls
-out of reconciliation for free." The gate then lists **guest destroy** among the
-irreversible ops that require an operator signature "*regardless of whether they arrive as a
-job or as a desired-state delta*." These collide: if healing a wedged guest requires
-destroy+recreate (corrupt rootfs, failed in-place restart, half-built guest from an
-interrupted provision), the reconciler hits a signature-gated op and **cannot proceed
-without an operator** — the loop either stalls or silently gives up, defeating "self-healing
-… tolerant of missed polls."
-**Why it matters:** This is the security-critical control model. A fuzzy benign/destructive
-line is unimplementable: either the reconciler can destroy (and a compromised hub's desired
-state can wipe guests — the exact threat §4 exists to stop), or it can't (and self-heal is a
-fiction for the crashed-guest case).
-**Grounding:** `03` §4 self-describes the gate as "security-critical"; §9/§10 already rely on
-the reconciler rolling back "a half-built guest" — which *is* a destroy of a customer-id-bound
-resource, contradicting the blanket "guest destroy needs a signature."
-**Suggested fix (crisp, implementable rule):** Scope the reconciler's destructive verbs by
-*provenance and data-bearing-ness*, not by verb:
-- The reconciler MAY, without a signature: (a) create/start/restart; (b) destroy resources it
-  **created earlier in the same journaled transaction** (compensating rollback, §10); (c)
-  destroy resources **tagged ephemeral/scratch** (restore-test scratch guests, §8).
-- Destroying or overwriting any resource that **holds the only/primary copy of customer data**
-  always needs an operator signature.
-- **Healing a crashed controller is non-destructive by construction:** the controller is
-  reconstructable from its image + the guest's persistent volume, so "redeploy" = restart the
-  LXC / `docker compose up -d` **inside the existing guest** — never a guest destroy. State
-  this explicitly so the two clauses stop colliding. (The v0.33 self-heal precedent is already
-  in-place restart: `watchdog.go` restarts stopped stacks, it never destroys the guest.)
-
-### B2 — vzdump bulk-exclusion: the rootfs-Docker-volume trap
-**Where:** `03` §7 ("Bulk external mounts are excluded from the guest's vzdump (a per-mount
-backup flag)").
-**What:** Two grounded problems:
-1. The flag is real but narrow. The pct manpage (verified): `backup=<boolean>` —
-   *"Whether to include the mount point in backups (**only used for volume mount points**)."*
-   It does **not** apply to bind mounts / device mounts (those are handled separately).
-2. The trap: `proxmox-platform.md` §4.3 (validated in `phase1-2` §2.2) proved that **Docker
-   named volumes live inside the LXC rootfs and ARE captured by vzdump** — a sentinel in
-   `pgdata` survived. The default Felhom app uses Docker named volumes. So unless bulk data is
-   deliberately placed on a **dedicated Proxmox volume mount point** (backup=0) or a bind
-   mount, a "bulk" volume will be an ordinary named volume in rootfs and will be **silently
-   swept into the whole-guest image** — exactly the 1 TB-media-in-every-backup outcome §7 says
-   it prevents.
-**Why it matters:** Backup size/cost and RPO blow up silently; the failure is invisible until
-a media drive fills the vzdump target. This is load-bearing for the §8 tier model.
-**Grounding:** pct manpage (fetched 2026); `proxmox-platform.md` §4.3; `phase1-2` §2.2.
-Not covered by any spike — `proxmox-platform.md` §6 "not yet validated" should gain this row.
-**Suggested fix:** Make the placement contract explicit: a `bulk` volume **must** be realized
-as a dedicated LXC mount point (volume mountpoint with `backup=0`, or an external bind mount),
-**never** a Docker named volume in rootfs. The per-volume placement component (`02` §5(2))
-must enforce this at deploy. Add a Phase-3 spike: create an LXC with a `backup=0` volume
-mountpoint + a bind mount, vzdump it, confirm both are excluded and the rootfs+`backup=1`
-volume are included.
-
-### B3 — The agent's Proxmox role is mis-grounded as "the Phase-1 minimal role"
-**Where:** `03` §2 ("scoped Proxmox API token (minimal role from Phase 1)"), §3 ("the
-Phase-1 minimal role is the API floor").
-**What:** Phase 1's minimal role (`FelhomSelfBackup` = `VM.Audit, VM.Snapshot, VM.Backup,
-Datastore.AllocateSpace, Datastore.Audit`) is the **narrow self-backup** role scoped to one
-guest, and Phase 1 explicitly proved it is **denied (403)** on create/allocate
-(`phase1-2` §1.3 call #7) — i.e. exactly the operator-tier ops the agent's whole job consists
-of (provision, restore, storage allocation). Worse, `01` appendix states that guest-side role
-"**is not used** — we chose the agent-mediated path." So `03` cites, as the agent's role
-floor, a role that (a) the architecture discarded and (b) is provably insufficient for the
-agent.
-**Why it matters:** The agent's actual operator-tier role is **undefined**. Provisioning,
-restore, and storage management cannot be built or hardened against an undefined privilege
-set, and §3's root-minimization argument ("the Phase-1 minimal role is the API floor")
-collapses because that floor can't create a guest.
-**Grounding:** `phase1-2` §1.3 (create CT = 403), §1.4 (role = self-backup only); `01`
-appendix ("not used … confirmed restore = operator-tier"); `proxmox-platform.md` §3.4.
-**Suggested fix:** Replace the Phase-1 reference with a **new agent operator role** to be
-defined and least-privilege-tested in a Phase-3 spike — minimally `VM.Allocate`, `VM.Config.*`,
-`VM.PowerMgmt`, `VM.Snapshot(.Rollback)`, `VM.Backup`, `VM.Audit`, `Datastore.Allocate(Space)`,
-`Datastore.Audit`, plus whatever storage-attach needs (see S4/root-boundary below). Keep §3's
-"API token, not root, where the API suffices" principle — that part is sound — but stop
-calling it the Phase-1 role.
-
----
-
-## Should-fix findings
-
-### S1 — No agent→controller channel for backup quiescing
-**Where:** `03` §6 (local API is controller→agent only) vs §8 ("the controller stops the app
-stack … before a guest vzdump where app-consistency matters").
-**What:** App-consistent LXC backup requires the controller to quiesce (no fsfreeze for LXC —
-`proxmox-platform.md` §4.2, `phase1-2` §2.1). But the §6 surface is entirely controller→agent;
-the box-initiated model forbids the hub calling in, and there is no agent→controller call
-defined. For a **hub/agent-scheduled** backup (schedule lives in the manifest `policy`, §7),
-the agent has no way to tell the controller "quiesce now."
-**Why it matters:** Either scheduled backups silently fall back to crash-consistent (relying
-on WAL recovery, which `phase1-2` §3 warns is unvalidated under write load), or the feature
-can't be built as drawn.
-**Suggested fix:** Make backups **controller-driven for app-consistency**: the controller
-learns due/policy via its own hub channel (or a `GET /backup/due` on the local API), quiesces,
-calls the existing `POST /backup`, then unquiesces on completion. Document that agent-initiated
-vzdump is crash-consistent only. (No inbound-to-guest channel needed — preserves §3/§5.)
-
-### S2 — Agent self-update revert authority unspecified
-**Where:** `03` §11 ("a watchdog reverts to last-good if the new binary fails to come up
-healthy").
-**What:** The agent is a single host systemd service with `Restart=always` (§3). If the new
-binary crashes on startup, systemd just restarts the **same bad binary** in a loop. "Revert
-to last-good" cannot be done *by* the thing that won't boot. §11 doesn't name the actor.
-**Why it matters:** A bad self-update can brick the crown-jewel host agent — the one component
-that recovers everything else — with no automatic recovery, requiring break-glass.
-**Suggested fix:** Put revert authority **outside** the swapped binary: e.g. an A/B symlink
-(`current → good|new`) where a separate systemd oneshot health-gate (`ExecStartPost` probe; on
-failure flip the symlink back and restart), or a tiny supervisor unit. Boot-into-last-good +
-explicit "commit" after a clean health window is the robust pattern. Add agent-update to the
-§10 journal so an interrupted swap is resumable.
-
-### S3 — Manifest schema omits live `StoragePath` fields without re-homing them
-**Where:** `03` §7 table vs `settings.go:90-103`.
-**What:** Today's `StoragePath` carries `Label`, `IsDefault`, `Schedulable`, `StoppedStacks`,
-`Decommissioned`/`DecommissionedAt`/`MigratedTo`. The manifest covers state (attached/
-disconnected/decommissioned) and durable_id, but drops: **Label** (human name, e.g. "Külső
-HDD 1TB" — UI), **Schedulable/IsDefault** (default placement target for new apps),
-**StoppedStacks** (which apps to restart on reconnect — app-domain), **MigratedTo** (decommission
-target pointer).
-**Why it matters:** `02` named this manifest as the contract that the `settings.StoragePath`
-reshape depends on. Silently dropped fields become lost behavior (no default-drive choice, no
-restart-after-reconnect list, no friendly labels).
-**Suggested fix:** Either add Label + a placement-default marker to the manifest, or explicitly
-state which fields re-home to the controller's `settings` (StoppedStacks and Label are
-plausibly controller-side; default/schedulable placement must live wherever placement decisions
-are made). Make the split explicit so neither side assumes the other owns it.
-
-### S4 — Geo-WAF ownership + Cloudflare API token placement unspecified
-**Where:** `03` covers `cloudflared` (tunnel) health but is silent on geo-restriction WAF; `02`
-§6 had `cloudflare/`+`geo` "blocked on tunnel placement"; `01` §3 lists the controller's creds
-as "hub API key + local-API token" only.
-**What:** Now that tunnel placement is locked (host), the **geo-restriction WAF** management
-(`cloudflare/` package: zone/waf/geosync) still has no home. It requires a Cloudflare **API
-token** (`config.go` InfrastructureConfig.cf_api_token) with zone-wide WAF edit rights. If geo
-stays in the controller (app-domain, per `02`), a **zone-wide Cloudflare token sits inside the
-customer guest** — a real blast-radius concern (compromise → edit/disable WAF for the whole
-zone, potentially other customers on the same zone).
-**Why it matters:** Trust-boundary gap. `01` §5's boundary table has no row for controller↔
-Cloudflare-API. Unspecified ownership blocks the `02` geo classification from being unblocked.
-**Suggested fix:** Decide geo-WAF ownership explicitly and add it to `01` §5. Options: (a) move
-WAF management to the **agent/hub** (operator-tier, token off the customer box); (b) keep it in
-the controller but scope the CF token per-zone/per-customer if the account model allows. Note
-this is now *unblocked* by the tunnel decision and should leave `02` §6's "blocked" state.
-
-### S5 — Cross-doc staleness on the now-locked tunnel placement
-**Where:** `01` §11 ("Cloudflare Tunnel placement: host vs guest (§7)") and `02` §6
-("`cloudflare/` + `api/geo.go` — blocked on tunnel placement") vs `03` §13 ("Resolved here:
-tunnel placement (host, agent-managed)") and the LOCKED list.
-**What:** `01` and `02` still present as OPEN/blocked a decision `03` and the locked set have
-resolved.
-**Why it matters:** A dev reading `01`/`02` would treat a settled decision as open (or a
-classification as blocked when only geo-ownership, S4, actually remains).
-**Suggested fix:** When folding this review in: update `01` §7/§11 to record tunnel=host
-(agent-managed systemd service); update `02` §6 to reduce the cloudflare item from "blocked on
-tunnel placement" to the narrower "blocked on geo-WAF ownership (S4)."
-
-### S6 — (self-correction) self-restore-test orchestration belongs to the agent, not the controller
-**Where:** `02` §5(3) said "Self-restore-test orchestration — *controller* asks the agent to
-restore to scratch guest, validates, reports." `03` §8 makes the **agent** drive it
-autonomously; §6 gives the controller only `GET /restore-test/status` (read-only).
-**What:** `03` is right and `02` overreached. Zero-knowledge means only the box/agent holds the
-PBS key (`03` §8); creating a scratch guest is operator-tier (create/allocate — `phase1-2`
-§1.3 #7); the controller cannot do either. The controller's only piece is surfacing status.
-**Why it matters:** Keeps the NEW-component list honest — this is not a controller component to
-build beyond a status read.
-**Suggested fix:** Amend `02` §5(3) to "self-restore-test **status display** (read-only); the
-agent owns orchestration."
-
----
-
-## Minor findings
-
-- **M1 (self-correction):** `02` §3 lists `UUID` among `settings.StoragePath` fields. It is
-  **not** there (`settings.go:91-103`: Path, Label, IsDefault, Schedulable, AddedAt,
-  Disconnected/At, StoppedStacks, Decommissioned/At, MigratedTo). UUID is derived at runtime
-  from fstab / `/host-dev/disk/by-uuid` by `system.ParseFstabUUID` and `watchdog.go`. The
-  classification (settings = MODIFY/split) is unaffected; the field list was wrong.
-- **M2:** Consequently `03` §7's "absorbs the disk-state fields `settings.StoragePath` carries
-  today" overstates: `durable_id`/UUID is *not* carried today, so the manifest **adds** durable
-  identity (a genuine improvement — today the controller re-derives UUID from fstab each boot,
-  which is fragile). Reword "absorbs" → "absorbs + adds durable_id."
-- **M3:** `03` §10 journals "provision, restore" but not **controller-update** (§11), which is
-  also a multi-step async op (snapshot→pull→redeploy→health→rollback). Add it so an agent crash
-  mid-controller-update is resume-or-rollback like the others.
-
----
-
-## Verified-correct (no action) — grounding that held up
-
-- LXC flags `nesting=1,keyctl=1` + overlayfs (`03` §9) match `proxmox-platform.md` §2.3 /
-  `phase0` §3. ✓
-- async `task exitstatus`, not POST return (`03` §8) matches `proxmox-platform.md` §3.5. ✓
-- stop-mode backup not requiring `VM.PowerMgmt` (`03` §8 "per Phase 1") matches
-  `proxmox-platform.md` §3.4. ✓ (applies to the agent role too.)
-- running-LXC snapshot on LVM-thin (`03` §6/§8/§11) matches `proxmox-platform.md` §4.5 /
-  `phase1-2` §1.6. ✓
-- `monitor/pinger.go` deprecation (`02` DELETE-obsolete) confirmed in `main.go:168,175`
-  ("legacy, will be removed" / "no longer used — monitoring is now handled by the Hub"). ✓
-- backup keep/delete **intra-file tear** (`02` hazard) confirmed: `backup.go` holds both
-  `RunDBDumps`/`DumpAppVolumes(Safe)` (keep) and `RunBackup`/`RunFullBackup` (restic, delete);
-  `restore.go` holds `RestoreApp` (restic) + `RestoreAppFromTier2` (app). The §7-8 backup
-  contract gives the extracted app-data-backup package a coherent destination. ✓
-- Control-plane-not-data-plane (`03` §2/§43): apps keep serving if the agent dies — consistent
-  with Docker-in-LXC running independently (`phase0` §3). ✓
-- §6 per-guest local-API authorization (token→guest map): sound; a leaked token acts only on
-  its own guest. Residual: a compromised controller can `POST /rollback` its **own** guest
-  (blast radius = self) — acceptable per design; worth a one-line note that rollback is
-  self-scoped and bounded.

docs/architecture/_hub-review.md

@@ -1,221 +0,0 @@
-# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
-
-Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
-cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
-breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
-
-The two highest-value catches (doc assumes something the code contradicts) are **S1** and **S2**.
-
----
-
-## Ranked summary
-
-| # | What | Where (doc → code) | Severity |
-|---|---|---|---|
-| S1 | §9/§11 name the **wrong infra-backup table as current** — `infra_backup_versions` is the live/primary one; `infra_backups` is the deprecated write-only mirror | 05 §9/§11 → `store.go:198-217,541-578` | should-fix (code-contradiction) |
-| S2 | §7 treats the CF token as **geo-only**; it is **dual-purpose (DNS-01/ACME + WAF)** and is injected into the generated `controller.yaml` | 05 §7 → `config_form.html:76-80`, `controller.yaml.default:26`, `configgen.go:28-37`, `configs.go:1041` | should-fix (code-contradiction / unverified assumption) |
-| S3 | §6 leans on the existing **"Push"**, but that is a hub→box **inbound** POST — forbidden by the box-initiated model; transport must invert to poll | 05 §6 → `configs.go:569-570,1148-1150`; Part 1 §4/§5/§11; Part 3 §5 | should-fix |
-| S4 | Part 1 §6 calls app inventory **"declarative"**; 05 §1 (LOCKED) says apps are mirrored, never declared/reconciled, restored from PBS | Part 1 §6 ↔ 05 §1/§9 | should-fix (cross-doc) |
-| S5 | §9 hands "guest inventory + snapshots" **from the prunable report mirror**; DR soundness actually rests on durable sources | 05 §9/§3 → `store.go:809-816` | should-fix (DR robustness) |
-| S6 | §4 says backup-deadline checker "maps onto host_reports' last-backup field"; today it is **event-based** and controller-emitted | 05 §4 → `deadline.go:31-86` | should-fix (mechanism) |
-| M1 | "60s staleness checker" conflates the 60s **cadence** with the 30m/1h **threshold** | 05 §4 → `main.go:207-217,99-102`, `staleness.go:33-37` | minor |
-| M2 | §2 `customer_configs` field list omits `api_key` — the very field the per-reporter plan retires | 05 §2 → `store.go:102-112` | minor |
-| M3 | §11 `reports`→`guest_reports` "rename" is really drop+create under the locked clean cutover | 05 §11 → `store.go:55-119` | minor |
-| M4 | Pre-existing weak authz on infra-backup GET / `/notify` (any valid key, not customer-scoped) | handler.go:407,536,568,596 | minor |
-
-No **blocking** findings — the data model and the two-driver framing are sound, and the LOCKED clean
-cutover absorbs most schema risk. The items below are gaps/contradictions worth fixing before the doc
-drives work.
-
----
-
-## Highest-value: doc assumes something the code contradicts
-
-### S1 — `infra_backups` vs `infra_backup_versions` is inverted (should-fix, code-contradiction)
-05 §9: *"`infra_backup_versions` retires; `infra_backups` is repurposed into the slim DR record."*
-§11 repeats: *"RETIRE `infra_backup_versions`; repurpose `infra_backups`."*
-
-The code is the other way round:
-- `infra_backup_versions` (added v0.7.0, `store.go:198-211`) is the **live/primary** table. **Every read
-  path hits it**: `GetInfraBackup` (`store.go:565-578`), `GetInfraBackupByID` (`store.go:581-593`),
-  `GetInfraBackupMeta` (`store.go:604`), `ListInfraBackupVersions` (`store.go:640`), and the recovery
-  endpoint (`handler.go:670-686`).
-- `infra_backups` (original single-row, `store.go:96-100`) is **deprecated**. It is now **written only
-  as a legacy mirror** ("for backward compatibility during rollback window", `store.go:552-558`) and is
-  **never read** except as the one-time migration *source* (`store.go:214-217`).
-
-So the doc proposes retiring the current table and repurposing the dead one. Under the LOCKED clean
-cutover both are discarded anyway, so blast radius is low — but an implementer following §9/§11
-literally would point the DR record at the wrong table.
-**Fix:** take §11's own alternative — *fold the slim DR record onto `hosts`* and **drop both**
-infra-backup tables. If a standalone table is kept, base it on `infra_backup_versions` (the one with the
-data/readers), and correct the "which is current" framing.
-
-### S2 — the CF API token is **not** geo-only; it is the ACME token too, and ships into `controller.yaml` (should-fix, code-contradiction)
-05 §7: *"The hub already holds the CF API token (the config form notes Zone WAF:Edit)… rather than
-pushing the token down to the controller… The controller no longer calls the CF API."*
-
-Grounding confirms the hub **does** hold the token and **does** have a remove-all path:
-`config_json → infrastructure.cf_api_token` (`configs.go:714-715,1041-1042,1089-1096`) →
-`cfClient.RemoveGeoRules(cfToken, cfg.Domain, …)` in `handleGeoDisable` (`configs.go:1112`), route
-`/customers/{id}/geo/disable` (`server.go:201-205`). ✓ The §7 framing of geo-enforcement-moves-to-hub
-is also consistent with Part 1 §5/§7 and Part 3 §2/§46.
-
-**But the doc's assumption that the token is *for geo* is contradicted by the code:** the same
-`cf_api_token` is **dual-purpose** —
-- the config-form hint says **"Zone DNS:Edit (ACME), Zone WAF:Edit (geo)"** (`config_form.html:80`),
-- `controller.yaml.default:26` documents it as the **"Cloudflare API token (DNS-01 challenge)"**,
-- and it is **deep-merged into the generated `controller.yaml`** via `configgen.Generate` (config_json
-  overrides, `configgen.go:28-37`), i.e. **today it is shipped down to the box** and served at
-  `/config/{id}` and `/recovery/{id}`.
-
-Consequences §7 must address:
-1. **"Token off the controller" is incomplete** if the box still does DNS-01/ACME. In the CF-Tunnel
-   model the box may no longer need a public cert at all (edge-terminated), making the ACME use moot —
-   but that is an assumption the doc must state, not skip. Either confirm ACME is gone, or the CF token
-   cannot fully come off the box.
-2. **`configgen` must stop emitting `cf_api_token` into `controller.yaml`** (or relocate it to a
-   hub-only field). As written, the generated config still carries it.
-
----
-
-## Should-fix
-
-### S3 — §6 "Push" is an inbound-to-box mechanism the new model forbids
-05 §6: *"the operator edits a customer's desired state (building on the existing config-form +
-Push/Pull/Diff)."* The form + diff/pull/push handlers exist — `handlePushConfig` (`configs.go:569`),
-`handlePullConfig` (`configs.go:952`), `handleConfigDiff` (`configs.go:861`), routes at
-`server.go:209-229`. ✓ So the UI base is real.
-
-The wrinkle: **"Push" today is a hub→controller outbound POST** (`handlePushConfig` "sends the generated
-YAML config to the controller", `configs.go:569-570`), as is the geo-disable notify
-(`notifyControllerGeoDisable` → `POST controllerURL/api/geo/settings`, `configs.go:1148-1153`). Both are
-the hub **connecting into the box** — explicitly disallowed by the box-initiated model (Part 1 §4
-"the hub never initiates inbound"; §5 row `agent↔hub`/`controller↔hub` = outbound poll; Part 3 §5 "The
-hub never connects inbound"). 05's own §5 already resolves this (desired state is **pulled** on poll
-with a `desired_generation`). So the doc is internally consistent in *mechanism* but loose in *wording*:
-**make §6 explicit that "Push" becomes "publish to desired state, delivered on the next agent/controller
-poll," not a reuse of the inbound push transport.** The form/diff UX carries over; the transport inverts.
-(Same applies to the geo-disable controller-notify path.)
-
-### S4 — "declarative app inventory" (Part 1 §6) vs "apps are mirrored, never reconciled" (05 §1)
-Part 1 §6 lists the durable record as including a **"declarative app inventory"** that survives box loss
-— wording that implies an operator-authored, re-deployable spec. 05 §1 (LOCKED two-driver model) is
-explicit the opposite way: *"Apps never enter the reconcile loop… the hub only mirrors the resulting
-inventory… the app/customer layer is mirrored,"* and 05 §9 restores apps **from the PBS guest snapshot**,
-not by re-deploying a declared inventory. These are reconcilable (the mirror *is* durable last-known
-truth) but the word "declarative" contradicts the locked framing and the §9 restore-from-snapshot path.
-**Fix (align the older doc to the locked model):** in Part 1 §6 change "declarative app inventory" →
-"mirrored / last-reported app inventory," and note apps are recovered from the guest snapshot, not
-re-declared. (Flagging an internal inconsistency, not relitigating the locked premise.)
-
-### S5 — §9 reads DR inputs from a prunable mirror; soundness rests on durable sources
-05 §9 hands the recovering agent *"identity, tunnel token, storage manifest, PBS namespace, guest
-inventory + snapshots."* §3 places "guest inventory" and "latest PBS snapshot pointers" in
-`host_reports` — the bottom-up mirror. But reports are **pruned** (`Prune` deletes rows older than
-`maxDays`, `store.go:809-816`; the doc keeps this), so after a long pre-DR outage the last `host_report`
-can be gone or stale. The actually-durable DR inputs are: desired-state on `hosts`/`guests` (§5), the
-slim DR record (PBS namespace + repo fingerprint + wrapped escrow key, §9/§11), and **PBS's own snapshot
-enumeration** (the agent lists snapshots once it has the namespace + unwrapped key). The mirrored
-inventory/pointers are convenience, not the source of record.
-**Fix:** state in §9 that DR reads from the durable sources (desired-state + DR record + PBS), **not**
-from prunable `host_reports`, so recovery doesn't degrade when the last report has aged out. This also
-keeps §1's two-driver discipline clean: DR must not depend on bottom-up mirror rows being retained.
-(Note: the `hosts` row legitimately mixes top-down intent columns with a few box-reported columns —
-repo fingerprint, wrapped escrow key. That is fine; just label them as box-reported so the §1 split
-stays legible at the column level.)
-
-### S6 — backup-deadline checker: doc says field-based, code is event-based (and re-emitter changes)
-05 §4: *"The existing backup-deadline checker maps onto `host_reports`' last-backup-per-target."* The
-existing checker is **event-based**, not field-based: `CheckBackupDeadlines` looks for
-`backup_completed` / `backup_failed` (and `db_dump_*`) **events** since Budapest midnight and emits
-`expected_backup_missed` if neither is present (`deadline.go:31-86`). Two changes the doc should make
-explicit:
-1. **Mechanism:** either keep it event-based (someone emits `backup_completed`) or genuinely move it to
-   a `host_reports.last_backup_per_target` field check — the doc says the latter but the impl is the
-   former.
-2. **Emitter:** today the **controller** emits backup events; in the de-privileged model the **agent**
-   owns backup/PBS (Part 3 §8), so the agent must now emit `backup_completed`/`backup_failed` (or the
-   host report carries last-backup-per-target). Without re-homing the emitter, the deadline check goes
-   silent after the controller stops doing backups.
-
----
-
-## Minor
-
-- **M1 — "60s staleness checker" (§4).** 60s is the **check cadence** (`main.go:207-217`,
-  `ticker := time.NewTicker(60 * time.Second)`); the **staleness threshold** is 30m (default,
-  `main.go:99-102`) with down at 2× = 60m (`staleness.go:33-37`; CLAUDE.md "OK <30m, DOWN >1h"). The
-  event-transition mechanism (`node_stale`/`node_down`/`node_recovered`) is described correctly
-  (`staleness.go:155-185`). Reword to "the staleness checker (60s cadence, 30m/1h thresholds)."
-- **M2 — `customer_configs` fields (§2).** The list ("identity, domain, email, retrieval_password,
-  status, config_json") omits **`api_key`** (`store.go:108`) — the field §2's per-reporter plan
-  actually retires. Worth noting `customer_configs.api_key` becomes unused once auth resolves via
-  `hosts.api_key` / `guests.api_key`.
-- **M3 — rename under clean cutover (§11).** `migrate()` is all `CREATE TABLE IF NOT EXISTS` +
-  idempotent `ALTER` (`store.go:55-119,146-149`). §11's claim "grounded in store.go's idempotent style"
-  is accurate. But a `reports`→`guest_reports` **rename** isn't part of that style; under the LOCKED
-  clean cutover (demo re-enrolls fresh, §2) it is really **drop `reports` + create `guest_reports`**
-  with no data migration. Name it as such to avoid implying an in-place rename + backfill.
-- **M4 — pre-existing weak authz.** `handleInfraBackupGet`/`Versions` and `handleNotify`/
-  `handleSavePreferences`/`handleInfraBackupPush` use `checkAuth` (global **or any** customer key,
-  `handler.go:63-66`), not customer-scoped `checkAuthCustomer`. Most retire with the infra-backup push
-  (§9); for any that carry over, the per-reporter model (§2) should scope them to the resolved
-  host/guest→customer. Not a regression the doc introduces — a cleanup the cutover enables.
-
----
-
-## Confirmed accurate (grounding that holds — so the rest of the doc can be trusted)
-
-- **§10 KEEP list** matches the schema exactly: `customer_configs`, `events`, `customer_notifications`,
-  `notification_log`, `app_telemetry`, `app_log_issues` all present (`store.go:74-189,102-135`). The
-  asset manager exists (`handler.go:57,834-867`). ✓
-- **§10 two-tier notifications** (operator English / customer Hungarian, Resend, cooldowns) match
-  `notify/dispatcher.go`: `processOperator` (1h cooldown, `FormatOperatorEmail`, gated by `operatorOn`,
-  `dispatcher.go:91-114`) + `processCustomer` (prefs-driven, default 6h, `FormatCustomerEmail`,
-  `dispatcher.go:116-158`); wired in `main.go:134`. ✓
-- **§8 enrollment / §11 configgen** — deep-merge + Hungarian passphrase base is real:
-  `configgen.deepMerge` (`configgen.go:76-91`), programmatic overrides + `hub.api_key = cfg.APIKey`
-  (`configgen.go:40-47`), retrieval-password gate (`handler.go:709-753`). The evolution to agent-first +
-  per-guest keys + key-pinning is a clean extension. ✓
-- **§2 auth extension** (Bearer → reporter → customer) is clean against today's
-  `checkAuthCustomer` (global key, else `GetCustomerConfigByAPIKey`, `handler.go:72-90`,
-  `store.go:913-935`); adding host/guest key lookups slots straight in. ✓
-- **§11 "idempotent style"** is accurate (`store.go:55-119`). New tables/columns (`hosts`, `guests`,
-  `host_reports`, `signed_ops`, `desired_generation`, `desired_spec_json`) follow the existing
-  `CREATE IF NOT EXISTS` / `ALTER … ` pattern cleanly.
-- **§9 escrow/custody** is consistent with Part 1 §8 (three-tier custody, zero-knowledge default,
-  recovery-code-wrapped PBS keyfile, operator can't open) and Part 3 §8 (live PBS key on the box for
-  backup + restore-test; hub holds only the wrapped escrow). The "customer recovery code is the
-  irreducible residual; operator-managed tier avoids it" matches Part 1 §8 verbatim in spirit. ✓
-- **§4 dead-man's-switch** (host-report recency = primary liveness) is consistent with Part 3 §5
-  ("the heartbeat *is* the liveness signal… first-class treatment hub-side"). ✓
-- **§5/§6 signed-op + desired-state** are consistent with Part 4 and Part 3 §4:
-  hub holds **no** signing key and queues opaque blobs (Part 4 §5; 05 §6 "The hub holds no signing
-  key"); agent runs the verify pipeline and is the authoritative guard (Part 4 §2.3, Part 3 §4; 05 §6
-  "the agent's classification is the authoritative guard"); hub classifies at edit-time for UX only.
-  05 §6's `signed_ops` columns are a consistent superset of Part 4 §2.1's blob
-  `{op, target:{host_id,guest_id}, params, nonce, issued_at, expires_at, key_id}` (05 adds hub-side
-  lifecycle states `delivered`/`rejected` — fine). The local-CLI hand-off (`felhom-sign --pending`)
-  matches Part 4 §5–6's `Signer`-on-the-workstation model. ✓
-
-## Two-driver soundness (axis 3) — holds
-No place in 05 has the hub **drive** box/customer-owned state. Desired-state (§5) is all infrastructure
-intent (guests, storage *policy*, versions, identity, tunnel) — top-down and legitimate. Apps are
-explicitly excluded from reconcile (§1, §5) and mirrored only. Storage is the handshake (detect →
-assign policy → reconcile policy onto the detected drive), matching Part 3 §7. The one nuance (S5): the
-`hosts` row holds both top-down intent and a few box-reported columns (repo fingerprint, wrapped escrow
-key) — acceptable, just label provenance per column. Reconcile (§5) never collides with app/storage
-reality because the reality columns (`durable_id` attached, snapshot pointers, app inventory) are
-mirror-only and never serve as desired state.
-
-## DR completeness (axis 4) — safe to retire the heavy push, with S5's clarification
-Retiring the controller's infra-backup push is safe **given** that DR reads from durable sources, not
-the prunable mirror (S5). What the old push carried — `deployed_stacks` + `disk_layout.mounts`
-(`store.go:768-795`, surfaced by `handleRecovery`, `handler.go:620-705`) — is reconstructible:
-storage layout/`durable_id`s from the storage manifest (desired-state, durable) + host-report mirror;
-app inventory from the guest **inside the PBS snapshot** (so it need not be separately stored); snapshot
-list from PBS itself. The one artifact only the box can produce — the recovery-code-wrapped PBS key — is
-explicitly escrowed (§9), zero-knowledge, consistent with Part 1 §8 / Part 3 §8. So nothing
-DR-essential is lost by removing the push **provided** §9 is amended per S5 to name durable sources and
-not lean on `host_reports` retention.

docs/proxmox-platform.md

@@ -1,385 +0,0 @@
-# Proxmox Platform Reference
-
-Authoritative, living reference for the Proxmox platform underneath `felhom-agent`.
-It records **facts about Proxmox and what we validated about it** — not Felhom design
-decisions. Where a design choice exists, this doc points to the (future) controller
-architecture document rather than making the choice here.
-
-**Evidence base** (raw, chronological spike logs — kept as the underlying record):
-- [tests/phase0-findings.md](tests/phase0-findings.md) — VM-vs-LXC overhead, Docker-in-LXC viability
-- [tests/phase1-2-findings.md](tests/phase1-2-findings.md) — privilege model, backup/restore round-trip
-- [tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md](tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md) — **superseded** pre-spike reference (contains a known privsep error; do not cite as authoritative)
-
-Every nontrivial claim links to its evidence section. Validated on a single host
-(`demo-felhom`, 192.168.0.162, 4 vCPU / 16 GB) on 2026-06-07; treat single-run timings and
-measurements as indicative, not benchmarks.
-
----
-
-## 1. Platform baseline
-
-Validated stack [[phase0 §1](tests/phase0-findings.md)]:
-
-| Component | Version |
-|---|---|
-| Proxmox VE (`pve-manager`) | **9.2.2** (`b9984c6d90a4bd80`) |
-| OS | Debian 13 (Trixie) |
-| Kernel | proxmox-kernel **7.0.2-6-pve** |
-| `pve-qemu-kvm` | 11.0.0-3 |
-| `qemu-server` | 9.1.15 |
-| `pve-container` | 6.1.10 |
-| `lxc-pve` / `lxcfs` | 7.0.0-2 / 7.0.0-pve1 |
-| `criu` | 4.1.1-1 |
-
-`pvesh get /version` → release 9.2. Always confirm the node name on the box
-(`pvesh get /nodes`) rather than hard-coding it.
-
-### 1.1 Storage backends
-Two backends were present and exercised [[phase0 §1](tests/phase0-findings.md), [phase1-2 §pre-flight](tests/phase1-2-findings.md)]:
-
-| Storage | Type | Path / VG | Content types | Holds |
-|---|---|---|---|---|
-| `local` | `dir` | `/var/lib/vz` | `iso, vztmpl, backup, import` | ISOs, CT templates, **vzdump archives** |
-| `local-lvm` | `lvmthin` | VG `pve`, thinpool `data` | `rootdir, images` | guest disk volumes |
-
-**Why backups cannot live on LVM-thin:** LVM-thin is a *block* backend — it allocates
-logical volumes for guest disks. Backup archives and templates are *files*, which require a
-file-level backend (`dir`, NFS, CIFS, or PBS). A `vzdump` target must therefore be a
-storage whose content types include `backup` (here, `local`); pointing `vzdump` at
-`local-lvm` is not valid. [[phase1-2 §pre-flight / §2.1](tests/phase1-2-findings.md)]
-
-### 1.2 Repositories
-PVE 9 uses **deb822** `.sources` files under `/etc/apt/sources.list.d/`. For a host
-without a subscription, the enterprise repos (`pve-enterprise.sources`,
-`ceph-*-enterprise.sources`) must be disabled (they return 401) and a no-subscription repo
-enabled. *The spike host arrived with the no-subscription repo already configured and the
-host updated [[phase0 baseline](tests/phase0-findings.md)]; the repo setup itself was not a
-spike deliverable* — the canonical no-subscription `.sources` is the standard Proxmox 9
-procedure (`/etc/apt/sources.list.d/pve-no-subscription.sources` with
-`Components: pve-no-subscription`). Treat the exact commands as standard setup, not
-spike-validated.
-
-**Docker repository (validated):** Docker's official apt repo **has a `trixie` channel**;
-no fallback to Debian's `docker.io` was needed. Installed Docker **29.5.3** from it in both
-guest types. [[phase0 §1](tests/phase0-findings.md)]
-
----
-
-## 2. Guest model (LXC vs VM) — validated facts
-
-Both guest types ran the **identical** workload (Debian 13, Docker 29.5.3, a
-postgres/redis/nginx compose stack) under identical resources (2 vCPU, 2048 MB, ~10 GB)
-[[phase0](tests/phase0-findings.md)].
-
-### 2.1 Isolation characteristic (fact, not recommendation)
-- **LXC** is an OS-level container: it **shares the host kernel**. Docker-in-LXC needs the
-  container configured for nesting (see §2.3).
-- **VM** runs its **own guest kernel** under KVM/QEMU, with full hardware-level isolation
-  and its own firmware.
-
-The trade-offs below follow directly from this difference.
-
-### 2.2 Resource overhead (measured)
-Host RAM used = `MemTotal − MemAvailable`, deltas vs a both-stopped baseline of 1702 MB;
-one guest measured at a time [[phase0 §2](tests/phase0-findings.md)]:
-
-| Metric | LXC | VM | Note |
-|---|---|---|---|
-| Idle host-RAM delta | **+211 MB** | **+2056 MB** | structural, see below |
-| Under-load host-RAM delta | **+410 MB** | **+2084 MB** | |
-| Per-guest attribution | cgroup `memory.current` 1961 MB¹ | KVM RSS ~2031 MB | |
-| Idle host CPU used | ~0.3 % | ~6.0 % | VM has an emulation/guest-kernel floor |
-| Under-load host CPU used | ~39.4 % | ~53.9 % | VM work shows as `%guest` (31.9 %) |
-| pgbench throughput | 2211 tps | 1820 tps | identical load, 0 failed both |
-| Disk used (host thin-LV) | ~2.67 GiB | ~2.94 GiB | of 10 GiB allocated |
-| Provisioning (create→ready) | ~10–15 s | ~60–75 s | template-extract vs qcow2-import+boot |
-
-¹ `cgroup memory.current` counts reclaimable page cache shared with the host and
-**overstates** the LXC's true incremental cost; the +211 MB host delta is the honest
-number [[phase0 §4.4](tests/phase0-findings.md)].
-
-**Why the RAM gap is structural** [[phase0 §4.3](tests/phase0-findings.md)]: LXC processes
-share the host kernel and page cache, so only the working set counts against the host. A VM
-with **no ballooning configured** has KVM back every guest-touched page (including the
-guest's own page cache), so its host cost ≈ the full RAM allocation and is largely
-load-independent. *Ballooning / KSM were not tested* and could change the VM figure.
-
-### 2.3 Docker-in-LXC viability (validated)
-Docker ran **cleanly in an *unprivileged* LXC** configured with
-`--features nesting=1,keyctl=1 --unprivileged 1` (PVE 9 syntax, accepted by `pct create`)
-[[phase0 §3](tests/phase0-findings.md)]:
-
-- `docker run hello-world` → success; full 3-container stack healthy.
-- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs`
-  fallback**. (Docker 29 names the overlay driver `overlayfs` via the containerd
-  snapshotter image store; same overlay technology as the legacy `overlay2`.)
-- Named volume persisted writes; multi-container networking + published port worked
-  (`curl localhost:8080` → 200); 0 failed transactions under load.
-- No privileged-container fallback was needed.
-
-### 2.4 Guest agent & app-consistency capability
-- **VM:** `qemu-guest-agent` installs and reports (`agent: 1`), enabling
-  `guest-fsfreeze`-based app-consistent `snapshot` backups [[phase0 §4.8](tests/phase0-findings.md)].
-  The Debian genericcloud image does **not** ship the agent — it must be installed
-  in-guest.
-- **LXC:** no guest agent exists → **no fsfreeze** (see §4.2).
-
----
-
-## 3. API & access control
-
-### 3.1 Fundamentals
-- **Base URL:** `https://<host>:8006/api2/json`. Every `pve*` CLI is a thin wrapper over
-  this REST API.
-- **Token auth header:** `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET`. The
-  secret is shown **once** at creation. Response envelope: `{"data": ...}`.
-- **TLS reality:** the host serves the default **self-signed** certificate. `curl` without
-  `-k` fails `SSL certificate problem: unable to get local issuer certificate`
-  [[phase1-2 §1.5](tests/phase1-2-findings.md)]. Production trust (pin the PVE CA / install
-  a real cert) is a separate, not-yet-decided concern.
-
-### 3.2 RBAC model
-An ACL entry is a triple **(path, principal, role)**; a role is a bundle of privileges,
-assigned at the most specific path. Paths include `/`, `/vms/<vmid>`, `/nodes/<node>`,
-`/storage/<store>`, `/pool/<pool>`, `/access/...`.
-
-Introspection (**corrected for PVE 9**) [[phase1-2 §1.1](tests/phase1-2-findings.md)]:
-- `pveum role list` — lists roles **with their privileges**.
-- ⚠️ `pveum role info <role>` **does not exist in PVE 9** (the old reference used it).
-- `pveum acl list`, `pveum user permissions <user> --path <path>`.
-
-### 3.3 Privilege-separated tokens — the intersection rule (corrected)
-> **A privsep token's (`--privsep 1`) effective permissions are the *intersection* of (a)
-> the backing user's permissions and (b) the token's own ACLs.** The role must therefore be
-> granted on **BOTH the user AND the token** for the same path. Granting it on the token
-> only yields an **empty intersection** and a **403 even on self-calls.**
-> [[phase1-2 §1.2](tests/phase1-2-findings.md)]
-
-This corrects the superseded reference (§3 there grants the ACL to the token only). The
-intersection is what keeps a privsep token ≤ its user while still being independently
-scopeable to a narrow path.
-
-Working pattern (validated):
-```bash
-pveum role add <Role> -privs "<priv> <priv> ..."          # NB: -privs is space-separated
-pveum user add <user>@pve
-pveum user token add <user>@pve <tokenid> --privsep 1     # capture SECRET (shown once)
-pveum acl modify <path> -user  '<user>@pve'         -role <Role>   # BOTH the user...
-pveum acl modify <path> -token '<user>@pve!<tokenid>' -role <Role> # ...AND the token
-```
-`pveum acl delete` **requires `--roles`** (a bare `-user`/`-token` path errors
-`400 roles: property is missing`). Deleting the token/user/role auto-invalidates the
-referencing ACLs. [[phase1-2 §5](tests/phase1-2-findings.md)]
-
-### 3.4 Validated minimal self-backup role
-A token scoped to **one VMID + the backup datastore** can audit, snapshot, and back up
-**only that guest**, and is denied on every other guest and on create/allocate
-[[phase1-2 §1.3–1.4](tests/phase1-2-findings.md)]:
-
-> **Minimal role for self-audit + self-snapshot + both `snapshot`- and `stop`-mode
-> self-backup:**
-> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit`
-
-⚠️ **`VM.PowerMgmt` is NOT required for stop-mode backup** — `vzdump` performs the guest
-shutdown/restart internally under `VM.Backup` (tested: stop-mode self-backup returned
-`exitstatus OK` without it) [[phase1-2 §1.4](tests/phase1-2-findings.md)]. This corrects the
-old reference's "likely yes" guess.
-
-Validated boundary (token scoped to `/vms/<self>` + `/storage/local`):
-
-| Operation | Result |
-|---|---|
-| `GET /version` | 200 |
-| `GET` self status, `POST` self snapshot, `POST` self vzdump | 200 / task `OK` |
-| `GET`/`POST` against **another** guest's vmid | **403** (read) / task **403** (backup) |
-| `POST /nodes/<node>/lxc` (create/allocate a guest) | **403** — create/allocate is operator-tier |
-
-### 3.5 Async tasks — trust `exitstatus`, not the POST
-Long operations (`vzdump`, `snapshot`, clone, restore) return a **UPID**, not a result.
-Poll `GET /nodes/<node>/tasks/<upid>/status` until `status: stopped`, then read
-`exitstatus` [[phase1-2 §1.3](tests/phase1-2-findings.md)].
-
-> ⚠️ **Authorization can surface at task execution, not at the HTTP POST.** A `vzdump`
-> against an unauthorized vmid returns **HTTP 200 + a UPID**, but the task then ends
-> `exitstatus: "403 Permission check failed (/vms/<id>, VM.Backup)"` and produces **no
-> archive**. A caller that trusts the 200 would wrongly believe the backup ran. Always poll
-> the task and check `exitstatus`.
-
-(The task owner — including a token — can read its own task status: 200.)
-
-### 3.6 Operator-tier agent role & root-vs-API boundary (validated)
-The operator-tier **host agent** (`03-host-agent.md`) needs a far broader role than the
-Phase-1 *guest self-backup* role (which is denied create/allocate — §3.4). The minimal role
-that drives the full guest lifecycle via an API token, validated by paring
-[[phase3 §B3](tests/phase3-findings.md)]:
-
-> **`FelhomAgent` (operator-tier, 16 privileges):**
-> `VM.Allocate, VM.Audit, VM.Config.Disk, VM.Config.CPU, VM.Config.Memory, VM.Config.Network,
-> VM.Config.Options, VM.PowerMgmt, VM.Snapshot, VM.Snapshot.Rollback, VM.Backup,
-> Datastore.Allocate, Datastore.AllocateSpace, Datastore.Audit, Sys.Audit, SDN.Use`
->
-> Paring proved: `SDN.Use` is **required** (PVE 9 gates bridge use; omitting it → `403
-> (/sdn/zones/localnetwork/vmbr0, SDN.Use)`); `Sys.Audit` required for host metrics
-> (`GET /nodes/<node>/status`); `VM.Config.Network`/`VM.Config.Options` required for NIC/onboot
-> config; `Datastore.AllocateTemplate` **not** needed (drop it). NB `VM.Config.CPUMemory` is
-> not a real privilege — it is `VM.Config.CPU` + `VM.Config.Memory`.
-
-**Root-vs-API boundary** [[phase3 §B3](tests/phase3-findings.md)] — nearly the entire guest
-lifecycle, **including restore**, is API-token-covered; the genuine OS-root residual is narrow:
-
-| Operation | Coverage |
-|---|---|
-| Create LXC (nesting-only), config, allocate, start/stop, snapshot/rollback, vzdump, **restore**, destroy, add storage definition, host metrics | **scoped API token** (the `FelhomAgent` role) |
-| ⚠️ **Create LXC with `keyctl=1`** (Docker needs it — §2.3) | **OS root `root@pam` only** |
-| USB physical mount-by-UUID / systemd mount unit / fstab; SMART/sensors | OS root / narrow sudoers |
-
-> ⚠️ **`keyctl=1` (and any feature flag except `nesting`) can be set only by an actual
-> `root@pam` session** — `changing feature flags (except nesting) is only allowed for
-> root@pam`. **No API token qualifies**, not even a non-privsep `root@pam` token (same 403).
-> So *fresh provisioning* of a Docker-capable LXC needs `pct create` as OS root (or a narrow
-> sudoers entry). **Restore is exempt:** a token-authorized `vzrestore` **preserves
-> `keyctl=1`** from the archive — the DR path needs no root.
-
----
-
-## 4. Backup & restore (`vzdump` / `pct restore`)
-
-### 4.1 Modes
-- **`stop`** — orderly guest shutdown → backup → restart. Highest consistency, defined
-  downtime. (For LXC the shutdown/restart is internal to `vzdump`; needs only `VM.Backup` —
-  §3.4.)
-- **`snapshot`** — lowest downtime; copies blocks while running. Consistency depends on the
-  guest cooperating (§4.2).
-- **`suspend`** — legacy/compat, not used.
-
-### 4.2 Consistency: crash-consistent vs quiesced, and no-fsfreeze-for-LXC
-> ⚠️ **An LXC has no guest agent, so `snapshot`-mode `vzdump` does NOT fsfreeze.** A
-> running-stack LXC backup is therefore **crash-consistent** (filesystem-level), not
-> app-consistent. App-consistency for an LXC is the caller's job: quiesce in-guest first
-> (stop the stack / flush DBs) or use `stop` mode. A **VM** with `qemu-guest-agent` gets
-> `guest-fsfreeze` around the copy → near-free app-consistency. [[phase1-2 §2.1](tests/phase1-2-findings.md), [phase0 §4.8](tests/phase0-findings.md)]
-
-**Validated restore behaviour** (LXC, Postgres) [[phase1-2 §2.2](tests/phase1-2-findings.md)]:
-- **Crash-consistent (running):** on first start Postgres ran **automatic WAL recovery**
-  (`database system was interrupted … not properly shut down; automatic recovery in
-  progress … redo done … ready to accept connections`) and the data was intact.
-- **Quiesced (stack stopped):** clean start, no recovery, data intact.
-- Both restored correctly here on an idle-at-backup DB; this is **not** a durability
-  guarantee under heavy write load (§6).
-
-### 4.3 What a backup captures
-A single LXC `vzdump` captures the container rootfs **including the Docker named volumes**
-(they live in the rootfs) — one backup = the whole guest and its data. Validated: a
-sentinel row survived both variants [[phase1-2 §2.2](tests/phase1-2-findings.md)].
-
-Sizes/timings (2.5 GiB source, zstd) [[phase1-2 §2.1–2.2](tests/phase1-2-findings.md)]:
-backup ~934 MB (~2.7:1) in ~22–25 s; restore in ~11–12 s.
-
-### 4.4 Restore = recreate-from-archive (identity is preserved)
-There is no single "restore" call — you recreate the guest from the archive into a **new
-VMID**:
-- **LXC:** `pct restore <newid> <archive> --storage <store>`
-- **VM:** `qmrestore <archive> <newid>` (or `POST /nodes/<node>/qemu` with `archive=`)
-
-> ⚠️ **`pct restore` preserves the source config — including the MAC address and
-> hostname.** Restoring while the original still runs causes a **MAC/hostname collision** on
-> the bridge; reset network identity (`pct set <id> -net0 name=eth0,bridge=vmbr0,ip=dhcp`
-> regenerates the MAC) before starting. [[phase1-2 §2.2](tests/phase1-2-findings.md)]
-
-**Restored config survives intact:** `unprivileged: 1` and `features: nesting=1,keyctl=1`
-are preserved, so Docker runs in the restored CT [[phase1-2 §2.2](tests/phase1-2-findings.md)].
-
-### 4.5 Snapshots
-A **running, unprivileged LXC can be snapshotted on LVM-thin** with no stop required
-(`exitstatus OK`; snapshot listed while the CT stays `running`)
-[[phase1-2 §1.6](tests/phase1-2-findings.md)]. This is the mechanism available for a
-snapshot-before-change rollback flow.
-
-### 4.6 PBS (Proxmox Backup Server)
-**Not yet validated.** No PBS datastore was configured or tested in the spike. All backup
-findings above are for `vzdump` to a `dir` storage. PBS (dedup, incremental, remote, dirty-
-bitmap) is pending.
-
-### 4.7 vzdump scope by LXC mount type (validated)
-A stop-mode `vzdump` includes/excludes each LXC mount point by **type and the `backup` flag**
-[[phase3 §B2](tests/phase3-findings.md)]. Validated three ways (vzdump log, archive grep,
-restore):
-
-| Location | `backup` flag | In the vzdump? |
-|---|---|---|
-| rootfs (and anything inside it) | — | **included** (always) |
-| **Docker named volume** (default driver) | — | **included** — it lives in the rootfs (`/var/lib/docker/volumes/<v>/_data`) |
-| volume mount point (`mpN`) | `backup=1` | included |
-| volume mount point (`mpN`) | `backup=0` | **excluded** (vol recreated empty on restore) |
-| bind mount point (`mpN: /host/path`) | n/a | **excluded** ("not a volume"); data is *not* in the archive |
-
-> ⚠️ **The `backup=<boolean>` flag is honoured ONLY for *volume* mount points.** A **Docker
-> named volume is in the rootfs and is always captured** — so a "bulk" volume left as a
-> default named volume is silently swept into the whole-guest image. To keep bulk data **out**,
-> realize it as a dedicated `backup=0` volume mount point (proven recipe:
-> `pct set <id> -mpN <storage>:<size>,mp=/mnt/bulk,backup=0` then
-> `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol`).
-> A **bind mount's** data is excluded from the archive entirely; on same-host restore it
-> reappears only because the bind config re-attaches the same host dir — on a *different* host
-> (true DR) it is gone unless backed up separately.
-
----
-
-## 5. Gotchas & operational notes (quick reference)
-
-| Gotcha | Detail | Evidence |
-|---|---|---|
-| **deb822 repos** | PVE 9 repos are `.sources` files; disable enterprise, enable no-subscription | standard setup |
-| **Privsep dual-grant** | privsep token needs the role on **both** user and token, else empty intersection → 403 | [phase1-2 §1.2](tests/phase1-2-findings.md) |
-| **Async authz** | `vzdump` POST returns 200+UPID even when unauthorized; the 403 is in the task `exitstatus`; poll it | [phase1-2 §1.3](tests/phase1-2-findings.md) |
-| **No fsfreeze for LXC** | running-LXC `snapshot` backup is crash-consistent only; quiesce or use `stop` for app-consistency | [phase1-2 §2.1](tests/phase1-2-findings.md) |
-| **Restore identity collision** | `pct restore` keeps source MAC + hostname; reset before starting alongside the original | [phase1-2 §2.2](tests/phase1-2-findings.md) |
-| **Restart policy for self-heal** | restored/rebooted containers come up `exited` with no restart policy; need a restart policy or an explicit `compose up -d` to return automatically | [phase1-2 §2.2/§3](tests/phase1-2-findings.md) |
-| **Self-signed TLS** | host cert is self-signed; `curl` needs `-k` until trust is set up | [phase1-2 §1.5](tests/phase1-2-findings.md) |
-| **`pveum role info` gone** | use `pveum role list` in PVE 9 | [phase1-2 §1.1](tests/phase1-2-findings.md) |
-| **`pveum acl delete` needs `--roles`** | bare `-user`/`-token` path errors `400 roles: property is missing` | [phase1-2 §5](tests/phase1-2-findings.md) |
-| **`VM.PowerMgmt` not needed** | stop-mode backup works under `VM.Backup` alone | [phase1-2 §1.4](tests/phase1-2-findings.md) |
-| **`keyctl=1` is root-only** | feature flags except `nesting` need a `root@pam` session; no API token (even root's) can set them; restore preserves them | [phase3 §B3](tests/phase3-findings.md) |
-| **`SDN.Use` gates bridge use** | PVE 9 needs `SDN.Use` to attach a NIC to `vmbr0`; omit it → 403 | [phase3 §B3](tests/phase3-findings.md) |
-| **Docker named vol = always backed up** | named volumes live in rootfs; only *volume mountpoints* honour `backup=0`; bulk must be a dedicated `backup=0` mp | [phase3 §B2](tests/phase3-findings.md) |
-
----
-
-## 6. Validated vs open
-
-### Validated by the spike
-| Fact | Evidence |
-|---|---|
-| PVE 9.2.2 / Debian 13 / kernel 7.0.2 baseline; `local` (dir) vs `local-lvm` (thin) roles | [phase0 §1](tests/phase0-findings.md), [phase1-2 pre-flight](tests/phase1-2-findings.md) |
-| Docker runs in an **unprivileged** LXC (`nesting=1,keyctl=1`), driver `overlayfs`, cgroup v2 | [phase0 §3](tests/phase0-findings.md) |
-| LXC vs VM overhead (idle host RAM +211 MB vs +2056 MB; CPU/throughput/provisioning) | [phase0 §2](tests/phase0-findings.md) |
-| Privsep token = intersection of user ∩ token ACLs (dual-grant required) | [phase1-2 §1.2](tests/phase1-2-findings.md) |
-| Minimal self-backup role; `VM.PowerMgmt` unnecessary | [phase1-2 §1.4](tests/phase1-2-findings.md) |
-| Token scoped to one VMID: self-ops succeed, cross-guest + create/allocate denied | [phase1-2 §1.3](tests/phase1-2-findings.md) |
-| Async UPID model; vzdump authz surfaces in `exitstatus`, not the POST | [phase1-2 §1.3](tests/phase1-2-findings.md) |
-| Running, unprivileged LXC snapshots on LVM-thin (no stop) | [phase1-2 §1.6](tests/phase1-2-findings.md) |
-| `vzdump` → `pct restore` round-trip; one backup captures Docker volumes; config survives | [phase1-2 §2](tests/phase1-2-findings.md) |
-| Crash-consistent restore recovers via Postgres WAL; quiesced restores clean | [phase1-2 §2.2](tests/phase1-2-findings.md) |
-| LXC vzdump scope by mount type; `backup=0` excludes volume mps; Docker named vols ride rootfs; proven bulk-exclusion recipe | [phase3 §B2](tests/phase3-findings.md) |
-| Operator agent role (16 privs); guest lifecycle incl. restore is API-token-covered; `keyctl` create is `root@pam`-only | [phase3 §B3](tests/phase3-findings.md) |
-
-### Not yet validated (do not assume)
-| Open item | Why it matters |
-|---|---|
-| **PBS** (dedup/incremental/remote backup) | the only backup path tested was `vzdump` to a `dir` |
-| **The real controller running inside an LXC** reaching `host:8006` | spike used `curl`/CLI, not the actual Go controller |
-| **App-consistency under heavy write load** | WAL recovery was validated only on an idle-at-backup DB |
-| **Live migration / restore to a different host** | single-node spike only |
-| **Ballooning / KSM** effect on VM RAM cost | VM RAM measured with neither configured |
-| **Cluster / HA** behaviour | single node |
-| **Production TLS trust** for the API | all calls used `-k` against a self-signed cert |
-| **deb822 no-subscription repo setup** as a controlled step | host arrived pre-configured |
-
----
-
-## 7. Scope boundary
-
-This document holds **platform facts only.** Felhom design decisions — e.g. which guest
-type is the default, whether to use privsep or non-privsep tokens, where PBS lives — are
-**out of scope** and belong in the controller-architecture document. Where this reference
-notes a decision exists, the decision itself is recorded there, not here.

docs/tests/Proxmox_Spike_-_API_&_Access-Control_Reference.md

@@ -1,176 +0,0 @@
-> ⚠️ **SUPERSEDED — spike evidence only, not authoritative.** This is the *pre-spike*
-> reference and contains at least one known error (the privsep/ACL mechanism in §3 — it
-> grants the ACL to the token only, which yields an empty intersection and a 403 even on
-> self-calls). For the corrected, validated facts read
-> [`../proxmox-platform.md`](../proxmox-platform.md). Kept here unchanged as the record of
-> what we believed going into the spike.
-
-# Proxmox Spike — API & Access-Control Reference
-
-Reference for the **controller-as-guest** architecture, synthesized from current
-Proxmox VE 9.x documentation (June 2026).
-
-Items marked **[confirm on box]** should be verified once PVE is installed —
-treat them as Phase 0/1 verification steps, not gospel. Every Proxmox CLI tool
-is a thin wrapper over the same REST API, so anything below is reachable from Go.
-
----
-
-## 1. API fundamentals
-
-- **Base URL:** `https://192.168.0.162:8006/api2/json`
-- **Auth (API token):** HTTP header
-  `Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET`
-  The secret is shown **once** at creation — capture it immediately, it can't be
-  retrieved again.
-- **Response shape:** `{ "data": ... }`; errors come back via HTTP status + body.
-- **Discovery (do this live on the box instead of trusting any doc):**
-  - `pvesh get /version`
-  - `pvesh ls /nodes/<node>/qemu/<vmid>`
-  - Full schema browser: `https://pve.proxmox.com/pve-docs/api-viewer/`
-  - "What call does the GUI make?" → perform the action in the web UI with
-    browser DevTools → Network open and read the request. Fastest way to find
-    the exact endpoint + params for anything.
-- **Async tasks:** long operations (backup, restore, clone) return a **UPID**
-  (task id), not a result. Poll `GET /nodes/<node>/tasks/<upid>/status` until
-  `status: stopped`, then check `exitstatus`. The controller must poll, not
-  block. **[confirm on box]** the exact polling/response shape.
-
----
-
-## 2. RBAC model — (path, principal, role)
-
-An ACL entry is a triple of **(path, user/group/token, role)**. A role is a
-bundle of privileges, assigned at the most specific path possible.
-
-- **Paths:** `/`, `/vms/<vmid>`, `/nodes/<node>`, `/storage/<store>`,
-  `/pool/<pool>`, `/access/...`
-- **Predefined roles include:** `PVEAuditor` (read-only), `PVEVMUser`,
-  `PVEVMAdmin`, `PVEDatastoreUser`, `PVEAdmin`, `PVEUserAdmin`.
-- **API tokens with privilege separation (`--privsep 1`):** the token's
-  effective permissions are the **intersection** of (a) the backing user's
-  permissions and (b) the token's own ACLs. A privsep token can therefore never
-  exceed its user, and you grant it a separate, minimal ACL. This is exactly the
-  property the in-guest controller needs.
-
-Introspection:
-```bash
-pveum role list
-pveum role info PVEVMAdmin
-pveum user permissions <user> --path /vms/<vmid>
-```
-
----
-
-## 3. Two-tier privilege model (our architecture decision)
-
-**Tier A — in-guest controller (customer-facing, NARROW).**
-Runs inside the customer's guest. Token scoped to *that guest's own VMID only*:
-read its own status/config, snapshot itself, back itself up, write the backup to
-the datastore. Cannot see or touch other guests. The LXC/VM's own privilege
-level is irrelevant here — reaching `host:8006` is just an HTTPS call + token.
-
-**Tier B — operator (provisioning, BROAD).**
-Creates/destroys guests, builds the golden template, attaches storage, wires PBS.
-Lives operator-side (hub / tooling), never on the customer box.
-
-### Phase 1 runbook — minimal self-backup role + scoped token
-
-```bash
-# 1. Custom least-privilege role: "back up / snapshot myself"
-#    [confirm on box: exact privilege names via `pveum role list` / api-viewer]
-pveum role add FelhomSelfBackup \
-  -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit"
-
-# 2. Dedicated API-only user in the PVE realm (no login password)
-pveum user add felhom-ctl@pve --comment "In-guest controller (self-backup)"
-
-# 3. Privsep token for that user (SECRET shown once)
-pveum user token add felhom-ctl@pve ctl --privsep 1
-
-# 4. Scope the TOKEN to one guest + the backup datastore only
-pveum acl modify /vms/<vmid>      -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
-pveum acl modify /storage/<store> -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
-
-# 5. Test FROM INSIDE the guest
-curl -k https://<host>:8006/api2/json/version \
-  -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=<SECRET>"
-
-curl -k -X POST https://<host>:8006/api2/json/nodes/<node>/vzdump \
-  -H "Authorization: PVEAPIToken=felhom-ctl@pve!ctl=<SECRET>" \
-  -d "vmid=<vmid>&storage=<store>&mode=snapshot"
-```
-
-**Pass criteria:** the token backs up its OWN vmid, and returns **403** on any
-other vmid. That single result validates the whole controller-as-guest design.
-
-**Open question to settle here:** does Tier A also need `VM.PowerMgmt` so it can
-stop/start its own guest for `stop`-mode backups? Likely yes — add it and re-test.
-
----
-
-## 4. Backup / restore (vzdump)
-
-**Modes:**
-- **`stop`** — orderly guest shutdown → live backup → resume. Highest
-  consistency, short defined downtime.
-- **`snapshot`** — lowest downtime; copies blocks while running. *Small
-  inconsistency risk* unless the guest cooperates (see below).
-- **`suspend`** — legacy/compat, longer downtime, not recommended.
-
-**App-consistency — the concrete version of the earlier warning:**
-- **VM:** install `qemu-guest-agent` in the guest and set `agent: 1`.
-  `snapshot`-mode vzdump then calls `guest-fsfreeze-freeze` / `-thaw` around the
-  copy → near-free filesystem consistency. **This is a real point in the VM's
-  favour over LXC.**
-- **LXC:** no guest agent → no fsfreeze. App-consistency becomes the
-  *controller's* job: quiesce in-guest first (stop stacks / flush DBs) **then**
-  vzdump, or use `stop` mode. Same lesson as the restic work, moved to the guest
-  layer.
-
-**CLI / API:**
-```bash
-vzdump <vmid> --mode snapshot --storage <store>                 # CLI
-# API (async → UPID):
-POST /api2/json/nodes/<node>/vzdump        params: vmid, storage, mode, ...
-```
-
-**Restore is NOT a single "restore" call** — you recreate the guest from the
-archive:
-- **VM:** `qmrestore <archive> <newvmid>`  /  `POST /nodes/<node>/qemu` with `archive=...`
-- **LXC:** `pct restore <newvmid> <archive>`  /  `POST /nodes/<node>/lxc` with the archive as source
-
-Phase 2's real-restore test = restore to a **fresh vmid** and boot it. Do not
-declare the backup "working" until a restored guest actually runs.
-
----
-
-## 5. Key REST endpoints (qemu shown; lxc is parallel under `/lxc`)
-
-```
-GET  /nodes
-GET  /nodes/<node>/qemu                          list VMs
-GET  /nodes/<node>/qemu/<vmid>/status/current    live status
-GET  /nodes/<node>/qemu/<vmid>/config            config
-POST /nodes/<node>/qemu/<vmid>/status/{start,stop,shutdown,reboot}
-POST /nodes/<node>/qemu/<vmid>/snapshot          (snapname, description)
-GET  /nodes/<node>/qemu/<vmid>/snapshot          list snapshots
-POST /nodes/<node>/qemu/<vmid>/snapshot/<snap>/rollback
-POST /nodes/<node>/vzdump                         backup (async, UPID)
-GET  /nodes/<node>/tasks/<upid>/status            poll async task
-```
-
-LXC: replace `/qemu/` with `/lxc/`. For **Docker-in-LXC** the container needs
-`features nesting=1,keyctl=1` (`pct set <vmid> -features nesting=1,keyctl=1`, or
-the `features` property on `POST /nodes/<node>/lxc`) — **[confirm on box]**.
-
----
-
-## 6. Phase 0 confirm-on-box checklist
-
-- [ ] PVE 9.2 installed; storage = LVM-thin (leave free space to also test dir/qcow2)
-- [ ] Exact privilege set for `FelhomSelfBackup` (`pveum role info`)
-- [ ] UPID task-polling response shape
-- [ ] Docker official apt repo has a `trixie` channel
-- [ ] LXC `features nesting=1,keyctl=1` syntax + Docker actually runs inside an LXC
-- [ ] Baseline idle + under-load RAM/CPU: one Debian VM vs one Debian LXC, identical resources

docs/tests/phase0-findings.md

@@ -1,331 +0,0 @@
-# Phase 0 — VM vs LXC Overhead Spike: Findings
-
-**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, Debian 13 (Trixie),
-kernel 7.0.2-6-pve, 4 vCPU, 16 GB RAM (15771 MB `MemTotal`).
-**Date:** 2026-06-07. **Measured one guest at a time, the other fully stopped.**
-
-> This document presents **data and observations only**. No recommendation or verdict —
-> the architecture decision is made elsewhere.
-
----
-
-## 1. Provenance
-
-### Platform
-| Component | Version |
-|---|---|
-| pve-manager | 9.2.2 (`b9984c6d90a4bd80`) |
-| kernel | proxmox-kernel 7.0.2-6-pve |
-| pve-qemu-kvm | 11.0.0-3 |
-| qemu-server | 9.1.15 |
-| pve-container | 6.1.10 |
-| lxc-pve / lxcfs | 7.0.0-2 / 7.0.0-pve1 |
-| criu | 4.1.1-1 |
-
-`pvesh get /version` → release 9.2, version 9.2.2.
-
-### Guest images
-| | LXC (9001) | VM (9000) |
-|---|---|---|
-| Source | `local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst` | `debian-13-genericcloud-amd64.qcow2` |
-| Build | Debian 13.1 standard CT template (downloaded via `pveam`, checksum verified) | cloud build **20260601-2496**; in-guest reports Debian **13.5** after `apt update` |
-| qcow2 | n/a | virtual 3 GiB, on-disk 323 MiB, compat 1.1/zlib |
-
-### Docker (identical in both guests)
-| | LXC | VM |
-|---|---|---|
-| Source | Docker official apt repo, **`trixie` channel** (confirmed present) | same |
-| Version | **29.5.3** build d1c06ef | **29.5.3** build d1c06ef |
-| Storage Driver | **`overlayfs`** (not vfs) | **`overlayfs`** (not vfs) |
-| Cgroup Version / Driver | **v2 / systemd** | **v2 / systemd** |
-| `hello-world` | OK | OK |
-
-> Docker's official repo **does** have a `trixie` channel — no fallback to Debian's
-> `docker.io` was needed. Docker 29 reports the driver as `overlayfs` (the containerd
-> snapshotter image store) rather than the legacy name `overlay2`; this is the same
-> overlay technology and is **not** a `vfs` fallback.
-
----
-
-## 2. Comparison table
-
-Baseline (both guests stopped): host RAM used **median 1702 MB** (range 1699–1703);
-host CPU **~0.1 % used** (99.9 % idle). All RAM deltas below are vs this baseline.
-Host RAM used = `MemTotal − MemAvailable`, 5 samples ~3 s apart (median reported).
-
-| Metric | LXC (9001) | VM (9000) | Δ (VM − LXC) |
-|---|---|---|---|
-| **Idle host-RAM delta** | **+211 MB** (1913) | **+2056 MB** (3758) | **+1845 MB** |
-| **Under-load host-RAM delta** | **+410 MB** (2112) | **+2084 MB** (3786) | **+1674 MB** |
-| **Per-guest mem attribution** | cgroup `memory.current` = **1961 MB**¹ | KVM process RSS = **2031 MB** (idle) / **2047 MB** (load) | — |
-| **Idle host CPU used** | **~0.3 %** (0.20 usr + 0.10 sys) | **~6.0 %** (3.37 usr + 2.31 sys + 0.29 guest) | **+5.7 pp** |
-| **Under-load host CPU used** | **~39.4 %** (17.1 usr + 7.5 sys + 14.5 iowait + 0.3 soft) | **~53.9 %** (31.9 guest + 16.4 iowait + 3.4 sys + 1.7 usr + 0.6 soft) | **+14.5 pp** |
-| **pgbench throughput** | **2211.7 tps**, lat 1.809 ms, 132 710 tx/60 s, 0 failed | **1819.6 tps**, lat 2.198 ms, 163 764 tx/90 s, 0 failed² | **−392 tps** |
-| **Disk allocated** | 10 GiB | 10 GiB | 0 |
-| **Disk used (host thin-LV)** | 26.73 % ≈ **2.67 GiB** | 29.33 % ≈ **2.94 GiB** | +0.27 GiB |
-| **Disk used (inside guest)** | 2.1 GiB / 9.7 GiB | 2.4 GiB / 9.7 GiB | +0.3 GiB |
-| **Provisioning (rough, create→ready)** | ~10–15 s³ | ~60–75 s³ | — |
-
-¹ `memory.current` counts reclaimable page cache shared with the host and therefore
-**overstates** the LXC's true incremental cost; the +211 MB host-RAM delta is the honest
-number. ² VM 60 s runs gave 1739 & 1759 tps — consistent with the 90 s definitive run.
-³ Guest-creation step only; see §4. Docker install + first image pull (~network-bound,
-~identical for both) is excluded.
-
-### Inside-guest `free -m` (context only — not the decisive number)
-| | total | used | buff/cache | available |
-|---|---|---|---|---|
-| LXC idle | 2048 | 125 | 1851 | 1922 |
-| VM idle | 1974 | 509 | 1524 | 1464 |
-
-The VM sees **1974 MB** usable of 2048 allocated (firmware/kernel reservation).
-
----
-
-## 3. Docker-in-LXC viability
-
-**Worked cleanly in an *unprivileged* LXC with `--features nesting=1,keyctl=1`. No
-privileged fallback was needed.**
-
-- `--features nesting=1,keyctl=1 --unprivileged 1` accepted by `pct create` (PVE 9
-  syntax confirmed via `pct help create`).
-- `docker run hello-world` → success.
-- **Storage driver: `overlayfs`** (cgroup v2, systemd cgroup driver) — **no `vfs`
-  fallback**.
-- Full 3-container stack (`postgres:17`, `redis:7`, `nginx:alpine`) came up healthy.
-- Named volume `pgdata` persisted a write (`SELECT count` returned 1 after table
-  create/insert).
-- Multi-container networking + published port worked: `curl localhost:8080` → **HTTP 200**.
-- 60 s pgbench load: **0 failed transactions**.
-
-No errors, no `dmesg`/`journalctl` anomalies, no workarounds. The privileged-LXC
-fallback path (step A5) was therefore **not exercised**.
-
----
-
-## 4. Observations & confounds
-
-1. **VM under-load CPU required a re-measurement (diagnosed, not hidden).** The first
-   VM-load sample showed host CPU ~5 % — identical to *idle* — while pgbench nonetheless
-   completed a full 60 s run (1739 tps). Root cause: the VM load was launched through a
-   **nested SSH + `nohup &`** layer (host→VM), which started pgbench *after* the sampling
-   window. The LXC path used local `pct exec` (no nested SSH) so its first sample was
-   valid. Re-running with pgbench held in the **foreground of a long-lived SSH channel**
-   (guaranteed active) and sampling during a confirmed window gave the true **53.9 %**
-   (`%guest`=31.9). **Confound:** the two guests' load was driven through different
-   plumbing (`pct exec` vs nested SSH); the *throughput* numbers are unaffected
-   (pgbench self-reports its own duration), but the CPU figures came from
-   methodologically asymmetric harnesses.
-2. **Baseline drift from residual page cache.** After stopping each guest, host RAM did
-   not snap back to 1702 MB immediately (e.g. 1895 MB just after the LXC stopped;
-   1965→1794 MB drifting down after the VM). This is reclaimable cache, not a leak.
-   Treat all RAM deltas as ±~100 MB.
-3. **The headline RAM gap is structural, not incidental.** LXC processes share the host
-   kernel and page cache, so only the working set counts against the host (+211 MB idle).
-   The VM, with **no ballooning configured**, has KVM back every guest-touched page —
-   including the guest's own 1.5 GB page cache — so the host cost ≈ the full 2 GB
-   allocation (KVM RSS ≈ 2031 MB) and is **largely load-independent** (3758 idle → 3786
-   load). Ballooning / KSM were not tested and could change this.
-4. **`cgroup memory.current` ≠ host cost.** For the LXC it read 1961 MB (near the 2 GB
-   limit) because it includes reclaimable page cache; the real incremental host cost was
-   +211 MB. Per the protocol, `MemTotal − MemAvailable` is the decisive metric.
-5. **VM idle CPU floor (~6 %) vs LXC (~0.3 %).** QEMU device emulation + a full guest
-   kernel's timer/housekeeping impose a small constant CPU cost even at rest.
-6. **Throughput vs CPU trade.** The VM did slightly *less* work (1820 vs 2211 tps) for
-   *more* host CPU (53.9 vs 39.4 %). The extra cost surfaces as `%guest` (31.9 %) — the
-   actual DB work *plus* virtualization overhead — whereas in the LXC the same DB work
-   appears directly as host `%usr`/`%sys`. iowait was comparable (~15–16 %, WAL fsync).
-7. **Workload fits in RAM.** pgbench scale `-s 10` (~150 MB) fits in cache in both
-   guests, so the test is commit/CPU-bound rather than disk-bound; a larger-than-RAM
-   dataset would stress the storage paths differently and is not covered here.
-8. **qemu-guest-agent confirmed on the VM** (`qm guest cmd 9000 ping` → OK). This enables
-   `guest-fsfreeze`-based app-consistent `snapshot`-mode vzdump for the VM — a capability
-   the LXC has no equivalent for. The genericcloud image does **not** ship the agent;
-   it had to be installed in-guest (and the VM IP had to be found via `nmap`/MAC until
-   the agent was up).
-9. **Provisioning asymmetry foreshadows cloning.** LXC create is template-extract-bound
-   (526 MiB at 387 MiB/s + SSH keygen, ~10–15 s). VM create is qcow2-import-bound (3 GiB
-   → LVM ≈ 30 s) plus a full firmware boot to SSH-ready (~30–45 s). Figures are rough,
-   single-run, and exclude the shared network-bound Docker install + first image pull.
-
----
-
-## 5. Raw command log (appendix)
-
-### 5.1 Provenance
-```
-$ pveversion -v | grep ...
-pve-manager: 9.2.2 (running version: 9.2.2/b9984c6d90a4bd80)
-proxmox-kernel-7.0: 7.0.2-6
-criu: 4.1.1-1
-lxc-pve: 7.0.0-2
-lxcfs: 7.0.0-pve1
-pve-container: 6.1.10
-pve-qemu-kvm: 11.0.0-3
-qemu-server: 9.1.15
-
-$ pvesm status
-local         dir      active  98497780  4333576  89114656  4.40%
-local-lvm  lvmthin    active 365760512        0 365760512  0.00%
-
-# Docker repo trixie channel:
-$ curl -fsSL https://download.docker.com/linux/debian/dists/ | grep -oE 'trixie|bookworm|bullseye'
-bookworm / bullseye / trixie        # trixie present
-
-# Cloud image:
-$ qemu-img info debian-13-genericcloud-amd64.qcow2
-virtual size: 3 GiB ; disk size: 323 MiB ; compat 1.1 ; build 20260601-2496
-```
-
-### 5.2 Baseline (both guests stopped)
-```
-$ for i in 1..5; awk MemTotal-MemAvailable /proc/meminfo ; sleep 3
-used=1699 MB / 1702 / 1702 / 1702 / 1703 MB      (median 1702)
-
-$ mpstat 1 5
-Average: all 0.05 usr 0.05 sys ... 99.90 idle
-```
-
-### 5.3 LXC 9001 — create + Docker
-```
-$ pct create 9001 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \
-    --hostname spike-lxc --cores 2 --memory 2048 --rootfs local-lvm:10 \
-    --net0 name=eth0,bridge=vmbr0,ip=dhcp --features nesting=1,keyctl=1 \
-    --unprivileged 1 --start 1
-  Logical volume "vm-9001-disk-0" created.
-  extracting archive ... Total bytes read: 551505920 (526MiB, 387MiB/s)
-  Creating SSH host key ... done
-=== exit: 0 ; status: running
-features: nesting=1,keyctl=1 ; unprivileged: 1 ; ip 192.168.0.115/24
-
-# Docker install (official repo, trixie stable): DOCKER-INSTALL-OK
-$ docker --version            -> Docker version 29.5.3, build d1c06ef
-$ docker run --rm hello-world -> Hello from Docker!
-$ docker info | grep -iE 'Storage Driver|Cgroup'
- Storage Driver: overlayfs
- Cgroup Driver: systemd
- Cgroup Version: 2
- Server Version: 29.5.3 ; Kernel: 7.0.2-6-pve ; OS: Debian GNU/Linux 13 (trixie)
-```
-
-### 5.4 LXC 9001 — stack health
-```
-$ docker compose ps
-spike-cache-1  running   Up
-spike-db-1     running   Up
-spike-web-1    running   Up
-$ curl -s -o /dev/null -w 'HTTP %{http_code}' localhost:8080   -> HTTP 200
-$ psql CREATE TABLE spike_persist; INSERT; SELECT count(*)     -> 1   (volume persists)
-```
-
-### 5.5 LXC 9001 — idle measurement
-```
-Host RAM used (5x3s): 1913 / 1914 / 1913 / 1914 / 1913 MB     (median 1913, Δ +211)
-cgroup memory.current: 2056036352 B = 1961 MB
-inside free -m: total 2048 used 125 buff/cache 1851 available 1922
-mpstat 1 5 Average: 0.20 usr 0.10 sys ... 99.70 idle   (~0.3% used)
-pct df 9001: rootfs 9.7G size, 2.1G used, 21.6%
-```
-
-### 5.6 LXC 9001 — under-load measurement
-```
-$ pgbench -i -s 10  -> done in 1.39 s
-$ pgbench -T 60 -c 4 (run concurrently with sampling):
-Host RAM used (5x3s): 2149 / 2143 / 2112 / 2086 / 2071 MB     (median 2112, Δ +410)
-cgroup memory.current: 2130382848 B = 2032 MB
-mpstat 1 5 Average: 17.10 usr 7.50 sys 14.50 iowait 0.31 soft 60.59 idle  (~39.4% used)
-pgbench result: scaling 10, clients 4, 60 s
-  transactions: 132710 ; failed 0 (0.000%)
-  latency average = 1.809 ms ; tps = 2211.713864
-host thin LV vm-9001-disk-0: 10240 MB, Data% 26.73  (≈2.67 GiB)
-```
-
-### 5.7 VM 9000 — create + cloud-init
-```
-$ qm create 9000 --name spike-vm --cores 2 --memory 2048 \
-    --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --agent 1
-$ qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2
-  transferred 3.0 GiB of 3.0 GiB (100.00%)
-  scsi0: successfully created disk 'local-lvm:vm-9000-disk-0,size=3G'
-$ qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0 --serial0 socket --vga serial0
-$ qm disk resize 9000 scsi0 10G        -> resized 3.00 -> 10.00 GiB
-$ qm set 9000 --ciuser spike --cipassword spike --sshkeys /root/spike-pubkey.pub --ipconfig0 ip=dhcp
-   # pubkey file = the two real keys from the host's /etc/pve/priv/authorized_keys
-   #   (incl. ssh-ed25519 ...kisfenyo@windows — the same workstation key)
-$ qm start 9000   -> start-ok
-```
-
-### 5.8 VM 9000 — IP discovery + guest agent + Docker
-```
-# genericcloud has no guest-agent at first boot -> qm guest cmd ping failed.
-# IP found via MAC on the bridge:
-$ nmap -sn 192.168.0.0/24 | grep -B2 BC:24:11:C7:41:87
-  Nmap scan report for 192.168.0.155 ; MAC BC:24:11:C7:41:87 (Proxmox)
-$ ssh -i /root/.ssh/id_rsa spike@192.168.0.155 'hostname; cat /etc/debian_version'
-  spike-vm ; 13.5
-# install qemu-guest-agent + Docker (official repo, trixie): VM-INSTALL-OK
-$ qm guest cmd 9000 ping            -> AGENT OK   (fsfreeze available)
-$ docker --version                  -> Docker version 29.5.3, build d1c06ef
-$ docker run --rm hello-world       -> Hello from Docker!
-$ docker info | grep -iE 'Storage Driver|Cgroup'
- Storage Driver: overlayfs ; Cgroup Driver: systemd ; Cgroup Version: 2
-```
-
-### 5.9 VM 9000 — stack health
-```
-$ docker compose ps -> spike-cache-1 / spike-db-1 / spike-web-1 all running
-$ curl ... localhost:8080 -> HTTP 200
-$ psql ... SELECT count(*) -> 1   (volume persists)
-```
-
-### 5.10 VM 9000 — idle measurement
-```
-Host RAM used (5x3s): 3758 / 3757 / 3754 / 3759 / 3758 MB     (median 3758, Δ +2056)
-KVM process RSS / VSZ: 2079988 / 3380896 KiB  (RSS = 2031 MB)
-inside free -m: total 1974 used 509 buff/cache 1524 available 1464
-mpstat 1 5 Average: 3.37 usr 2.31 sys 0.29 guest ... 94.04 idle  (~6.0% used)
-qm config: scsi0 local-lvm:vm-9000-disk-0,size=10G
-host thin LV vm-9000-disk-0: 10240 MB, Data% 29.33  (≈2.94 GiB)
-inside df -h /: 9.7G size, 2.4G used, 25%
-```
-
-### 5.11 VM 9000 — under-load measurement (definitive, load confirmed active)
-```
-# First attempt (nested-ssh + nohup &) launched pgbench AFTER the sample window ->
-# host CPU read a false ~5% (identical to idle). Diagnosed; re-run below holds
-# pgbench in the foreground of a long-lived SSH channel and samples during it.
-
-$ pgbench -T 90 -c 4 (foreground, channel held):
-  transactions: 163764 ; failed 0 (0.000%)
-  latency average = 2.198 ms ; tps = 1819.602345
-  (60 s confirmation runs: 1739 & 1759 tps)
-
-# Sampled 10 s into the confirmed-active load:
-Host RAM used (5x3s): 3784 / 3786 / 3786 / 3786 / 3786 MB     (median 3786, Δ +2084)
-KVM process RSS / VSZ: 2096508 / 4495008 KiB  (RSS = 2047 MB)
-guest uptime: load average 1.71 (2 vCPU)  -> vCPUs busy
-mpstat 1 8 Average:
-  1.70 usr  3.40 sys  16.35 iowait  0.58 soft  31.89 guest  46.08 idle   (~53.9% used)
-```
-
-### 5.12 Teardown state
-```
-$ qm list  -> 9000 spike-vm stopped
-$ pct list -> 9001 spike-lxc stopped
-# both present, both stopped (numbers can be re-checked)
-```
-
----
-
-## 6. Teardown — destroy commands (NOT run)
-
-Both guests were left **stopped but present**. To remove them:
-
-```bash
-qm destroy 9000 --purge            # VM   (also removes cloudinit + disks)
-pct destroy 9001 --purge           # LXC
-# optional spike artifacts on the host:
-rm -f /var/lib/vz/template/qcow2/debian-13-genericcloud-amd64.qcow2
-rm -f /root/spike-pubkey.pub /root/vm-install.sh
-# (Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst)
-```

docs/tests/phase1-2-findings.md

@@ -1,315 +0,0 @@
-# Phase 1 + 2 — Privilege Model & Backup/Restore Round-Trip: Findings
-
-**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via
-`pvesh get /nodes` → `demo-felhom`. Storage: `local` (dir, content
-`iso,vztmpl,backup,import`), `local-lvm` (LVM-thin, `rootdir,images`).
-**Subject:** LXC `9001` (`spike-lxc`, unprivileged, `nesting=1,keyctl=1`, Docker +
-postgres/redis/nginx stack). **Date:** 2026-06-07.
-
-> Data and observations only — **no recommendation or verdict**.
-
-## Hypotheses — verdicts at a glance
-| | Hypothesis | Result |
-|---|---|---|
-| **H1** | Backup scopes to one VMID; restore/create needs node/pool allocate → denied to narrow token | **CONFIRMED** (create CT = 403) |
-| **H2** | An LXC vzdump captures the Docker volumes (they live in the container rootfs) | **CONFIRMED** (sentinel survived both restores) |
-| **H3** | Crash-consistent (running) *and* quiesced (stopped) backups both restore cleanly | **CONFIRMED** (A via WAL recovery, B clean start) |
-| **H4** | Running unprivileged LXC snapshots on LVM-thin; restored CT keeps unprivileged+nesting/keyctl | **CONFIRMED** (live snapshot OK; config survived) |
-
----
-
-## 1. Phase 1 — Privilege model
-
-### 1.1 Setup (operator side, root)
-```
-pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit"
-pveum user add felhom-ctl@pve --comment "spike in-guest controller"
-pveum user token add felhom-ctl@pve ctl --privsep 1   # secret: b6547d9d-... (ephemeral, spike-only)
-pveum acl modify /vms/9001      -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
-pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup
-```
-Privilege names were verified against `PVEVMAdmin` / `PVEDatastoreUser` via
-`pveum role list` first. **Note:** the reference doc's introspection command
-`pveum role info <role>` **does not exist in PVE 9** — only `pveum role list` works.
-
-### 1.2 ⚠️ Privsep gotcha — the doc's runbook is incomplete
-With `--privsep 1`, a token's effective rights are the **intersection of the backing
-user's permissions AND the token's own ACLs**. The reference doc (§3) grants ACLs to the
-**token only**. With the user `felhom-ctl@pve` holding **no** permissions, the
-intersection was **empty** — the first self-audit call returned:
-```
-HTTP 403  {"message":"Permission check failed (/vms/9001, VM.Audit)\n"}
-```
-**Fix applied:** also grant the user the role on the same paths
-(`pveum acl modify /vms/9001 -user felhom-ctl@pve -role FelhomSelfBackup`, same for
-`/storage/local`). After that the self-calls succeeded. **A privsep token needs the
-permission present on *both* the user and the token** (the token ACL is what keeps the
-token ≤ user / narrowly scoped). This must be reflected in the controller provisioning.
-
-### 1.3 Test matrix (every call run from **inside** the unprivileged LXC, `pct exec 9001`)
-`H=192.168.0.162  N=demo-felhom  AUTH="PVEAPIToken=felhom-ctl@pve!ctl=<secret>"`
-
-| # | Call | Expected | **Actual** | Notes |
-|---|---|---|---|---|
-| 1 | `GET /version` | 200 | **200** | reachable + auth from inside LXC (no privilege needed) |
-| 2 | `GET /nodes/$N/lxc/9001/status/current` | 200 | **200**¹ | self audit (after privsep fix) |
-| 3 | `POST /nodes/$N/lxc/9001/snapshot snapname=spk1` | 200/UPID→OK | **200, task exitstatus OK** | **running-LXC self-snapshot (H4)** |
-| 4 | `POST /nodes/$N/vzdump vmid=9001 storage=local mode=snapshot` | 200/UPID→OK | **200, task exitstatus OK** | self backup, archive produced |
-| 5 | `GET /nodes/$N/qemu/9000/status/current` | 403 | **403** | `Permission check failed (/vms/9000, VM.Audit)` |
-| 6 | `POST /nodes/$N/vzdump vmid=9000 storage=local` | 403 | **200 POST → task exitstatus 403**² | see note |
-| 7 | `POST /nodes/$N/lxc` (create CT) | 403 | **403** | `Permission check failed` — **proves create/allocate is operator-tier (H1)** |
-
-¹ before the privsep fix this was 403; see §1.2.
-² **Important nuance:** the `vzdump` endpoint accepts the POST and returns a UPID even for
-an unauthorized vmid; the authorization failure surfaces at **task execution**, not at the
-HTTP layer. Polled from root:
-`exitstatus: "403 Permission check failed (/vms/9000, VM.Backup)"`, and **no 9000 archive
-was created**. The boundary holds — but a controller must **poll the task exitstatus**, not
-trust the POST's 200, to know a cross-guest backup was actually refused.
-
-**Pass criteria met:** self-ops (1–4) succeed; cross-guest read (5), cross-guest backup
-(6, at task level), and create/allocate (7) are denied. The controller-as-guest boundary
-and the two-tier split are validated.
-
-### 1.4 Final minimal role — `VM.PowerMgmt` **not** required
-The doc's open question ("does Tier A need `VM.PowerMgmt` for stop-mode backups? Likely
-yes"). **Tested and refuted:** a **stop-mode** self-vzdump submitted by the token
-(`vmid=9001 mode=stop`) completed with **`exitstatus: OK`** using the role *without*
-`VM.PowerMgmt`. `vzdump` performs the guest shutdown/restart internally under
-`VM.Backup`; no separate power privilege is needed.
-
-> **Final minimal role (`FelhomSelfBackup`) — satisfies self-audit, self-snapshot, and
-> both `snapshot`- and `stop`-mode self-backup:**
-> `VM.Audit, VM.Snapshot, VM.Backup, Datastore.AllocateSpace, Datastore.Audit`
-> (`VM.PowerMgmt` deliberately omitted — confirmed unnecessary.)
-
-### 1.5 TLS observation
-From inside the LXC, `curl` **without** `-k`:
-```
-curl: (60) SSL certificate problem: unable to get local issuer certificate
-```
-The host serves the default self-signed PVE cert; all tests used `-k`. Production trust
-(pin the PVE CA / issue a proper cert) is a separate design decision, flagged here.
-
-### 1.6 Running-LXC snapshot (H4)
-Call #3 snapshotted the **running** unprivileged LXC on LVM-thin (`exitstatus OK`).
-`pct listsnapshot 9001` shows `spk1` with `pct status 9001 = running`. **No stop
-required** — the snapshot-before-update rollback flow is viable on a live container.
-
----
-
-## 2. Phase 2 — Backup → real restore round-trip
-
-Sentinel written pre-flight into the `pgdata` volume:
-`restore_check(42,'phase2-sentinel')` → clean read `42|phase2-sentinel`.
-
-### 2.1 Backups (operator/root side)
-| Variant | Mode | Stack state | Task time | Wall | Archive | Size (zstd) |
-|---|---|---|---|---|---|---|
-| **A — crash-consistent** | `snapshot` | **running** | 00:00:24 | 25 s | `vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst` | **934 MB** (979,718,569 B) |
-| **B — quiesced** | `snapshot` | **stopped** (`docker compose stop`) | 00:00:21 | 22 s | `vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst` | **934 MB** (979,671,582 B) |
-
-Both from a 2.5 GiB source; zstd → ~934 MB (~2.7:1). The stack was restarted after
-Variant B. **LXC snapshot-mode vzdump does *not* fsfreeze** (no guest agent in an LXC —
-consistent with the Phase 0 finding) → Variant A is genuinely crash-consistent.
-
-### 2.2 Restore → fresh VMID → boot → verify
-| Check | 9002 (Variant A) | 9003 (Variant B) |
-|---|---|---|
-| Restore time (`pct restore … --storage local-lvm`) | **12 s** | **11 s** |
-| `unprivileged: 1` survived | **yes** | **yes** |
-| `features: nesting=1,keyctl=1` survived | **yes** | **yes** |
-| Containers after boot | `exited` (no restart policy) → `docker compose up -d` | same |
-| 3 containers healthy | **yes** | **yes** |
-| `curl localhost:8080` | **HTTP 200** | **HTTP 200** |
-| **Sentinel `(42,'phase2-sentinel')`** | **PRESENT** | **PRESENT** |
-| Postgres first-start | **WAL crash recovery** (see below) | **clean start, no recovery** |
-
-> Restored CTs inherit 9001's fixed `hwaddr`. To avoid a MAC clash with the still-running
-> 9001 on `vmbr0`, `net0` was reset to auto-generate a fresh MAC before boot. All
-> verification (stack health, `curl localhost`, sentinel) is guest-internal and needs no
-> external network — and the Docker images are inside the restored rootfs, so no pulls.
-
-**Variant A — Postgres automatic WAL recovery on 9002 (verbatim, post-restore boot):**
-```
-LOG:  database system was interrupted; last known up at 2026-06-07 18:13:21 UTC
-LOG:  database system was not properly shut down; automatic recovery in progress
-LOG:  redo starts at 0/CB12838
-LOG:  invalid record length at 0/CB12870: expected at least 24, got 0   # normal end-of-WAL
-LOG:  redo done at 0/CB12838 ...
-LOG:  checkpoint starting: end-of-recovery immediate wait
-LOG:  database system is ready to accept connections
-```
-**Variant B — clean start on 9003 (verbatim, post-restore boot):**
-```
-LOG:  database system was shut down at 2026-06-07 18:14:39 UTC
-LOG:  database system is ready to accept connections
-```
-
-**H2 confirmed:** one LXC vzdump captured the whole customer including the Docker named
-volume — the sentinel data restored in both guests. **H3 confirmed:** both variants
-restored to a bootable guest with intact data; the crash-consistent one recovered via WAL
-with no manual intervention, the quiesced one started clean. **H4 confirmed:** restored
-config preserved `unprivileged` + `nesting/keyctl`, so Docker ran in the restored CT.
-
----
-
-## 3. Observations & confounds
-1. **Privsep token needs perms on user *and* token** (§1.2) — the single most important
-   correction to the reference runbook; without it every scoped call 403s.
-2. **vzdump authorization is task-level, not POST-level** (§1.3 note ²) — a 200 + UPID
-   does **not** mean authorized. The controller must poll `exitstatus`. This is also the
-   general async-task lesson: every backup/snapshot/restore returns a UPID and the real
-   result is in the task status.
-3. **`pveum role info` is gone in PVE 9** — use `pveum role list`. Minor doc drift.
-4. **`VM.PowerMgmt` not needed for stop-mode backup** (§1.4) — narrower role than the doc
-   assumed.
-5. **No fsfreeze for LXC** — Variant A relied on Postgres's own WAL crash recovery, which
-   worked here for an idle-at-backup DB. Under heavy write load, app-consistency for LXC
-   still rests on the controller quiescing first (or stop-mode), exactly as the reference
-   warned. This single test is not a durability guarantee under load.
-6. **Restore MAC collision** (§2.2) — `pct restore` preserves the source `hwaddr`;
-   restoring while the original runs needs a MAC reset (or the original stopped). The
-   controller's restore flow must handle identity (MAC/hostname/IP) to avoid clashes.
-7. **No restart policy on the compose services** — restored containers came up `exited`;
-   `docker compose up -d` (or a restart policy / systemd unit) is required for the stack
-   to return automatically after a restore or guest reboot.
-8. **Restore is fast, backup dominated by I/O** — restores were 11–12 s (extract at
-   ~524 MiB/s); backups ~22–25 s (read 2.5 GiB at ~108–119 MiB/s + zstd). Single runs,
-   idle host, ~150 MB DB; not a throughput benchmark.
-9. **Sequencing artifact:** a Phase-1 stop-mode self-backup ran before Phase 2 and
-   stopped/started 9001; the stack was brought back up and the sentinel re-verified
-   before the Variant A/B backups, so it does not affect the round-trip results.
-
----
-
-## 4. Raw command log (appendix)
-
-### 4.1 Pre-flight
-```
-$ pvesh get /nodes  -> node: demo-felhom
-$ cat /etc/pve/storage.cfg
-dir: local   ... content iso,vztmpl,backup,import        # 'backup' present
-lvmthin: local-lvm ... content rootdir,images            # no backup (expected)
-$ pct start 9001 ; docker compose up -d  -> 3 containers Started
-$ curl localhost:8080  -> HTTP 200
-# sentinel:
-CREATE TABLE ; INSERT 0 1 ; SELECT count -> 1 ; SELECT * -> 42 | phase2-sentinel
-```
-
-### 4.2 Phase 1 — role/user/token/ACL
-```
-$ pveum role add FelhomSelfBackup -privs "VM.Audit VM.Snapshot VM.Backup Datastore.AllocateSpace Datastore.Audit"  -> role-ok
-$ pveum user add felhom-ctl@pve --comment "spike in-guest controller"  -> user-ok
-$ pveum user token add felhom-ctl@pve ctl --privsep 1
-  {"full-tokenid":"felhom-ctl@pve!ctl","info":{"privsep":"1"},"value":"b6547d9d-08ec-4f22-beb8-a551dc2cd69d"}
-$ pveum acl modify /vms/9001 -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup   -> ok
-$ pveum acl modify /storage/local -token 'felhom-ctl@pve!ctl' -role FelhomSelfBackup -> ok
-$ pveum role list | grep FelhomSelfBackup
-  FelhomSelfBackup | Datastore.AllocateSpace,Datastore.Audit,VM.Audit,VM.Backup,VM.Snapshot
-$ pveum role info FelhomSelfBackup   -> ERROR: unknown command 'pveum role info'   # PVE9 has no 'role info'
-```
-
-### 4.3 Phase 1 — matrix (from inside LXC)
-```
-# TLS without -k:
-curl: (60) SSL certificate problem: unable to get local issuer certificate
-
-# BEFORE privsep fix:
-#2 GET self status -> HTTP 403 {"message":"Permission check failed (/vms/9001, VM.Audit)\n"}
-
-# privsep fix:
-$ pveum acl modify /vms/9001 -user 'felhom-ctl@pve' -role FelhomSelfBackup  -> ok
-$ pveum acl modify /storage/local -user 'felhom-ctl@pve' -role FelhomSelfBackup -> ok
-
-# AFTER fix:
-#1 GET /version                         -> HTTP 200
-#2 GET /nodes/.../lxc/9001/status/current -> HTTP 200 {"data":{...,"status":"running",...}}
-#5 GET /nodes/.../qemu/9000/status/current -> HTTP 403 (/vms/9000, VM.Audit)
-#6 POST vzdump vmid=9000 -> HTTP 200 {"data":"UPID:...vzdump:9000:felhom-ctl@pve!ctl:"}
-   root poll: exitstatus="403 Permission check failed (/vms/9000, VM.Backup)"
-   task log: TASK ERROR: 403 Permission check failed (/vms/9000, VM.Backup)
-   /var/lib/vz/dump: no 9000 archive created
-#7 POST /nodes/.../lxc (create CT vmid=9009) -> HTTP 403 {"message":"Permission check failed\n"}
-
-#3 POST lxc/9001/snapshot snapname=spk1 -> HTTP 200 UPID:...vzsnapshot:9001...
-   root: exitstatus "OK" ; pct listsnapshot 9001 -> spk1 ; pct status 9001 -> running
-#4 POST vzdump vmid=9001 storage=local mode=snapshot -> HTTP 200 UPID:...vzdump:9001...
-   root: exitstatus "OK"
-   token can read own task status: HTTP 200 {"...exitstatus":"OK"}   # earlier poll TIMEOUTs were a shell-quoting bug in the helper, not a perms issue
-
-# stop-mode self-backup (VM.PowerMgmt test):
-$ token POST vzdump vmid=9001 storage=local mode=stop -> HTTP 200 UPID:...vzdump:9001...
-   root poll: exitstatus "OK"     # SUCCEEDED without VM.PowerMgmt in the role
-```
-
-### 4.4 Phase 2 — backups
-```
-# Variant A (running):
-$ vzdump 9001 --mode snapshot --storage local --compress zstd
-INFO: Total bytes written: 2585589760 (2.5GiB, 108MiB/s)
-INFO: archive file size: 934MB
-INFO: Finished Backup of VM 9001 (00:00:24)   ; WALL_SECONDS=25
--> vzdump-lxc-9001-2026_06_07-20_13_43.tar.zst  (979718569 B)
-
-# Variant B (stopped):
-$ docker compose stop   (cache,db,web Stopped)
-$ vzdump 9001 --mode snapshot --storage local --compress zstd
-INFO: Total bytes written: 2585825280 (2.5GiB, 119MiB/s)
-INFO: Finished Backup of VM 9001 (00:00:21)   ; WALL_SECONDS=22
--> vzdump-lxc-9001-2026_06_07-20_14_40.tar.zst  (979671582 B)
-$ docker compose start   (db,cache,web Started)
-```
-
-### 4.5 Phase 2 — restores + verification
-```
-# A -> 9002:
-$ pct restore 9002 .../20_13_43.tar.zst --storage local-lvm
-  Total bytes read: 2585589760 (2.5GiB, 524MiB/s) ; RESTORE_A_SECONDS=12
-$ pct config 9002 -> features: nesting=1,keyctl=1 ; unprivileged: 1
-$ pct set 9002 -net0 name=eth0,bridge=vmbr0,ip=dhcp   # fresh MAC BC:24:11:E3:F4:64
-$ pct start 9002 ; docker compose up -d -> 3 running ; curl -> HTTP 200
-$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel
-  db log: "was interrupted ... not properly shut down; automatic recovery in progress
-           redo starts/redo done ... database system is ready to accept connections"
-
-# B -> 9003:
-$ pct restore 9003 .../20_14_40.tar.zst --storage local-lvm
-  Total bytes read: 2585825280 (2.5GiB, 524MiB/s) ; RESTORE_B_SECONDS=11
-$ pct config 9003 -> features: nesting=1,keyctl=1 ; unprivileged: 1
-$ pct set 9003 -net0 ... (fresh MAC) ; pct start 9003 ; docker compose up -d -> 3 running ; curl 200
-$ psql SELECT * FROM restore_check -> 42 | phase2-sentinel
-  db log: "database system was shut down at ... ; database system is ready to accept connections"  # clean
-```
-
----
-
-## 5. Teardown (executed)
-Restore targets destroyed; Phase 1 objects and spike artifacts removed; `9000`/`9001`
-left **stopped-but-present**. Verified clean: `felhom-ctl@pve` deleted, no spike ACLs,
-empty `dump/`, `spk1` removed.
-
-> **Correction:** `pveum acl delete` **requires `--roles`** (a bare `-user`/`-token`
-> path errors `400 roles: property is missing`). In practice the explicit ACL deletes
-> are unnecessary — deleting the token/user/role **auto-invalidates** the referencing
-> ACLs (PVE logs `ignore invalid acl token …` and drops them).
-
-```bash
-pct stop 9002 ; pct stop 9003 ; pct destroy 9002 --purge ; pct destroy 9003 --purge
-# correct ACL-delete syntax (needs --roles), or just let user/role deletion clean them:
-pveum acl delete /vms/9001      --roles FelhomSelfBackup --users  'felhom-ctl@pve'
-pveum acl delete /vms/9001      --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl'
-pveum acl delete /storage/local --roles FelhomSelfBackup --users  'felhom-ctl@pve'
-pveum acl delete /storage/local --roles FelhomSelfBackup --tokens 'felhom-ctl@pve!ctl'
-pveum user token remove felhom-ctl@pve ctl ; pveum user delete felhom-ctl@pve ; pveum role delete FelhomSelfBackup
-pct delsnapshot 9001 spk1
-rm -f /var/lib/vz/dump/vzdump-lxc-9001-*.tar.zst /var/lib/vz/dump/vzdump-lxc-9001-*.log
-pct stop 9001     # back to stopped-but-present
-```
-
-## 6. To destroy 9000/9001 later (NOT run — left stopped-but-present)
-```bash
-qm destroy 9000 --purge        # VM  (Phase 0 subject)
-pct destroy 9001 --purge       # LXC (Phase 0/1/2 subject)
-# Debian 13 CT template left in place: local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst
-```

docs/tests/phase3-findings.md

@@ -1,234 +0,0 @@
-# Phase 3 — vzdump exclusion (B2) & agent operator role + root boundary (B3): Findings
-
-**Host:** `demo-felhom` (192.168.0.162) — Proxmox VE 9.2.2, node confirmed via
-`pvesh get /nodes` → `demo-felhom`. **Date:** 2026-06-08. Throwaway resources (VMIDs
-9010-9023, role/user `FelhomAgent`/`felhom-agent@pve`); all torn down (only the pre-existing
-9000/9001 remain, stopped). Every Proxmox op polled to `task exitstatus` (not the POST
-return).
-
-> Validates the two items the design review (`_design-review.md`) flagged as unvalidated:
-> **B2** (what vzdump includes/excludes per LXC mount type + how to keep bulk out) and **B3**
-> (the least-privilege operator role + the root-vs-API boundary). Data only.
-
----
-
-## B2 — vzdump inclusion/exclusion matrix
-
-**Setup:** one unprivileged LXC `9010` (`nesting=1,keyctl=1`, overlayfs), Docker 29.5.3
-installed, with five sentinel locations:
-
-| # | location | config |
-|---|---|---|
-| 1 | rootfs file `/SENTINEL_ROOTFS` | rootfs (`local-lvm:8`) |
-| 2 | Docker **named** volume `b2vol` → `SENTINEL_DOCKERVOL` | default driver |
-| 3 | `mp1` volume mount `/mnt/mp1` `SENTINEL_MP1` | `local-lvm:1,backup=1` |
-| 4 | `mp2` volume mount `/mnt/mp2` `SENTINEL_MP2` | `local-lvm:1,backup=0` |
-| 5 | `mp3` **bind** mount `/mnt/mp3` `SENTINEL_MP3` | host `/root/b2-bindsrc` |
-| 6 | bulk Docker vol `bulkvol` bound onto mp2 → `SENTINEL_BULK` | `--driver local -o type=none -o o=bind -o device=/mnt/mp2` |
-
-**The "trap" confirmed at setup:** the Docker named volume's on-disk path is
-`/var/lib/docker/volumes/b2vol/_data` — **inside the LXC rootfs**.
-
-### Result matrix (stop-mode vzdump → `local`, verified 3 ways: vzdump log, archive grep, restore to 9011)
-
-| Sentinel | location | flag | **in archive?** | restored 9011 |
-|---|---|---|---|---|
-| `SENTINEL_ROOTFS` | rootfs | — | **INCLUDED** | present |
-| `SENTINEL_DOCKERVOL` | Docker named vol (in rootfs) | — | **INCLUDED** ⚠️ the trap | present |
-| `SENTINEL_MP1` | volume mp | `backup=1` | **INCLUDED** | present |
-| `SENTINEL_MP2` | volume mp | `backup=0` | **EXCLUDED** | absent (vol recreated empty) |
-| `SENTINEL_MP3` | bind mount | n/a | **EXCLUDED** | reappears via re-bind only¹ |
-| `SENTINEL_BULK` | Docker vol on mp2 | `backup=0` | **EXCLUDED** | absent |
-
-¹ The bind-mount **data is not in the archive** (archive grep shows no mp3 path). It
-reappears in the restored 9011 only because `pct restore` preserves the bind config
-`mp3: /root/b2-bindsrc` and re-attaches the **same host dir**. On a *different* host (true DR)
-the bind data would be gone unless backed up separately — important for DR planning.
-
-**vzdump log (verbatim) — the authoritative per-mount decision:**
-```
-INFO: including mount point rootfs ('/') in backup
-INFO: including mount point mp1 ('/mnt/mp1') in backup
-INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled)
-INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume)
-```
-**Archive contents (verbatim) — `tar --zstd -tf … | grep SENTINEL`:**
-```
-./var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL
-./SENTINEL_ROOTFS
-./mnt/mp1/SENTINEL_MP1
-```
-**Restore verification (verbatim) — sentinels in restored 9011:**
-```
-PRESENT : /SENTINEL_ROOTFS
-PRESENT : /var/lib/docker/volumes/b2vol/_data/SENTINEL_DOCKERVOL
-PRESENT : /mnt/mp1/SENTINEL_MP1
-ABSENT  : /mnt/mp2/SENTINEL_MP2
-ABSENT  : /mnt/mp2/SENTINEL_BULK
-PRESENT : /mnt/mp3/SENTINEL_MP3   # via re-bind to same host dir, NOT from archive
-```
-
-### Proven bulk-exclusion recipe
-A "bulk" Docker volume is kept out of the guest vzdump by binding it onto a **volume
-mountpoint with `backup=0`**:
-1. Attach a Proxmox volume mountpoint with the flag:
-   `pct set <id> -mpN <storage>:<size>,mp=/mnt/bulk,backup=0`
-2. Realize the Docker volume on that path:
-   `docker volume create --driver local -o type=none -o o=bind -o device=/mnt/bulk bulkvol`
-   (or a compose bind to `/mnt/bulk`).
-3. Data written through `bulkvol` lands on the `backup=0` mountpoint → **excluded** from
-   vzdump, while rootfs/hot sentinels are **included**. Verified: `SENTINEL_BULK` absent from
-   archive and restore; `SENTINEL_ROOTFS` present.
-
-### The trap, stated for the placement component
-`backup=<boolean>` is **only honoured for volume mount points** (confirmed: pct manpage +
-vzdump log "excluding volume mount point … (disabled)"). A Docker **named volume uses the
-default driver and lands in the rootfs**, which is **always backed up** — so a "bulk" volume
-left as an ordinary named volume is **silently swept into the whole-guest image**. The
-per-volume placement component **must** realize every `bulk` volume as a dedicated `backup=0`
-mountpoint (or external bind mount), never a default named volume.
-
----
-
-## B3 — agent operator role + root-vs-API boundary
-
-**Caveat applied (Phase 1):** privsep token needs the role on **both** user and token. Setup:
-user `felhom-agent@pve` + privsep token `agent`, role `FelhomAgent`, dual-granted at `/`.
-All ops driven **as the token** via the REST API; task `exitstatus` polled.
-
-> ⚠️ **Terminology:** the Phase-1 `FelhomSelfBackup` role is the discarded **guest-side
-> self-backup** role (scoped to one guest, *denied* create/allocate). `FelhomAgent` here is
-> its **operator-tier replacement** — a different, broader role. Do not conflate.
-
-### Op matrix (as the scoped token)
-
-| # | Operation | API call | Result |
-|---|---|---|---|
-| read | host status | `GET /nodes/$N/status` | **200** (needs `Sys.Audit`) |
-| read | storage list | `GET /storage` | **200** (`Datastore.Audit`) |
-| 1 | **create LXC, `nesting=1,keyctl=1`** | `POST /nodes/$N/lxc` | **403** — `changing feature flags (except nesting) is only allowed for root@pam` |
-| 1′ | create LXC, **nesting-only** | `POST /nodes/$N/lxc` | **200 / OK** |
-| 2 | set config (mem/cpu/options + mountpoint w/ `backup` flag) | `PUT /nodes/$N/lxc/<id>/config` | **200** |
-| 3 | allocate volume | `POST /nodes/$N/storage/local-lvm/content` | **200** (`Datastore.AllocateSpace`) |
-| 4 | start | `POST …/status/start` | **OK** (`VM.PowerMgmt`) |
-| 5 | stop | `POST …/status/stop` | **OK** |
-| 6a | snapshot | `POST …/snapshot` | **OK** (`VM.Snapshot`) |
-| 6b | rollback | `POST …/snapshot/s1/rollback` | **OK** (`VM.Snapshot.Rollback`) |
-| 7 | stop-mode backup | `POST /nodes/$N/vzdump mode=stop` | **OK** (`VM.Backup`) |
-| 8 | restore → fresh vmid | `POST /nodes/$N/lxc restore=1` | **OK** — and **restored CT kept `features: nesting=1,keyctl=1`** |
-| 9 | destroy CT | `DELETE /nodes/$N/lxc/<id>?purge=1` | **OK** (`VM.Allocate`) |
-| 9b | add storage definition (dir) | `POST /storage` | **200** (`Datastore.Allocate`, **no root**) |
-
-**The two headline results:**
-1. **`keyctl=1` on create is `root@pam`-only.** Verbatim:
-   `Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)`.
-   Confirmed this is **not** token-fixable: a **non-privsep `root@pam` token** got the **same
-   403**. Only an actual `root@pam` session (OS root / `pct create` as root) can set it.
-   `nesting` alone is allowed for a scoped token.
-2. **Restore preserves `keyctl`.** A token-authorized `vzrestore` of a keyctl archive produced
-   `9021` with `features: nesting=1,keyctl=1, unprivileged: 1`. So the **DR/restore path is
-   fully token-covered**; only *fresh provisioning* needs root for the keyctl flag.
-
-### Paring (each drop shown to still pass, or proven needed)
-
-| Privilege | Verdict | Evidence |
-|---|---|---|
-| `Datastore.AllocateTemplate` | **DROP** (unnecessary) | create-from-template succeeded without it (200/OK) |
-| `Sys.Audit` | **KEEP** | `GET /nodes/$N/status` → **403** without it (host metrics, `03` §5) |
-| `VM.Config.Network` | **KEEP** | create with `net0` → **403 (/vms/…, VM.Config.Network)** without it |
-| `VM.Config.Options` | **KEEP** | config `onboot=1` → **403 (/vms/…, VM.Config.Options)** without it |
-| `SDN.Use` | **KEEP (added vs review sketch)** | create → **403 (/sdn/zones/localnetwork/vmbr0, SDN.Use)** without it |
-
-> Corrections to the review's candidate sketch: `VM.Config.CPUMemory` is **not a real
-> privilege** — split into `VM.Config.CPU` + `VM.Config.Memory`. `SDN.Use` was **missing** and
-> is **required** (PVE 9 gates bridge use behind it). `Datastore.AllocateTemplate` is **not
-> needed**.
-
-### Final minimal `FelhomAgent` role (proven sufficient for ops 1′–9b)
-```
-VM.Allocate  VM.Audit  VM.Config.Disk  VM.Config.CPU  VM.Config.Memory
-VM.Config.Network  VM.Config.Options  VM.PowerMgmt  VM.Snapshot  VM.Snapshot.Rollback
-VM.Backup  Datastore.Allocate  Datastore.AllocateSpace  Datastore.Audit  Sys.Audit  SDN.Use
-```
-(16 privileges. `Datastore.Allocate` is for the storage-definition add; drop it if the agent
-never creates Proxmox storage entries via the API. `VM.PowerMgmt` is for start/stop lifecycle
-— not for the backup itself, consistent with `proxmox-platform.md` §3.4.)
-
-### Root-vs-API boundary table (answers `03` §3)
-
-| Agent host operation | Coverage | Notes |
-|---|---|---|
-| Create unprivileged LXC, **nesting-only** | **API token** | `VM.Allocate`+`VM.Config.*`+`Datastore.AllocateSpace`+`SDN.Use` |
-| **Create with `keyctl=1` (Docker needs it — Phase 0)** | **OS root `root@pam`** (`pct create` as root / sudoers) | no API token works, incl. a root@pam token |
-| Set config (mem/cpu/net/options/mountpoint + `backup` flag) | API token | |
-| Allocate guest volume | API token | `Datastore.AllocateSpace` |
-| Start / stop / snapshot / rollback | API token | `VM.PowerMgmt` / `VM.Snapshot(.Rollback)` |
-| vzdump backup (stop/snapshot mode) | API token | `VM.Backup` |
-| **Restore from vzdump (preserves keyctl)** | **API token** | DR path needs no root |
-| Destroy guest (scratch + compensating rollback, B1) | API token | `VM.Allocate` |
-| Add Proxmox **storage definition** (dir/nfs/cifs/pbs) | API token | `Datastore.Allocate`; the *definition* only |
-| Host status / metrics report | API token | `Sys.Audit` |
-| **USB physical mount-by-UUID / systemd mount unit / fstab** | **OS root / narrow sudoers** | not a Proxmox API op (host-level mount; not tested here) |
-| **SMART / hardware sensors** | OS root | not API-exposed |
-
-**Boundary summary:** nearly the entire guest lifecycle — including **restore** — is covered
-by the scoped token. The genuine OS-root residual is narrow: **(1) fresh creation of a
-Docker-capable LXC (the `keyctl` flag), (2) physical USB mount-by-UUID / systemd mount units /
-fstab, (3) hardware/SMART.** This supports `03` §3's "non-root service + scoped token + narrow
-sudoers" model — with the **specific** sudoers/root entries being: `pct create` (or just the
-keyctl-setting step) and the host mount operations.
-
----
-
-## Raw command log (appendix)
-
-### B2
-```
-pct create 9010 ... --features nesting=1,keyctl=1 --unprivileged 1   # rootfs local-lvm:8
-pct set 9010 -mp1 local-lvm:1,mp=/mnt/mp1,backup=1
-pct set 9010 -mp2 local-lvm:1,mp=/mnt/mp2,backup=0
-pct set 9010 -mp3 /root/b2-bindsrc,mp=/mnt/mp3
-# docker named vol: docker volume inspect b2vol -> /var/lib/docker/volumes/b2vol/_data
-# bulk: docker volume create --driver local -o type=none -o o=bind -o device=/mnt/mp2 bulkvol
-vzdump 9010 --mode stop --storage local --compress zstd
-#   INFO: including mount point rootfs ('/') in backup
-#   INFO: including mount point mp1 ('/mnt/mp1') in backup
-#   INFO: excluding volume mount point mp2 ('/mnt/mp2') from backup (disabled)
-#   INFO: excluding bind mount point mp3 ('/mnt/mp3') from backup (not a volume)
-tar --zstd -tf <archive> | grep SENTINEL   # -> rootfs, dockervol, mp1 only
-pct restore 9011 <archive> --storage local-lvm   # -> mp2/bulk absent, mp3 via re-bind
-```
-
-### B3
-```
-pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Sys.Audit"   # candidate (pre-SDN)
-pveum user add felhom-agent@pve ; pveum user token add felhom-agent@pve agent --privsep 1
-pveum acl modify / -user  'felhom-agent@pve'        -role FelhomAgent
-pveum acl modify / -token 'felhom-agent@pve!agent'  -role FelhomAgent
-
-# token create with keyctl:
-POST /nodes/demo-felhom/lxc ... features=nesting=1,keyctl=1
-  -> 403 "changing feature flags (except nesting) is only allowed for root@pam"
-# + SDN.Use missing initially:
-  -> 403 "Permission check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)"
-# root@pam non-privsep token, keyctl create:
-  -> 403 (same "only allowed for root@pam")   # tokens never qualify
-
-# token nesting-only create / config(PUT) / start / stop / snapshot / rollback /
-# vzdump(stop) / restore->9021 (kept keyctl) / destroy / POST /storage  -> all 200/OK
-
-# paring:
-GET /nodes/$N/status  without Sys.Audit            -> 403   (KEEP)
-create net0           without VM.Config.Network     -> 403   (KEEP)
-config onboot=1       without VM.Config.Options      -> 403   (KEEP)
-create from template  without Datastore.AllocateTemplate -> OK (DROP)
-```
-
-### Teardown
-```
-pct destroy 9010 9011 9021 --purge   # 9020/9022/9023 already destroyed during tests
-pveum user token remove felhom-agent@pve agent ; pveum user delete felhom-agent@pve
-pveum role delete FelhomAgent        # ACLs at / auto-invalidated
-rm -f /var/lib/vz/dump/vzdump-lxc-9010-* /var/lib/vz/dump/vzdump-lxc-9020-*
-# verified: only 9000/9001 remain (stopped-but-present); no felhom-agent user/role; dump dir empty
-```

docs/tests/phase4-signing-findings.md

@@ -1,257 +0,0 @@
-# Phase 4 — Control-plane signing primitive (SSHSIG + Go verify): Findings
-
-**Where run:** build server `192.168.0.180` (Debian 13, **Go 1.24.4**, **OpenSSH 10.0p2**),
-no Proxmox. **Date:** 2026-06-08. Throwaway key generated, used, and **deleted** — no private
-key, passphrase, or `.sig` committed.
-
-> De-risks the signing primitive *before* it is written into `04-control-plane-authorization.md`
-> or the agent's verify code. **Verdict up front: the approach works cleanly and is key-type-
-> agnostic — no fallback needed.** Go verifies the armored `SSHSIG` format, every tamper/replay/
-> authorization case is rejected, and a synthetic FIDO2 `sk-ssh-ed25519` signature verifies
-> through the **unchanged** code path (true hardware drop-in).
-
----
-
-## 0. Result at a glance — 14/14 checks pass
-
-```
-== Step 2: SSHSIG signature verification (key-type-agnostic path) ==
-  PASS  correct                verified, op="guest_destroy"
-  PASS  wrong key              rejected: signer not in allowed set
-  PASS  tampered blob          rejected: signature invalid: ssh: signature did not verify
-  PASS  wrong namespace        rejected: namespace mismatch: got "felhom-op-wrong" want "felhom-op-v1"
-
-== Step 3: anti-replay / authorization (valid signature, still rejected) ==
-  PASS  first use              verified, op="guest_destroy"
-  PASS  replay (same nonce)    rejected: replay: nonce a1b2c3d4...8f90 already seen
-  PASS  expired                rejected: expired (expires_at=2020-01-02 ..., now=2026-06-08 ...)
-  PASS  not-yet-valid          rejected: not yet valid (issued_at=2030-01-01 ...)
-  PASS  retargeted host        rejected: target mismatch: blob=demo-felhom/9001 this=other-host/9001
-  PASS  retargeted guest       rejected: target mismatch: blob=demo-felhom/9001 this=demo-felhom/8888
-
-== Step 4: key-type-agnosticism — FIDO2 sk-ssh-ed25519 (synthetic, no device) ==
-  PASS  parses sk pubkey       type="sk-ssh-ed25519@openssh.com"
-  PASS  authorized_keys form   sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5...
-  PASS  sk end-to-end verify   verified, op="guest_destroy"
-```
-
----
-
-## 1. Software round-trip (baseline, CLI)
-
-- Key: `ssh-keygen -t ed25519 -f felhom-op -N '<passphrase>' -C felhom-operator`.
-  (Signing non-interactively used an `SSH_ASKPASS` helper + `setsid -w`; in production the
-  operator key lives behind an agent or a FIDO2 device, so the at-sign passphrase prompt is a
-  non-issue. The passphrase mechanics are **not** what this spike de-risks.)
-- Sign with a **domain-separated namespace**:
-  `ssh-keygen -Y sign -f felhom-op -n felhom-op-v1 blob.json` → `blob.json.sig`
-  (armored `-----BEGIN SSH SIGNATURE-----`).
-- Baseline verify (CLI sanity) with an allow-list:
-  ```
-  allowed_signers:  felhom-operator namespaces="felhom-op-v1" ssh-ed25519 AAAAC3...
-  $ ssh-keygen -Y verify -f allowed_signers -I felhom-operator -n felhom-op-v1 \
-        -s blob.json.sig < blob.json
-  Good "felhom-op-v1" signature for felhom-operator with ED25519 key SHA256:y0Lj8dIYTM6...
-  ```
-
-## 2. Canonical op blob spec (documented)
-
-The signature covers **these exact bytes**; the operator CLI (also Go) must reproduce them
-byte-for-byte. **Canonical form: JSON, keys sorted lexicographically at every level, no
-insignificant whitespace, no trailing newline, UTF-8.**
-
-```json
-{"expires_at":"<RFC3339 UTC>","issued_at":"<RFC3339 UTC>","key_id":"<id>","nonce":"<128-bit hex>","op":"<op>","params":{...},"target":{"guest_id":"<vmid>","host_id":"<node>"}}
-```
-
-| field | meaning |
-|---|---|
-| `op` | the operation, e.g. `guest_destroy`, `storage_detach`, `restore_overwrite` |
-| `target.host_id` / `target.guest_id` | the box + guest the op is bound to (anti-retarget) |
-| `params` | op-specific arguments (themselves canonical-sorted) |
-| `nonce` | unique per op (anti-replay); ≥128-bit random |
-| `issued_at` / `expires_at` | validity window (short — minutes) |
-| `key_id` | which operator key (for rotation / audit) |
-
-Exact test blob (236 bytes): `{"expires_at":"2026-06-09T00:00:00Z","issued_at":"2026-06-08T00:00:00Z","key_id":"felhom-op-1","nonce":"a1b2c3d4e5f60718293a4b5c6d7e8f90","op":"guest_destroy","params":{"purge":true},"target":{"guest_id":"9001","host_id":"demo-felhom"}}`
-
-> Note: the SSHSIG **namespace** (`felhom-op-v1`) is the cryptographic domain separator and is
-> a **fixed constant in the verifier**, never caller-supplied — a signature minted for any
-> other namespace must not verify (proven: "wrong namespace" rejected).
-
-## 3. Go SSHSIG verify — approach + implementation cost
-
-**It is not a one-call verify, but it is clean — no hand-rolled crypto.** The only manual work
-is SSHSIG *framing*; all crypto and key-type dispatch is the library's. Steps:
-
-1. `pem.Decode` the armor → `block.Type == "SSH SIGNATURE"`, `block.Bytes` is the binary SSHSIG.
-   *(Go's `encoding/pem` parses the armor directly — no manual base64/line handling.)*
-2. Strip the literal 6-byte `SSHSIG` magic preamble (it is **not** length-prefixed).
-3. `ssh.Unmarshal` the rest into a struct `{Version uint32; PublicKey, Namespace, Reserved,
-   HashAlgo, Signature string}` — library does the SSH wire parsing.
-4. `ssh.ParsePublicKey([]byte(PublicKey))` → an `ssh.PublicKey`.
-5. Recompute the signed data per spec: `"SSHSIG" || string(namespace) || string(reserved) ||
-   string(hash_algorithm) || string(H(message))`, where `H` is the **named** hash
-   (`sha256`/`sha512`) — built with one `ssh.Marshal`.
-6. `ssh.Unmarshal([]byte(Signature))` into `ssh.Signature`, then **`pub.Verify(signed, &sig)`** —
-   which **dispatches on the key's own algorithm** (this is what makes it key-agnostic).
-
-**Cost verdict:** ~40 lines of framing in one file, zero crypto implemented by us. Well within
-the agent's budget; **no reason to fall back** to a different primitive.
-
-## 4. Anti-replay / authorization layer (on top of signature validity)
-
-Enforced in `VerifySignedOp` *after* the signature check, each proven to reject **even with a
-valid signature** (Step 3 output above):
-
-- **replay** — nonce already recorded in the window → reject;
-- **expired / not-yet-valid** — `now ∉ [issued_at, expires_at]` → reject (both sides shown);
-- **retargeted** — `target.host_id`/`guest_id` ≠ this box/guest → reject (both shown).
-
-(Order matters: signature → namespace → allow-list → crypto verify → target → time → nonce, so
-a replayed *but otherwise valid* op is still caught, and an invalid sig never consumes a nonce.)
-
-## 5. Key-type-agnosticism — **TRUE DROP-IN** (no box change for FIDO2 later)
-
-No FIDO2 device was used (by choice). Instead the spike **emulated the authenticator exactly**:
-
-- Synthesized a well-formed `sk-ssh-ed25519@openssh.com` public key; `ssh.ParsePublicKey` parses
-  it and `ssh.MarshalAuthorizedKey` round-trips it.
-- Constructed a real `SSHSIG` whose inner signature follows the sk scheme (per OpenSSH
-  `PROTOCOL.u2f`): `ed25519` over `sha256(application) || flags || counter || sha256(signed_data)`,
-  with the blob `string(format) string(ed25519_sig) byte(flags) uint32(counter)` — i.e. exactly
-  what a FIDO2 key emits.
-- Ran it through the **unchanged `VerifySignedOp`** → **verified** (`op="guest_destroy"`).
-
-**Verdict: true drop-in.** `pub.Verify` for `sk-ssh-ed25519` is implemented in
-`golang.org/x/crypto/ssh` **v0.52.0** (it reconstructs `appDigest‖flags‖counter‖dataDigest` and
-`ed25519.Verify`s it). Introducing a hardware operator key later is a **no-op on the boxes** —
-the agent's verify code is identical; only the operator's signer key (and the allowed-signers
-set entry) changes. No sk-specific handler is needed.
-
-> Because verification dispatches on the key type embedded in the signature, the same path also
-> accepts `ssh-ed25519`, `rsa-sha2-*`, `ecdsa-sha2-*`, etc. — algorithm choice is the operator's,
-> not the agent's.
-
-## 6. Fallback (not taken) and its cost
-
-A fallback would be a **raw Ed25519 detached signature** (or `minisign`): trivially one
-`ed25519.Verify` call, no SSHSIG framing. **Rejected** because it **loses the clean FIDO2 path** —
-a raw-Ed25519 verifier cannot consume an `sk-ssh-ed25519` signature (which carries flags+counter
-and a different signed-data construction), so the future hardware swap would require **changing
-the verifier on every box**. SSHSIG buys exactly the key-type-agnosticism (§5) that a raw scheme
-forfeits, at a one-file framing cost (§3). **No fallback is warranted.**
-
-## 7. Reference verifier (seed of the agent's verify code)
-
-Verified working on Go 1.24.4 / `x/crypto` v0.52.0. (Test harness omitted; this is the verify
-core + SSHSIG framing + anti-replay/authz.)
-
-```go
-const Namespace = "felhom-op-v1"   // FIXED domain separator, never caller-supplied
-const sshsigMagic = "SSHSIG"
-
-type Target struct{ HostID, GuestID string }
-type OpBlob struct {
-	Op        string          `json:"op"`
-	Target    Target          `json:"target"`
-	Params    json.RawMessage `json:"params"`
-	Nonce     string          `json:"nonce"`
-	IssuedAt  time.Time       `json:"issued_at"`
-	ExpiresAt time.Time       `json:"expires_at"`
-	KeyID     string          `json:"key_id"`
-}
-// (Target needs json tags host_id/guest_id in the real struct.)
-
-type NonceStore interface{ SeenOrRecord(nonce string, exp time.Time) bool }
-
-type sshsigBlob struct {
-	Version                                       uint32
-	PublicKey, Namespace, Reserved, HashAlgo, Signature string
-}
-
-func hashByName(n string) (hash.Hash, error) {
-	switch n {
-	case "sha256": return sha256.New(), nil
-	case "sha512": return sha512.New(), nil
-	}
-	return nil, fmt.Errorf("unsupported SSHSIG hash %q", n)
-}
-
-func parseArmoredSSHSIG(armored []byte) (*sshsigBlob, error) {
-	block, _ := pem.Decode(armored)
-	if block == nil || block.Type != "SSH SIGNATURE" {
-		return nil, errors.New("not an SSH SIGNATURE armor")
-	}
-	if len(block.Bytes) < 6 || string(block.Bytes[:6]) != sshsigMagic {
-		return nil, errors.New("missing SSHSIG magic")
-	}
-	var sb sshsigBlob
-	if err := ssh.Unmarshal(block.Bytes[6:], &sb); err != nil { return nil, err }
-	if sb.Version != 1 { return nil, fmt.Errorf("bad version %d", sb.Version) }
-	return &sb, nil
-}
-
-func signedData(sb *sshsigBlob, msg []byte) ([]byte, error) {
-	h, err := hashByName(sb.HashAlgo); if err != nil { return nil, err }
-	h.Write(msg); md := h.Sum(nil)
-	body := ssh.Marshal(struct{ Namespace, Reserved, HashAlgo string; Hash []byte }{
-		sb.Namespace, sb.Reserved, sb.HashAlgo, md})
-	return append([]byte(sshsigMagic), body...), nil
-}
-
-// VerifySignedOp: key-type-agnostic signature verify + anti-replay/authorization.
-// allowedSigners is the trusted operator set (one key now; a quorum set later).
-func VerifySignedOp(blob, sigArmored []byte, allowedSigners []ssh.PublicKey,
-	thisHostID, thisGuestID string, seenNonces NonceStore) (string, error) {
-
-	sb, err := parseArmoredSSHSIG(sigArmored)
-	if err != nil { return "", err }
-	if sb.Namespace != Namespace {
-		return "", fmt.Errorf("namespace mismatch: got %q want %q", sb.Namespace, Namespace)
-	}
-	pub, err := ssh.ParsePublicKey([]byte(sb.PublicKey))
-	if err != nil { return "", err }
-	allowed := false
-	for _, a := range allowedSigners {
-		if bytes.Equal(a.Marshal(), pub.Marshal()) { allowed = true; break }
-	}
-	if !allowed { return "", errors.New("signer not in allowed set") }
-
-	signed, err := signedData(sb, blob)
-	if err != nil { return "", err }
-	var inner ssh.Signature
-	if err := ssh.Unmarshal([]byte(sb.Signature), &inner); err != nil { return "", err }
-	if err := pub.Verify(signed, &inner); err != nil {     // dispatches on key algorithm
-		return "", fmt.Errorf("signature invalid: %w", err)
-	}
-
-	var op OpBlob
-	if err := json.Unmarshal(blob, &op); err != nil { return "", err }
-	if op.Target.HostID != thisHostID || op.Target.GuestID != thisGuestID {
-		return "", fmt.Errorf("target mismatch")
-	}
-	now := time.Now().UTC()
-	if now.Before(op.IssuedAt) { return "", errors.New("not yet valid") }
-	if now.After(op.ExpiresAt) { return "", errors.New("expired") }
-	if seenNonces.SeenOrRecord(op.Nonce, op.ExpiresAt) {
-		return "", fmt.Errorf("replay: nonce %s already seen", op.Nonce)
-	}
-	return op.Op, nil
-}
-```
-
-## 8. Inputs to the design doc (`04-control-plane-authorization.md`)
-
-- **Primitive confirmed:** SSHSIG (`ssh-keygen -Y sign` / armored `BEGIN SSH SIGNATURE`),
-  verified in Go via `pem.Decode` + `ssh.Unmarshal` + `ssh.ParsePublicKey` + `pub.Verify`. Low
-  implementation cost; no crypto hand-rolled.
-- **Hub cannot forge:** the operator private key never touches the hub; the hub only queues the
-  opaque armored blob (matches `03` §4).
-- **Key-type-agnostic / hardware-ready:** software `ed25519` now, FIDO2 `sk-ssh-ed25519` later is
-  a **box no-op** (proven end-to-end). The verifier hardcodes neither key type nor algorithm.
-- **`allowedSigners` is a set:** single signer today; **threshold/quorum is just set sizing** plus
-  an N-of-M policy on top (out of scope here).
-- **Anti-replay/authz are mandatory and cheap:** namespace (fixed), allow-list, then crypto,
-  then target-binding, time-window, nonce — all enforced and tested.
-- **Canonical blob (§2)** is the shared contract between the operator CLI and the agent verifier.

go.mod

@@ -0,0 +1,7 @@
+module gitea.dooplex.hu/admin/felhom-agent
+
+go 1.25.0
+
+require golang.org/x/crypto v0.52.0
+
+require golang.org/x/sys v0.45.0 // indirect

go.sum

@@ -0,0 +1,6 @@
+golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
+golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
+golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
+golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
+golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=

internal/authz/blob.go

@@ -0,0 +1,81 @@
+package authz
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// CanonicalBlob builds the canonical OpBlob bytes (phase4 §2 field order: keys sorted at
+// every level, no insignificant whitespace, no trailing newline, UTF-8). This is the SINGLE
+// production source of the signed bytes — the operator signing CLI (cmd/felhom-opsign) and the
+// in-Go test minting both call it, so the signer can NEVER drift from what the verifier expects
+// (the verifier authenticates over the RAW received bytes, so these bytes ARE the contract).
+//
+// params is canonicalized internally (parsed + re-marshaled → object keys sorted, whitespace
+// stripped) so the same op+params always yields identical bytes; "" → "{}". Returns an error
+// only when params is not valid JSON.
+func CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) ([]byte, error) {
+	params := strings.TrimSpace(paramsJSON)
+	if params == "" {
+		params = "{}"
+	}
+	var pv interface{}
+	if err := json.Unmarshal([]byte(params), &pv); err != nil {
+		return nil, fmt.Errorf("authz: params is not valid JSON: %w", err)
+	}
+	pc, err := json.Marshal(pv) // Go marshals object keys sorted, compact
+	if err != nil {
+		return nil, fmt.Errorf("authz: canonicalizing params: %w", err)
+	}
+	return []byte(fmt.Sprintf(
+		`{"expires_at":%q,"issued_at":%q,"key_id":%q,"nonce":%q,"op":%q,"params":%s,"target":{"guest_id":%q,"host_id":%q}}`,
+		expires.UTC().Format(time.RFC3339), issued.UTC().Format(time.RFC3339),
+		keyID, nonce, op, pc, guestID, hostID)), nil
+}
+
+// Target binds an op to a specific box (and optionally a guest) — the anti-retarget
+// field. The §7 reference omitted the json tags; production needs them so the
+// signed canonical bytes decode correctly.
+type Target struct {
+	HostID  string `json:"host_id"`
+	GuestID string `json:"guest_id"`
+}
+
+// OpBlob is the canonical signed object (phase4 §2). The signature covers the
+// EXACT bytes of this object's canonical JSON (keys sorted at every level, no
+// insignificant whitespace, no trailing newline, UTF-8) — produced by the
+// operator CLI / hub, verified here over the raw received bytes.
+type OpBlob struct {
+	Op        string          `json:"op"`
+	Target    Target          `json:"target"`
+	Params    json.RawMessage `json:"params"`
+	Nonce     string          `json:"nonce"`
+	IssuedAt  time.Time       `json:"issued_at"`
+	ExpiresAt time.Time       `json:"expires_at"`
+	KeyID     string          `json:"key_id"`
+}
+
+// VerifiedOp is the authenticated, parsed op returned on success — everything the
+// reconcile layer (slice 4) needs to route and execute, not just the op string.
+type VerifiedOp struct {
+	Op     string // the operation, e.g. "guest_destroy"
+	HostID string // target host (== this agent's host)
+	// GuestID is non-empty for a guest-scoped op; the caller routes by it. "" =
+	// host-scoped op. The verifier does NOT need to know all guest ids.
+	GuestID   string
+	Params    json.RawMessage
+	Nonce     string
+	IssuedAt  time.Time
+	ExpiresAt time.Time
+
+	// KeyID is the blob's self-declared key id — ADVISORY / audit only, never an
+	// authz input. Authz is the key-material allow-list match (Signer below).
+	KeyID string
+	// Signer is the allow-listed key whose material matched the signature.
+	Signer AllowedSigner
+	// KeyIDMatchesSigner is false when the blob's advisory KeyID disagrees with
+	// the matched signer's id (a benign audit signal, not a rejection).
+	KeyIDMatchesSigner bool
+}

internal/authz/doc.go

@@ -0,0 +1,37 @@
+// Package authz is the control-plane-authorization layer: it verifies
+// operator-signed destructive ops before the agent executes them. It is what the
+// reconcile loop (slice 4) calls to gate destructive desired-state deltas and
+// signed one-shot jobs (03 §4, 04). The signing mechanism is proven (Phase 4,
+// 14/14) — this package is its production form: a key-type-agnostic SSHSIG
+// verifier, the full anti-replay/authorization pipeline, and a durable,
+// crash-safe nonce store.
+//
+// # Mechanism (LOCKED — do not redesign)
+//
+//   - SSHSIG via golang.org/x/crypto/ssh; no hand-rolled crypto, no raw-Ed25519
+//     fallback. pub.Verify dispatches on the key's own algorithm, so the same path
+//     accepts ed25519 / sk-ssh-ed25519 (FIDO2) / rsa / ecdsa — a hardware operator
+//     key later is a box no-op (Phase 4 §5/§6, doc 04 §7).
+//   - Fixed namespace felhom-op-v1 (package constant, never caller-supplied).
+//   - The verifier verifies over the RAW received blob bytes and never
+//     canonicalizes — the canonical form (sorted-key, whitespace-free JSON) is the
+//     signer's contract, shared by the hub and the felhom-sign CLI.
+//
+// # Pipeline order (load-bearing — Verify)
+//
+//	parse armor → namespace → parse pubkey → allow-list (by key MATERIAL, not
+//	key_id) → crypto verify → parse blob → target → time window → nonce LAST
+//
+// Each post-crypto stage rejects even with an otherwise-valid signature. The nonce
+// is recorded last, so an invalid signature can never consume a nonce. key_id is
+// advisory/audit only — authz is the key-material allow-list match.
+//
+// # Shared-contract dependency (flag for later, not built here)
+//
+// Signatures only verify if the op-generator (hub) and the felhom-sign CLI produce
+// BYTE-IDENTICAL canonical JSON (keys sorted at every level, no insignificant
+// whitespace, no trailing newline, UTF-8 — Phase 4 §2). The verifier deliberately
+// does NOT re-canonicalize, so a divergence between those two producers surfaces as
+// a crypto failure here. A shared canonicalizer that both import would be the right
+// home for that contract; it is out of scope for this slice.
+package authz

internal/authz/errors.go

@@ -0,0 +1,27 @@
+package authz
+
+import "errors"
+
+// Typed rejection sentinels — one per pipeline stage so the reconcile layer can
+// distinguish "rejected" (a real signed op that failed a check) from "malformed"
+// from a future "not yet signed". All are errors.Is-friendly: Verify wraps them
+// with %w plus context.
+var (
+	// ErrMalformed: the armor/SSHSIG/blob could not be parsed (not a rejection of
+	// a well-formed op — bad input).
+	ErrMalformed = errors.New("authz: malformed signature or blob")
+	// ErrNamespace: SSHSIG namespace != the fixed felhom-op-v1 domain separator.
+	ErrNamespace = errors.New("authz: namespace mismatch")
+	// ErrUnknownSigner: the signing key's material is not in the pinned allow-list.
+	ErrUnknownSigner = errors.New("authz: signer not in allowed set")
+	// ErrBadSignature: cryptographic verification failed (tamper / wrong key).
+	ErrBadSignature = errors.New("authz: signature did not verify")
+	// ErrTarget: target.host_id is not this box.
+	ErrTarget = errors.New("authz: target mismatch")
+	// ErrExpired: now > expires_at.
+	ErrExpired = errors.New("authz: op expired")
+	// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
+	ErrNotYetValid = errors.New("authz: op not yet valid")
+	// ErrReplay: the nonce was already recorded in the window.
+	ErrReplay = errors.New("authz: replay (nonce already seen)")
+)

internal/authz/mint_test.go

@@ -0,0 +1,103 @@
+package authz
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/pem"
+	"testing"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// Test helpers that MINT armored SSHSIGs in-Go (hermetic) — the inverse of the
+// production framing. They reuse the production signedData()/sshsigBlob so a test
+// can never drift from the verifier's notion of the signed bytes.
+
+// canonicalBlob delegates to the production CanonicalBlob (so the in-Go test minting can never
+// drift from the real signed-bytes path). Panics on a params error — tests pass valid JSON.
+func canonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) []byte {
+	b, err := CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON, issued, expires)
+	if err != nil {
+		panic(err)
+	}
+	return b
+}
+
+// mintArmor builds an armored SSHSIG over message, using sign to produce the inner
+// ssh.Signature over the recomputed SSHSIG signed-data.
+func mintArmor(t *testing.T, pubMarshaled []byte, namespace, hashName string, message []byte, sign func([]byte) ssh.Signature) []byte {
+	t.Helper()
+	sb := &sshsigBlob{Version: 1, PublicKey: string(pubMarshaled), Namespace: namespace, Reserved: "", HashAlgo: hashName}
+	signed, err := signedData(sb, message)
+	if err != nil {
+		t.Fatalf("signedData: %v", err)
+	}
+	sig := sign(signed)
+	sb.Signature = string(ssh.Marshal(&sig))
+	raw := append([]byte(sshsigMagic), ssh.Marshal(sb)...)
+	return pem.EncodeToMemory(&pem.Block{Type: "SSH SIGNATURE", Bytes: raw})
+}
+
+// newEd25519Signer returns an ssh.PublicKey + a sign closure for a fresh ed25519 key.
+func newEd25519Signer(t *testing.T) (ssh.PublicKey, func([]byte) ssh.Signature) {
+	t.Helper()
+	pub, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sshPub, err := ssh.NewPublicKey(pub)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sign := func(signed []byte) ssh.Signature {
+		return ssh.Signature{Format: ssh.KeyAlgoED25519, Blob: ed25519.Sign(priv, signed)}
+	}
+	return sshPub, sign
+}
+
+// newSyntheticSKSigner emulates a FIDO2 sk-ssh-ed25519@openssh.com key with NO
+// hardware (Phase 4 §5). It builds a spec-faithful sk public key and an sk-format
+// signature: ed25519 over sha256(application)‖flags‖counter‖sha256(signed_data),
+// sig.Blob = the raw ed25519 signature, sig.Rest = flags‖counter. It must verify
+// through the UNCHANGED Verify path.
+func newSyntheticSKSigner(t *testing.T) (ssh.PublicKey, func([]byte) ssh.Signature) {
+	t.Helper()
+	edPub, edPriv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	const application = "ssh:"
+	skBlob := ssh.Marshal(struct {
+		Name        string
+		KeyBytes    []byte
+		Application string
+	}{"sk-ssh-ed25519@openssh.com", []byte(edPub), application})
+	skPub, err := ssh.ParsePublicKey(skBlob)
+	if err != nil {
+		t.Fatalf("parse synthetic sk pubkey: %v", err)
+	}
+	if skPub.Type() != "sk-ssh-ed25519@openssh.com" {
+		t.Fatalf("sk pubkey type = %q", skPub.Type())
+	}
+
+	sign := func(signed []byte) ssh.Signature {
+		const flagUserPresence = byte(0x01) // required, else Verify rejects
+		const counter = uint32(1)
+		appDigest := sha256.Sum256([]byte(application))
+		dataDigest := sha256.Sum256(signed)
+		// original = appDigest ‖ flags ‖ counter(BE) ‖ dataDigest (x/crypto layout)
+		var original []byte
+		original = append(original, appDigest[:]...)
+		original = append(original, flagUserPresence)
+		original = binary.BigEndian.AppendUint32(original, counter)
+		original = append(original, dataDigest[:]...)
+		edSig := ed25519.Sign(edPriv, original)
+		// sig.Rest = skFields{Flags, Counter} = flags ‖ counter(BE)
+		rest := append([]byte{flagUserPresence}, binary.BigEndian.AppendUint32(nil, counter)...)
+		return ssh.Signature{Format: "sk-ssh-ed25519@openssh.com", Blob: edSig, Rest: rest}
+	}
+	return skPub, sign
+}

internal/authz/noncestore.go

@@ -0,0 +1,195 @@
+package authz
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sync"
+	"time"
+)
+
+// MemoryNonceStore is a non-durable NonceStore for tests. Replay protection does
+// NOT survive process restart — never use it on a real host.
+type MemoryNonceStore struct {
+	mu   sync.Mutex
+	seen map[string]time.Time
+}
+
+// NewMemoryNonceStore builds an empty in-memory store.
+func NewMemoryNonceStore() *MemoryNonceStore {
+	return &MemoryNonceStore{seen: make(map[string]time.Time)}
+}
+
+// SeenOrRecord reports whether nonce was already recorded, recording it if not.
+func (m *MemoryNonceStore) SeenOrRecord(nonce string, exp time.Time) bool {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if _, ok := m.seen[nonce]; ok {
+		return true
+	}
+	m.seen[nonce] = exp
+	return false
+}
+
+// FileNonceStore is the durable, crash-safe NonceStore for the host. Mechanism:
+// an fsync'd append-only JSONL log with an in-memory index, periodic compaction,
+// and expiry-only pruning.
+//
+// Durability guarantee: a nonce is on disk AND fsync'd before SeenOrRecord returns
+// false, so the caller acting on a verified op always does so AFTER the durable
+// record. A crash between verify and execute therefore drops the op (fail-safe
+// direction) and never enables a replay. Replay protection survives restarts: the
+// log is replayed into the index on Open.
+//
+// Pruning: a nonce is dropped only after its exp (compaction), never before —
+// pruning before expiry would reopen the replay window. (An expired nonce can't be
+// replayed anyway: the time-window check rejects an expired op before the nonce
+// check, so pruning is housekeeping, not an authz hole.)
+//
+// Concurrency: a single mutex guards the file handle and index (single-process; the
+// agent is concurrent — 03 §10).
+type FileNonceStore struct {
+	mu           sync.Mutex
+	path         string
+	f            *os.File
+	idx          map[string]time.Time
+	sinceCompact int
+	now          func() time.Time
+
+	// CompactEvery is the append count that triggers a compaction (default 1000).
+	CompactEvery int
+}
+
+type nonceRecord struct {
+	Nonce string    `json:"n"`
+	Exp   time.Time `json:"e"`
+}
+
+// OpenFileNonceStore opens (or creates) the durable store at path, replaying any
+// existing log into the index.
+func OpenFileNonceStore(path string) (*FileNonceStore, error) {
+	s := &FileNonceStore{
+		path:         path,
+		idx:          make(map[string]time.Time),
+		now:          func() time.Time { return time.Now().UTC() },
+		CompactEvery: 1000,
+	}
+	if err := s.load(); err != nil {
+		return nil, err
+	}
+	f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+	if err != nil {
+		return nil, err
+	}
+	s.f = f
+	syncDir(filepath.Dir(path)) // make a freshly-created file's dir entry durable
+	return s, nil
+}
+
+func (s *FileNonceStore) load() error {
+	b, err := os.ReadFile(s.path)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil
+	}
+	if err != nil {
+		return err
+	}
+	for _, line := range bytes.Split(b, []byte("\n")) {
+		line = bytes.TrimSpace(line)
+		if len(line) == 0 {
+			continue
+		}
+		var r nonceRecord
+		if json.Unmarshal(line, &r) != nil {
+			continue // skip a torn trailing line from a crash mid-append
+		}
+		s.idx[r.Nonce] = r.Exp
+	}
+	return nil
+}
+
+// SeenOrRecord durably records an unseen nonce before returning false. On any I/O
+// failure it returns true (fail-safe: the op is NOT executed rather than risk an
+// unrecorded nonce enabling a later replay).
+func (s *FileNonceStore) SeenOrRecord(nonce string, exp time.Time) bool {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if _, ok := s.idx[nonce]; ok {
+		return true
+	}
+	rec, _ := json.Marshal(nonceRecord{Nonce: nonce, Exp: exp})
+	rec = append(rec, '\n')
+	if _, err := s.f.Write(rec); err != nil {
+		return true
+	}
+	if err := s.f.Sync(); err != nil {
+		return true
+	}
+	s.idx[nonce] = exp
+	s.sinceCompact++
+	s.maybeCompact()
+	return false
+}
+
+// Close releases the file handle.
+func (s *FileNonceStore) Close() error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.f != nil {
+		return s.f.Close()
+	}
+	return nil
+}
+
+// maybeCompact rewrites the log keeping only non-expired entries once enough
+// appends have accrued. Caller holds the mutex. Compaction is housekeeping: the
+// recorded nonce is already durable, so a compaction failure never fails the op.
+func (s *FileNonceStore) maybeCompact() {
+	if s.CompactEvery <= 0 || s.sinceCompact < s.CompactEvery {
+		return
+	}
+	s.sinceCompact = 0
+	now := s.now()
+
+	live := make(map[string]time.Time, len(s.idx))
+	var buf bytes.Buffer
+	for n, e := range s.idx {
+		if e.Before(now) {
+			continue // prune AFTER expiry only — safe
+		}
+		live[n] = e
+		rec, _ := json.Marshal(nonceRecord{Nonce: n, Exp: e})
+		buf.Write(rec)
+		buf.WriteByte('\n')
+	}
+
+	tmp := s.path + ".tmp"
+	if err := os.WriteFile(tmp, buf.Bytes(), 0o600); err != nil {
+		return // keep using the existing handle; nonce already durable
+	}
+	if tf, err := os.OpenFile(tmp, os.O_WRONLY, 0o600); err == nil {
+		_ = tf.Sync()
+		_ = tf.Close()
+	}
+	if s.f != nil {
+		_ = s.f.Close()
+	}
+	if err := os.Rename(tmp, s.path); err != nil {
+		s.f, _ = os.OpenFile(s.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+		return
+	}
+	syncDir(filepath.Dir(s.path))
+	s.f, _ = os.OpenFile(s.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+	s.idx = live
+}
+
+// syncDir best-effort fsyncs a directory so a create/rename is durable.
+func syncDir(dir string) {
+	if d, err := os.Open(dir); err == nil {
+		_ = d.Sync()
+		_ = d.Close()
+	}
+}

internal/authz/noncestore_test.go

@@ -0,0 +1,95 @@
+package authz
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestMemoryNonceStore(t *testing.T) {
+	m := NewMemoryNonceStore()
+	exp := time.Now().Add(time.Hour)
+	if m.SeenOrRecord("a", exp) {
+		t.Fatal("first record should be unseen")
+	}
+	if !m.SeenOrRecord("a", exp) {
+		t.Fatal("second record should be seen")
+	}
+	if m.SeenOrRecord("b", exp) {
+		t.Fatal("distinct nonce should be unseen")
+	}
+}
+
+func TestFileNonceStore_RecordAndReload(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	exp := refNow.Add(time.Hour)
+
+	s1, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if s1.SeenOrRecord("dead", exp) {
+		t.Fatal("first record should be unseen")
+	}
+	if err := s1.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Reopen: the recorded nonce must still be seen (durable across restart).
+	s2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if !s2.SeenOrRecord("dead", exp) {
+		t.Fatal("nonce not durable across reopen")
+	}
+}
+
+func TestFileNonceStore_CompactionPrunesExpiredOnly(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	s, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	s.now = func() time.Time { return refNow }
+	s.CompactEvery = 2 // force a compaction after two appends
+
+	s.SeenOrRecord("expired", refNow.Add(-time.Hour)) // exp in the past
+	s.SeenOrRecord("live", refNow.Add(time.Hour))     // triggers compaction
+	if err := s.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Reopen: the live nonce survived, the expired one was pruned (housekeeping;
+	// an expired op is rejected by the time check before the nonce check anyway).
+	s2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if !s2.SeenOrRecord("live", refNow.Add(time.Hour)) {
+		t.Error("live nonce should have survived compaction")
+	}
+	if s2.SeenOrRecord("expired", refNow.Add(-time.Hour)) {
+		t.Error("expired nonce should have been pruned (was still present)")
+	}
+}
+
+func TestFileNonceStore_SkipsTornLine(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "nonces.log")
+	// a valid record line + a torn/garbage trailing line from a hypothetical crash
+	content := `{"n":"good","e":"` + refNow.Add(time.Hour).Format(time.RFC3339Nano) + `"}` + "\n" + `{"n":"tor`
+	if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	s, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatalf("open with torn line should not fail: %v", err)
+	}
+	defer s.Close()
+	if !s.SeenOrRecord("good", refNow.Add(time.Hour)) {
+		t.Error("valid record before the torn line should have loaded")
+	}
+}

internal/authz/signer_test.go

@@ -0,0 +1,36 @@
+package authz
+
+import (
+	"os"
+	"testing"
+)
+
+func TestNewAllowedSigner(t *testing.T) {
+	line, err := os.ReadFile("testdata/operator.pub")
+	if err != nil {
+		t.Fatal(err)
+	}
+	s, err := NewAllowedSigner("felhom-op-1", RoleOperational, string(line))
+	if err != nil {
+		t.Fatalf("NewAllowedSigner: %v", err)
+	}
+	if s.KeyID != "felhom-op-1" || s.Role != RoleOperational || s.PublicKey == nil {
+		t.Errorf("signer = %+v", s)
+	}
+	if s.PublicKey.Type() != "ssh-ed25519" {
+		t.Errorf("key type = %q", s.PublicKey.Type())
+	}
+}
+
+func TestNewAllowedSigner_BadRole(t *testing.T) {
+	line, _ := os.ReadFile("testdata/operator.pub")
+	if _, err := NewAllowedSigner("k", "bogus", string(line)); err == nil {
+		t.Fatal("invalid role should error")
+	}
+}
+
+func TestNewAllowedSigner_BadLine(t *testing.T) {
+	if _, err := NewAllowedSigner("k", RoleOperational, "not a key"); err == nil {
+		t.Fatal("malformed key line should error")
+	}
+}

internal/authz/sshsig.go

@@ -0,0 +1,81 @@
+package authz
+
+import (
+	"crypto/sha256"
+	"crypto/sha512"
+	"encoding/pem"
+	"fmt"
+	"hash"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// SSHSIG framing — ported verbatim-in-spirit from phase4-signing-findings.md §7.
+// The only manual work is SSHSIG *framing*; all crypto and key-type dispatch is
+// x/crypto/ssh's (pub.Verify dispatches on the key's own algorithm, which is what
+// makes the verifier key-type-agnostic — ed25519 / sk-ssh-ed25519 / rsa / ecdsa).
+// No hand-rolled crypto.
+
+const sshsigMagic = "SSHSIG"
+
+// sshsigBlob is the binary SSHSIG body (after the 6-byte magic). Field order is
+// the SSH wire order — do not reorder.
+type sshsigBlob struct {
+	Version   uint32
+	PublicKey string
+	Namespace string
+	Reserved  string
+	HashAlgo  string
+	Signature string
+}
+
+func hashByName(n string) (hash.Hash, error) {
+	switch n {
+	case "sha256":
+		return sha256.New(), nil
+	case "sha512":
+		return sha512.New(), nil
+	}
+	return nil, fmt.Errorf("%w: unsupported SSHSIG hash %q", ErrMalformed, n)
+}
+
+// parseArmoredSSHSIG decodes the `-----BEGIN SSH SIGNATURE-----` armor into the
+// SSHSIG body: pem.Decode → strip the literal 6-byte magic (not length-prefixed)
+// → ssh.Unmarshal.
+func parseArmoredSSHSIG(armored []byte) (*sshsigBlob, error) {
+	block, _ := pem.Decode(armored)
+	if block == nil || block.Type != "SSH SIGNATURE" {
+		return nil, fmt.Errorf("%w: not an SSH SIGNATURE armor", ErrMalformed)
+	}
+	if len(block.Bytes) < len(sshsigMagic) || string(block.Bytes[:len(sshsigMagic)]) != sshsigMagic {
+		return nil, fmt.Errorf("%w: missing SSHSIG magic", ErrMalformed)
+	}
+	var sb sshsigBlob
+	if err := ssh.Unmarshal(block.Bytes[len(sshsigMagic):], &sb); err != nil {
+		return nil, fmt.Errorf("%w: %v", ErrMalformed, err)
+	}
+	if sb.Version != 1 {
+		return nil, fmt.Errorf("%w: bad SSHSIG version %d", ErrMalformed, sb.Version)
+	}
+	return &sb, nil
+}
+
+// signedData recomputes the bytes the signature actually covers, per the SSHSIG
+// spec: "SSHSIG" || ssh.Marshal(namespace, reserved, hash_algorithm, H(message)),
+// where H is the named hash. The message is the RAW received blob bytes — the
+// verifier never canonicalizes (the canonical form is the signer's contract).
+func signedData(sb *sshsigBlob, msg []byte) ([]byte, error) {
+	h, err := hashByName(sb.HashAlgo)
+	if err != nil {
+		return nil, err
+	}
+	h.Write(msg)
+	md := h.Sum(nil)
+	body := ssh.Marshal(struct {
+		Namespace string
+		Reserved  string
+		HashAlgo  string
+		Hash      []byte
+	}{sb.Namespace, sb.Reserved, sb.HashAlgo, md})
+	return append([]byte(sshsigMagic), body...), nil
+}

internal/authz/testdata/op_blob.json

@@ -0,0 +1 @@
+{"expires_at":"2026-06-09T00:00:00Z","issued_at":"2026-06-08T00:00:00Z","key_id":"felhom-op-1","nonce":"a1b2c3d4e5f60718293a4b5c6d7e8f90","op":"guest_destroy","params":{"purge":true},"target":{"guest_id":"9001","host_id":"demo-felhom"}}

internal/authz/testdata/op_blob.sig

@@ -0,0 +1,6 @@
+-----BEGIN SSH SIGNATURE-----
+U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgNXOOuMvD3Fh9MJYspBRWLXyQAd
+WVeBICspeB9eL1xfIAAAAMZmVsaG9tLW9wLXYxAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz
+aC1lZDI1NTE5AAAAQG+bj+GNodNw7cfGYg3HWTDyJiu3g/5Aez1xlZQ540JGUIG9FV7vv8
+wrgN0r+rNh+ytEAM6UTOyI7g3LOjuVJgY=
+-----END SSH SIGNATURE-----

internal/authz/testdata/operator.pub

@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDVzjrjLw9xYfTCWLKQUVi18kAHVlXgSArKXgfXi9cXy felhom-op-1

internal/authz/verifier.go

@@ -0,0 +1,191 @@
+package authz
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// Namespace is the FIXED SSHSIG domain separator. It is a package constant, never
+// caller-supplied (phase4 §2.2): a signature minted for any other namespace must
+// not verify.
+const Namespace = "felhom-op-v1"
+
+// DefaultClockSkew tolerates operator/host clock drift on the not-yet-valid check
+// only (issued_at may be up to this far in the future). Expiry is NOT extended —
+// the validity window stays an honest upper bound.
+const DefaultClockSkew = 2 * time.Minute
+
+// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
+type KeyRole string
+
+const (
+	// RoleOperational signs ordinary destructive ops (the "master stamp").
+	RoleOperational KeyRole = "operational"
+	// RoleRecovery is the cold key; authorizes ONLY key-rotation/break-glass ops.
+	// Role-scoping is enforced by the consuming layer (slice 4), not here.
+	RoleRecovery KeyRole = "recovery"
+)
+
+// AllowedSigner is one pinned operator public key.
+type AllowedSigner struct {
+	KeyID     string
+	Role      KeyRole
+	PublicKey ssh.PublicKey // parsed; allow-list match is by PublicKey.Marshal()
+	Comment   string        // from the authorized_keys line, if any
+}
+
+// NewAllowedSigner parses a standard authorized_keys line ("ssh-ed25519 AAAA… [comment]"
+// or "sk-ssh-ed25519@openssh.com AAAA… …") into an AllowedSigner with the given id+role.
+func NewAllowedSigner(keyID string, role KeyRole, authorizedKeyLine string) (AllowedSigner, error) {
+	pub, comment, _, _, err := ssh.ParseAuthorizedKey([]byte(authorizedKeyLine))
+	if err != nil {
+		return AllowedSigner{}, fmt.Errorf("authz: parsing pinned key %q: %w", keyID, err)
+	}
+	if role != RoleOperational && role != RoleRecovery {
+		return AllowedSigner{}, fmt.Errorf("authz: pinned key %q has invalid role %q", keyID, role)
+	}
+	return AllowedSigner{KeyID: keyID, Role: role, PublicKey: pub, Comment: comment}, nil
+}
+
+// NonceStore records seen nonces for anti-replay. SeenOrRecord reports whether the
+// nonce was already recorded; if not, it records it (durably, in the host impl)
+// before returning false. See noncestore.go.
+type NonceStore interface {
+	SeenOrRecord(nonce string, exp time.Time) (seen bool)
+}
+
+// Verifier authenticates operator-signed destructive ops. Construct with New.
+type Verifier struct {
+	signers []AllowedSigner
+	store   NonceStore
+	hostID  string
+
+	// ClockSkew tolerance for the not-yet-valid check (default DefaultClockSkew).
+	ClockSkew time.Duration
+	// Logger, if set, emits a warning when a blob's advisory key_id disagrees with
+	// the matched signer. Never affects the verdict.
+	Logger *slog.Logger
+
+	now func() time.Time // injectable for tests
+}
+
+// New builds a Verifier over the pinned signer set, a nonce store, and this box's
+// host id. allowedSigners is a set (single signer today; quorum is just sizing).
+func New(signers []AllowedSigner, store NonceStore, hostID string) *Verifier {
+	return &Verifier{
+		signers:   signers,
+		store:     store,
+		hostID:    hostID,
+		ClockSkew: DefaultClockSkew,
+		now:       func() time.Time { return time.Now().UTC() },
+	}
+}
+
+// Verify runs the LOCKED pipeline (phase4 §4 / doc 04 §2.3) and returns the
+// authenticated op. Order is load-bearing and each post-crypto stage rejects even
+// with an otherwise-valid signature:
+//
+//	parse armor → namespace → parse pubkey → allow-list (by key MATERIAL, not
+//	key_id) → crypto verify (over the RAW received blob bytes) → parse blob →
+//	target → time window → nonce SeenOrRecord (LAST)
+//
+// The nonce is recorded last, so an invalid signature can never consume a nonce
+// (DoS / replay-priming safe). Errors wrap the typed sentinels in errors.go.
+func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
+	// 1. parse armor
+	sb, err := parseArmoredSSHSIG(sigArmored)
+	if err != nil {
+		return nil, err
+	}
+
+	// 2. namespace (fixed domain separator)
+	if sb.Namespace != Namespace {
+		return nil, fmt.Errorf("%w: got %q want %q", ErrNamespace, sb.Namespace, Namespace)
+	}
+
+	// 3. parse the embedded public key
+	pub, err := ssh.ParsePublicKey([]byte(sb.PublicKey))
+	if err != nil {
+		return nil, fmt.Errorf("%w: parsing signature public key: %v", ErrMalformed, err)
+	}
+
+	// 4. allow-list match by KEY MATERIAL (pub.Marshal equality) — NOT by key_id
+	matched, ok := v.matchSigner(pub)
+	if !ok {
+		return nil, ErrUnknownSigner
+	}
+
+	// 5. crypto verify over the RAW received bytes (never re-serialized)
+	signed, err := signedData(sb, blob)
+	if err != nil {
+		return nil, err
+	}
+	var inner ssh.Signature
+	if err := ssh.Unmarshal([]byte(sb.Signature), &inner); err != nil {
+		return nil, fmt.Errorf("%w: %v", ErrMalformed, err)
+	}
+	if err := pub.Verify(signed, &inner); err != nil { // dispatches on the key's algorithm
+		return nil, fmt.Errorf("%w: %v", ErrBadSignature, err)
+	}
+
+	// 6. parse the (now authenticated) blob bytes
+	var op OpBlob
+	if err := json.Unmarshal(blob, &op); err != nil {
+		return nil, fmt.Errorf("%w: decoding op blob: %v", ErrMalformed, err)
+	}
+
+	// 7. target binding — host must be this box. guest_id is surfaced, not matched
+	// here (the verifier doesn't enumerate guests; the caller routes by it).
+	if op.Target.HostID != v.hostID {
+		return nil, fmt.Errorf("%w: blob host_id=%q this=%q", ErrTarget, op.Target.HostID, v.hostID)
+	}
+
+	// 8. time window (clock-skew tolerance on not-yet-valid only)
+	now := v.now()
+	if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
+		return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
+	}
+	if now.After(op.ExpiresAt) {
+		return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
+	}
+
+	// 9. nonce LAST — only now is it durably recorded.
+	if v.store.SeenOrRecord(op.Nonce, op.ExpiresAt) {
+		return nil, fmt.Errorf("%w: nonce %s", ErrReplay, op.Nonce)
+	}
+
+	// advisory key_id audit (never a verdict input)
+	keyIDMatches := op.KeyID == matched.KeyID
+	if !keyIDMatches && v.Logger != nil {
+		v.Logger.Warn("authz: blob key_id does not match the matched signer (advisory)",
+			"blob_key_id", op.KeyID, "matched_signer", matched.KeyID)
+	}
+
+	return &VerifiedOp{
+		Op:                 op.Op,
+		HostID:             op.Target.HostID,
+		GuestID:            op.Target.GuestID,
+		Params:             op.Params,
+		Nonce:              op.Nonce,
+		IssuedAt:           op.IssuedAt,
+		ExpiresAt:          op.ExpiresAt,
+		KeyID:              op.KeyID,
+		Signer:             matched,
+		KeyIDMatchesSigner: keyIDMatches,
+	}, nil
+}
+
+func (v *Verifier) matchSigner(pub ssh.PublicKey) (AllowedSigner, bool) {
+	pm := pub.Marshal()
+	for _, s := range v.signers {
+		if s.PublicKey != nil && bytes.Equal(s.PublicKey.Marshal(), pm) {
+			return s, true
+		}
+	}
+	return AllowedSigner{}, false
+}

internal/authz/verify_test.go

@@ -0,0 +1,248 @@
+package authz
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"golang.org/x/crypto/ssh"
+)
+
+// fixed reference instant used across in-Go tests (deterministic time window).
+var refNow = time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC)
+
+func atRefNow(v *Verifier) *Verifier { v.now = func() time.Time { return refNow }; return v }
+
+// rejects asserts a Verify error matches the expected sentinel.
+func rejects(t *testing.T, err, want error) {
+	t.Helper()
+	if !errors.Is(err, want) {
+		t.Fatalf("want %v, got %v", want, err)
+	}
+}
+
+// signerSet builds a one-key operational allow-list around an ssh.PublicKey.
+func signerSet(pub ssh.PublicKey, keyID string) []AllowedSigner {
+	return []AllowedSigner{{KeyID: keyID, Role: RoleOperational, PublicKey: pub}}
+}
+
+// validBlob is an op blob valid at refNow.
+func validBlob(host, guest, keyID, nonce string) []byte {
+	return canonicalBlob("guest_destroy", host, guest, keyID, nonce, `{"purge":true}`,
+		refNow.Add(-time.Hour), refNow.Add(time.Hour))
+}
+
+// --- Real OpenSSH interop: committed ssh-keygen fixture ---
+
+func TestVerify_RealSSHKeygenFixture(t *testing.T) {
+	blob := readFile(t, "testdata/op_blob.json")
+	sig := readFile(t, "testdata/op_blob.sig")
+	pubLine := readFile(t, "testdata/operator.pub")
+
+	signer, err := NewAllowedSigner("felhom-op-1", RoleOperational, string(pubLine))
+	if err != nil {
+		t.Fatalf("NewAllowedSigner: %v", err)
+	}
+	v := New([]AllowedSigner{signer}, NewMemoryNonceStore(), "demo-felhom")
+	v.now = func() time.Time { return time.Date(2026, 6, 8, 12, 0, 0, 0, time.UTC) } // inside fixture window
+
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("real fixture did not verify: %v", err)
+	}
+	if op.Op != "guest_destroy" || op.HostID != "demo-felhom" || op.GuestID != "9001" {
+		t.Errorf("unexpected op: %+v", op)
+	}
+	if op.KeyID != "felhom-op-1" || !op.KeyIDMatchesSigner {
+		t.Errorf("key_id audit wrong: %q matches=%v", op.KeyID, op.KeyIDMatchesSigner)
+	}
+}
+
+// --- Happy path (in-Go ed25519) ---
+
+func TestVerify_HappyPath(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-happy-0001")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+
+	v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("Verify: %v", err)
+	}
+	if op.Op != "guest_destroy" || op.Signer.KeyID != "op" {
+		t.Errorf("op = %+v", op)
+	}
+}
+
+// --- Per-stage rejection, each with an otherwise-valid signature ---
+
+func TestVerify_RejectsPerStage(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	other, _ := newEd25519Signer(t)
+
+	t.Run("wrong namespace", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-ns-1")
+		sig := mintArmor(t, pub.Marshal(), "felhom-op-wrong", "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrNamespace)
+	})
+
+	t.Run("signer not in set", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-unk-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(other, "other"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrUnknownSigner)
+	})
+
+	t.Run("tampered blob (crypto)", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-tamper-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		tampered := append([]byte{}, blob...)
+		tampered[len(tampered)-2] = '!' // mutate inside the JSON
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(tampered, sig)
+		rejects(t, err, ErrBadSignature)
+	})
+
+	t.Run("retargeted host", func(t *testing.T) {
+		blob := validBlob("other-host", "9001", "op", "n-target-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrTarget)
+	})
+
+	t.Run("expired", func(t *testing.T) {
+		blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
+			refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrExpired)
+	})
+
+	t.Run("not yet valid", func(t *testing.T) {
+		blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-nyv-1", "{}",
+			refNow.Add(time.Hour), refNow.Add(2*time.Hour))
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrNotYetValid)
+	})
+
+	t.Run("replay", func(t *testing.T) {
+		blob := validBlob("demo-felhom", "9001", "op", "n-replay-1")
+		sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+		v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+		if _, err := v.Verify(blob, sig); err != nil {
+			t.Fatalf("first use: %v", err)
+		}
+		_, err := v.Verify(blob, sig)
+		rejects(t, err, ErrReplay)
+	})
+}
+
+// --- THE anti-replay invariant: an invalid-sig attempt must NOT burn the nonce ---
+
+func TestVerify_InvalidSigDoesNotBurnNonce(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	store := NewMemoryNonceStore()
+	const nonce = "n-not-burned-cafe"
+
+	blobV := validBlob("demo-felhom", "9001", "op", nonce)
+	validSig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blobV, sign)
+
+	// Attacker reuses the SAME nonce but a signature that fails crypto (valid key,
+	// signed over different bytes) — passes namespace + allow-list, fails at the
+	// crypto stage, which is BEFORE the nonce stage.
+	badSig := mintArmor(t, pub.Marshal(), Namespace, "sha512", []byte(`{"different":"bytes"}`), sign)
+
+	v := atRefNow(New(signerSet(pub, "op"), store, "demo-felhom"))
+	if _, err := v.Verify(blobV, badSig); !errors.Is(err, ErrBadSignature) {
+		t.Fatalf("invalid attempt: want ErrBadSignature, got %v", err)
+	}
+	// The genuine valid op with the same nonce must still succeed — proving the
+	// failed attempt did NOT burn the nonce (nonce-recorded-last).
+	if _, err := v.Verify(blobV, validSig); err != nil {
+		t.Fatalf("valid op after invalid attempt should succeed, got %v", err)
+	}
+}
+
+// --- Persistence across restart (durable nonce store) ---
+
+func TestVerify_ReplayRejectedAcrossRestart(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-persist-1")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+	path := filepath.Join(t.TempDir(), "nonces.log")
+
+	store1, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	v1 := atRefNow(New(signerSet(pub, "op"), store1, "demo-felhom"))
+	if _, err := v1.Verify(blob, sig); err != nil {
+		t.Fatalf("first use: %v", err)
+	}
+	if err := store1.Close(); err != nil {
+		t.Fatal(err)
+	}
+
+	// Fresh store + verifier over the SAME path — simulates an agent restart.
+	store2, err := OpenFileNonceStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store2.Close()
+	v2 := atRefNow(New(signerSet(pub, "op"), store2, "demo-felhom"))
+	_, err = v2.Verify(blob, sig)
+	rejects(t, err, ErrReplay)
+}
+
+// --- Key-type-agnostic: synthetic FIDO2 sk-ssh-ed25519 through the unchanged path ---
+
+func TestVerify_KeyTypeAgnostic_SK(t *testing.T) {
+	skPub, skSign := newSyntheticSKSigner(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-sk-1")
+	sig := mintArmor(t, skPub.Marshal(), Namespace, "sha512", blob, skSign)
+
+	v := atRefNow(New(signerSet(skPub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	op, err := v.Verify(blob, sig)
+	if err != nil {
+		t.Fatalf("sk verify through unchanged path failed: %v", err)
+	}
+	if op.Op != "guest_destroy" {
+		t.Errorf("op = %q", op.Op)
+	}
+}
+
+// --- Byte-exactness: a re-serialized blob is NOT re-canonicalized (fails crypto) ---
+
+func TestVerify_ByteExactNoRecanonicalization(t *testing.T) {
+	pub, sign := newEd25519Signer(t)
+	blob := validBlob("demo-felhom", "9001", "op", "n-bytes-1")
+	sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
+
+	// Same fields, different whitespace + key order — what a non-identical producer
+	// canonicalizer would emit. The verifier verifies raw bytes, so this fails crypto.
+	reserialized := []byte(`{ "op":"guest_destroy", "target":{"host_id":"demo-felhom","guest_id":"9001"}, "params":{"purge":true}, "nonce":"n-bytes-1", "issued_at":"` +
+		refNow.Add(-time.Hour).Format(time.RFC3339) + `", "expires_at":"` + refNow.Add(time.Hour).Format(time.RFC3339) + `", "key_id":"op" }`)
+
+	v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
+	_, err := v.Verify(reserialized, sig)
+	rejects(t, err, ErrBadSignature)
+}
+
+func readFile(t *testing.T, path string) []byte {
+	t.Helper()
+	b, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return b
+}

internal/backup/archive_completeness_test.go

@@ -0,0 +1,243 @@
+package backup
+
+import (
+	"bytes"
+	"context"
+	"log/slog"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// F-CRIT-2 (Campaign 8): a failed backup must not look like a fresh one.
+//
+// Every fixture below is a VERBATIM shape captured from the live PVE API on 2026-07-28
+// (`pvesh get /nodes/<node>/storage/<store>/content`), not a hand-invented struct. That matters:
+// the `unparseable` path in this package went untested for months behind a JSON shape that did not
+// match production, and the whole point of this fix is that presence != validity.
+
+// phantomEntry is the artefact a PBS daemon killed mid-upload leaves behind: listed as a restorable
+// backup, 1 byte, NEWEST, and carrying no `verification`/`encrypted`/`notes` at all because it has
+// no manifest (`index.json.blob` is absent on disk).
+func phantomEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z",
+		Content: "backup",
+		Format:  "pbs-ct",
+		Size:    1,
+		CTime:   1785216674,
+		VMID:    9201,
+	}
+}
+
+// goodPBSEntry is a real, complete offsite snapshot (demo-hp, 2026-07-28T03:40:42Z).
+func goodPBSEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "felhom-pbs:backup/ct/9201/2026-07-28T03:40:42Z",
+		Content: "backup",
+		Format:  "pbs-ct",
+		Size:    4353457559,
+		CTime:   1785210042,
+		VMID:    9201,
+	}
+}
+
+// goodLocalEntry is a real, complete LOCAL vzdump (demo-hp). Note it legitimately has no
+// `verification` and no `encrypted` on the wire — a dir storage has no such concept — which is
+// exactly why those fields must never be used as completeness discriminators.
+func goodLocalEntry() proxmox.StorageContent {
+	return proxmox.StorageContent{
+		VolID:   "local:backup/vzdump-lxc-9201-2026_07_28-07_29_54.tar.zst",
+		Content: "backup",
+		Format:  "tar.zst",
+		Size:    1590431865,
+		CTime:   1785216594,
+		VMID:    9201,
+	}
+}
+
+func runnerWithContent(t *testing.T, buf *bytes.Buffer, content []proxmox.StorageContent) *BackupRunner {
+	t.Helper()
+	lg := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
+	return NewBackupRunner(&fakeBackupAPI{content: content}, "felhom-pbs", proxmox.ModeSnapshot, "", "", lg)
+}
+
+// Group A — the phantom must NOT set tier freshness, even though it is the newest entry.
+//
+// RED-PROOF: restore the old predicate in NewestArchiveTime
+// (`if e.Content == "backup" && e.VMID == vmid && e.CTime > best`) → the phantom's ctime
+// (1785216674) wins over the good snapshot's (1785210042) and this test fails with
+// "got 1785216674, want 1785210042" — i.e. the exact F-CRIT-2 defect.
+func TestNewestArchiveTime_PhantomIsNotCounted(t *testing.T) {
+	var buf bytes.Buffer
+	// phantom deliberately listed FIRST and is also the newest by ctime.
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
+
+	got, found, err := r.NewestArchiveTime(context.Background(), 9201)
+	if err != nil {
+		t.Fatalf("NewestArchiveTime: %v", err)
+	}
+	if !found {
+		t.Fatal("found=false — the GOOD snapshot must still be counted; rejecting everything is the thrash path")
+	}
+	if got.Unix() != goodPBSEntry().CTime {
+		t.Errorf("freshness came from the wrong entry: got ctime %d, want %d (the good snapshot)", got.Unix(), goodPBSEntry().CTime)
+	}
+	if got.Unix() == phantomEntry().CTime {
+		t.Error("the 1-byte manifest-less phantom set tier freshness — this is F-CRIT-2")
+	}
+}
+
+// Group A — with ONLY a phantom present the tier must report "no backup", not a fresh one.
+// That is what lets the controller see age_state=absent and fire its first-backup valve.
+func TestNewestArchiveTime_OnlyPhantomReportsNotFound(t *testing.T) {
+	var buf bytes.Buffer
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry()})
+
+	_, found, err := r.NewestArchiveTime(context.Background(), 9201)
+	if err != nil {
+		t.Fatalf("NewestArchiveTime: %v", err)
+	}
+	if found {
+		t.Error("found=true with only a phantom present — the tier would report fresh and go silent for a full cadence")
+	}
+}
+
+// Group B — THE SCENARIO-D GUARD. A valid snapshot on EITHER tier must still be counted.
+//
+// This is what makes Group A safe. A filter that is too aggressive does not merely lose safety
+// margin: the tier reports absent on every poll, backs up every cycle, and the R-88 breaker cannot
+// save it because those backups SUCCEED. That is a continuous multi-GB write loop across the fleet.
+//
+// RED-PROOF: make archivePlausiblyComplete return `false, "reject everything"` unconditionally →
+// both subtests fail with found=false.
+func TestNewestArchiveTime_ValidSnapshotsAreStillCounted(t *testing.T) {
+	for _, tc := range []struct {
+		name  string
+		entry proxmox.StorageContent
+	}{
+		{"pbs offsite (has verification+encrypted on the wire)", goodPBSEntry()},
+		{"local dir vzdump (has NEITHER verification NOR encrypted — and must still count)", goodLocalEntry()},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			var buf bytes.Buffer
+			r := runnerWithContent(t, &buf, []proxmox.StorageContent{tc.entry})
+
+			got, found, err := r.NewestArchiveTime(context.Background(), 9201)
+			if err != nil {
+				t.Fatalf("NewestArchiveTime: %v", err)
+			}
+			if !found {
+				t.Fatalf("a REAL %s backup was rejected — this is the backup-thrash path, not extra safety", tc.name)
+			}
+			if got.Unix() != tc.entry.CTime {
+				t.Errorf("got ctime %d, want %d", got.Unix(), tc.entry.CTime)
+			}
+			if strings.Contains(buf.String(), "INCOMPLETE archive") {
+				t.Errorf("a valid archive was announced as incomplete:\n%s", buf.String())
+			}
+		})
+	}
+}
+
+// Group B — the smallest REAL backup measured anywhere on the fleet (612,397,450 B, a guest-9100
+// vzdump) must clear the floor with room to spare. If someone ever raises
+// minPlausibleArchiveBytes past this, that is the fleet-thrash bug and this test is the tripwire.
+func TestMinPlausibleArchiveBytes_LeavesHeadroomBelowTheSmallestRealBackup(t *testing.T) {
+	const smallestObservedRealBackup int64 = 612397450 // fleet survey 2026-07-28
+	if minPlausibleArchiveBytes >= smallestObservedRealBackup {
+		t.Fatalf("floor %d B is not below the smallest real backup ever observed (%d B) — this WILL reject real archives",
+			minPlausibleArchiveBytes, smallestObservedRealBackup)
+	}
+	if ratio := smallestObservedRealBackup / minPlausibleArchiveBytes; ratio < 100 {
+		t.Errorf("floor %d B leaves only %dx headroom below the smallest real backup (%d B) — too tight",
+			minPlausibleArchiveBytes, ratio, smallestObservedRealBackup)
+	}
+}
+
+// Group C — UNDECIDABLE ⇒ NOT COUNTED (the fail-safe direction).
+//
+// A zero/absent size is not evidence of a good backup; it is absence of evidence. Erring toward
+// "not fresh" costs one extra backup. Erring the other way is F-CRIT-2.
+//
+// RED-PROOF: flip the comparison in archivePlausiblyComplete to `e.Size > minPlausibleArchiveBytes
+// || e.Size == 0` (i.e. treat unknown as complete) → the size-0 case reports ok=true and this fails.
+func TestArchivePlausiblyComplete_UndecidableIsNotCounted(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		size int64
+	}{
+		{"the observed phantom", 1},
+		{"absent size field (unmarshals to 0)", 0},
+		{"just under the floor", minPlausibleArchiveBytes - 1},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			e := phantomEntry()
+			e.Size = tc.size
+			ok, why := archivePlausiblyComplete(e)
+			if ok {
+				t.Errorf("size %d counted as a complete backup — undecidable must fail safe", tc.size)
+			}
+			if why == "" {
+				t.Error("rejection carried no reason — a silent rejection is a new quiet path")
+			}
+		})
+	}
+	if ok, why := archivePlausiblyComplete(goodPBSEntry()); !ok {
+		t.Errorf("a real snapshot was rejected: %s", why)
+	}
+}
+
+// Group D — the rejection is announced ONCE per snapshot, not once per due-check.
+//
+// The due-check runs every 5 minutes and a phantom persists indefinitely (server-side prune does
+// not collect it), so per-poll logging would emit ~288 identical lines a day and bury the signal.
+//
+// RED-PROOF: delete the `if seen { return }` guard in warnRejectedArchiveOnce → this test reports
+// "logged 5 times, want 1".
+func TestNewestArchiveTime_RejectionLoggedOncePerSnapshot(t *testing.T) {
+	var buf bytes.Buffer
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
+
+	const polls = 5
+	for i := 0; i < polls; i++ {
+		if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
+			t.Fatalf("poll %d: %v", i, err)
+		}
+	}
+
+	n := strings.Count(buf.String(), "INCOMPLETE archive")
+	if n != 1 {
+		t.Errorf("rejection logged %d times across %d polls, want exactly 1:\n%s", n, polls, buf.String())
+	}
+	out := buf.String()
+	if !strings.Contains(out, phantomEntry().VolID) {
+		t.Errorf("the log line does not NAME the rejected snapshot:\n%s", out)
+	}
+	if !strings.Contains(out, "below the") {
+		t.Errorf("the log line does not say WHY it was rejected:\n%s", out)
+	}
+	if !strings.Contains(out, "level=WARN") {
+		t.Errorf("rejection was not logged at WARN:\n%s", out)
+	}
+}
+
+// Group D — a SECOND, distinct phantom is announced separately. The dedupe must be per snapshot,
+// not a one-shot latch that hides every later phantom.
+func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
+	var buf bytes.Buffer
+	second := phantomEntry()
+	second.VolID = "felhom-pbs:backup/ct/9201/2026-07-29T05:31:14Z"
+	second.CTime = phantomEntry().CTime + 86400
+	r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), second, goodPBSEntry()})
+
+	for i := 0; i < 3; i++ {
+		if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
+			t.Fatalf("poll %d: %v", i, err)
+		}
+	}
+	if n := strings.Count(buf.String(), "INCOMPLETE archive"); n != 2 {
+		t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
+	}
+}

internal/backup/backup_test.go

@@ -0,0 +1,253 @@
+package backup
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"log/slog"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// fakeBackupAPI is a synthetic BackupAPI.
+type fakeBackupAPI struct {
+	vzdumpUPID string
+	vzdumpErr  error
+	waitErr    error
+	cfg        proxmox.GuestConfig
+	cfgErr     error
+	content    []proxmox.StorageContent
+	contentErr error
+	storages   []proxmox.Storage // returned by ListStorage (the local-prune scope gate)
+	storageErr error
+	vzdumps    []proxmox.VzdumpOptions
+	logLines   []string      // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
+	waitGate   chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
+}
+
+func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) {
+	f.vzdumps = append(f.vzdumps, o)
+	return f.vzdumpUPID, f.vzdumpErr
+}
+func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOptions) (proxmox.TaskStatus, error) {
+	if f.waitGate != nil {
+		<-f.waitGate
+	}
+	return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
+}
+func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
+	return f.cfg, f.cfgErr
+}
+func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
+	return f.content, f.contentErr
+}
+func (f *fakeBackupAPI) ListStorage(_ context.Context) ([]proxmox.Storage, error) {
+	return f.storages, f.storageErr
+}
+func (f *fakeBackupAPI) TaskLogTail(_ context.Context, _ string, _ int) ([]string, error) {
+	return f.logLines, nil
+}
+
+// guestCfgWithMounts builds a GuestConfig whose Extra carries the given mpN strings.
+func guestCfgWithMounts(mps map[string]string) proxmox.GuestConfig {
+	extra := map[string]json.RawMessage{}
+	for k, v := range mps {
+		b, _ := json.Marshal(v)
+		extra[k] = b
+	}
+	return proxmox.GuestConfig{Extra: extra}
+}
+
+func TestBackup_SuccessResolvesArchiveAndBulkGap(t *testing.T) {
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:vzdump:1",
+		cfg: guestCfgWithMounts(map[string]string{
+			"mp0": "local-lvm:8,mp=/mnt/bulk,backup=0", // explicit opt-out → uncovered
+			"mp1": "local-lvm:4,mp=/mnt/db,backup=1",   // covered
+			"mp2": "local-lvm:2,mp=/mnt/scratch",       // UNSET → uncovered (opt-in default)
+		}),
+		content: []proxmox.StorageContent{
+			{VolID: "local:backup/old-9001.tar.zst", Content: "backup", VMID: 9001, Size: 100, CTime: 100},
+			{VolID: "local:backup/new-9001.tar.zst", Content: "backup", VMID: 9001, Size: 524288000, CTime: 200},
+			{VolID: "local:backup/other-9002.tar.zst", Content: "backup", VMID: 9002, Size: 7, CTime: 999},
+		},
+	}
+	r := NewBackupRunner(api, "local", "", "felhom test", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err != nil {
+		t.Fatalf("Backup: %v", err)
+	}
+	if !rec.Success || !rec.CrashConsistent {
+		t.Errorf("record = %+v, want success + crash_consistent", rec)
+	}
+	if rec.Archive != "local:backup/new-9001.tar.zst" || rec.SizeBytes != 524288000 {
+		t.Errorf("resolved wrong archive/size: %+v", rec)
+	}
+	if rec.Mode != string(proxmox.ModeSnapshot) {
+		t.Errorf("mode = %q, want snapshot (default)", rec.Mode)
+	}
+	// Bulk gap: mp0 (backup=0) AND mp2 (unset) are uncovered; mp1 (backup=1) is NOT.
+	if got := rec.UncoveredVolumes; len(got) != 2 || !has(got, "/mnt/bulk") || !has(got, "/mnt/scratch") {
+		t.Errorf("uncovered = %v, want [/mnt/bulk /mnt/scratch] (unset is uncovered too)", got)
+	}
+	if has(rec.UncoveredVolumes, "/mnt/db") {
+		t.Error("backup=1 mountpoint must NOT be reported uncovered")
+	}
+	// Notes-template threaded through.
+	if len(api.vzdumps) != 1 || api.vzdumps[0].Notes != "felhom test" {
+		t.Errorf("vzdump opts = %+v", api.vzdumps)
+	}
+}
+
+func TestBackup_ReportsActualModeFromTaskLog(t *testing.T) {
+	// Requested snapshot, but PVE used stop (stopped guest) — the report must reflect ACTUAL.
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:vzdump:1",
+		content:    []proxmox.StorageContent{{VolID: "v", Content: "backup", VMID: 9001, Size: 10, CTime: 1}},
+		logLines:   []string{"INFO: CT Name: spike", "INFO: backup mode: stop", "INFO: Finished"},
+	}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if rec.Mode != "stop" {
+		t.Errorf("mode = %q, want the ACTUAL %q from the task log (not the requested snapshot)", rec.Mode, "stop")
+	}
+}
+
+func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
+	api := &fakeBackupAPI{vzdumpErr: errors.New("vzdump boom")}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	rec, err := r.Backup(context.Background(), 9001)
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if rec.Success || rec.Error == "" {
+		t.Errorf("failed backup must produce a Success=false record with an Error: %+v", rec)
+	}
+}
+
+func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
+	const big = 4 << 30 // a plausible whole-guest archive
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		{VolID: "a", Content: "backup", CTime: 10, Size: big},
+		{VolID: "b", Content: "backup", CTime: 99, Size: big},
+		{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
+	}}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	vol, err := r.PickRestoreCandidate(context.Background())
+	if err != nil || vol != "b" {
+		t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
+	}
+	// no backups → "".
+	api.content = []proxmox.StorageContent{{VolID: "iso", Content: "iso"}}
+	if vol, _ := r.PickRestoreCandidate(context.Background()); vol != "" {
+		t.Errorf("no backup → empty, got %q", vol)
+	}
+}
+
+// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
+// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
+// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
+// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
+//
+// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
+// PickSettledRestoreCandidateOn and this fails with
+// `pick = "phantom" want the newest COMPLETE archive 'real'`.
+func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
+		{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
+	}}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+	vol, err := r.PickRestoreCandidate(context.Background())
+	if err != nil || vol != "real" {
+		t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
+	}
+}
+
+// --- scheduler ---
+
+type fakeRTRunner struct {
+	res  reconcile.RestoreTestResult
+	runs int
+}
+
+func (f *fakeRTRunner) RunRestoreTest(_ context.Context, _ reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
+	f.runs++
+	return f.res
+}
+
+func TestScheduler_TickRunsAndRecords(t *testing.T) {
+	store := NewStore()
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick:   func(context.Context) (string, error) { return "vol", nil },
+		Store:  store,
+		Spec: func(context.Context, string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence: time.Hour,
+		Logger:  quiet(),
+	})
+	s.tick(context.Background())
+	if rt.runs != 1 {
+		t.Fatalf("tick should run the restore-test once, got %d", rt.runs)
+	}
+	got := store.RestoreTests(context.Background())
+	if len(got) != 1 || !got[0].Pass || got[0].SourceArchive != "vol" {
+		t.Fatalf("store should have the recorded result: %+v", got)
+	}
+}
+
+func TestScheduler_TickNoBackupNoOp(t *testing.T) {
+	rt := &fakeRTRunner{}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt, Pick: func(context.Context) (string, error) { return "", nil },
+		Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background())
+	if rt.runs != 0 {
+		t.Errorf("no backup available → no restore-test run, got %d", rt.runs)
+	}
+}
+
+func TestScheduler_DisabledRunReturnsOnCancel(t *testing.T) {
+	s := NewScheduler(SchedulerOptions{Cadence: 0, Logger: quiet()})
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- s.Run(ctx) }()
+	cancel()
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatalf("disabled scheduler Run should return nil on cancel, got %v", err)
+		}
+	case <-time.After(time.Second):
+		t.Fatal("disabled scheduler did not return on cancel")
+	}
+}
+
+func TestStore_Reporters(t *testing.T) {
+	s := NewStore()
+	if len(s.Backups(context.Background())) != 0 || len(s.RestoreTests(context.Background())) != 0 {
+		t.Fatal("empty store must report empty (non-nil) slices")
+	}
+}
+
+func has(ss []string, want string) bool {
+	for _, s := range ss {
+		if s == want {
+			return true
+		}
+	}
+	return false
+}

internal/backup/doc.go

@@ -0,0 +1,23 @@
+// Package backup is the guest-level backup + self-restore-test layer (doc 03 §8, slice 6
+// Phase A). It orchestrates a crash-consistent vzdump to a LOCAL target, resolves the
+// produced archive, and drives the self-restore-test (restore → boot → verify → teardown)
+// through the reconcile engine so it inherits the journal / per-guest serialization /
+// crash-safe recovery.
+//
+// Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): it reuses the
+// slice-4 classifier/gate/journal via reconcile — no new destructive class, no new crypto.
+// Restore is to a NEW guest only (no overwrite this slice). PBS / offsite / zero-knowledge
+// is Phase B.
+//
+// Layout:
+//   - runner.go   — BackupRunner: vzdump + archive-volid/size resolve + the bulk-volume gap;
+//     restore-candidate picker.
+//   - store.go    — in-memory latest-backup-per-target + latest-restore-test, implementing
+//     the hub BackupReporter / RestoreTestReporter seams (point-in-time state
+//     the collector reads; re-populated each cadence/selftest run).
+//   - schedule.go — the restore-test cadence goroutine (default 24h; disabled when 0).
+//
+// hub does NOT import this package (the report types live in hub; this package imports hub
+// for them, mirroring the slice-5 storage seam). This package may import reconcile + hub +
+// proxmox (acyclic).
+package backup

internal/backup/inflight.go

@@ -0,0 +1,69 @@
+package backup
+
+import "sync"
+
+// InFlight is the host-wide "one heavy guest operation at a time" gate.
+//
+// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
+// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
+// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
+// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
+// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
+// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
+// up recorded as failed.
+//
+// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
+// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
+//
+// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
+// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
+// coverage; cancelling a running backup costs the backup.
+//
+// CORRECTED 2026-07-28 (F-A1). That "DEFERS" was true of the restore-test caller and NOT of the
+// backup caller, and the comment did not say so. The controller's start path had no 409 branch, so
+// a refusal here was recorded as a tier FAILURE: the R-88 breaker armed and the operator was
+// emailed "Whole-guest backup FAILED" about a backup that was merely waiting its turn. Campaign 8
+// observed it on both demo boxes in the same minute.
+//
+// Fixed on the CONTROLLER side (v0.179.0), which is where the misreading lived — this gate's
+// behaviour was correct throughout and is unchanged. The controller now maps HTTP 409 to a
+// contention path: it defers the tier, keeps it DUE, and alarms only if contention outlives the
+// agent's own restore-test ceiling. Nothing here needs to change; the claim above is simply now
+// true of both callers.
+type InFlight struct {
+	mu   sync.Mutex
+	what string // "" = idle
+}
+
+// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
+// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
+func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
+	if g == nil {
+		// Not wired (older call sites, tests) → no gating, previous behaviour.
+		return func() {}, "", true
+	}
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.what != "" {
+		return nil, g.what, false
+	}
+	g.what = what
+	var once sync.Once
+	return func() {
+		once.Do(func() {
+			g.mu.Lock()
+			g.what = ""
+			g.mu.Unlock()
+		})
+	}, "", true
+}
+
+// Busy reports what currently holds the gate ("" = idle).
+func (g *InFlight) Busy() string {
+	if g == nil {
+		return ""
+	}
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	return g.what
+}

internal/backup/prune_test.go

@@ -0,0 +1,99 @@
+package backup
+
+import (
+	"context"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/config"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// localTarget is a non-PBS dir storage; pbsTarget is a PBS storage — for the scope gate.
+var (
+	localTargetStores = []proxmox.Storage{{Storage: "local", Type: "dir", Content: "backup"}}
+	pbsTargetStores   = []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", Content: "backup"}}
+)
+
+func okAPI(stores []proxmox.Storage) *fakeBackupAPI {
+	return &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: stores}
+}
+
+// TestPrune_LocalCarriesKeepLast: a LOCAL-target backup with retention carries `--prune-backups
+// keep-last=3` on the vzdump. Companion: the SAME runner built with no retention ("") emits NO prune
+// option — proving the flag only rides when retention is set (a no-prune build accumulates).
+func TestPrune_LocalCarriesKeepLast(t *testing.T) {
+	api := okAPI(localTargetStores)
+	r := NewBackupRunner(api, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201) // archive-resolution may fail in the fake; we assert the captured vzdump opts
+	if len(api.vzdumps) != 1 || api.vzdumps[0].PruneBackups != "keep-last=3" {
+		t.Fatalf("local backup must carry prune-backups keep-last=3, got %q", api.vzdumps[0].PruneBackups)
+	}
+
+	// COMPANION: no retention → no prune option (dumps would accumulate).
+	api2 := okAPI(localTargetStores)
+	r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "", quiet())
+	_, _ = r2.Backup(context.Background(), 9201)
+	if api2.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("a no-retention runner must NOT prune, got %q", api2.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_NeverPrunesPBS is the scope rule (§9): retention is NOT applied when the target is a PBS
+// storage (offsite retention is a separate lifecycle). Companion: the identical retention on a LOCAL
+// target IS applied — proving the gate keys on storage type, not luck.
+func TestPrune_NeverPrunesPBS(t *testing.T) {
+	api := okAPI(pbsTargetStores)
+	r := NewBackupRunner(api, "felhom-pbs", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if api.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("a PBS target must NEVER be pruned by the per-run flag, got %q", api.vzdumps[0].PruneBackups)
+	}
+
+	// COMPANION: same retention, local target → applied.
+	api2 := okAPI(localTargetStores)
+	r2 := NewBackupRunner(api2, "local", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r2.Backup(context.Background(), 9201)
+	if api2.vzdumps[0].PruneBackups != "keep-last=3" {
+		t.Fatalf("control: a local target with the same retention MUST be pruned, got %q", api2.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_FailSafeOnUnknownTarget: if the target's type can't be confirmed (lookup error / not in the
+// list), the run SKIPS pruning rather than risk pruning a PBS/unknown storage.
+func TestPrune_FailSafeOnUnknownTarget(t *testing.T) {
+	// target not present in the list → skip.
+	api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1", storages: localTargetStores}
+	r := NewBackupRunner(api, "some-other-store", proxmox.ModeStop, "", "keep-last=3", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if api.vzdumps[0].PruneBackups != "" {
+		t.Fatalf("an unknown target must skip pruning (fail-safe), got %q", api.vzdumps[0].PruneBackups)
+	}
+}
+
+// TestPrune_KeepLastClamp (§7-B): a 0/negative/unset LocalBackupRetention clamps to ≥1 (default 3) so the
+// vzdump NEVER prunes the archive it just made. Companion: a no-clamp impl that returns 0 would emit
+// keep-last=0 → PVE prunes everything → FAILS the "≥1" assertion.
+func TestPrune_KeepLastClamp(t *testing.T) {
+	cases := []struct {
+		set  int
+		want int
+	}{
+		{0, 3},   // unset → default
+		{-5, 3},  // negative → default
+		{1, 1},   // honored
+		{3, 3},   // honored
+		{10, 10}, // honored
+	}
+	for _, c := range cases {
+		b := config.BackupConfig{LocalBackupRetention: c.set}
+		if got := b.KeepLast(); got != c.want {
+			t.Errorf("KeepLast(%d) = %d, want %d", c.set, got, c.want)
+		}
+		if b.KeepLast() < 1 {
+			t.Fatalf("keep-last must NEVER be < 1 (would prune the fresh backup), got %d for %d", b.KeepLast(), c.set)
+		}
+	}
+	if spec := (config.BackupConfig{}).PruneBackupsSpec(); spec != "keep-last=3" {
+		t.Fatalf("default PruneBackupsSpec = %q, want keep-last=3", spec)
+	}
+}

internal/backup/restoretest_due.go

@@ -0,0 +1,168 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"time"
+)
+
+// R-86 — a restore-test follows the BACKUP, not the clock.
+//
+// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
+//
+// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
+// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
+// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
+// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
+// sometimes twice on the same archive.
+//
+// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
+//
+// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
+// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
+// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
+// The naive rule silently switches restore-testing off for the tier that matters most, and it is
+// the version a reasonable person would write. It has a red-proof of its own
+// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
+//
+// The rule implemented here:
+//
+//	Let A = the newest archive on this tier that is at least `settle` old.
+//	The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
+//
+//	daily tier    → A is yesterday's archive; a new one settles each day  → proved once per day
+//	weekly tier   → A is last week's until the next settles               → proved once per week
+//	newborn tier  → A does not exist                                      → UNKNOWN, never a fault
+//
+// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
+// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
+// produce a cadence nobody can predict from either.
+//
+// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
+//
+// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
+// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
+// the trigger changed.
+
+// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
+// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
+type DueVerdict struct {
+	Target string // the tier's storage target id
+
+	// Due is true only when Archive is set and has not been proven.
+	Due bool
+	// Archive is the settled candidate A ("" when the tier holds none).
+	Archive string
+	// Landed is when A landed on the tier (zero when Archive is "").
+	Landed time.Time
+	// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
+	ProvenArchive string
+	// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
+	// NEVER reported as "not due", which would silently retire a tier the moment its storage
+	// stopped answering. Due stays false (we have no archive to test) and the error travels.
+	Err error
+	// Reason is the one-line human account of this verdict.
+	Reason string
+}
+
+// String renders a verdict for the operator log / selftest output.
+func (v DueVerdict) String() string {
+	return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
+}
+
+// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
+//
+// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
+// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
+// still never be starved — a tier that has waited longest is served first — and keeping it as the
+// order rather than as the trigger is the whole of this change.
+func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
+	if !s.rotating() {
+		return nil
+	}
+	order := s.tiers
+	if s.rtState != nil {
+		order = s.rtState.OldestFirst(s.tiers)
+	}
+	cutoff := s.settleCutoff()
+	out := make([]DueVerdict, 0, len(order))
+	for _, target := range order {
+		out = append(out, s.evaluateTier(ctx, target, cutoff))
+	}
+	return out
+}
+
+// settleCutoff is the newest landing time an archive may have and still count as settled.
+func (s *Scheduler) settleCutoff() time.Time {
+	if s.settle <= 0 {
+		return time.Time{} // no settle requirement configured → any archive is a candidate
+	}
+	return s.now().Add(-s.settle)
+}
+
+// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
+// unit-tested directly rather than inferred from whether a fake runner happened to be called.
+func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
+	v := DueVerdict{Target: target}
+	archive, landed, err := s.tierPick(ctx, target, cutoff)
+	if err != nil {
+		// UNKNOWN, never "not due", and never silent.
+		v.Err = err
+		v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
+		return v
+	}
+	v.Archive, v.Landed = archive, landed
+	if archive == "" {
+		v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
+		return v
+	}
+	proven, ok := "", false
+	if s.rtState != nil {
+		proven, ok = s.rtState.ProvenArchive(target)
+	}
+	v.ProvenArchive = proven
+	if ok && proven == archive {
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
+		return v
+	}
+	v.Due = true
+	switch {
+	case !ok && proven == "":
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
+	default:
+		v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
+	}
+	return v
+}
+
+// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
+// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
+func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
+	return s.evaluateTier(ctx, target, s.settleCutoff())
+}
+
+// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
+//
+// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
+// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
+// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
+// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
+// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
+func (s *Scheduler) verdictSummary(ctx context.Context) string {
+	out := ""
+	for _, v := range s.EvaluateDue(ctx) {
+		if out != "" {
+			out += "; "
+		}
+		switch {
+		case v.Err != nil:
+			out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
+		default:
+			out += v.Target + ": " + v.Reason
+		}
+	}
+	if out == "" {
+		return "no tiers configured"
+	}
+	return out
+}

internal/backup/restoretest_due_test.go

@@ -0,0 +1,593 @@
+package backup
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-86 — the restore-test follows the BACKUP, not the clock.
+//
+// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
+// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
+// asserted "no error" would pass against a scheduler that never ran anything at all — which is
+// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
+// archive, and did a second evaluation correctly run NOTHING.
+
+// ── the fake tier storage ────────────────────────────────────────────────────────────────────
+
+// archiveStub is one archive on a tier: its volid and when it landed.
+type archiveStub struct {
+	volid  string
+	landed time.Time
+}
+
+// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
+// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
+// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
+// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
+// picker; that the real picker obeys the same rule is asserted there, not here.
+type tierStorage struct {
+	archives map[string][]archiveStub
+	err      map[string]error // target → lookup failure
+}
+
+func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
+	if e, ok := ts.err[target]; ok && e != nil {
+		return "", time.Time{}, e
+	}
+	var best archiveStub
+	for _, a := range ts.archives[target] {
+		if !notAfter.IsZero() && a.landed.After(notAfter) {
+			continue // not settled yet
+		}
+		if best.volid == "" || a.landed.After(best.landed) {
+			best = a
+		}
+	}
+	return best.volid, best.landed, nil
+}
+
+// dueHarness is a scheduler with a driven clock over a fake tier storage.
+type dueHarness struct {
+	s     *Scheduler
+	rr    *rotRunner
+	st    *RestoreTestState
+	ts    *tierStorage
+	clock time.Time
+	path  string
+}
+
+func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
+	t.Helper()
+	return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
+}
+
+func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
+	t.Helper()
+	h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
+	h.st = NewRestoreTestState(statePath)
+	h.s = NewScheduler(SchedulerOptions{
+		Runner: h.rr,
+		Store:  NewStore(),
+		Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence:  time.Hour,
+		Settle:   settle,
+		Logger:   quiet(),
+		Tiers:    tiers,
+		TierPick: ts.pick,
+		State:    h.st,
+		InFlight: &InFlight{},
+	})
+	h.s.now = func() time.Time { return h.clock }
+	return h
+}
+
+// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
+func (h *dueHarness) advance(step, total time.Duration) {
+	for elapsed := time.Duration(0); elapsed < total; elapsed += step {
+		h.clock = h.clock.Add(step)
+		h.s.tick(context.Background())
+	}
+}
+
+var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
+
+// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
+func dailyArchives(tier string, n int) []archiveStub {
+	out := make([]archiveStub, 0, n)
+	for d := 0; d < n; d++ {
+		out = append(out, archiveStub{
+			volid:  fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
+			landed: day0.AddDate(0, 0, d),
+		})
+	}
+	return out
+}
+
+// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
+//
+// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
+// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
+// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
+// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
+// the tier that matters most.
+//
+// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
+// was replaced by the naive age rule:
+//
+//   - if ok && proven == archive { … not due … }
+//   - if s.now().Sub(landed) < s.settle { … not due … }   // and the proven-archive check deleted
+//
+// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
+// old enough". Result:
+//
+//	--- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
+//	    restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
+//
+// Zero runs — restore-testing off. Restored immediately afterwards.
+func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
+	h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	// Five days, evaluated hourly.
+	h.advance(time.Hour, 5*24*time.Hour)
+
+	got := h.rr.seen()
+	if len(got) != 5 {
+		t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
+	}
+	// And each run must be on the archive that settled that day — day0's on day 1, and so on.
+	for i, a := range got {
+		want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
+		if a != want {
+			t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
+		}
+	}
+	// The newest archive is NEVER the one tested: it has not settled.
+	if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
+		t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
+	}
+}
+
+// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
+func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
+		{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
+		{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
+		{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
+	}}}
+	h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+
+	// Three weeks, evaluated every 6 hours — 84 evaluations.
+	h.advance(6*time.Hour, 21*24*time.Hour)
+
+	got := h.rr.seen()
+	want := []string{
+		"felhom-pbs:backup/ct/9201/w0",
+		"felhom-pbs:backup/ct/9201/w1",
+		"felhom-pbs:backup/ct/9201/w2",
+	}
+	if len(got) != len(want) {
+		t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
+		}
+	}
+}
+
+// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
+//
+// This is the defect a person actually notices: today every deploy restarts the ticker, so a
+// restore-test runs one interval after each deploy regardless of what has already been proven.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
+// ProvenArchive ignore the stored archive —
+//
+//   - if !ok || p.Archive == "" { return "", false }
+//   - return "", false     // per-tier time only, the pre-R-86 state
+//
+// →	--- FAIL: TestDue_RestartRunsNothing
+//
+//	restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
+//	produced 4 run(s)
+//
+// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
+// today's behaviour with the ticker's phase reset by the deploy. Restored.
+func TestDue_RestartRunsNothing(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
+	start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
+
+	h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
+	}
+
+	// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
+	total := 0
+	for i := 0; i < 2; i++ {
+		h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+		h2.s.tick(context.Background())
+		h2.s.tick(context.Background())
+		total += len(h2.rr.seen())
+	}
+	if total != 0 {
+		t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
+	}
+}
+
+// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
+func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	h.s.tick(context.Background()) // proves day0's archive
+	h.s.tick(context.Background()) // nothing new has settled → nothing
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
+	}
+
+	// Day 2, 03:00 — day1's archive has now settled.
+	h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
+	h.s.tick(context.Background())
+
+	got := h.rr.seen()
+	if len(got) != 2 {
+		t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
+	}
+	if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
+		t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
+	}
+}
+
+// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
+//
+// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
+//
+//   - if rt.Pass && s.rtState != nil && target != "" {
+//   - if s.rtState != nil && target != "" {
+//
+// →	--- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
+//
+//	restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
+//	evaluations
+//
+// A single failure would have retired the archive as proven — a permanently broken DR tier looking
+// freshly verified, which is the loudest signal this system produces going silent. Restored.
+func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
+
+	for i := 0; i < 3; i++ {
+		h.s.tick(context.Background())
+	}
+
+	got := h.rr.seen()
+	if len(got) != 3 {
+		t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
+	}
+	if _, ok := h.st.ProvenArchive("local"); ok {
+		t.Fatal("a FAILED restore-test must not record the archive as proven")
+	}
+	if _, ok := h.st.LastSuccess("local"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+}
+
+// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
+func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{
+		"local":      {{volid: "local:backup/a.tar.zst", landed: day0}},
+		"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
+	}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+
+	// Both tiers are due at this instant.
+	due := h.s.EvaluateDue(context.Background())
+	if len(due) != 2 || !due[0].Due || !due[1].Due {
+		t.Fatalf("precondition: both tiers should be due; got %v", due)
+	}
+
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
+	}
+
+	// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
+	h.s.tick(context.Background())
+	got := h.rr.seen()
+	if len(got) != 2 || got[0] == got[1] {
+		t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
+	}
+}
+
+// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
+func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	gate := &InFlight{}
+	h.s.inFlight = gate
+	release, _, _ := gate.TryAcquire("backup:felhom-pbs")
+
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
+	}
+	if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
+		t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
+	}
+	release()
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 1 {
+		t.Fatalf("must resume once the gate frees; got %d run(s)", n)
+	}
+}
+
+// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
+func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
+	h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+
+	due := h.s.EvaluateDue(context.Background())
+	if len(due) != 1 {
+		t.Fatalf("want one verdict, got %v", due)
+	}
+	v := due[0]
+	if v.Due || v.Err != nil || v.Archive != "" {
+		t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
+	}
+	if v.Reason == "" {
+		t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
+	}
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
+	}
+}
+
+// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
+func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
+	h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
+
+	if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
+		t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
+	}
+	h.s.tick(context.Background())
+	if n := len(h.rr.seen()); n != 0 {
+		t.Fatalf("nothing settled → no run; got %d", n)
+	}
+}
+
+// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
+// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
+// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
+func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
+	boom := errors.New("storage unreachable")
+	ts := &tierStorage{
+		archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
+		err:      map[string]error{"felhom-pbs": boom},
+	}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+
+	var pbs DueVerdict
+	for _, v := range h.s.EvaluateDue(context.Background()) {
+		if v.Target == "felhom-pbs" {
+			pbs = v
+		}
+	}
+	if pbs.Err == nil {
+		t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
+	}
+	if pbs.Due {
+		t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
+	}
+	if pbs.Reason == "" {
+		t.Fatal("the failure must be explained, not merely flagged")
+	}
+
+	// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
+	// tier its proof.
+	h.s.tick(context.Background())
+	if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
+		t.Fatalf("the readable tier must still be proved; got %v", got)
+	}
+}
+
+// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
+
+// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
+// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
+// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
+// invented by a migration.
+func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
+	if err := writeFileForTest(path, legacy); err != nil {
+		t.Fatal(err)
+	}
+
+	st := NewRestoreTestState(path)
+	if _, ok := st.ProvenArchive("local"); ok {
+		t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
+	}
+	at, ok := st.LastSuccess("local")
+	if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
+		t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
+	}
+	// Ordering still works off the legacy times.
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		t.Fatalf("oldest-first must still order legacy records; got %v", got)
+	}
+}
+
+// The new shape round-trips, archive and all.
+func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	now := time.Now().UTC().Truncate(time.Second)
+	st := NewRestoreTestState(path)
+	if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
+		t.Fatal(err)
+	}
+	re := NewRestoreTestState(path)
+	got, ok := re.ProvenArchive("felhom-pbs")
+	if !ok || got != "felhom-pbs:backup/ct/9201/x" {
+		t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
+	}
+	at, ok := re.LastSuccess("felhom-pbs")
+	if !ok || !at.Equal(now) {
+		t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
+	}
+}
+
+// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
+func writeFileForTest(path, content string) error {
+	return os.WriteFile(path, []byte(content), 0o600)
+}
+
+// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
+// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
+// quiet journal is equally consistent with a healthy loop and a dead goroutine.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
+// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
+// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
+func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{
+		"local":      {{volid: "local:backup/a.tar.zst", landed: day0}},
+		"felhom-pbs": nil, // no archive at all
+	}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+	// Prove the local tier so NOTHING is due.
+	if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
+		t.Fatal(err)
+	}
+
+	// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
+	// test would pass against a tick that never calls it.
+	var logbuf strings.Builder
+	h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
+	h.s.tick(context.Background())
+	got := logbuf.String()
+	for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
+		if !strings.Contains(got, want) {
+			t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
+		}
+	}
+}
+
+// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
+// reads as "nothing due" is the silence this rule exists to prevent.
+func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
+	ts := &tierStorage{
+		archives: map[string][]archiveStub{"local": nil},
+		err:      map[string]error{"felhom-pbs": errors.New("storage unreachable")},
+	}
+	h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
+	got := h.s.verdictSummary(context.Background())
+	if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
+		t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
+	}
+}
+
+// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
+//
+// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
+// agent will not repeat the work. So the persisted record has to be able to become a host-report
+// entry — without inventing anything it does not know.
+//
+// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
+// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
+//
+//	--- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
+//	    restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
+//	    per-tier proof on it); got [{... SourceTier: ...}]
+//
+// Restored.
+func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "rt.json")
+	// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
+	// file has ever had, which is what a real box carries after two upgrades.
+	legacy := `{
+	  "old-v1":     "2026-07-30T02:11:07Z",
+	  "old-v2":     {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
+	  "felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
+	}`
+	if err := writeFileForTest(path, legacy); err != nil {
+		t.Fatal(err)
+	}
+
+	got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
+	if len(got) != 1 {
+		t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
+	}
+	e := got[0]
+	if e.SourceTier != "pbs" {
+		t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
+	}
+	if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
+		t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
+	}
+	if e.TestedAt != "2026-08-03T13:25:14Z" {
+		t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
+	}
+	if e.Verified != "boot+running" {
+		t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
+	}
+	// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
+	if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
+		t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
+			e.DurationSeconds, e.ScratchVMID)
+	}
+	// The legacy records still serve the DUE-check, which is a separate question from reporting.
+	if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
+		t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
+	}
+}
+
+// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
+// the production path, not a hand-built fixture.
+func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
+	// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
+	h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+		return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
+	}
+	h.s.tick(context.Background())
+
+	got := h.st.ProvenRestoreTests(context.Background())
+	if len(got) != 1 {
+		t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
+	}
+	if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
+		t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
+	}
+}
+
+// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
+func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
+	ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
+	h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
+	h.s.tick(context.Background())
+	if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
+		t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
+			"would outlive the fault; got %+v", got)
+	}
+}

internal/backup/restoretest_state.go

@@ -0,0 +1,289 @@
+package backup
+
+import (
+	"context"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"sort"
+	"sync"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
+//
+// R-85 (1.4). This one genuinely needs PERSISTENCE, unlike R-84 — and the difference is worth
+// stating, because the two look like the same problem and are not:
+//
+//   - R-84 (backup freshness) had a GROUND TRUTH to consult: the archive is still on the storage,
+//     so the agent could ask "when did a backup last land?" and never persist anything. That is
+//     strictly better, because a pruned archive correctly stops counting.
+//   - A restore-test leaves NO artifact — the scratch guest is destroyed as its final act. There is
+//     nothing to query. "Did we prove this tier restores?" exists only as remembered state, so it
+//     must be written down or it is lost.
+//
+// Why it must survive a restart: rotation is oldest-first (the operator ruling), so an in-memory map
+// would reset every tier to "never tested" on each restart. Ordering would then depend on map
+// iteration order, and one tier could be starved indefinitely while the other is re-tested — with
+// agent deploys as routine as they are, that is not a corner case.
+//
+// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
+// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
+// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
+// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
+// ever becoming due again. The rule earns its keep twice now.
+//
+// R-86 (1.2) — WHICH ARCHIVE, not just when.
+//
+// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
+// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
+// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
+// a tier is due once it holds a settled archive that has not been proven — so the identity of the
+// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
+//
+// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
+// here it records a result, but not WHICH result, and that is just as unable to answer the question
+// being asked of it.
+type RestoreTestState struct {
+	path string
+	mu   sync.Mutex
+	last map[string]provenTier // target id → what was last PROVEN on that tier
+}
+
+// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
+// and when.
+//
+// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
+// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
+// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
+// hub can stay ignorant of a real success until the next archive generation.
+//
+// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
+// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
+// at report-building time — a network call that can fail, on a path where failing means mis-labelling
+// a proof. Store what you knew when you knew it.
+type provenTier struct {
+	Archive  string    // volid of the archive that PASSED; "" = a legacy record with no archive
+	Tier     string    // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
+	Verified string    // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
+	At       time.Time // when that run passed (UTC)
+}
+
+// reportable reports whether this record can be re-reported to the hub as a restore-test result.
+//
+// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
+// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
+// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
+// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
+func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
+
+// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
+//
+//	v1 (pre-R-86)  "<target>": "<RFC3339>"                     — a time, no archive
+//	v2 (R-86)      "<target>": {archive, proven_at}            — due-check usable, not reportable
+//	v3 (R-189)     "<target>": {archive, tier, verified, …}    — both
+//
+// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
+// readers above test for — the migration needs no version number because the absence IS the answer.
+type provenTierJSON struct {
+	Archive  string `json:"archive"`
+	Tier     string `json:"tier,omitempty"`
+	Verified string `json:"verified,omitempty"`
+	ProvenAt string `json:"proven_at"`
+}
+
+// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
+// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
+// corrupt file from wedging the daemon.
+//
+// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
+// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
+// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
+// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
+// read a legacy time as proof of whatever archive happens to be current, which would mark an
+// unproven archive proven — inventing a guarantee out of a migration.
+func NewRestoreTestState(path string) *RestoreTestState {
+	s := &RestoreTestState{path: path, last: map[string]provenTier{}}
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return s
+	}
+	var raw map[string]json.RawMessage
+	if json.Unmarshal(data, &raw) != nil {
+		return s
+	}
+	for target, msg := range raw {
+		// Legacy shape: a bare RFC3339 string.
+		var legacy string
+		if json.Unmarshal(msg, &legacy) == nil {
+			if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
+				s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
+			}
+			continue
+		}
+		var cur provenTierJSON
+		if json.Unmarshal(msg, &cur) != nil {
+			continue // one unreadable entry must not lose the others
+		}
+		t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
+		if perr != nil {
+			continue
+		}
+		s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
+	}
+	return s
+}
+
+// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
+// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
+// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
+//
+// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
+// the next reader will notice failures are absent and try to "fix" it:
+//
+//	a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
+//	the system quietly less tested than it believes. It must survive a restart.
+//
+//	a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
+//	so a lost failure heals itself within one interval. Persisting it would do the opposite of
+//	helping: a healed tier would keep reporting a failure that is no longer true.
+func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
+	if target == "" {
+		return nil
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
+	return s.saveLocked()
+}
+
+// LastSuccess returns when this tier was last proven (ok=false = never).
+func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	p, ok := s.last[target]
+	return p.At, ok
+}
+
+// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
+// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
+// not this one has not been proven.
+func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	p, ok := s.last[target]
+	if !ok || p.Archive == "" {
+		return "", false
+	}
+	return p.Archive, true
+}
+
+// Snapshot returns a copy of the last-proven TIMES.
+//
+// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
+// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
+// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
+// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
+// only the times; if it acquires none, delete it rather than let it claim a purpose again.
+func (s *RestoreTestState) Snapshot() map[string]time.Time {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make(map[string]time.Time, len(s.last))
+	for k, v := range s.last {
+		out[k] = v.At
+	}
+	return out
+}
+
+// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
+//
+// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
+// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
+//
+//   - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
+//   - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
+//   - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
+//     is not a claim; a fabricated one would be.
+//
+// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
+// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
+// worse defect than the one this fixes.
+func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make([]hub.RestoreTest, 0, len(s.last))
+	for _, p := range s.last {
+		if !p.reportable() {
+			continue
+		}
+		out = append(out, hub.RestoreTest{
+			SourceArchive: p.Archive,
+			SourceTier:    p.Tier,
+			Pass:          true,
+			Verified:      p.Verified,
+			TestedAt:      p.At.UTC().Format(time.RFC3339),
+		})
+	}
+	// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
+	// iteration is randomised.
+	sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
+	return out
+}
+
+// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
+//
+// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
+// naturally prioritises a tier that has never been restore-tested at all — which on this fleet was
+// the offsite tier, unproven for its entire existence.
+//
+// Ties break on target id so the order is deterministic; without that, two tiers proven in the same
+// second would rotate by map iteration order, which is randomised in Go and would make the
+// behaviour untestable and occasionally starving.
+func (s *RestoreTestState) OldestFirst(targets []string) []string {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := append([]string(nil), targets...)
+	sort.SliceStable(out, func(i, j int) bool {
+		pi, oki := s.last[out[i]]
+		pj, okj := s.last[out[j]]
+		ti, tj := pi.At, pj.At
+		switch {
+		case !oki && !okj:
+			return out[i] < out[j] // both never proven → deterministic
+		case !oki:
+			return true // never proven wins
+		case !okj:
+			return false
+		case !ti.Equal(tj):
+			return ti.Before(tj)
+		default:
+			return out[i] < out[j]
+		}
+	})
+	return out
+}
+
+func (s *RestoreTestState) saveLocked() error {
+	raw := make(map[string]provenTierJSON, len(s.last))
+	for target, p := range s.last {
+		raw[target] = provenTierJSON{
+			Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
+			ProvenAt: p.At.UTC().Format(time.RFC3339),
+		}
+	}
+	data, err := json.MarshalIndent(raw, "", "  ")
+	if err != nil {
+		return err
+	}
+	if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
+		return err
+	}
+	tmp := s.path + ".tmp"
+	if err := os.WriteFile(tmp, data, 0o600); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, s.path)
+}

internal/backup/rotation_test.go

@@ -0,0 +1,354 @@
+package backup
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-85 Phase 2 — tier rotation, persisted state, and the one-heavy-operation gate.
+//
+// The failure this prevents is not hypothetical: demo-hp's DR tier reported `applied` with ZERO
+// snapshots for five days and nobody noticed, because the scheduler could only ever see the primary
+// tier. Rotation is what makes the offsite tier testable at all.
+
+// rotRunner records which archives it was asked to restore.
+type rotRunner struct {
+	mu       sync.Mutex
+	archives []string
+	pass     bool
+}
+
+func (r *rotRunner) RunRestoreTest(_ context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.archives = append(r.archives, spec.Archive)
+	return reconcile.RestoreTestResult{
+		Archive: spec.Archive, SourceTier: spec.SourceTier,
+		Pass: r.pass, Verified: "boot+running",
+	}
+}
+func (r *rotRunner) seen() []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]string(nil), r.archives...)
+}
+
+// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
+// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
+// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
+var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
+
+// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
+func archiveFor(m map[string]string) TierPicker {
+	return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
+		a := m[target]
+		if a == "" {
+			return "", time.Time{}, nil
+		}
+		return a, testLanded, nil
+	}
+}
+
+func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPicker, gate *InFlight) *Scheduler {
+	t.Helper()
+	return NewScheduler(SchedulerOptions{
+		Runner: rr,
+		Store:  NewStore(),
+		Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
+			return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
+		},
+		Cadence:  time.Hour,
+		Logger:   quiet(),
+		Tiers:    []string{"local", "felhom-pbs"},
+		TierPick: pick,
+		State:    st,
+		InFlight: gate,
+	})
+}
+
+// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
+//
+// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
+// it was a faithful statement of the defect. It read:
+//
+//	4 ticks → 4 runs, and consecutive runs must hit different tiers
+//
+// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
+// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
+// newest settled archive is already proven is not re-tested just because time passed. So the
+// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
+// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
+//
+// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
+// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
+// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
+// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
+func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
+	}), &InFlight{})
+	s.now = func() time.Time { return time.Now().UTC() }
+
+	for i := 0; i < 4; i++ {
+		s.tick(context.Background())
+	}
+
+	got := rr.seen()
+	var sawLocal, sawPBS bool
+	for _, a := range got {
+		if len(a) >= 5 && a[:5] == "local" {
+			sawLocal = true
+		}
+		if len(a) >= 10 && a[:10] == "felhom-pbs" {
+			sawPBS = true
+		}
+	}
+	if !sawLocal || !sawPBS {
+		t.Fatalf("both tiers must be exercised; got %v", got)
+	}
+	// Exactly one run per tier: the archives never changed, so nothing became due a second time.
+	if len(got) != 2 {
+		t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
+	}
+	if got[0] == got[1] {
+		t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
+	}
+}
+
+// A tier with NO archive is skipped, not failed, and the other tier still runs. A brand-new offsite
+// tier legitimately has nothing to restore; turning that into a failure would make every fresh box
+// look broken for its first week.
+func TestRotation_EmptyTierSkippedNotFailed(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs": "", // provisioned but empty
+	}), &InFlight{})
+
+	s.tick(context.Background())
+	got := rr.seen()
+	if len(got) != 1 || got[0][:5] != "local" {
+		t.Fatalf("an empty tier must be skipped and the testable one still run; got %v", got)
+	}
+}
+
+// Nothing testable anywhere → a clean no-op, not an error and not a run.
+func TestRotation_NoArchivesAnywhereIsANoOp(t *testing.T) {
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{}), &InFlight{})
+	s.tick(context.Background())
+	if got := rr.seen(); len(got) != 0 {
+		t.Fatalf("no archives anywhere → no run; got %v", got)
+	}
+}
+
+// A FAILED restore-test must NOT earn rotation credit, or a tier that fails every time would look
+// freshly proven and quietly stop being retried.
+func TestRotation_FailureEarnsNoCredit(t *testing.T) {
+	rr := &rotRunner{pass: false}
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	s := rotScheduler(t, rr, st, archiveFor(map[string]string{
+		"local":      "local:backup/x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
+	}), &InFlight{})
+	s.tick(context.Background())
+	if _, ok := st.LastSuccess("local"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+	if _, ok := st.LastSuccess("felhom-pbs"); ok {
+		t.Fatal("a FAILED restore-test must not stamp the tier as proven")
+	}
+}
+
+// ── SCENARIO E — rotation survives a restart ─────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed): make the state in-memory (construct a fresh
+// `NewRestoreTestState` on a DIFFERENT path for the second scheduler, i.e. lose the file) and this
+// fails with "after a restart the OTHER tier must be next; got felhom-pbs" — the same tier repeats
+// and the other is starved indefinitely, which with agent deploys as routine as they are is not a
+// corner case.
+func TestRotation_SurvivesRestart(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	picks := archiveFor(map[string]string{
+		"local":      "local:backup/x.tar.zst",
+		"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
+	})
+
+	// First process: the OFFSITE tier is tested (never-proven sorts first).
+	rr1 := &rotRunner{pass: true}
+	st1 := NewRestoreTestState(path)
+	s1 := rotScheduler(t, rr1, st1, picks, &InFlight{})
+	s1.tick(context.Background())
+	first := rr1.seen()
+	if len(first) != 1 {
+		t.Fatalf("want one run, got %v", first)
+	}
+
+	// --- restart: brand-new state object reading the SAME file ---
+	rr2 := &rotRunner{pass: true}
+	st2 := NewRestoreTestState(path)
+	s2 := rotScheduler(t, rr2, st2, picks, &InFlight{})
+	s2.tick(context.Background())
+	second := rr2.seen()
+	if len(second) != 1 {
+		t.Fatalf("want one run after restart, got %v", second)
+	}
+
+	if second[0] == first[0] {
+		t.Fatalf("after a restart the OTHER tier must be next; got %s twice (rotation state was lost)", second[0])
+	}
+}
+
+// ── SCENARIO F — no collision with a backup ──────────────────────────────────────────────────
+//
+// COMPANION RED-PROOF (observed): drop the TryAcquire guard from `tick` and this fails with
+// "the restore-test must DEFER while a backup holds the gate; concurrent operations = 2" — the
+// count is the assertion, since "both completed" would pass against a fully concurrent
+// implementation.
+func TestRotation_DefersWhileABackupHoldsTheGate(t *testing.T) {
+	gate := &InFlight{}
+	release, _, ok := gate.TryAcquire("backup:felhom-pbs")
+	if !ok {
+		t.Fatal("precondition: the gate should have been free")
+	}
+	defer release()
+
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
+
+	s.tick(context.Background())
+
+	concurrent := 1 + len(rr.seen()) // the backup holding the gate, plus anything the tick started
+	if concurrent != 1 {
+		t.Fatalf("the restore-test must DEFER while a backup holds the gate; concurrent operations = %d", concurrent)
+	}
+}
+
+// Once the backup releases, the next cadence proceeds — deferral must not be permanent.
+func TestRotation_ResumesAfterTheGateFrees(t *testing.T) {
+	gate := &InFlight{}
+	release, _, _ := gate.TryAcquire("backup:local")
+
+	rr := &rotRunner{pass: true}
+	s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
+		archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
+
+	s.tick(context.Background())
+	if len(rr.seen()) != 0 {
+		t.Fatal("should have deferred while held")
+	}
+	release()
+	s.tick(context.Background())
+	if len(rr.seen()) != 1 {
+		t.Fatalf("must resume once the gate frees; got %v", rr.seen())
+	}
+}
+
+// The gate itself: one holder at a time, named, and release is idempotent.
+func TestInFlight_Semantics(t *testing.T) {
+	g := &InFlight{}
+	rel, busy, ok := g.TryAcquire("backup:local")
+	if !ok || busy != "" {
+		t.Fatalf("first acquire must succeed; ok=%v busy=%q", ok, busy)
+	}
+	if _, busy2, ok2 := g.TryAcquire("restore-test"); ok2 || busy2 != "backup:local" {
+		t.Fatalf("second acquire must fail and NAME the holder; ok=%v busy=%q", ok2, busy2)
+	}
+	rel()
+	rel() // idempotent — a double release must not free someone else's later claim
+	if g.Busy() != "" {
+		t.Fatalf("gate should be idle after release; busy=%q", g.Busy())
+	}
+	if _, _, ok3 := g.TryAcquire("restore-test"); !ok3 {
+		t.Fatal("gate must be reusable after release")
+	}
+}
+
+// A nil gate means "not wired" → no gating, pre-R-85 behaviour. Keeps every existing caller working.
+func TestInFlight_NilIsUngated(t *testing.T) {
+	var g *InFlight
+	rel, _, ok := g.TryAcquire("x")
+	if !ok {
+		t.Fatal("a nil gate must not block")
+	}
+	rel()
+	if g.Busy() != "" {
+		t.Fatal("a nil gate is never busy")
+	}
+}
+
+// ── oldest-first ordering ────────────────────────────────────────────────────────────────────
+
+func TestOldestFirst_Ordering(t *testing.T) {
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	now := time.Now().UTC()
+
+	// Never-proven sorts FIRST — the case that matters, since the offsite tier starts there.
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		// both never proven → deterministic tie-break by id
+		if got[0] != "felhom-pbs" && got[0] != "local" {
+			t.Fatalf("unexpected: %v", got)
+		}
+	}
+	_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
+		t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
+	}
+	_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
+	if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
+		t.Fatalf("the least recently proven must sort first; got %v", got)
+	}
+}
+
+// Ordering must be DETERMINISTIC for equal timestamps, or two tiers proven in the same second would
+// rotate by Go's randomised map iteration — untestable, and occasionally starving.
+func TestOldestFirst_DeterministicOnTies(t *testing.T) {
+	st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
+	now := time.Now().UTC()
+	_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
+	_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
+	for i := 0; i < 20; i++ {
+		if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
+			t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
+		}
+	}
+}
+
+// The state file round-trips, and a corrupt file degrades to "nothing proven" rather than wedging.
+func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "rt.json")
+	now := time.Now().UTC().Truncate(time.Second)
+
+	st := NewRestoreTestState(path)
+	if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
+		t.Fatal(err)
+	}
+	reopened := NewRestoreTestState(path)
+	got, ok := reopened.LastSuccess("felhom-pbs")
+	if !ok || !got.Equal(now) {
+		t.Fatalf("state must round-trip; got %v ok=%v want %v", got, ok, now)
+	}
+
+	bad := filepath.Join(dir, "corrupt.json")
+	if err := os.WriteFile(bad, []byte("{{{not json"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	c := NewRestoreTestState(bad)
+	if _, ok := c.LastSuccess("felhom-pbs"); ok {
+		t.Fatal("a corrupt state file must degrade to 'nothing proven', not invent a timestamp")
+	}
+}

internal/backup/runner.go

@@ -0,0 +1,524 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// BackupAPI is the read+backup proxmox surface the runner needs. *proxmox.Client satisfies it.
+type BackupAPI interface {
+	Vzdump(ctx context.Context, opts proxmox.VzdumpOptions) (string, error)
+	WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
+	GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
+	StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
+	// ListStorage enumerates storages (name+type) — used to scope local-only retention (never prune PBS).
+	ListStorage(ctx context.Context) ([]proxmox.Storage, error)
+	// TaskLogTail reads trailing task-log lines — used to read the ACTUAL vzdump mode
+	// (PVE may downgrade a requested snapshot to stop for a stopped guest — spike B1).
+	TaskLogTail(ctx context.Context, upid string, limit int) ([]string, error)
+}
+
+// BackupRunner orchestrates a crash-consistent vzdump to a local target and reports the
+// result (incl. the bulk-volume gap). An agent-initiated vzdump is crash-consistent only
+// (no fsfreeze); the report says so.
+type BackupRunner struct {
+	api    BackupAPI
+	target string             // backup storage (content=backup)
+	mode   proxmox.BackupMode // default ModeSnapshot
+	notes  string             // optional notes-template
+	// retention is the per-run `--prune-backups` spec (e.g. "keep-last=3") applied to a LOCAL target after
+	// each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune
+	// (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target.
+	retention string
+	// waitTimeout bounds the WaitTask poll on this runner's vzdump. Per-TIER since R-82: 30m is
+	// right for a local vzdump and badly wrong for an offsite PBS upload (see the 2026-07-26 live
+	// failure recorded on config.BackupTargetConfig.WaitTimeoutSeconds). 0 → 30m (legacy).
+	waitTimeout time.Duration
+	// allowPBSPrune permits `--prune-backups` on a PBS-type target. OFF by default and ON only for
+	// an ADDITIONAL tier whose keep_last was set explicitly (operator ruling 2026-07-26: keep two
+	// weeks of weekly offsite backups).
+	//
+	// The blanket PBS refusal it replaces existed for a real reason and still applies to the
+	// PRIMARY tier: BackupTarget() DEFAULTS to "felhom-pbs" and KeepLast() DEFAULTS to 3, so a box
+	// with neither key set would silently prune its offsite DR to 3 restore points. An additional
+	// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
+	// there is a deliberate act.
+	allowPBSPrune bool
+	logger        *slog.Logger
+	now           func() time.Time
+	// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
+	// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
+	// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
+	// due-check is served from the local-API handler goroutines.
+	rejectedMu sync.Mutex
+	rejected   map[string]struct{}
+}
+
+// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
+// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
+// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
+func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner {
+	return NewBackupRunnerWithWait(api, target, mode, notes, retention, 0, logger)
+}
+
+// NewBackupRunnerWithWait is NewBackupRunner plus an explicit vzdump wait bound (0 → 30m).
+func NewBackupRunnerWithWait(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, logger *slog.Logger) *BackupRunner {
+	return NewBackupRunnerFull(api, target, mode, notes, retention, waitTimeout, false, logger)
+}
+
+// NewBackupRunnerFull is the full constructor. allowPBSPrune must be true ONLY for an additional
+// tier with an explicitly configured keep_last — see BackupRunner.allowPBSPrune.
+func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, allowPBSPrune bool, logger *slog.Logger) *BackupRunner {
+	if mode == "" {
+		mode = proxmox.ModeSnapshot
+	}
+	if logger == nil {
+		logger = slog.Default()
+	}
+	if waitTimeout <= 0 {
+		waitTimeout = 30 * time.Minute
+	}
+	return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention,
+		waitTimeout: waitTimeout, allowPBSPrune: allowPBSPrune, logger: logger,
+		now: func() time.Time { return time.Now().UTC() }}
+}
+
+// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
+// applies the configured retention ONLY when the target is a non-PBS storage — PBS offsite retention is a
+// separate lifecycle and must never be pruned by the per-run flag (§9). Fail-safe: if the target's type
+// can't be confirmed (lookup error / not found), it SKIPS pruning rather than risk pruning PBS — the
+// host_disk + storage_fill detectors remain the safety net.
+func (r *BackupRunner) localPruneSpec(ctx context.Context) string {
+	if r.retention == "" {
+		return ""
+	}
+	stores, err := r.api.ListStorage(ctx)
+	if err != nil {
+		r.logger.Warn("backup: could not resolve target storage type — skipping local prune this run", "target", r.target, "err", err)
+		return ""
+	}
+	for _, s := range stores {
+		if s.Storage == r.target {
+			if s.Type == "pbs" && !r.allowPBSPrune {
+				// Not opted in → never prune the offsite DR (the pre-R-82 rule, and still the rule
+				// for the primary tier, whose target+retention both DEFAULT and could prune by
+				// accident).
+				return ""
+			}
+			return r.retention
+		}
+	}
+	r.logger.Warn("backup: target storage not found in list — skipping local prune this run (fail-safe)", "target", r.target)
+	return ""
+}
+
+// snapshotMarker is the vzdump task-log line that signals the storage snapshot has been created
+// and the backup is now reading from it — the point after which resuming the guest's app cannot
+// affect the backup (slice 8B.2; validated on PVE 9.2.2: `INFO: create storage snapshot 'vzdump'`).
+// It only appears in snapshot mode (stop mode takes no storage snapshot), so its presence ⟹
+// snapshot mode — the basis for the controller's early resume.
+const snapshotMarker = "create storage snapshot"
+
+// snapshotWatchInterval is how often watchForSnapshot polls the task log. A package var so tests
+// can shrink it (production: poll once a second — the marker appears in the first ~1s, §0).
+var snapshotWatchInterval = time.Second
+
+// Backup runs one vzdump of vmid to the local target and returns the report record.
+func (r *BackupRunner) Backup(ctx context.Context, vmid int) (hub.Backup, error) {
+	return r.backup(ctx, vmid, nil)
+}
+
+// BackupWithSnapshotHook is Backup plus an onSnapshot callback invoked ONCE, mid-backup, when the
+// storage snapshot has been taken (snapshot mode only) — the 8B.2 early-resume signal. In
+// stop/downgraded mode the marker never appears, so onSnapshot is never called (the caller then
+// resumes at completion). onSnapshot must be cheap + non-blocking (it runs on a watcher goroutine).
+func (r *BackupRunner) BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) {
+	return r.backup(ctx, vmid, onSnapshot)
+}
+
+// backup is the shared body. A failure is returned BOTH as an error and as a
+// Backup{Success:false,...} so the caller can record the failed attempt.
+func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) {
+	start := r.now()
+	rec := hub.Backup{
+		TargetID:        r.target,
+		VMID:            vmid,
+		Mode:            string(r.mode),
+		CrashConsistent: true, // always, this slice (no controller quiesce)
+		StartedAt:       start.Format(time.RFC3339),
+	}
+	if r.target == "" {
+		rec.Error = "no backup target configured"
+		return rec, fmt.Errorf("backup: %s", rec.Error)
+	}
+
+	// Bulk-volume gap: which mountpoints the vzdump will EXCLUDE (best-effort; a config-read
+	// failure just leaves the gap unknown, never fails the backup).
+	if cfg, err := r.api.GuestConfig(ctx, vmid); err == nil {
+		rec.UncoveredVolumes = uncoveredMountpoints(cfg.MountPoints())
+	} else {
+		r.logger.Warn("backup: could not read guest config for bulk-gap", "vmid", vmid, "err", err)
+		rec.UncoveredVolumes = []string{}
+	}
+
+	upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
+		VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
+		PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
+	})
+	if err != nil {
+		rec.Error = err.Error()
+		rec.DurationSeconds = time.Since(start).Seconds()
+		return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
+	}
+	if upid != "" {
+		// 8B.2: while the backup runs, watch the task log for the storage-snapshot marker and
+		// fire onSnapshot once (snapshot mode only) so the controller can resume its app early.
+		if onSnapshot != nil {
+			watchCtx, stopWatch := context.WithCancel(ctx)
+			defer stopWatch()
+			go r.watchForSnapshot(watchCtx, upid, onSnapshot)
+		}
+		if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
+			rec.Error = err.Error()
+			rec.DurationSeconds = time.Since(start).Seconds()
+			return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
+		}
+		// Report the ACTUAL mode PVE used (it may downgrade snapshot→stop for a stopped
+		// guest — spike B1), read from the task log; fall back to the requested mode.
+		if lines, err := r.api.TaskLogTail(ctx, upid, 200); err == nil {
+			if actual := parseBackupMode(lines); actual != "" {
+				rec.Mode = actual
+			}
+		}
+	}
+
+	// Resolve the produced archive (volid + size) — the task status carries no result volid.
+	vol, size, err := r.latestArchive(ctx, vmid)
+	if err != nil {
+		rec.Error = fmt.Sprintf("backup succeeded but archive not resolved: %v", err)
+		rec.DurationSeconds = time.Since(start).Seconds()
+		return rec, fmt.Errorf("backup: resolve archive vmid %d: %w", vmid, err)
+	}
+	rec.Archive = vol
+	rec.SizeBytes = size
+	rec.Success = true
+	rec.DurationSeconds = time.Since(start).Seconds()
+	r.logger.Info("backup: completed", "vmid", vmid, "target", r.target, "archive", vol, "size_bytes", size,
+		"uncovered_volumes", len(rec.UncoveredVolumes))
+	return rec, nil
+}
+
+// watchForSnapshot polls the running backup's task log until it sees the storage-snapshot marker
+// (→ onSnapshot once) or the requested mode is reported as `stop` (→ downgraded; the marker will
+// never come, so stop watching) or ctx is cancelled (backup finished). Best-effort: a log-read
+// error is retried on the next tick; onSnapshot fires at most once.
+func (r *BackupRunner) watchForSnapshot(ctx context.Context, upid string, onSnapshot func()) {
+	ticker := time.NewTicker(snapshotWatchInterval)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			lines, err := r.api.TaskLogTail(ctx, upid, 200)
+			if err != nil {
+				continue
+			}
+			// A stop-mode (or downgraded) backup never creates a storage snapshot → never resume early.
+			if m := parseBackupMode(lines); m != "" && m != string(proxmox.ModeSnapshot) {
+				return
+			}
+			for _, ln := range lines {
+				if strings.Contains(ln, snapshotMarker) {
+					onSnapshot()
+					return
+				}
+			}
+		}
+	}
+}
+
+// PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
+// when there is none — the restore-test then no-ops cleanly.
+func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
+	return r.PickRestoreCandidateOn(ctx, r.target)
+}
+
+// PickRestoreCandidateOn is PickRestoreCandidate for an ARBITRARY tier's storage (R-85 1.2), so the
+// scheduler can rotate across tiers instead of only ever seeing this runner's own target.
+//
+// Contract preserved: "" + nil error when the storage holds no archive. **A tier with nothing to
+// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
+// that into a failure would make every fresh box look broken for its first week.
+func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
+	archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
+	return archive, err
+}
+
+// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
+// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
+// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
+// one-line call into this and its contract is untouched (one scan, one owner).
+//
+// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
+// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
+// restore-testing the archive a backup is still writing proves nothing about the backup that
+// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
+// on that: the tier is due when a settled archive exists that has not been proven.
+//
+// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
+// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
+// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
+// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
+// the archive generation rather than by the evaluation interval.
+//
+// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
+// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
+// turning that into a failure would make every fresh box look broken for its first week.
+func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
+	if target == "" {
+		return "", time.Time{}, nil
+	}
+	contents, err := r.api.StorageContent(ctx, target)
+	if err != nil {
+		return "", time.Time{}, err
+	}
+	var best string
+	var bestCTime int64 = -1
+	for _, e := range contents {
+		if e.Content != "backup" {
+			continue
+		}
+		if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
+			continue // not settled yet — a newer archive is not a reason to re-prove an older one
+		}
+		if ok, why := archivePlausiblyComplete(e); !ok {
+			r.warnRejectedArchiveOnce(e, why)
+			continue
+		}
+		if e.CTime > bestCTime {
+			bestCTime, best = e.CTime, e.VolID
+		}
+	}
+	if best == "" {
+		return "", time.Time{}, nil
+	}
+	return best, time.Unix(bestCTime, 0).UTC(), nil
+}
+
+// latestArchive finds the newest backup archive volid + size for vmid on the target.
+func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int64, error) {
+	contents, err := r.api.StorageContent(ctx, r.target)
+	if err != nil {
+		return "", 0, err
+	}
+	var vol string
+	var size, bestCTime int64 = 0, -1
+	for _, e := range contents {
+		if e.Content == "backup" && e.VMID == vmid && e.CTime > bestCTime {
+			bestCTime, vol, size = e.CTime, e.VolID, e.Size
+		}
+	}
+	if vol == "" {
+		return "", 0, fmt.Errorf("no backup archive found for vmid %d on %s", vmid, r.target)
+	}
+	return vol, size, nil
+}
+
+// NewestArchiveTime reports when this guest's newest backup archive LANDED ON THIS TARGET, from the
+// storage itself. ok=false means the target genuinely holds no archive for this guest.
+//
+// R-84: this is the cure for the redundant-backup-after-restart problem. The agent's backup Store is
+// in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due
+// reported "no successful backup recorded yet" and the controller dutifully took another one. On the
+// local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
+// every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
+// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
+// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
+//
+// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
+//
+//	smallest REAL backup anywhere on the fleet ...  612,397,450 B  (~584 MiB, a guest-9100 vzdump)
+//	demo-hp      local / PBS ..................... 1.59 GB / 4.35-4.37 GB
+//	demo-felhom  local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
+//	the phantom left by a PBS daemon killed mid-upload ....... 1 B
+//
+// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
+// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
+// archive — which is the property that matters, because a floor set too HIGH does not merely lose
+// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
+const minPlausibleArchiveBytes int64 = 1 << 20
+
+// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
+// why. Pure, so the contract is unit-testable without a storage.
+//
+// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
+// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
+// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
+//
+//   - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
+//     storage has no verification concept — and absent on a good PBS snapshot until verify-new
+//     catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
+//     one: continuous re-backup across the fleet.
+//   - `encrypted` fails the same way, and for the same reason.
+//   - `notes` happens to be present on both good tiers today only because the agent sets it; an
+//     archive written by any other path lacks it. Too fragile to gate freshness on.
+//
+// Size is the only signal that means the same thing on every tier.
+//
+// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
+// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
+// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
+// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
+func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
+	if e.Size < minPlausibleArchiveBytes {
+		return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
+			e.Size, minPlausibleArchiveBytes)
+	}
+	return true, ""
+}
+
+// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
+//
+// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
+// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
+// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
+// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
+// logging per poll would emit ~288 identical lines a day and bury the one that matters.
+func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
+	r.rejectedMu.Lock()
+	if r.rejected == nil {
+		r.rejected = map[string]struct{}{}
+	}
+	_, seen := r.rejected[e.VolID]
+	if !seen {
+		r.rejected[e.VolID] = struct{}{}
+	}
+	r.rejectedMu.Unlock()
+	if seen {
+		return
+	}
+	r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
+		"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
+}
+
+// demo-felhom in a single afternoon of deploys (2026-07-26).
+//
+// Asking the STORAGE rather than persisting the store is deliberate:
+//   - it is ground truth, not remembered state — if an archive was pruned or deleted it correctly
+//     stops counting, whereas a persisted record would keep claiming a backup that no longer exists;
+//   - it needs no new on-disk state and no migration;
+//   - it is the same source `latestArchive` already trusts to build the post-backup record.
+//
+// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
+// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
+// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
+// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
+// richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
+// synthesized record would put invented numbers into the host-report.
+func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
+	contents, err := r.api.StorageContent(ctx, r.target)
+	if err != nil {
+		return time.Time{}, false, err
+	}
+	var best int64 = -1
+	for _, e := range contents {
+		if e.Content != "backup" || e.VMID != vmid {
+			continue
+		}
+		if ok, why := archivePlausiblyComplete(e); !ok {
+			r.warnRejectedArchiveOnce(e, why)
+			continue
+		}
+		if e.CTime > best {
+			best = e.CTime
+		}
+	}
+	if best < 0 {
+		return time.Time{}, false, nil
+	}
+	return time.Unix(best, 0).UTC(), true, nil
+}
+
+// parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>`
+// (e.g. "INFO: backup mode: stop"). Returns "" if not found.
+func parseBackupMode(lines []string) string {
+	const marker = "backup mode:"
+	for _, ln := range lines {
+		if i := strings.Index(ln, marker); i >= 0 {
+			return strings.TrimSpace(ln[i+len(marker):])
+		}
+	}
+	return ""
+}
+
+// uncoveredMountpoints returns the mountpoint paths the guest vzdump EXCLUDES. LXC mount
+// points are OPT-IN to vzdump: a mpN with `backup=1` is covered; ANY other state — the
+// `backup=` token absent OR `backup=0` — is excluded. We deliberately treat unset as
+// uncovered (the safe DR direction: never imply an unprotected volume is backed up).
+func uncoveredMountpoints(mps map[string]string) []string {
+	var out []string
+	for key, cfg := range mps {
+		if mountpointCovered(cfg) {
+			continue
+		}
+		out = append(out, mountpointLabel(key, cfg))
+	}
+	sort.Strings(out)
+	if out == nil {
+		return []string{}
+	}
+	return out
+}
+
+// mountpointCovered reports whether a mpN config string is included in the vzdump — true
+// ONLY when it carries an explicit `backup=1`.
+func mountpointCovered(cfg string) bool {
+	for _, tok := range strings.Split(cfg, ",") {
+		if v, ok := strings.CutPrefix(tok, "backup="); ok {
+			return v == "1"
+		}
+	}
+	return false // no backup= token → opt-out by default → not covered
+}
+
+// mountpointLabel prefers the mp=/path token, falling back to the mpN key.
+func mountpointLabel(key, cfg string) string {
+	for _, tok := range strings.Split(cfg, ",") {
+		if p, ok := strings.CutPrefix(tok, "mp="); ok && p != "" {
+			return p
+		}
+	}
+	return key
+}
+
+// ToHubRestoreTest maps a reconcile restore-test result to the hub wire record (the backup
+// package owns the reconcile→hub mapping so reconcile need not import hub for the result).
+func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.RestoreTest {
+	rt := hub.RestoreTest{
+		SourceArchive:      res.Archive,
+		SourceTier:         res.SourceTier,
+		ScratchVMID:        res.ScratchVMID,
+		Pass:               res.Pass,
+		Verified:           res.Verified,
+		TestedAt:           testedAt.Format(time.RFC3339),
+		DurationSeconds:    res.Duration.Seconds(),
+		Warnings:           res.StartWarnings,
+		WarningsRecognized: res.WarningsRecognized,
+		MountParity:        res.MountParity,
+		MountInventory:     res.MountInventory,
+	}
+	if res.Err != nil {
+		rt.Error = res.Err.Error()
+	}
+	return rt
+}

internal/backup/runner_snapshot_test.go

@@ -0,0 +1,72 @@
+package backup
+
+import (
+	"context"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// 8B.2: in snapshot mode, the runner fires onSnapshot when the storage-snapshot marker appears in
+// the task log — mid-backup, before completion.
+func TestBackupWithSnapshotHook_FiresOnMarker(t *testing.T) {
+	old := snapshotWatchInterval
+	snapshotWatchInterval = 2 * time.Millisecond
+	defer func() { snapshotWatchInterval = old }()
+
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:backup",
+		waitGate:   make(chan struct{}), // hold the backup open so the watcher gets to poll
+		logLines:   []string{"INFO: backup mode: snapshot", "INFO: create storage snapshot 'vzdump'"},
+		content:    []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
+	}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+
+	var fired int32
+	done := make(chan struct{})
+	go func() {
+		_, _ = r.BackupWithSnapshotHook(context.Background(), 9001, func() { atomic.StoreInt32(&fired, 1) })
+		close(done)
+	}()
+
+	// the watcher should fire onSnapshot well before we release the backup
+	deadline := time.Now().Add(time.Second)
+	for time.Now().Before(deadline) && atomic.LoadInt32(&fired) == 0 {
+		time.Sleep(2 * time.Millisecond)
+	}
+	if atomic.LoadInt32(&fired) != 1 {
+		t.Fatal("onSnapshot did not fire on the storage-snapshot marker")
+	}
+	close(api.waitGate) // let the backup complete
+	<-done
+}
+
+// Stop/downgraded mode → no storage-snapshot marker → onSnapshot never fires (the 8B.2 fallback).
+func TestBackupWithSnapshotHook_StopMode_NeverFires(t *testing.T) {
+	old := snapshotWatchInterval
+	snapshotWatchInterval = 2 * time.Millisecond
+	defer func() { snapshotWatchInterval = old }()
+
+	api := &fakeBackupAPI{
+		vzdumpUPID: "UPID:backup",
+		waitGate:   make(chan struct{}),
+		logLines:   []string{"INFO: backup mode: stop"}, // downgraded; no snapshot marker
+		content:    []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9001-x", Content: "backup", VMID: 9001, Size: 100, CTime: 1}},
+	}
+	r := NewBackupRunner(api, "local", "", "", "", quiet())
+
+	var fired int32
+	done := make(chan struct{})
+	go func() {
+		_, _ = r.BackupWithSnapshotHook(context.Background(), 9001, func() { atomic.StoreInt32(&fired, 1) })
+		close(done)
+	}()
+	time.Sleep(40 * time.Millisecond) // the watcher polls several times + sees stop mode → returns
+	if atomic.LoadInt32(&fired) != 0 {
+		t.Fatal("onSnapshot fired in stop mode (must not)")
+	}
+	close(api.waitGate)
+	<-done
+}

internal/backup/schedule.go

@@ -0,0 +1,297 @@
+package backup
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// RestoreTestRunner is the reconcile-engine seam the scheduler drives (*reconcile.Engine
+// satisfies it). Kept narrow so the scheduler is unit-testable with a fake.
+type RestoreTestRunner interface {
+	RunRestoreTest(ctx context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult
+}
+
+// CandidatePicker resolves the archive volid to restore-test (newest backup), or "" when
+// there is none yet (the tick then no-ops).
+type CandidatePicker func(ctx context.Context) (string, error)
+
+// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
+//
+// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
+// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
+// the resulting value reused for every run for the lifetime of the process. Two consequences:
+//   - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
+//   - it was a latent staleness bug in its own right — a storage-type or config change did not take
+//     effect until the daemon restarted.
+//
+// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
+// the configured target: deriving it from config is what produced the 600 s false failure when a
+// PBS archive was classified "local" and got the 10-minute local wait.
+type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
+
+// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
+// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
+// satisfies it. A zero notAfter means "no settle requirement".
+//
+// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
+// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
+// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
+// brand-new offsite tier legitimately has nothing to restore yet.
+type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
+
+// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
+// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
+// AND a valid scratch band is configured (validated by the caller before construction).
+type Scheduler struct {
+	runner RestoreTestRunner
+	pick   CandidatePicker
+	store  *Store
+	spec   SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
+	// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
+	// longer the thing that decides a test happens; see restoretest_due.go.
+	cadence time.Duration
+	// settle is how long an archive must have sat before it is a candidate (R-86).
+	settle time.Duration
+	logger *slog.Logger
+	now    func() time.Time
+
+	// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
+	// (single tier via `pick`), which keeps every existing caller and test working untouched.
+	tiers    []string          // configured tier target ids, primary first
+	tierPick TierPicker        // newest archive on a named tier
+	rtState  *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
+	inFlight *InFlight         // shared with the backup path — Scenario F
+}
+
+// SchedulerOptions configures a Scheduler.
+type SchedulerOptions struct {
+	Runner RestoreTestRunner
+	Pick   CandidatePicker
+	Store  *Store
+	// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
+	// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
+	Spec SpecBuilder
+	// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
+	// (R-86). 0 → disabled.
+	Cadence time.Duration
+	// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
+	// 0 → no settle requirement (any archive is a candidate).
+	Settle time.Duration
+	Logger *slog.Logger
+
+	// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
+	// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
+	// named tier; State persists last-successful-per-tier; InFlight is the shared one-heavy-op gate.
+	Tiers    []string
+	TierPick TierPicker
+	State    *RestoreTestState
+	InFlight *InFlight
+}
+
+// NewScheduler builds a Scheduler.
+func NewScheduler(opts SchedulerOptions) *Scheduler {
+	logger := opts.Logger
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Scheduler{
+		runner:   opts.Runner,
+		pick:     opts.Pick,
+		store:    opts.Store,
+		spec:     opts.Spec,
+		cadence:  opts.Cadence,
+		settle:   opts.Settle,
+		logger:   logger,
+		now:      func() time.Time { return time.Now().UTC() },
+		tiers:    append([]string(nil), opts.Tiers...),
+		tierPick: opts.TierPick,
+		rtState:  opts.State,
+		inFlight: opts.InFlight,
+	}
+}
+
+// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
+// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
+//
+// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
+// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
+// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
+//
+// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
+// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
+// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
+// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
+// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
+//
+// Returns nil on ctx cancellation.
+func (s *Scheduler) Run(ctx context.Context) error {
+	if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
+		s.logger.Info("backup: restore-test cadence disabled")
+		<-ctx.Done()
+		return nil
+	}
+	s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
+		"eval_interval", s.cadence, "settle", s.settle)
+	t := time.NewTicker(s.cadence)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err())
+			return nil
+		case <-t.C:
+			s.tick(ctx)
+		}
+	}
+}
+
+// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
+// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
+// is already in flight. Deterministic given s.now — tests call it directly.
+//
+// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
+// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
+func (s *Scheduler) tick(ctx context.Context) {
+	if s.spec == nil {
+		// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
+		// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
+		// spec must cost a restore-test, never the agent.
+		s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
+		return
+	}
+	// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
+	//
+	// It used to be the other way round, and correctly so: the gate was held for one heavy run a
+	// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
+	// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
+	// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
+	// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
+	// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
+	// not be able to manufacture that.
+	//
+	// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
+	// before anything heavy begins.
+	archive, target, err := s.pickForThisRun(ctx)
+	if err != nil {
+		s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
+		return
+	}
+	if archive == "" {
+		// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
+		//
+		// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
+		// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
+		// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
+		// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
+		// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
+		// nothing run last night?" is in the log rather than in a re-derivation.
+		s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
+		return
+	}
+
+	// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
+	// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
+	// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
+	// never cancel what is already running: a deferred restore-test costs hours of coverage, a
+	// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
+	release, busy, ok := s.inFlight.TryAcquire("restore-test")
+	if !ok {
+		s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
+			"busy", busy, "target", target, "archive", archive)
+		return
+	}
+	defer release()
+	// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
+	spec := s.spec(ctx, archive)
+	spec.Archive = archive
+	res := s.runner.RunRestoreTest(ctx, spec)
+	if res.Skipped {
+		return // already logged by the engine (no free scratch VMID)
+	}
+	rt := ToHubRestoreTest(res, s.now())
+	s.store.RecordRestoreTest(rt)
+	// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
+	// that fails every time would look freshly proven and quietly stop being retried.
+	if rt.Pass && s.rtState != nil && target != "" {
+		// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
+		// not-due until a NEWER archive settles, and what makes a proof survive a restart.
+		// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
+		// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
+		// this run was actually judged as, and deriving it later would need a storage lookup that
+		// can fail on the one path where failing means mislabelling a proof.
+		if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
+			s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
+		}
+	}
+	switch {
+	case !rt.Pass:
+		// A failing restore-test is the loudest DR signal there is.
+		s.logger.Error("backup: scheduled restore-test FAILED", "archive", rt.SourceArchive, "err", rt.Error)
+	case len(res.StartWarnings) == 0:
+		s.logger.Info("backup: scheduled restore-test passed", "archive", rt.SourceArchive, "duration_s", rt.DurationSeconds)
+	case res.WarningsRecognized:
+		// Passed; the only warnings are the known-benign (e.g. systemd-nesting) advisory.
+		s.logger.Info("backup: scheduled restore-test passed with warnings (recognized)",
+			"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
+	default:
+		// Passed liveness, but an UNRECOGNIZED start warning stood out — worth an operator look.
+		s.logger.Warn("backup: scheduled restore-test passed with UNRECOGNIZED warnings",
+			"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
+	}
+}
+
+// rotating reports whether multi-tier rotation is wired.
+func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
+
+// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
+// order.
+//
+// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
+// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
+// due?", and "nothing" is a normal, frequent and correct answer.
+//
+// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
+// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
+// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
+// that a test happens at all.
+//
+// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
+// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
+// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
+// storage being unreadable must not cost the other tier its proof, and must not be silent either.
+//
+// Returns ("", "", nil) when nothing anywhere is due.
+func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
+	if !s.rotating() {
+		// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
+		// identity and no persisted proof here, so there is nothing to compare an archive against
+		// and no due-check is possible. It runs on every evaluation, exactly as it always did.
+		a, perr := s.pick(ctx)
+		return a, "", perr
+	}
+	var firstErr error
+	for _, v := range s.EvaluateDue(ctx) {
+		if v.Err != nil {
+			s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
+				"target", v.Target, "err", v.Err)
+			if firstErr == nil {
+				firstErr = v.Err
+			}
+			continue
+		}
+		if !v.Due {
+			s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
+			continue
+		}
+		s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
+			"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
+		return v.Archive, v.Target, nil
+	}
+	if firstErr != nil {
+		return "", "", firstErr
+	}
+	return "", "", nil
+}

internal/backup/schedule_spec_test.go

@@ -0,0 +1,124 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// R-85 (1.1) — the spec is built PER RUN, never frozen at construction.
+//
+// It used to be an immediately-invoked function at daemon start, so storageTier() and
+// restoreTaskTimeout() were evaluated ONCE and the value reused for every run for the process
+// lifetime. That is what made an offsite restore-test impossible to schedule at all, and it was a
+// latent staleness bug besides: a storage-type or config change did not take effect until restart.
+
+type specSpy struct {
+	mu       sync.Mutex
+	calls    int
+	archives []string
+	tiers    []string // what the builder decided, per call
+}
+
+func (sp *specSpy) build(_ context.Context, archive string) reconcile.RestoreTestSpec {
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	sp.calls++
+	sp.archives = append(sp.archives, archive)
+	// Decide the tier from the ARCHIVE, exactly as main.go does (the v0.100.0 rule).
+	tier := "local"
+	if len(archive) > 10 && archive[:10] == "felhom-pbs" {
+		tier = "pbs"
+	}
+	sp.tiers = append(sp.tiers, tier)
+	return reconcile.RestoreTestSpec{
+		RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: tier,
+	}
+}
+
+// COMPANION RED-PROOF (observed): change Scheduler.spec back to a frozen
+// `reconcile.RestoreTestSpec` value captured at construction → this fails with
+// "the spec builder must run ONCE PER RUN, got 1 call(s) across 3 ticks", because a frozen value is
+// evaluated exactly once no matter how many ticks fire. Restored.
+func TestScheduler_SpecIsBuiltPerRun(t *testing.T) {
+	sp := &specSpy{}
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
+	n := 0
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick: func(context.Context) (string, error) {
+			n++
+			return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", n), nil
+		},
+		Store:   NewStore(),
+		Spec:    sp.build,
+		Cadence: time.Hour,
+		Logger:  quiet(),
+	})
+
+	for i := 0; i < 3; i++ {
+		s.tick(context.Background())
+	}
+
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	if sp.calls != 3 {
+		t.Fatalf("the spec builder must run ONCE PER RUN, got %d call(s) across 3 ticks", sp.calls)
+	}
+	// And it must see the archive THIS run picked — not a stale one.
+	for i, a := range sp.archives {
+		want := fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", i+1)
+		if a != want {
+			t.Fatalf("run %d: builder saw archive %q, want %q — the spec is not tracking the picked archive", i+1, a, want)
+		}
+	}
+}
+
+// The tier must follow the ARCHIVE across runs. A builder that saw only the configured target would
+// return the same tier every time — which is exactly the v0.100.0 defect that killed a 14.46 GB WAN
+// restore at the 10-minute local bound.
+func TestScheduler_SpecTierFollowsTheArchive(t *testing.T) {
+	sp := &specSpy{}
+	rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
+	archives := []string{
+		"local:backup/vzdump-lxc-9201-x.tar.zst",
+		"felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
+	}
+	i := 0
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick: func(context.Context) (string, error) {
+			a := archives[i%len(archives)]
+			i++
+			return a, nil
+		},
+		Store: NewStore(), Spec: sp.build, Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background())
+	s.tick(context.Background())
+
+	sp.mu.Lock()
+	defer sp.mu.Unlock()
+	if len(sp.tiers) != 2 || sp.tiers[0] != "local" || sp.tiers[1] != "pbs" {
+		t.Fatalf("the tier must follow the archive per run; got %v", sp.tiers)
+	}
+}
+
+// A nil spec builder must SKIP loudly, not panic — a wiring bug costs a restore-test, never the
+// daemon goroutine.
+func TestScheduler_NilSpecSkipsInsteadOfPanicking(t *testing.T) {
+	rt := &fakeRTRunner{}
+	s := NewScheduler(SchedulerOptions{
+		Runner: rt,
+		Pick:   func(context.Context) (string, error) { return "vol", nil },
+		Store:  NewStore(), Cadence: time.Hour, Logger: quiet(),
+	})
+	s.tick(context.Background()) // must not panic
+	if rt.runs != 0 {
+		t.Fatalf("a nil spec must not run a restore-test; got %d run(s)", rt.runs)
+	}
+}

internal/backup/store.go

@@ -0,0 +1,75 @@
+package backup
+
+import (
+	"context"
+	"sync"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// Store holds the agent's LATEST backup result per target and the latest restore-test
+// result — the point-in-time state the host-report surfaces. It is updated by the backup
+// runner + the restore-test scheduler/selftest and read by the collector via the hub
+// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
+// collector vs scheduler access.
+//
+// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
+// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
+// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
+// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
+// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
+// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
+// consecutive host-reports carried `0 restore-tests`.
+//
+// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
+// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
+// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
+// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
+// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
+type Store struct {
+	mu       sync.Mutex
+	byTarget map[string]hub.Backup // latest backup per target id
+	lastTest *hub.RestoreTest
+}
+
+// NewStore builds an empty Store.
+func NewStore() *Store {
+	return &Store{byTarget: map[string]hub.Backup{}}
+}
+
+// RecordBackup stores the latest backup for its target.
+func (s *Store) RecordBackup(b hub.Backup) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.byTarget[b.TargetID] = b
+}
+
+// RecordRestoreTest stores the latest restore-test result.
+func (s *Store) RecordRestoreTest(r hub.RestoreTest) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	cp := r
+	s.lastTest = &cp
+}
+
+// Backups implements hub.BackupReporter — the latest backup per target (stable order by
+// target id is not guaranteed; the hub does not depend on order).
+func (s *Store) Backups(context.Context) []hub.Backup {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make([]hub.Backup, 0, len(s.byTarget))
+	for _, b := range s.byTarget {
+		out = append(out, b)
+	}
+	return out
+}
+
+// RestoreTests implements hub.RestoreTestReporter — the latest restore-test result (0 or 1).
+func (s *Store) RestoreTests(context.Context) []hub.RestoreTest {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.lastTest == nil {
+		return []hub.RestoreTest{}
+	}
+	return []hub.RestoreTest{*s.lastTest}
+}

internal/capability/manifest.go

@@ -0,0 +1,195 @@
+// Package capability is the agent's privileged-capability self-check (slice 1 of agent
+// self-health). It declares the MANIFEST — the (binary, representative-arg-vector) pairs the
+// non-root agent depends on running via `sudo -n` — and a PROBE that lists each against the live
+// sudoers policy (`sudo -n -l`, never executing) + checks the binary exists. The result is a
+// snapshot the agent attaches to its hub report; the hub owns the ok→degraded transition + alert.
+//
+// Why this exists: the 2026-06-28 root→non-root cutover dropped several grants from
+// configs/felhom-agent.sudoers (lxc-info, make-private, restart dnsmasq, …). Each broke a feature
+// silently until a user hit it (the multi-drive flapping incident, audit 2026-06-29). A non-root
+// agent that can't run a command it depends on is DEGRADED and must SAY so — at cutover, not days
+// later. The companion build-time test (manifest_test.go) asserts every manifest vector is covered
+// by a sudoers pattern, catching authoring gaps in CI before they ship.
+package capability
+
+import (
+	"strings"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
+)
+
+// Capability is one privileged command the agent depends on. Name is a stable id; Feature is the
+// human-readable thing that breaks if the grant is missing (used in logs + the operator alert).
+// Binary is the absolute path the runner invokes; ReprArgs is a CONCRETE argument vector that
+// matches the corresponding sudoers glob (e.g. a vmid "9201" matches `[0-9]*`, a device "/dev/sda"
+// matches `/dev/*`). Critical marks the user-facing ones — the hub alerts only when a Critical
+// capability is degraded (non-critical degradations still ride the report snapshot + agent log).
+type Capability struct {
+	Name     string
+	Feature  string
+	Binary   string
+	ReprArgs []string
+	Critical bool
+	// GatedBy names the config gate this capability sits behind ("" = always active). A healthy
+	// probe of a gated capability whose gate is OFF reports StatusInactive instead of ok — see
+	// Prober.GateActive. Broken plumbing stays degraded regardless of the gate.
+	GatedBy string
+}
+
+// GatePBSDR is the gate id for the PBS-DR tier: active when this box carries a live (enabled)
+// pbs_dr descriptor or a previously-converged marker (pbsdr.Manager.DRConfigured).
+const GatePBSDR = "pbs_dr"
+
+// Manifest is the required set, seeded from the 2026-06-29 sudoers audit (felhom-agent/REPORT.md):
+// the OK + newly-CLOSED rows. The SURFACED/DEFERRED rows are deliberately EXCLUDED — they are not
+// required capabilities: the general `pct exec <vmid> -- *` (controller-swap; arbitrary exec, an
+// open operator decision), `pct create` (golden build, maintenance, no daemon caller), `mount
+// UUID=…` (legacy/unreferenced), and the callerless `sensors -j`. Adding them here would assert
+// grants the agent neither has nor should depend on.
+//
+// Each ReprArgs is a representative instance; the probe LISTS it (`sudo -n -l`) and never runs it,
+// so even mkfs/pct-set entries are side-effect-free to probe.
+//
+// Gating (v0.86.0): every `pbsdr-*` capability is behind GatePBSDR — the whole pbsdr vector IS
+// the DR-tier apply path, so the gate rides the stable name prefix (a new pbsdr-* op is gated
+// automatically; nothing else ever is). Applied here, not in the literals, to keep the manifest
+// entries positional.
+func Manifest() []Capability {
+	out := make([]Capability, len(manifest))
+	copy(out, manifest)
+	for i := range out {
+		if strings.HasPrefix(out[i].Name, "pbsdr-") {
+			out[i].GatedBy = GatePBSDR
+		}
+	}
+	return out
+}
+
+// reprFingerprint is a shape-valid all-zero SHA-256 colon fingerprint for list-mode repr vectors
+// (matches the wrapper's fingerprint validation; never executed).
+const reprFingerprint = "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00"
+
+var manifest = []Capability{
+	// ---- Intermediary drive model (the multi-drive path — mostly Critical) ----
+	{"guest-init-pid", "drive-gate guest-sees check (multi-drive concurrency)", "/usr/bin/lxc-info", []string{"-n", "9201", "-p", "-H"}, true, ""},
+	{"parent-self-bind", "intermediary shared-parent self-bind", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-drives", "/mnt/felhom-drives"}, true, ""},
+	{"parent-make-shared", "intermediary shared-parent propagation", "/usr/bin/mount", []string{"--make-shared", "/mnt/felhom-drives"}, true, ""},
+	{"parent-make-private", "intermediary shared-parent peer-group isolation", "/usr/bin/mount", []string{"--make-private", "/mnt/felhom-drives"}, true, ""},
+	{"drive-bind", "drive attach (felhom-data bind under parent)", "/usr/bin/mount", []string{"--bind", "/mnt/felhom-usb/felhom-data", "/mnt/felhom-drives/felhom-usb"}, true, ""},
+	{"drive-umount", "drive detach (fail-closed unmount)", "/usr/bin/umount", []string{"/mnt/felhom-drives/felhom-usb"}, true, ""},
+	{"drives-mkdir-parent", "stable parent dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives"}, false, ""},
+	{"drives-mkdir-sub", "per-drive stable dir create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-drives/felhom-usb"}, false, ""},
+	{"drives-mkdir-data", "felhom-data namespace create", "/usr/bin/mkdir", []string{"-p", "/mnt/felhom-usb/felhom-data"}, false, ""},
+	{"drives-chown-data", "felhom-data guest-root chown", "/usr/bin/chown", []string{"100000:100000", "/mnt/felhom-usb/felhom-data"}, false, ""},
+	{"parent-script-install", "shared-parent boot script install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-shared-parent-123456789.sh", "/usr/local/sbin/felhom-shared-parent.sh"}, false, ""},
+	{"parent-unit-install", "shared-parent boot unit install", "/usr/bin/install", []string{"-m", "0644", "--", "/tmp/felhom-shared-parent-123456789.service", "/etc/systemd/system/felhom-shared-parent.service"}, false, ""},
+	{"parent-unit-enable", "shared-parent boot-persistence enable", "/usr/bin/systemctl", []string{"enable", "felhom-shared-parent.service"}, false, ""},
+	{"parent-bind-mp8", "parent bind into guest at provision", "/usr/sbin/pct", []string{"set", "9201", "-mp8", "/mnt/felhom-drives"}, false, ""},
+
+	// ---- Disk inspect / format gate (Critical: the data-bearing classifier + format) ----
+	{"disk-blkid", "disk data-bearing classify (format gate)", "/usr/sbin/blkid", []string{"-p", "-o", "export", "/dev/sda"}, true, ""},
+	{"disk-lsblk", "disk topology read (format gate)", "/usr/bin/lsblk", []string{"-J", "-o", "NAME,FSTYPE,PTTYPE,MOUNTPOINT", "/dev/sda"}, true, ""},
+	{"disk-mkfs-ext4", "guarded format (ext4)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "ext4"}, true, ""},
+	{"disk-mkfs-xfs", "guarded format (xfs)", "/usr/local/sbin/felhom-mkfs-guarded", []string{"/dev/sda", "xfs"}, false, ""},
+	{"disk-smart", "disk SMART health read", "/usr/sbin/smartctl", []string{"-a", "-j", "/dev/sda"}, false, ""},
+	{"disk-lvs", "thin-pool usage read", "/usr/sbin/lvs", []string{"--reportformat", "json", "--units", "b", "-o", "lv_name,data_percent,metadata_percent", "--", "pve/data"}, false, ""},
+
+	// ---- Storage mount units (watchdog re-mount) ----
+	{"mount-unit-install", "fs-UUID mount unit install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0644", "--", "/var/lib/felhom-agent/units/felhom-x.mount", "/etc/systemd/system/felhom-x.mount"}, false, ""},
+	{"mount-daemon-reload", "systemd reload after unit write", "/usr/bin/systemctl", []string{"daemon-reload"}, false, ""},
+	{"mount-unit-enable", "mount unit enable", "/usr/bin/systemctl", []string{"enable", "--now", "--", "felhom-x.mount"}, false, ""},
+	{"mount-unit-disable", "mount unit disable", "/usr/bin/systemctl", []string{"disable", "--", "felhom-x.mount"}, false, ""},
+	{"mount-unit-stop", "mount unit stop", "/usr/bin/systemctl", []string{"stop", "--", "felhom-x.mount"}, false, ""},
+
+	// ---- Network storage re-arm + cleanup (CAMPAIGN-3 F10/F1) ----
+	{"netmount-reset-failed", "NAS automount re-arm after start-limit (F10)", "/usr/bin/systemctl", []string{"reset-failed", "--", "mnt-felhom\\x2ddrives-media.automount"}, false, ""},
+	{"netmount-rmdir", "removed-share mountpoint cleanup (F1)", "/usr/bin/rmdir", []string{"/mnt/felhom-drives/media"}, false, ""},
+
+	// ---- Node self-heal (CAMPAIGN-3 F12-class, appliance-gated in code) ----
+	{"selfheal-networking-start", "appliance networking recovery at boot (F12 defense in depth)", "/usr/bin/systemctl", []string{"start", "networking.service"}, false, ""},
+
+	// ---- Provisioning back-half ----
+	{"provision-chown", "bootstrap mount guest-root chown", "/usr/bin/chown", []string{"-R", "100000:100000", "/var/lib/felhom-agent/guests/9201"}, false, ""},
+	{"provision-config-mount", "bootstrap config bind mount", "/usr/sbin/pct", []string{"set", "9201", "-mp0", "/var/lib/felhom-agent/guests/9201"}, false, ""},
+	{"provision-onboot", "customer guest autostart (onboot)", "/usr/sbin/pct", []string{"set", "9201", "-onboot", "1"}, false, ""},
+
+	// ---- Pre-start self-heal hook + guest lifecycle ----
+	{"guesthook-install", "pre-start hook snippet install", "/usr/bin/install", []string{"-m", "0755", "--", "/tmp/felhom-guest-hook-123456789.sh", "/var/lib/vz/snippets/felhom-guest-hook.sh"}, false, ""},
+	{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
+	{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
+	{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
+
+	// ---- LAN split-horizon resolver (dnsmasq) ----
+	{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
+	{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
+	{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
+	{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
+	{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
+	{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
+	{"dnsmasq-guest-ip", "guest LAN IP discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "-4", "-o", "addr", "show", "dev", "eth0"}, false, ""},
+	{"dnsmasq-guest-domain", "guest domain discovery", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "exec", "felhom-controller", "cat", "/opt/docker/felhom-controller/controller.yaml"}, false, ""},
+
+	// ---- Guest-network watchdog (FELHOM_GUESTNET, R-54, v0.92.0) ----
+	// Deliberately NOT Critical: a host that has not yet taken the new sudoers file should show
+	// degraded (visible, diagnosable) without paging an operator for every box on rollout day —
+	// the R-50b lesson that an amber fleet is background noise. The address read is covered by
+	// dnsmasq-guest-ip above and is not duplicated here (one command, one capability row).
+	{"guestnet-route", "guest default-route probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "ip", "route", "show", "default"}, false, ""},
+	{"guestnet-ifaces", "guest interface-mode read", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/network/interfaces"}, false, ""},
+	{"guestnet-dhclient-probe", "guest DHCP-client liveness probe", "/usr/sbin/pct", []string{"exec", "9201", "--", "pgrep", "-x", "dhclient"}, false, ""},
+	{"guestnet-heal", "guest DHCP-client restart (the 2026-07-20 heal)", "/usr/sbin/pct", []string{"exec", "9201", "--", "dhclient", "-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"}, false, ""},
+
+	// ---- Controller-swap / managed auto-update (FELHOM_CONTROLLERSWAP, v0.45.0; Critical: a
+	// silently-broken fleet auto-update is operator-alert-worthy) ----
+	{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
+	{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
+	{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
+	{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
+	{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
+
+	// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
+	// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
+	{"stalelock-unlock", "reboot-during-backup stale-lock recovery", "/usr/sbin/pct", []string{"unlock", "9201"}, true, ""},
+
+	// ---- Offsite WG tunnel (FELHOM_WG, S3/v0.64.0; Critical FLIPPED in S4/v0.66.0 — offsite
+	// backups now RIDE the tunnel, so a degraded tunnel capability is operator-alert-worthy: the
+	// conf install, unit enable/restart and the handshake read gate the backup path. apt-install
+	// (one-time bootstrap) and disable (revocation, a deliberate teardown) stay non-critical. The
+	// handshake read is the ONLY wg invocation (never `dump`). ----
+	{"wg-tools-install", "wireguard-tools package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "wireguard-tools"}, false, ""},
+	{"wg-conf-install", "wg-felhom conf install", "/usr/bin/install", []string{"-o", "root", "-g", "root", "-m", "0600", "--", "/var/lib/felhom-agent/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"}, true, ""},
+	{"wg-enable", "wg-quick@wg-felhom enable", "/usr/bin/systemctl", []string{"enable", "--now", "wg-quick@wg-felhom"}, true, ""},
+	{"wg-restart", "wg-quick@wg-felhom restart (conf change)", "/usr/bin/systemctl", []string{"restart", "wg-quick@wg-felhom"}, true, ""},
+	{"wg-disable", "wg-quick@wg-felhom disable (revocation)", "/usr/bin/systemctl", []string{"disable", "--now", "wg-quick@wg-felhom"}, false, ""},
+	{"wg-handshake-read", "tunnel handshake-age read", "/usr/bin/wg", []string{"show", "wg-felhom", "latest-handshakes"}, true, ""},
+
+	// ---- PBS DR tier apply (FELHOM_PBSDR, slice 2). NON-critical (the selfupdate rationale):
+	// applying the tier is an occasional hub-driven provisioning op, not a steady-state serving
+	// path — a degraded grant means "can't provision/reconcile the PBS entry" (the bridge reports
+	// loudly anyway), not a serving outage. The steady-state backup path is covered by the wg +
+	// storage capabilities. List-mode representations only; never executed. ----
+	{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
+	{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
+	{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false, ""},
+	// R-39 leg (b), v0.91.0: the credential READ path. Its absence is what made the PBS verify loop
+	// permanently blind to an applied-but-401 tier, so a host missing this verb is DEGRADED in a way
+	// that matters — it cannot detect the failure this whole tier exists to survive.
+	{"pbsdr-read", "PBS DR credential read (verify-loop auth probe)", "/usr/local/sbin/felhom-pbs-apply", []string{"read", "felhom-pbs", "/etc/pve/priv/storage"}, false, ""},
+
+	// ---- Escrow ceremony (FELHOM_ESCROW, controller-driven, v0.88.0). Critical: the customer
+	// wizard's whole run path IS this one grant — a dropped line silently breaks every ceremony.
+	// GatedBy is set EXPLICITLY (the name deliberately says "escrow", not "pbsdr-": the feature is
+	// the ceremony, but it only exists behind the DR tier — no PBS key, no ceremony). ReprArgs is
+	// the SHARED argv constant (internal/escrow/ceremony.go) — the exec runner uses the same one,
+	// so runner ↔ manifest can't drift, and TestManifestCoveredBySudoers locks manifest ↔ sudoers.
+	// List-mode probe only (`sudo -n -l`), spike-confirmed side-effect-free — never a real ceremony.
+	{"escrow-ceremony", "customer recovery-code ceremony (controller-driven)", escrow.CeremonyBinary, escrow.CeremonyArgs(), true, GatePBSDR},
+
+	// ---- Agent self-update (FELHOM_SELFUPDATE, D1). NON-critical: self-update is an occasional
+	// operator-driven op, not a steady-state serving path — a degraded grant means "can't
+	// self-update" (fall back to a manual SSH deploy), not a serving outage. The apply repr uses a
+	// staging-dir path + a placeholder sha (list-mode never runs it). ----
+	{"selfupdate-apply", "agent self-update apply (A/B flip)", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"apply", "/var/lib/felhom-agent/selfupdate/felhom-agent-0.0.0", "0000000000000000000000000000000000000000000000000000000000000000"}, false, ""},
+	{"selfupdate-commit", "agent self-update commit", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"commit"}, false, ""},
+	{"selfupdate-rollback", "agent self-update rollback", "/usr/local/sbin/felhom-selfupdate-guarded", []string{"rollback"}, false, ""},
+}

internal/capability/manifest_test.go

@@ -0,0 +1,296 @@
+package capability
+
+import (
+	"os"
+	"reflect"
+	"regexp"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
+)
+
+// sudoersPath is the in-repo allowlist, relative to this test file (internal/capability/).
+const sudoersPath = "../../configs/felhom-agent.sudoers"
+
+// parseSudoersEntries returns every command pattern from the Cmnd_Alias blocks, with the sudoers
+// escapes (`\,` `\:`) unescaped. It joins continuation lines and splits the alias RHS on commas
+// that are NOT backslash-escaped (escaped commas are literal arg chars, e.g. the lvs `-o` list).
+func parseSudoersEntries(t *testing.T, text string) []string {
+	t.Helper()
+	// 1. Collapse line continuations, keeping only Cmnd_Alias RHS text.
+	var rhs strings.Builder
+	lines := strings.Split(text, "\n")
+	inAlias := false
+	for _, ln := range lines {
+		trimmed := strings.TrimSpace(ln)
+		if strings.HasPrefix(trimmed, "#") {
+			continue
+		}
+		if strings.HasPrefix(trimmed, "Cmnd_Alias ") {
+			inAlias = true
+			if eq := strings.IndexByte(trimmed, '='); eq >= 0 {
+				trimmed = trimmed[eq+1:]
+			}
+		} else if !inAlias {
+			continue
+		}
+		// The final NOPASSWD line ("felhom-agent ALL=...") ends the alias region.
+		if strings.Contains(trimmed, "ALL=(") {
+			inAlias = false
+			continue
+		}
+		cont := strings.HasSuffix(trimmed, "\\")
+		rhs.WriteString(strings.TrimSuffix(trimmed, "\\"))
+		rhs.WriteString(" ")
+		if !cont {
+			// A non-continued line is the last entry of this alias. Emit a comma so it does not
+			// merge with the next alias's first entry when all RHS text is concatenated.
+			rhs.WriteString(", ")
+			inAlias = false
+		}
+	}
+	// 2. Split on unescaped commas → individual command entries.
+	raw := rhs.String()
+	var entries []string
+	var cur strings.Builder
+	for i := 0; i < len(raw); i++ {
+		c := raw[i]
+		if c == '\\' && i+1 < len(raw) {
+			cur.WriteByte(raw[i+1]) // unescape: keep the next char literally (\, → , ; \: → :)
+			i++
+			continue
+		}
+		if c == ',' {
+			entries = appendTrimmed(entries, cur.String())
+			cur.Reset()
+			continue
+		}
+		cur.WriteByte(c)
+	}
+	entries = appendTrimmed(entries, cur.String())
+	return entries
+}
+
+func appendTrimmed(entries []string, s string) []string {
+	if t := strings.Join(strings.Fields(s), " "); t != "" {
+		return append(entries, t)
+	}
+	return entries
+}
+
+// globToRegex translates a sudoers fnmatch pattern to an anchored regex. It is NOT a perfect sudo
+// emulator — it only needs to catch a removed/renamed grant (the real failure mode). `*` → `.*`,
+// `[...]` char classes pass through (valid regex), regex metachars are escaped.
+func globToRegex(pat string) *regexp.Regexp {
+	var b strings.Builder
+	b.WriteString("^")
+	for i := 0; i < len(pat); i++ {
+		c := pat[i]
+		switch {
+		case c == '*':
+			b.WriteString(".*")
+		case c == '[': // copy the char class verbatim (valid in regex too)
+			if j := strings.IndexByte(pat[i:], ']'); j > 0 {
+				b.WriteString(pat[i : i+j+1])
+				i += j
+				continue
+			}
+			b.WriteString("\\[")
+		case strings.IndexByte(`.+()|{}^$\?`, c) >= 0:
+			b.WriteByte('\\')
+			b.WriteByte(c)
+		default:
+			b.WriteByte(c)
+		}
+	}
+	b.WriteString("$")
+	return regexp.MustCompile(b.String())
+}
+
+// matchesAny reports whether cmdline matches at least one sudoers entry pattern.
+func matchesAny(cmdline string, entries []string) bool {
+	for _, e := range entries {
+		if globToRegex(e).MatchString(cmdline) {
+			return true
+		}
+	}
+	return false
+}
+
+// TestManifestCoveredBySudoers is the headline build-time gate: EVERY manifest capability's
+// representative command line must be permitted by at least one sudoers pattern. This is exactly
+// the check that would have caught the lxc-info / make-private grants being dropped at the
+// 2026-06-28 cutover — in CI, before shipping.
+func TestManifestCoveredBySudoers(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers %s: %v", sudoersPath, err)
+	}
+	entries := parseSudoersEntries(t, string(data))
+	if len(entries) < 20 {
+		t.Fatalf("parsed only %d sudoers entries — parser likely broke", len(entries))
+	}
+	for _, c := range Manifest() {
+		cmdline := strings.TrimSpace(c.Binary + " " + strings.Join(c.ReprArgs, " "))
+		if !matchesAny(cmdline, entries) {
+			t.Errorf("capability %q (%s) NOT covered by any sudoers grant:\n  %s",
+				c.Name, c.Feature, cmdline)
+		}
+	}
+}
+
+// TestRedProof_DroppedGrantFailsCheck is the companion red-proof: with the lxc-info line removed
+// from an in-memory copy of the sudoers, the coverage check for guest-init-pid MUST fail. Proves
+// the build gate actually catches the regression (a green test that can never go red is hollow).
+func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers: %v", err)
+	}
+	// Drop the lxc-info grant line.
+	var kept []string
+	for _, ln := range strings.Split(string(data), "\n") {
+		if strings.Contains(ln, "lxc-info") {
+			continue
+		}
+		kept = append(kept, ln)
+	}
+	mutated := strings.Join(kept, "\n")
+	if strings.Contains(mutated, "lxc-info") {
+		t.Fatal("setup: lxc-info line not removed")
+	}
+	entries := parseSudoersEntries(t, mutated)
+
+	var guestInit Capability
+	for _, c := range Manifest() {
+		if c.Name == "guest-init-pid" {
+			guestInit = c
+		}
+	}
+	if guestInit.Name == "" {
+		t.Fatal("manifest missing guest-init-pid")
+	}
+	cmdline := guestInit.Binary + " " + strings.Join(guestInit.ReprArgs, " ")
+	if matchesAny(cmdline, entries) {
+		t.Errorf("red-proof FAILED: guest-init-pid still matches after dropping the lxc-info grant — the build gate would NOT catch the regression")
+	}
+
+	// Sanity: the UNMUTATED file MUST cover it (so the failure above is specific to the drop).
+	full := parseSudoersEntries(t, string(data))
+	if !matchesAny(cmdline, full) {
+		t.Errorf("guest-init-pid should be covered by the real sudoers")
+	}
+}
+
+// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
+// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
+// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
+// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
+func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
+	data, err := os.ReadFile(sudoersPath)
+	if err != nil {
+		t.Fatalf("read sudoers: %v", err)
+	}
+	var kept []string
+	for _, ln := range strings.Split(string(data), "\n") {
+		if strings.Contains(ln, "tee /etc/felhom-controller-image") {
+			continue
+		}
+		kept = append(kept, ln)
+	}
+	mutated := strings.Join(kept, "\n")
+	entries := parseSudoersEntries(t, mutated)
+
+	var write Capability
+	for _, c := range Manifest() {
+		if c.Name == "controllerswap-write" {
+			write = c
+		}
+	}
+	if write.Name == "" {
+		t.Fatal("manifest missing controllerswap-write")
+	}
+	cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
+	if matchesAny(cmdline, entries) {
+		t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
+	}
+	if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
+		t.Errorf("controllerswap-write should be covered by the real sudoers")
+	}
+}
+
+// TestEscrowCeremonyArgvPinned locks the ceremony argv copies together (Scenario G, v0.88.0).
+// The exec runner and the manifest entry both consume escrow.CeremonyArgs() (one shared source),
+// and TestManifestCoveredBySudoers proves manifest ⊆ sudoers — so pinning the shared source to
+// the EXPECTED literal here transitively locks all three: runner == manifest == sudoers.
+// Red-proof: mutate one element of the argv in internal/escrow/ceremony.go and THIS test fails
+// (and so does the sudoers coverage); a sudoers-side mutation is caught by the existing
+// TestRedProof_* machinery.
+func TestEscrowCeremonyArgvPinned(t *testing.T) {
+	wantBinary := "/usr/local/bin/felhom-agent"
+	wantArgs := []string{"--config", "/etc/felhom-agent/agent.json", "--selftest=escrow-create", "--upload", "--output=json"}
+
+	if escrow.CeremonyBinary != wantBinary {
+		t.Errorf("escrow.CeremonyBinary = %q, want %q", escrow.CeremonyBinary, wantBinary)
+	}
+	if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
+		t.Errorf("escrow.CeremonyArgs() = %q, want %q (the sudoers line + manifest entry must stay byte-identical)", got, wantArgs)
+	}
+
+	var entry Capability
+	for _, c := range Manifest() {
+		if c.Name == "escrow-ceremony" {
+			entry = c
+		}
+	}
+	if entry.Name == "" {
+		t.Fatal("manifest missing escrow-ceremony")
+	}
+	if entry.Binary != escrow.CeremonyBinary || !reflect.DeepEqual(entry.ReprArgs, escrow.CeremonyArgs()) {
+		t.Errorf("manifest escrow-ceremony argv diverged from the shared constant: %s %q", entry.Binary, entry.ReprArgs)
+	}
+	if !entry.Critical {
+		t.Error("escrow-ceremony must be Critical (the wizard's whole run path is this one grant)")
+	}
+	if entry.GatedBy != GatePBSDR {
+		t.Errorf("escrow-ceremony GatedBy = %q, want %q (no PBS key → no ceremony; inactive, never red, on a DR-off box)", entry.GatedBy, GatePBSDR)
+	}
+	// CeremonyArgs must return a COPY — a caller mutating its slice must not poison the source.
+	mutated := escrow.CeremonyArgs()
+	mutated[0] = "--poisoned"
+	if got := escrow.CeremonyArgs(); !reflect.DeepEqual(got, wantArgs) {
+		t.Error("escrow.CeremonyArgs() shares its backing array — callers can mutate the source")
+	}
+}
+
+// TestWGCapabilityCriticality pins the exact S4 (v0.66.0) Critical set for the FELHOM_WG entries:
+// the backup path (conf install, unit enable/restart, handshake read) is operator-alert-worthy now
+// that offsite backups ride the tunnel; the one-time apt install and the deliberate disable
+// (revocation) are NOT. Red-proof: flip any one entry's Critical in manifest.go and this fails.
+func TestWGCapabilityCriticality(t *testing.T) {
+	wantCritical := map[string]bool{
+		"wg-tools-install":  false,
+		"wg-conf-install":   true,
+		"wg-enable":         true,
+		"wg-restart":        true,
+		"wg-disable":        false,
+		"wg-handshake-read": true,
+	}
+	seen := map[string]bool{}
+	for _, c := range Manifest() {
+		want, ok := wantCritical[c.Name]
+		if !ok {
+			continue
+		}
+		seen[c.Name] = true
+		if c.Critical != want {
+			t.Errorf("%s: Critical = %v, want %v", c.Name, c.Critical, want)
+		}
+	}
+	for name := range wantCritical {
+		if !seen[name] {
+			t.Errorf("manifest missing wg capability %q", name)
+		}
+	}
+}

internal/capability/probe.go

@@ -0,0 +1,122 @@
+package capability
+
+import (
+	"context"
+	"os"
+)
+
+// Status is one capability's live result — the wire shape the agent attaches to its hub report
+// (HostReport.Capabilities). The hub mirrors this struct field-for-field and keys its alert on
+// Critical+degraded. Reason is empty when ok.
+//
+// "inactive" (v0.86.0, DR-tier-by-default): a config-GATED capability whose plumbing is HEALTHY
+// (binary present, sudo granted) but whose gating feature is disabled by configuration. Distinct
+// from degraded on purpose — disabled ≠ broken; the hub renders it as a neutral chip, never red.
+// Broken plumbing (binary missing / grant denied) stays DEGRADED even when the gate is off: an
+// un-migrated box must never look deliberately disabled.
+type Status struct {
+	Name     string `json:"name"`
+	Feature  string `json:"feature"`
+	Critical bool   `json:"critical"`
+	Status   string `json:"status"` // "ok" | "degraded" | "inactive"
+	Reason   string `json:"reason,omitempty"`
+}
+
+const (
+	StatusOK       = "ok"
+	StatusDegraded = "degraded"
+	StatusInactive = "inactive"
+)
+
+// ReasonInactive is the fixed reason string for the inactive state (the hub + operator docs
+// reference it verbatim).
+const ReasonInactive = "disabled by configuration"
+
+// Runner is the minimal exec seam the probe needs (satisfied by proxmox.ExecRunner). The probe
+// runs `sudo -n -l -- <binary> <args…>` LITERALLY — a sudo POLICY LIST that never executes the
+// command — so the Runner MUST be a DIRECT runner (RunnerDirect), not the sudo-prepending one
+// (else it would double-sudo). exit 0 ⇔ the command is permitted under the NOPASSWD allowlist.
+type Runner interface {
+	Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
+}
+
+// Prober checks the manifest against the live host. Exists defaults to an os.Stat check on the
+// absolute binary path (what `command -v` would resolve for an absolute path) when nil.
+// GateActive answers "is the feature behind this gate id configured on?" for GATED capabilities
+// (Capability.GatedBy). nil, or a gate it answers true for, keeps the historical behavior; false
+// downgrades a HEALTHY probe to StatusInactive (broken plumbing stays degraded regardless).
+type Prober struct {
+	Runner     Runner
+	Exists     func(path string) bool  // nil → os.Stat
+	GateActive func(gate string) bool // nil → every gate treated active
+}
+
+// Probe lists every manifest capability against the sudo policy and checks its binary exists,
+// mapping to ok/degraded (§8 of the spec). It NEVER executes a probed command and NEVER returns a
+// fatal error (serve-degraded): a probe failure is reported, not raised. If sudo itself is
+// unavailable for the agent (the drop-in is missing / the user has no sudo at all), it collapses
+// to ONE aggregate degraded signal instead of N identical ones.
+func (p Prober) Probe(ctx context.Context) []Status {
+	exists := p.Exists
+	if exists == nil {
+		exists = func(path string) bool { _, err := os.Stat(path); return err == nil }
+	}
+	caps := Manifest()
+
+	// Preflight: a bare `sudo -n -l` lists the user's allowed commands. For our NOPASSWD service
+	// user it exits 0; if it fails, the drop-in isn't installed (or sudo is gone) and EVERY vector
+	// would individually fail — collapse to one aggregate signal so the operator gets one alert.
+	if p.Runner != nil {
+		if _, _, err := p.Runner.Run(ctx, "sudo", "-n", "-l"); err != nil {
+			return []Status{{
+				Name:     "sudo",
+				Feature:  "the entire privileged surface (mount/format/pct/dnsmasq/lxc-info)",
+				Critical: true,
+				Status:   StatusDegraded,
+				Reason:   "sudoers drop-in not installed / sudo unavailable",
+			}}
+		}
+	}
+
+	out := make([]Status, 0, len(caps))
+	for _, c := range caps {
+		s := Status{Name: c.Name, Feature: c.Feature, Critical: c.Critical, Status: StatusOK}
+		switch {
+		case !exists(c.Binary):
+			s.Status, s.Reason = StatusDegraded, "binary not found"
+		case p.Runner != nil && !p.granted(ctx, c):
+			s.Status, s.Reason = StatusDegraded, "sudo policy denied"
+		}
+		// Config gate (v0.86.0): only a HEALTHY probe is downgraded to inactive — a degraded one
+		// stays degraded (missing binary/grant = un-migrated or mis-installed box, never "off").
+		if s.Status == StatusOK && c.GatedBy != "" && p.GateActive != nil && !p.GateActive(c.GatedBy) {
+			s.Status, s.Reason = StatusInactive, ReasonInactive
+		}
+		out = append(out, s)
+	}
+	return out
+}
+
+// granted reports whether `sudo -n -l -- <binary> <reprArgs…>` is permitted (exit 0). List-mode is
+// side-effect-free — the command is matched against the policy, never run.
+func (p Prober) granted(ctx context.Context, c Capability) bool {
+	args := append([]string{"-n", "-l", "--", c.Binary}, c.ReprArgs...)
+	_, _, err := p.Runner.Run(ctx, "sudo", args...)
+	return err == nil
+}
+
+// Summarize returns (okCount, total, degraded) for logging. degraded lists DEGRADED statuses
+// only — inactive is a deliberate, healthy state and must not land in the error log (it is
+// counted via len(statuses)-ok-len(degraded) by callers that want it).
+func Summarize(statuses []Status) (ok, total int, degraded []Status) {
+	total = len(statuses)
+	for _, s := range statuses {
+		switch s.Status {
+		case StatusOK:
+			ok++
+		case StatusDegraded:
+			degraded = append(degraded, s)
+		}
+	}
+	return ok, total, degraded
+}

internal/capability/probe_test.go

@@ -0,0 +1,202 @@
+package capability
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+)
+
+// fakeRunner returns a canned error per (command line) and records calls. deny holds binaries (or
+// the bare "sudo -n -l" preflight) that should fail; everything else exits 0.
+type fakeRunner struct {
+	preflightErr error
+	denyBinary   map[string]bool // binary path → policy-denied
+	calls        int
+	executedReal bool // set if a probed command was ever run WITHOUT -l (must never happen)
+}
+
+func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	f.calls++
+	// Preflight is `sudo -n -l` (exactly 2 args, no `--`).
+	if name == "sudo" && len(args) == 2 && args[0] == "-n" && args[1] == "-l" {
+		return nil, nil, f.preflightErr
+	}
+	// Every real probe must be a LIST: `sudo -n -l -- <binary> …`.
+	if name != "sudo" || len(args) < 4 || args[0] != "-n" || args[1] != "-l" || args[2] != "--" {
+		f.executedReal = true
+		return nil, nil, nil
+	}
+	binary := args[3]
+	if f.denyBinary[binary] {
+		return nil, nil, errors.New("sudo: a password is required")
+	}
+	return nil, nil, nil
+}
+
+func find(statuses []Status, name string) Status {
+	for _, s := range statuses {
+		if s.Name == name {
+			return s
+		}
+	}
+	return Status{}
+}
+
+// §7-A: all grants present + binaries exist → every capability ok.
+func TestProbe_AllOK(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+	ok, total, degraded := Summarize(statuses)
+	if total != len(Manifest()) {
+		t.Fatalf("total=%d want %d", total, len(Manifest()))
+	}
+	if ok != total || len(degraded) != 0 {
+		t.Fatalf("expected all ok, got %d/%d (degraded: %+v)", ok, total, degraded)
+	}
+	if r.executedReal {
+		t.Fatal("probe executed a command without -l (must be list-only)")
+	}
+}
+
+// §7-B: one grant denied → that capability degraded "sudo policy denied", others ok. Serve-degraded.
+func TestProbe_OneDenied(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{"/usr/bin/lxc-info": true}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+
+	gi := find(statuses, "guest-init-pid")
+	if gi.Status != StatusDegraded || gi.Reason != "sudo policy denied" {
+		t.Fatalf("guest-init-pid = %+v, want degraded/sudo policy denied", gi)
+	}
+	if !gi.Critical {
+		t.Fatal("guest-init-pid should be Critical")
+	}
+	// A sibling stays ok.
+	if s := find(statuses, "drive-bind"); s.Status != StatusOK {
+		t.Fatalf("drive-bind = %+v, want ok", s)
+	}
+	ok, total, _ := Summarize(statuses)
+	if ok != total-1 {
+		t.Fatalf("expected exactly one degraded, got ok=%d total=%d", ok, total)
+	}
+}
+
+// §7-D: binary missing but policy granted → degraded "binary not found".
+func TestProbe_BinaryMissing(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(path string) bool { return path != "/usr/bin/lxc-info" }}
+	statuses := p.Probe(context.Background())
+	gi := find(statuses, "guest-init-pid")
+	if gi.Status != StatusDegraded || gi.Reason != "binary not found" {
+		t.Fatalf("guest-init-pid = %+v, want degraded/binary not found", gi)
+	}
+}
+
+// §8 aggregate: sudo itself unavailable for the user → ONE aggregate degraded, not N.
+func TestProbe_SudoUnavailableAggregates(t *testing.T) {
+	r := &fakeRunner{preflightErr: errors.New("Sorry, user felhom-agent may not run sudo"), denyBinary: map[string]bool{}}
+	p := Prober{Runner: r, Exists: func(string) bool { return true }}
+	statuses := p.Probe(context.Background())
+	if len(statuses) != 1 {
+		t.Fatalf("expected 1 aggregate status, got %d", len(statuses))
+	}
+	s := statuses[0]
+	if s.Name != "sudo" || s.Status != StatusDegraded || !s.Critical || !strings.Contains(s.Reason, "drop-in not installed") {
+		t.Fatalf("aggregate = %+v, want critical degraded sudo-unavailable", s)
+	}
+}
+
+// Probe must never raise — even with a nil runner (e.g. a dev path) it returns statuses.
+func TestProbe_NilRunnerNoPanic(t *testing.T) {
+	p := Prober{Runner: nil, Exists: func(string) bool { return true }}
+	if got := len(p.Probe(context.Background())); got != len(Manifest()) {
+		t.Fatalf("nil-runner probe returned %d statuses, want %d", got, len(Manifest()))
+	}
+}
+
+// ── DR-tier gate (v0.86.0) ─────────────────────────────────────────────────────────────────────
+
+// Gate OFF + healthy plumbing → the gated pbsdr-* capabilities report INACTIVE (the neutral
+// "disabled by configuration" state), NOT ok and NOT degraded — Scenario B of the DR-by-default
+// spec. Ungated siblings are untouched. Red-proof partner: collapse inactive into ok (drop the
+// gate branch in Probe) → this fails while TestProbe_GateOffBinaryMissingStaysDegraded passes.
+func TestProbe_GateOffHealthyIsInactive(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{
+		Runner:     r,
+		Exists:     func(string) bool { return true },
+		GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
+	}
+	statuses := p.Probe(context.Background())
+	// v0.88.0: escrow-ceremony joins the gate EXPLICITLY (non-pbsdr name, GatedBy literal) —
+	// the ceremony only exists behind the DR tier (no PBS key, no ceremony).
+	// v0.91.0: pbsdr-read (the R-39 credential-read verb) rides the same `pbsdr-` prefix gate — a new
+	// pbsdr-* op is gated by construction, which is exactly the property this list is here to hold.
+	for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant", "pbsdr-read", "escrow-ceremony"} {
+		s := find(statuses, name)
+		if s.Status != StatusInactive || s.Reason != ReasonInactive {
+			t.Fatalf("%s = %+v, want inactive/%q", name, s, ReasonInactive)
+		}
+	}
+	// An ungated sibling stays plain ok.
+	if s := find(statuses, "drive-bind"); s.Status != StatusOK {
+		t.Fatalf("drive-bind = %+v, want ok (ungated)", s)
+	}
+	// Summarize must NOT count inactive as degraded (it is not error-log-worthy).
+	ok, total, degraded := Summarize(statuses)
+	if len(degraded) != 0 {
+		t.Fatalf("inactive leaked into degraded: %+v", degraded)
+	}
+	if ok != total-5 {
+		t.Fatalf("ok=%d total=%d, want exactly the 5 gated ones non-ok", ok, total)
+	}
+}
+
+// Gate OFF + BROKEN plumbing (binary missing) → DEGRADED stays degraded. An un-migrated
+// pre-v1.15.0 box must never masquerade as deliberately disabled ("never silently pretend").
+func TestProbe_GateOffBinaryMissingStaysDegraded(t *testing.T) {
+	r := &fakeRunner{denyBinary: map[string]bool{}}
+	p := Prober{
+		Runner:     r,
+		Exists:     func(path string) bool { return path != "/usr/local/sbin/felhom-pbs-apply" },
+		GateActive: func(gate string) bool { return gate != GatePBSDR }, // DR tier OFF
+	}
+	statuses := p.Probe(context.Background())
+	for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant"} {
+		s := find(statuses, name)
+		if s.Status != StatusDegraded || s.Reason != "binary not found" {
+			t.Fatalf("%s = %+v, want degraded/binary not found even with the gate off", name, s)
+		}
+	}
+}
+
+// Gate ON (DR configured) + healthy plumbing → plain ok, exactly the pre-v0.86.0 behavior.
+// A nil GateActive must behave the same (fails ACTIVE).
+func TestProbe_GateOnOrNilIsOK(t *testing.T) {
+	for _, gate := range []func(string) bool{nil, func(string) bool { return true }} {
+		r := &fakeRunner{denyBinary: map[string]bool{}}
+		p := Prober{Runner: r, Exists: func(string) bool { return true }, GateActive: gate}
+		statuses := p.Probe(context.Background())
+		if s := find(statuses, "pbsdr-create"); s.Status != StatusOK {
+			t.Fatalf("pbsdr-create = %+v, want ok (gate active/nil)", s)
+		}
+	}
+}
+
+// The gate covers exactly the pbsdr-* entries (name-prefix mechanism) PLUS escrow-ceremony (an
+// explicit GatedBy literal — v0.88.0: the ceremony only exists behind the DR tier, but its name
+// says what the feature is). Nothing else may be gated (a regression here would silently un-gate
+// the tier or gate an unrelated capability).
+func TestManifest_ExactlyPBSDRGated(t *testing.T) {
+	for _, c := range Manifest() {
+		wantGated := strings.HasPrefix(c.Name, "pbsdr-") || c.Name == "escrow-ceremony"
+		if gated := c.GatedBy == GatePBSDR; gated != wantGated {
+			t.Fatalf("%s: GatedBy=%q, want gated=%v", c.Name, c.GatedBy, wantGated)
+		}
+		if c.GatedBy != "" && c.GatedBy != GatePBSDR {
+			t.Fatalf("%s: unknown gate id %q", c.Name, c.GatedBy)
+		}
+	}
+}

internal/config/backup_tiers_test.go

@@ -0,0 +1,182 @@
+package config
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+)
+
+// R-82 Slice A.1 — per-target cadence + retention resolution.
+//
+// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
+// resolve to exactly one tier that behaves as it does now. The second property is that a
+// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
+// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
+// R-82 exists to fix.
+
+func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
+	// Exactly the shape live on demo-felhom today.
+	var b BackupConfig
+	raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
+	if err := json.Unmarshal([]byte(raw), &b); err != nil {
+		t.Fatal(err)
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(warnings) != 0 {
+		t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
+	}
+	if len(tiers) != 1 {
+		t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
+	}
+	got := tiers[0]
+	if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
+		t.Fatalf("legacy tier changed: %+v", got)
+	}
+}
+
+// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
+// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
+func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
+	tiers, warnings := BackupConfig{}.BackupTiers()
+	if len(warnings) != 0 || len(tiers) != 1 {
+		t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
+	}
+	if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
+		t.Fatalf("zero-config defaults changed: %+v", tiers[0])
+	}
+}
+
+// The whole point: local daily + PBS weekly, each with its OWN retention.
+func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
+	var b BackupConfig
+	raw := `{
+	  "local_backup_target":"local",
+	  "local_backup_retention":3,
+	  "backup_cadence_seconds":86400,
+	  "backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
+	}`
+	if err := json.Unmarshal([]byte(raw), &b); err != nil {
+		t.Fatal(err)
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(warnings) != 0 {
+		t.Fatalf("unexpected warnings: %v", warnings)
+	}
+	if len(tiers) != 2 {
+		t.Fatalf("want 2 tiers, got %+v", tiers)
+	}
+	if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
+		t.Fatalf("primary tier wrong: %+v", tiers[0])
+	}
+	if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
+		t.Fatalf("PBS tier wrong: %+v", tiers[1])
+	}
+	// THE knob-sharing check: the two retentions are independent values, not one shared number.
+	if tiers[0].KeepLast == tiers[1].KeepLast {
+		t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
+	}
+}
+
+// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
+// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
+func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(tiers) != 1 {
+		t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
+	}
+	if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
+		t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
+	}
+	if !strings.Contains(warnings[0], "felhom-pbs") {
+		t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
+	}
+}
+
+func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets: []BackupTargetConfig{
+			{TargetID: "", CadenceSeconds: 3600},
+			{TargetID: "local", CadenceSeconds: 3600},        // repeats the primary
+			{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
+			{TargetID: "felhom-pbs", CadenceSeconds: 99},     // duplicate
+		},
+	}
+	tiers, warnings := b.BackupTiers()
+	if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
+		t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
+	}
+	if len(warnings) != 3 {
+		t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
+	}
+}
+
+// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
+// itself because someone forgot a field.
+func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
+	}
+	tiers, _ := b.BackupTiers()
+	if len(tiers) != 2 {
+		t.Fatalf("got %+v", tiers)
+	}
+	if tiers[1].KeepLast != 0 {
+		t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
+	}
+	// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
+	b.ExtraTargets[0].KeepLast = -5
+	tiers, _ = b.BackupTiers()
+	if tiers[1].KeepLast != 0 {
+		t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
+	}
+}
+
+// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
+func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
+	for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
+		b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
+		tiers, _ := b.BackupTiers()
+		if tiers[0].KeepLast != tc.want {
+			t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
+		}
+	}
+}
+
+// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
+// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
+// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
+// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
+// makes the next attempt collide with the guest lock vzdump still holds.
+func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
+	b := BackupConfig{
+		LocalBackupTarget: "local",
+		ExtraTargets:      []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
+	}
+	tiers, _ := b.BackupTiers()
+	if len(tiers) != 2 {
+		t.Fatalf("got %+v", tiers)
+	}
+	if tiers[0].WaitTimeout != 30*time.Minute {
+		t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
+	}
+	if tiers[1].WaitTimeout != 12*time.Hour {
+		t.Fatalf("an offsite tier must default to a GENEROUS wait (operator ruling: let the first backup run as long as needed) — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
+	}
+	// And it must be overridable per tier.
+	b.ExtraTargets[0].WaitTimeoutSeconds = 3600
+	tiers, _ = b.BackupTiers()
+	if tiers[1].WaitTimeout != time.Hour {
+		t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
+	}
+	// The two tiers must NOT share one bound.
+	if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
+		t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
+	}
+}

internal/config/config_test.go

@@ -0,0 +1,216 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+// TestRestoreTestPBSRestoreTimeout mirrors the BackupCadence accessor contract: positive as-is,
+// 0 → default (120m), negative → default.
+func TestRestoreTestPBSRestoreTimeout(t *testing.T) {
+	cases := []struct {
+		secs int
+		want time.Duration
+	}{
+		{0, 120 * time.Minute},
+		{-5, 120 * time.Minute},
+		{1800, 30 * time.Minute},
+		{7200, 120 * time.Minute},
+	}
+	for _, c := range cases {
+		got := BackupConfig{RestoreTestPBSRestoreTimeoutSeconds: c.secs}.RestoreTestPBSRestoreTimeout()
+		if got != c.want {
+			t.Errorf("RestoreTestPBSRestoreTimeout(secs=%d) = %v, want %v", c.secs, got, c.want)
+		}
+	}
+}
+
+func TestRedactedMasksSecret(t *testing.T) {
+	c := Default()
+	c.Proxmox.Token = "felhom-agent@pve!agent=b6547d9d-08ec-4f22-beb8-a551dc2cd69d"
+	got := c.Redacted().Proxmox.Token
+	if strings.Contains(got, "b6547d9d") {
+		t.Fatalf("secret leaked in redacted token: %q", got)
+	}
+	if !strings.HasPrefix(got, "felhom-agent@pve!agent=") {
+		t.Errorf("redacted token lost its public prefix: %q", got)
+	}
+	// The original must be untouched (Redacted returns a copy).
+	if !strings.Contains(c.Proxmox.Token, "b6547d9d") {
+		t.Errorf("Redacted mutated the original config")
+	}
+}
+
+func TestValidate(t *testing.T) {
+	c := Default()
+	c.Proxmox.Node = "demo-felhom"
+	c.Proxmox.Token = "felhom-agent@pve!agent=secret"
+	if err := c.Validate(); err != nil {
+		t.Fatalf("valid config rejected: %v", err)
+	}
+	c.Proxmox.Token = "no-bang-no-eq"
+	if err := c.Validate(); err == nil {
+		t.Errorf("malformed token accepted")
+	}
+}
+
+func TestRedactedMasksHubKey(t *testing.T) {
+	c := Default()
+	c.Hub.APIKey = "hub-secret-abcdef"
+	if got := c.Redacted().Hub.APIKey; got == "hub-secret-abcdef" || got == "" {
+		t.Fatalf("hub key not masked: %q", got)
+	}
+	if !strings.Contains(c.Hub.APIKey, "abcdef") {
+		t.Error("Redacted mutated the original hub key")
+	}
+}
+
+func TestHubConfigValidate(t *testing.T) {
+	base := HubConfig{URL: "https://hub.felhom.eu", HostID: "h1", APIKey: "k"}
+	if err := base.Validate(); err != nil {
+		t.Fatalf("valid hub config rejected: %v", err)
+	}
+	bad := []HubConfig{
+		{HostID: "h", APIKey: "k"},                              // no URL
+		{URL: "https://x", APIKey: "k"},                         // no host
+		{URL: "https://x", HostID: "h"},                         // no key
+		{URL: "http://hub.felhom.eu", HostID: "h", APIKey: "k"}, // http non-loopback
+		{URL: "ftp://x", HostID: "h", APIKey: "k"},              // bad scheme
+	}
+	for i, h := range bad {
+		if err := h.Validate(); err == nil {
+			t.Errorf("case %d: expected validation error for %+v", i, h)
+		}
+	}
+	// http is allowed for loopback (tests).
+	if err := (HubConfig{URL: "http://127.0.0.1:8443", HostID: "h", APIKey: "k"}).Validate(); err != nil {
+		t.Errorf("http loopback should be allowed: %v", err)
+	}
+}
+
+func TestHubEnvOverlayAndDefaults(t *testing.T) {
+	t.Setenv("FELHOM_AGENT_HUB_URL", "https://hub.example")
+	t.Setenv("FELHOM_AGENT_HUB_HOST_ID", "env-host")
+	t.Setenv("FELHOM_AGENT_HUB_API_KEY", "env-key")
+	t.Setenv("FELHOM_AGENT_HUB_POLL_SECONDS", "120")
+	cfg, err := Load("")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Hub.URL != "https://hub.example" || cfg.Hub.HostID != "env-host" || cfg.Hub.APIKey != "env-key" {
+		t.Errorf("hub env overlay failed: %+v", cfg.Hub)
+	}
+	if cfg.Hub.PollSeconds != 120 {
+		t.Errorf("poll seconds = %d, want 120", cfg.Hub.PollSeconds)
+	}
+	// withDefaults fills zero timeout.
+	if (HubConfig{}).WithDefaults().TimeoutSeconds != 30 {
+		t.Error("WithDefaults should set TimeoutSeconds=30")
+	}
+}
+
+func TestLoadFileThenEnvOverride(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "agent.json")
+	if err := os.WriteFile(path, []byte(`{"proxmox":{"node":"file-node","token":"u@pve!t=filesecret"}}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("FELHOM_AGENT_PROXMOX_NODE", "env-node")
+	cfg, err := Load(path)
+	if err != nil {
+		t.Fatalf("Load: %v", err)
+	}
+	if cfg.Proxmox.Node != "env-node" {
+		t.Errorf("env did not override node: %q", cfg.Proxmox.Node)
+	}
+	if cfg.Proxmox.Token != "u@pve!t=filesecret" {
+		t.Errorf("token from file lost: %q", cfg.Proxmox.Token)
+	}
+	if cfg.Proxmox.Endpoint != "https://127.0.0.1:8006" {
+		t.Errorf("default endpoint lost: %q", cfg.Proxmox.Endpoint)
+	}
+}
+
+// CAMPAIGN-3 Part 6: deployment_mode gates node self-heal, and it is FAIL-SAFE to byo — absent or any
+// unknown value is byo, ONLY the exact "appliance" unlocks the remedy.
+func TestIsAppliance_FailSafeToByo(t *testing.T) {
+	cases := []struct {
+		mode string
+		want bool
+	}{
+		{"appliance", true},
+		{"byo", false},
+		{"", false},          // absent field → byo (fail-safe)
+		{"Appliance", false}, // case-sensitive — a typo must not unlock the remedy
+		{"garbage", false},
+	}
+	for _, c := range cases {
+		cfg := &Config{DeploymentMode: c.mode}
+		if got := cfg.IsAppliance(); got != c.want {
+			t.Errorf("IsAppliance(mode=%q) = %t, want %t", c.mode, got, c.want)
+		}
+	}
+}
+
+// The env overlay can set deployment_mode (FELHOM_AGENT_DEPLOYMENT_MODE).
+func TestDeploymentModeEnvOverlay(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "agent.json")
+	if err := os.WriteFile(path, []byte(`{"proxmox":{"node":"n","token":"u@pve!t=s"},"deployment_mode":"byo"}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("FELHOM_AGENT_DEPLOYMENT_MODE", "appliance")
+	cfg, err := Load(path)
+	if err != nil {
+		t.Fatalf("Load: %v", err)
+	}
+	if !cfg.IsAppliance() {
+		t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode)
+	}
+}
+
+// R-50: the island NIC fields are all-or-nothing and the guest addr must be a CIDR. A half-set or
+// malformed island must fail at config load (a botched install) rather than silently fall back to
+// LAN-only, which would leave a guest with an island bind and no island NIC — the exact silent break
+// R-50 exists to kill. Covers LocalAPIConfig.Validate + IslandEnabled.
+func TestLocalAPIConfig_IslandValidation(t *testing.T) {
+	base := LocalAPIConfig{Enable: true, ListenAddr: "169.254.253.1:8443"}
+
+	// both empty → fine (pre-R-50 default), IslandEnabled false
+	if err := base.Validate(); err != nil {
+		t.Errorf("no island config must validate: %v", err)
+	}
+	if base.IslandEnabled() {
+		t.Errorf("IslandEnabled must be false when unset")
+	}
+	// both set, valid CIDR → fine, IslandEnabled true
+	ok := base
+	ok.IslandBridge, ok.IslandGuestAddr = "vmbr9", "169.254.253.2/30"
+	if err := ok.Validate(); err != nil {
+		t.Errorf("valid island config must validate: %v", err)
+	}
+	if !ok.IslandEnabled() {
+		t.Errorf("IslandEnabled must be true when both set")
+	}
+	// bridge only → rejected (all-or-nothing)
+	half := base
+	half.IslandBridge = "vmbr9"
+	if err := half.Validate(); err == nil {
+		t.Errorf("half-set island (bridge only) must be rejected")
+	}
+	// guest addr only → rejected
+	half2 := base
+	half2.IslandGuestAddr = "169.254.253.2/30"
+	if err := half2.Validate(); err == nil {
+		t.Errorf("half-set island (guest addr only) must be rejected")
+	}
+	// both set but guest addr is not a CIDR → rejected
+	bad := base
+	bad.IslandBridge, bad.IslandGuestAddr = "vmbr9", "169.254.253.2" // missing /30
+	if err := bad.Validate(); err == nil {
+		t.Errorf("island guest addr without a CIDR mask must be rejected")
+	}
+}

internal/desired/consumer_test.go

@@ -0,0 +1,90 @@
+package desired
+
+// S3 Group C — the raw-consumer fan-out seam: called on generation advance, NOT on no-advance,
+// and a panicking consumer is contained (the guest reconcile path must never break).
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"sync"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+type recordingConsumer struct {
+	mu    sync.Mutex
+	calls []*hub.DesiredStateResponse
+	panic bool
+}
+
+func (r *recordingConsumer) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
+	r.mu.Lock()
+	r.calls = append(r.calls, resp)
+	r.mu.Unlock()
+	if r.panic {
+		panic("consumer exploded")
+	}
+}
+
+func (r *recordingConsumer) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.calls) }
+
+type stubFetcher struct{ resp *hub.DesiredStateResponse }
+
+func (s *stubFetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
+	return s.resp, nil
+}
+
+func testResp(gen int64) *hub.DesiredStateResponse {
+	return &hub.DesiredStateResponse{
+		Generation: gen,
+		DesiredState: hub.WireDesiredState{
+			Guests:    []hub.WireDesiredGuest{},
+			Wireguard: &hub.WireWireguard{Pubkey: "PK", AssignedIP: "10.77.0.2/32"},
+		},
+	}
+}
+
+func TestSyncer_ConsumerCalledOnAdvanceOnly(t *testing.T) {
+	provider := reconcile.NewCachingProvider()
+	f := &stubFetcher{resp: testResp(2)}
+	s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	c := &recordingConsumer{}
+	s.AddConsumer(c)
+
+	// Advance → fetch → consumer called with the raw doc (wireguard block intact).
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
+	if c.count() != 1 {
+		t.Fatalf("consumer calls = %d, want 1", c.count())
+	}
+	if c.calls[0].DesiredState.Wireguard == nil || c.calls[0].DesiredState.Wireguard.Pubkey != "PK" {
+		t.Fatalf("consumer got %+v — the raw wireguard block must ride through", c.calls[0].DesiredState.Wireguard)
+	}
+
+	// No advance → no fetch → no consumer call (the negative).
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 2})
+	if c.count() != 1 {
+		t.Errorf("consumer called without a generation advance: %d", c.count())
+	}
+}
+
+func TestSyncer_PanickingConsumerContained(t *testing.T) {
+	provider := reconcile.NewCachingProvider()
+	f := &stubFetcher{resp: testResp(1)}
+	s := NewSyncer(f, provider, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	bomb := &recordingConsumer{panic: true}
+	after := &recordingConsumer{}
+	s.AddConsumer(bomb)
+	s.AddConsumer(after)
+
+	// Must not panic out; the second consumer still runs; the provider still updated.
+	s.OnEnvelope(context.Background(), &hub.ControlEnvelope{DesiredGeneration: 1})
+	if after.count() != 1 {
+		t.Errorf("consumer after the panicking one not called: %d", after.count())
+	}
+	if provider.Generation() != 1 {
+		t.Errorf("provider generation = %d, want 1 (guest path unaffected)", provider.Generation())
+	}
+}

internal/desired/syncer.go

@@ -0,0 +1,137 @@
+// Package desired bridges the hub's "Down" channel (the control-envelope generation signal +
+// the desired-state fetch) to the reconcile engine's provider (slice 10A). It implements
+// hub.EnvelopeObserver: on each heartbeat it inspects the envelope's DesiredGeneration and, only
+// when it has ADVANCED past the cached one, fetches the full desired-state and updates the
+// engine's CachingProvider. So the heartbeat stays light; the heavy state moves on change.
+//
+// It lives in its own package because it imports BOTH hub (the wire client + types) and reconcile
+// (the domain DesiredState + CachingProvider). hub does not import it (the loop sees only the
+// hub.EnvelopeObserver seam) and reconcile does not import it — so there is no import cycle.
+package desired
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+// Fetcher fetches this host's desired-state from the hub. Satisfied by *hub.Client.
+type Fetcher interface {
+	FetchDesiredState(ctx context.Context) (*hub.DesiredStateResponse, error)
+}
+
+// RawConsumer receives the FULL fetched desired-state document after each successful
+// generation-advance fetch (S3 seam — internal/wgtunnel consumes its wireguard block this way
+// without the reconcile engine learning about tunnels). Implementations must not block: do the
+// cheap store-and-nudge, never network/exec inline.
+type RawConsumer interface {
+	OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse)
+}
+
+// Syncer keeps the engine's CachingProvider in step with the hub's authoritative desired-state.
+type Syncer struct {
+	fetcher   Fetcher
+	provider  *reconcile.CachingProvider
+	consumers []RawConsumer
+	logger    *slog.Logger
+}
+
+// AddConsumer registers a raw desired-state consumer (nil-safe no-op). Not concurrency-safe —
+// call during wiring, before the hub loop starts.
+func (s *Syncer) AddConsumer(c RawConsumer) {
+	if c != nil {
+		s.consumers = append(s.consumers, c)
+	}
+}
+
+// NewSyncer builds a Syncer over the hub fetcher and the engine's provider.
+func NewSyncer(fetcher Fetcher, provider *reconcile.CachingProvider, logger *slog.Logger) *Syncer {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Syncer{fetcher: fetcher, provider: provider, logger: logger}
+}
+
+// OnEnvelope implements hub.EnvelopeObserver. It fetches + caches the desired-state ONLY when the
+// envelope's generation advances past the provider's cached generation — otherwise it is a no-op
+// (the cached state is already current). A fetch failure keeps the last-known state (the engine
+// keeps reconciling toward it) and is retried on the next advance signal.
+func (s *Syncer) OnEnvelope(ctx context.Context, env *hub.ControlEnvelope) {
+	if env == nil || s.provider == nil {
+		return
+	}
+	have := s.provider.Generation()
+	if env.DesiredGeneration <= have {
+		return // cached: the heavy desired-state moves only on a generation advance
+	}
+	s.logger.Debug("desired: generation advanced — fetching desired-state",
+		"have_generation", have, "envelope_generation", env.DesiredGeneration)
+	start := time.Now()
+	resp, err := s.fetcher.FetchDesiredState(ctx)
+	if err != nil {
+		s.logger.Warn("desired: fetch failed; keeping cached desired-state",
+			"have_generation", have, "envelope_generation", env.DesiredGeneration, "err", err)
+		return
+	}
+	s.logger.Debug("desired: fetched", "generation", resp.Generation,
+		"duration_ms", time.Since(start).Milliseconds())
+	state := mapWire(resp.DesiredState, s.logger)
+	// Cache against the FETCHED generation (not the envelope's) — robust to a generation that
+	// advanced again between the heartbeat and this fetch (we won't re-fetch the same state).
+	s.provider.Update(resp.Generation, state)
+	s.logger.Info("desired: updated from hub",
+		"generation", resp.Generation, "guests", len(state.Guests))
+	// S3: fan the raw document out to registered consumers (wgtunnel etc). A panicking consumer
+	// is contained — the guest reconcile path must never break over a tunnel add-on.
+	for _, c := range s.consumers {
+		s.notifyConsumer(ctx, c, resp)
+	}
+	if env.HasSignedOps {
+		// 10A only notes the flag; fetching + verifying + executing signed ops is slice 10B.
+		s.logger.Info("desired: hub reports pending signed ops (fetch/execute is slice 10B)")
+	}
+}
+
+// notifyConsumer delivers one raw document with panic containment.
+func (s *Syncer) notifyConsumer(ctx context.Context, c RawConsumer, resp *hub.DesiredStateResponse) {
+	defer func() {
+		if r := recover(); r != nil {
+			s.logger.Error("desired: raw consumer panicked (contained)", "panic", r)
+		}
+	}()
+	c.OnDesiredState(ctx, resp)
+}
+
+// mapWire maps the hub wire desired-state to the reconcile domain. 10A acts only on guests; the
+// forward-compat fields (restore_directive — 10D — etc.) are carried on the wire and logged, but
+// not translated into actions here.
+func mapWire(w hub.WireDesiredState, logger *slog.Logger) reconcile.DesiredState {
+	guests := make(map[int]reconcile.DesiredGuest, len(w.Guests))
+	for _, g := range w.Guests {
+		dg := reconcile.DesiredGuest{
+			VMID:         g.VMID,
+			Spec:         g.Spec,
+			Description:  g.Description,
+			Decommission: g.Decommission,
+		}
+		switch g.Run {
+		case "running":
+			dg.Run = reconcile.RunRunning
+		case "stopped":
+			dg.Run = reconcile.RunStopped
+		default:
+			dg.Run = reconcile.RunUnspecified // unknown/empty → unmanaged (planner leaves run alone)
+		}
+		guests[g.VMID] = dg
+	}
+	if w.RestoreDirective != nil {
+		// The reconcile mapping does NOT act on the directive; the DR consumer (raw-consumer seam,
+		// S5 internal/dr) surfaces it as an inspectable restore PLAN — no restore is executed here.
+		logger.Info("desired: restore_directive present (surfaced as a restore PLAN by the DR consumer; not acted on in the reconcile mapping)",
+			"mode", w.RestoreDirective.Mode)
+	}
+	return reconcile.DesiredState{Guests: guests}
+}

internal/desired/syncer_test.go

@@ -0,0 +1,117 @@
+package desired
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
+)
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// fakeFetcher counts FetchDesiredState calls and returns a fixed response (or error).
+type fakeFetcher struct {
+	resp  *hub.DesiredStateResponse
+	err   error
+	calls int
+}
+
+func (f *fakeFetcher) FetchDesiredState(context.Context) (*hub.DesiredStateResponse, error) {
+	f.calls++
+	return f.resp, f.err
+}
+
+func env(gen int64, signed bool) *hub.ControlEnvelope {
+	return &hub.ControlEnvelope{DesiredGeneration: gen, HasSignedOps: signed}
+}
+
+// The headline caching behaviour: desired-state is fetched ONCE when the generation advances, and
+// NOT re-fetched while the generation is unchanged (the heartbeat stays light).
+func TestSyncer_FetchesOnceOnGenerationAdvance(t *testing.T) {
+	run := "running"
+	f := &fakeFetcher{resp: &hub.DesiredStateResponse{
+		Generation: 1,
+		DesiredState: hub.WireDesiredState{Guests: []hub.WireDesiredGuest{
+			{VMID: 100, Run: run},
+			{VMID: 200, Decommission: true},
+		}},
+	}}
+	p := reconcile.NewCachingProvider()
+	s := NewSyncer(f, p, quiet())
+	ctx := context.Background()
+
+	// Generation 0 (fresh host, no intent) → NO fetch.
+	s.OnEnvelope(ctx, env(0, false))
+	if f.calls != 0 {
+		t.Fatalf("fetched %d times at generation 0, want 0", f.calls)
+	}
+
+	// Generation advances to 1 → fetch exactly once, cache updated.
+	s.OnEnvelope(ctx, env(1, false))
+	if f.calls != 1 {
+		t.Fatalf("fetched %d times on advance, want 1", f.calls)
+	}
+	if p.Generation() != 1 {
+		t.Errorf("cached generation = %d, want 1", p.Generation())
+	}
+	st, _ := p.Desired(ctx)
+	if st.Guests[100].Run != reconcile.RunRunning {
+		t.Errorf("guest 100 run = %q, want running", st.Guests[100].Run)
+	}
+	if !st.Guests[200].Decommission {
+		t.Errorf("guest 200 decommission = false, want true")
+	}
+
+	// Same generation on the next heartbeats → NO re-fetch (cached).
+	s.OnEnvelope(ctx, env(1, false))
+	s.OnEnvelope(ctx, env(1, false))
+	if f.calls != 1 {
+		t.Errorf("re-fetched on an unchanged generation (calls=%d, want 1)", f.calls)
+	}
+
+	// A further advance → one more fetch.
+	f.resp = &hub.DesiredStateResponse{Generation: 2, DesiredState: hub.WireDesiredState{}}
+	s.OnEnvelope(ctx, env(2, false))
+	if f.calls != 2 || p.Generation() != 2 {
+		t.Errorf("second advance: calls=%d gen=%d, want 2/2", f.calls, p.Generation())
+	}
+}
+
+// A fetch failure keeps the last-known cache (the engine keeps reconciling toward it) and does not
+// advance the cached generation (so the next heartbeat retries).
+func TestSyncer_FetchFailureKeepsCache(t *testing.T) {
+	p := reconcile.NewCachingProvider()
+	p.Update(1, reconcile.DesiredState{Guests: map[int]reconcile.DesiredGuest{100: {VMID: 100, Run: reconcile.RunRunning}}})
+	f := &fakeFetcher{err: errors.New("hub down")}
+	s := NewSyncer(f, p, quiet())
+
+	s.OnEnvelope(context.Background(), env(5, false)) // generation jumped, but fetch fails
+	if p.Generation() != 1 {
+		t.Errorf("generation advanced to %d despite fetch failure, want kept at 1", p.Generation())
+	}
+	st, _ := p.Desired(context.Background())
+	if st.Guests[100].Run != reconcile.RunRunning {
+		t.Errorf("cache lost on fetch failure: %+v", st.Guests)
+	}
+}
+
+// The fetched generation (not the envelope's) is what gets cached — robust to a generation that
+// advanced again between the heartbeat and the fetch.
+func TestSyncer_CachesFetchedGeneration(t *testing.T) {
+	f := &fakeFetcher{resp: &hub.DesiredStateResponse{Generation: 7, DesiredState: hub.WireDesiredState{}}}
+	p := reconcile.NewCachingProvider()
+	s := NewSyncer(f, p, quiet())
+	s.OnEnvelope(context.Background(), env(5, false)) // envelope said 5, fetch returned 7
+	if p.Generation() != 7 {
+		t.Errorf("cached generation = %d, want 7 (the fetched generation)", p.Generation())
+	}
+	// A later envelope at generation 6 must NOT trigger a re-fetch (we already have 7).
+	s.OnEnvelope(context.Background(), env(6, false))
+	if f.calls != 1 {
+		t.Errorf("re-fetched at generation 6 when cache is 7 (calls=%d)", f.calls)
+	}
+}

internal/dr/plan.go

@@ -0,0 +1,134 @@
+// Package dr consumes the host-loss restore_directive (slice 10D / S5) into an inspectable restore
+// PLAN. It is DERIVE-AND-SURFACE only: the plan is logged (and exposed for the report), never
+// executed — the destructive restore is a separate, operator-present, STOP-gated step. The Consumer
+// has NO restore/destroy API by construction, so "execute nothing" is a structural guarantee.
+package dr
+
+import (
+	"context"
+	"log/slog"
+	"sync"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// RestorePlan is the derived-but-not-executed host-loss plan: per guest → restore coords + sizing;
+// per drive → durable_id → expected mount. No secrets (coordinates/identifiers/sizes only).
+type RestorePlan struct {
+	Mode   string          `json:"mode"`
+	Guests []PlannedGuest  `json:"guests"`
+	Drives []PlannedDrive  `json:"drives"`
+	PBS    *hub.DRPBSCoord `json:"pbs,omitempty"` // WHERE the offsite backups live (repo/ns/latest snapshot)
+}
+
+// PlannedGuest is one guest to restore in place, from the offsite datastore, at its original sizing.
+type PlannedGuest struct {
+	VMID          int    `json:"vmid"`
+	Archive       string `json:"archive,omitempty"` // explicit archive from the directive; "" → resolve latest at restore time
+	TargetStorage string `json:"target_storage"`    // where the restored volumes land (e.g. local-lvm)
+	Cores         int    `json:"cores"`
+	MemoryBytes   int64  `json:"memory_bytes"`
+	DiskBytes     int64  `json:"disk_bytes"`
+}
+
+// PlannedDrive is one data drive to re-attach BY DURABLE_ID (the wrong-disk guard: a match attaches,
+// a non-match is refused — the matcher, exercised in the Part-4 spike, never resolves to a near disk).
+type PlannedDrive struct {
+	DurableID     string `json:"durable_id"`
+	ExpectedMount string `json:"expected_mount"`
+	Intent        string `json:"intent"`
+}
+
+// BuildRestorePlan derives the plan from a host_loss directive + the live DR recipe. Returns
+// (nil,false) for a guest_loss/absent directive or a nil recipe (nothing to plan). PURE: reads
+// nothing, executes nothing — the whole point of this slice's safe half.
+func BuildRestorePlan(directive *hub.WireRestoreDirective, recipe *hub.DRRecipeHostHalf, restoreStorage string) (*RestorePlan, bool) {
+	if directive == nil || directive.Mode != "host_loss" || recipe == nil {
+		return nil, false
+	}
+	plan := &RestorePlan{Mode: directive.Mode, PBS: recipe.PBS}
+	for _, g := range recipe.Guests {
+		pg := PlannedGuest{
+			VMID:          g.VMID,
+			TargetStorage: restoreStorage,
+			Cores:         g.Cores,
+			MemoryBytes:   g.MemoryBytes,
+			DiskBytes:     g.DiskBytes,
+		}
+		// The directive may name an explicit archive for a specific guest (else the restore step
+		// resolves the latest snapshot from the PBS coord at execution time).
+		if directive.Archive != "" && (directive.VMID == 0 || directive.VMID == g.VMID) {
+			pg.Archive = directive.Archive
+		}
+		plan.Guests = append(plan.Guests, pg)
+	}
+	for _, d := range recipe.Drives {
+		plan.Drives = append(plan.Drives, PlannedDrive{
+			DurableID:     d.DurableID,
+			ExpectedMount: d.MountPath,
+			Intent:        d.Intent,
+		})
+	}
+	return plan, true
+}
+
+// RecipeFunc yields the current DR recipe (the agent-derived scaffolding). It is called ONLY when a
+// host_loss directive is present (a rare DR event), so an on-demand Collect is acceptable.
+type RecipeFunc func(ctx context.Context) *hub.DRRecipeHostHalf
+
+// Consumer implements desired.RawConsumer: on a host_loss restore_directive it builds + SURFACES the
+// plan (structured log + LastPlan for the report/inspection) and executes NOTHING. A guest_loss or
+// absent directive clears the plan. It holds no restore/destroy dependency — surfacing is all it can do.
+type Consumer struct {
+	recipe         RecipeFunc
+	restoreStorage string
+	logger         *slog.Logger
+
+	mu       sync.Mutex
+	lastPlan *RestorePlan
+}
+
+// NewConsumer builds the DR plan consumer. recipe may be nil (then no plan can be built — logged).
+func NewConsumer(recipe RecipeFunc, restoreStorage string, logger *slog.Logger) *Consumer {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &Consumer{recipe: recipe, restoreStorage: restoreStorage, logger: logger}
+}
+
+// OnDesiredState implements desired.RawConsumer. Non-host_loss → clear + no-op.
+func (c *Consumer) OnDesiredState(ctx context.Context, resp *hub.DesiredStateResponse) {
+	if resp == nil {
+		return
+	}
+	dir := resp.DesiredState.RestoreDirective
+	if dir == nil || dir.Mode != "host_loss" {
+		c.mu.Lock()
+		c.lastPlan = nil
+		c.mu.Unlock()
+		return
+	}
+	var recipe *hub.DRRecipeHostHalf
+	if c.recipe != nil {
+		recipe = c.recipe(ctx)
+	}
+	plan, ok := BuildRestorePlan(dir, recipe, c.restoreStorage)
+	if !ok {
+		c.logger.Warn("dr: host_loss restore_directive present but no DR recipe available yet — cannot build a plan",
+			"directive_vmid", dir.VMID)
+		return
+	}
+	c.mu.Lock()
+	c.lastPlan = plan
+	c.mu.Unlock()
+	// SURFACE only — the destructive restore is a separate, operator-present step.
+	c.logger.Warn("dr: host_loss RESTORE PLAN derived (NOT executed — supervised in-place restore is a separate, gated step)",
+		"mode", plan.Mode, "guests", len(plan.Guests), "drives", len(plan.Drives), "plan", plan)
+}
+
+// LastPlan returns the most recently derived plan (nil if none / cleared). For the report + tests.
+func (c *Consumer) LastPlan() *RestorePlan {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.lastPlan
+}

internal/dr/plan_test.go

@@ -0,0 +1,92 @@
+package dr
+
+import (
+	"context"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+func sampleRecipe() *hub.DRRecipeHostHalf {
+	return &hub.DRRecipeHostHalf{
+		RecipeVersion: 1,
+		Guests:        []hub.DRGuest{{VMID: 9201, Cores: 2, MemoryBytes: 12 << 30, DiskBytes: 32 << 30}},
+		PBS:           &hub.DRPBSCoord{RepoID: "felhom-offsite", Namespace: "demo-felhom-01", LatestSnapshotID: "9201"},
+		Drives:        []hub.DRDrive{{DurableID: "uuid:abc", MountPath: "/mnt/felhom-drives/photos", Intent: "enrolled", TotalBytes: 500 << 30}},
+	}
+}
+
+// TestBuildRestorePlan_HostLoss: a host_loss directive + recipe yields per-guest {vmid, archive,
+// target, sizing} + per-drive {durable_id → mount} + the offsite PBS coord.
+func TestBuildRestorePlan_HostLoss(t *testing.T) {
+	dir := &hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201, Archive: "felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z"}
+	plan, ok := BuildRestorePlan(dir, sampleRecipe(), "local-lvm")
+	if !ok || plan == nil {
+		t.Fatal("host_loss must yield a plan")
+	}
+	if plan.Mode != "host_loss" || len(plan.Guests) != 1 || len(plan.Drives) != 1 {
+		t.Fatalf("plan shape = %+v", plan)
+	}
+	g := plan.Guests[0]
+	if g.VMID != 9201 || g.TargetStorage != "local-lvm" || g.Cores != 2 || g.DiskBytes != 32<<30 {
+		t.Errorf("planned guest = %+v", g)
+	}
+	if g.Archive != dir.Archive {
+		t.Errorf("planned guest archive = %q, want the directive's %q", g.Archive, dir.Archive)
+	}
+	d := plan.Drives[0]
+	if d.DurableID != "uuid:abc" || d.ExpectedMount != "/mnt/felhom-drives/photos" {
+		t.Errorf("planned drive (durable_id→mount) = %+v", d)
+	}
+	if plan.PBS == nil || plan.PBS.RepoID != "felhom-offsite" {
+		t.Errorf("plan must carry the offsite PBS coord, got %+v", plan.PBS)
+	}
+}
+
+// TestBuildRestorePlan_NoPlanCases is the red-proof anchor: guest_loss / absent / nil-recipe yield
+// NO plan (execute-nothing on the wrong mode). Relaxing the mode gate → the guest_loss case fails.
+func TestBuildRestorePlan_NoPlanCases(t *testing.T) {
+	if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "guest_loss", VMID: 9201}, sampleRecipe(), "local-lvm"); ok {
+		t.Error("guest_loss must NOT yield a host-loss plan")
+	}
+	if _, ok := BuildRestorePlan(nil, sampleRecipe(), "local-lvm"); ok {
+		t.Error("absent directive must NOT yield a plan")
+	}
+	if _, ok := BuildRestorePlan(&hub.WireRestoreDirective{Mode: "host_loss"}, nil, "local-lvm"); ok {
+		t.Error("nil recipe must NOT yield a plan")
+	}
+}
+
+// TestConsumer_SurfacesPlanNeverExecutes: the consumer surfaces the plan on host_loss, consults the
+// recipe only then, and clears it otherwise. It has NO restore/destroy dependency (execute-nothing
+// is structural — the type literally cannot call a restore).
+func TestConsumer_SurfacesPlanNeverExecutes(t *testing.T) {
+	recipeCalls := 0
+	c := NewConsumer(func(context.Context) *hub.DRRecipeHostHalf { recipeCalls++; return sampleRecipe() }, "local-lvm", nil)
+	ds := func(d *hub.WireRestoreDirective) *hub.DesiredStateResponse {
+		return &hub.DesiredStateResponse{DesiredState: hub.WireDesiredState{RestoreDirective: d}}
+	}
+
+	// non-host_loss → no plan, recipe NOT consulted.
+	c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "guest_loss"}))
+	if c.LastPlan() != nil {
+		t.Error("guest_loss set a plan")
+	}
+	if recipeCalls != 0 {
+		t.Errorf("recipe consulted on a non-host_loss directive (%d calls)", recipeCalls)
+	}
+	// host_loss → plan surfaced, recipe consulted once.
+	c.OnDesiredState(context.Background(), ds(&hub.WireRestoreDirective{Mode: "host_loss", VMID: 9201}))
+	p := c.LastPlan()
+	if p == nil || len(p.Guests) != 1 || p.Guests[0].VMID != 9201 {
+		t.Fatalf("host_loss plan = %+v", p)
+	}
+	if recipeCalls != 1 {
+		t.Errorf("recipe calls = %d, want 1", recipeCalls)
+	}
+	// absent directive clears the plan.
+	c.OnDesiredState(context.Background(), ds(nil))
+	if c.LastPlan() != nil {
+		t.Error("absent directive did not clear the plan")
+	}
+}

internal/escrow/ceremony.go

@@ -0,0 +1,49 @@
+package escrow
+
+// Controller-driven ceremony contract (v0.88.0, TASK 2026-07-13; mechanics validated by
+// felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md). The agent's local API
+// re-invokes the agent binary as root via `sudo -n` with ONE fixed argument vector; sudoers
+// matches that vector byte-for-byte (spike §2.2: any alteration — value, extra flag, order,
+// config path — is refused), so the argv below is the SINGLE SOURCE OF TRUTH shared by the
+// exec (localapi), the capability manifest entry, and (byte-identically) the FELHOM_ESCROW
+// sudoers line. Never build it with flag helpers and never normalize `--` to `-` — Go's flag
+// package would accept either spelling, sudoers only the literal one.
+
+// CeremonyBinary is the installed agent binary path the sudoers line pins.
+const CeremonyBinary = "/usr/local/bin/felhom-agent"
+
+// ceremonyArgv is the fixed vector. --config is pinned explicitly: `sudo -n` env_reset strips
+// FELHOM_AGENT_CONFIG, and the pin closes env-injection of an alternate config (spike probe (e)).
+var ceremonyArgv = []string{
+	"--config", "/etc/felhom-agent/agent.json",
+	"--selftest=escrow-create",
+	"--upload",
+	"--output=json",
+}
+
+// CeremonyArgs returns a fresh copy of the fixed argv (callers must not be able to mutate the
+// shared source).
+func CeremonyArgs() []string {
+	out := make([]string, len(ceremonyArgv))
+	copy(out, ceremonyArgv)
+	return out
+}
+
+// CeremonyOutput is the --output=json wire contract (version 1): the ONE JSON object json mode
+// emits on stdout — nothing else lands there; every human/info line goes to stderr. RecoveryCode
+// is the only secret field: the consumer must extract it, hand it to the one-shot claim holder,
+// and zero both the parsed struct and the raw stdout buffer. (Best-effort — Go's GC may hold
+// stale copies; the discipline still shrinks the exposure window.)
+type CeremonyOutput struct {
+	Version           int     `json:"version"`
+	RecoveryCode      string  `json:"recovery_code"`
+	KeyFingerprint    string  `json:"key_fingerprint"`
+	EntropyBits       float64 `json:"entropy_bits"`
+	BlobBytes         int     `json:"blob_bytes"`
+	IdentityBlobBytes int     `json:"identity_blob_bytes"`
+	ResticPwSealed    bool    `json:"restic_pw_sealed"`
+	Uploaded          bool    `json:"uploaded"`
+}
+
+// CeremonyOutputVersion is the current CeremonyOutput.Version value.
+const CeremonyOutputVersion = 1