Compare commits
70 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 7581f8140a | |||
| 3d0a1d615d | |||
| 77e2cc4583 | |||
| cd1b087db7 | |||
| 53d0c6bfc4 | |||
| 4d82591052 | |||
| 4618169036 | |||
| 1b14cfd0b4 | |||
| 0db77666c6 | |||
| dd2d1feb6e | |||
| 9dfd89cb94 | |||
| 4bb84fc3ca | |||
| cd6e26785a | |||
| 587dbb43fe | |||
| eb99144509 | |||
| 2c4efed5de | |||
| 75245a467c | |||
| 054e85a2bf | |||
| 4663df7ff3 | |||
| 14642e3c7b | |||
| 6b5dade4dc | |||
| 1c8a67eece | |||
| 1913e12031 | |||
| 966d8f41ff | |||
| 6be168d1a0 | |||
| d4eb259da2 | |||
| 21b0164fad | |||
| 2f4ccab166 | |||
| a58239f6de | |||
| b58d7bcf39 | |||
| 58b598b697 | |||
| 958e54f6a6 | |||
| 38176ada9d | |||
| d5c769173b | |||
| 50751b8901 | |||
| ff7f68e089 | |||
| 88b3cf03dd | |||
| f27f7a2659 | |||
| 8db92947cd | |||
| 367a503a0f | |||
| a18b18e5de | |||
| af1c21abc4 | |||
| c9a5cc664a | |||
| 023655370b | |||
| 9842c52853 | |||
| 1c2664b0c1 | |||
| 5bca7bfc9a | |||
| 043c7622bc | |||
| 765d8b3168 | |||
| edde8a01ca | |||
| a7ef497cc4 | |||
| 5acf1033a2 | |||
| e4f22f4c4f | |||
| 13ca2d96b2 | |||
| 005083b558 | |||
| 0fabc15896 | |||
| a7421b09c7 | |||
| 3d955e4edd | |||
| a667c269c7 | |||
| 68bcebe493 | |||
| 739b3c3b58 | |||
| dfd5d731ee | |||
| 36ed6594d4 | |||
| 271aa3d9ed | |||
| ed97232598 | |||
| 643899c191 | |||
| 21fee69154 | |||
| c230258542 | |||
| eba040d0be | |||
| a452dc3314 |
@@ -0,0 +1,116 @@
|
||||
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
|
||||
# pushed or what they typed.
|
||||
#
|
||||
# *** THIS REPORTS. IT CANNOT REFUSE. ***
|
||||
#
|
||||
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
|
||||
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
|
||||
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
|
||||
# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
|
||||
#
|
||||
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
|
||||
# the runner is a host-mode container with python3 and git and nothing else (see
|
||||
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
|
||||
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
|
||||
#
|
||||
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
|
||||
# layer up. That is the last step, and it runs ONLY on failure.
|
||||
name: gates
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
gates:
|
||||
runs-on: felhom-gates
|
||||
steps:
|
||||
- name: Fetch the pushed commit and the sibling clone it needs
|
||||
# This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
|
||||
# door and is deliberately never copied here — so CI has to reproduce the workspace's
|
||||
# sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
|
||||
# needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
|
||||
run: |
|
||||
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
|
||||
# which can move under us if two pushes race.
|
||||
mkdir -p ws/felhom-agent
|
||||
cd ws/felhom-agent
|
||||
git init -q .
|
||||
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
|
||||
git fetch -q --depth 1 origin "$GITHUB_SHA"
|
||||
git checkout -q FETCH_HEAD
|
||||
echo "checked out $(git rev-parse HEAD)"
|
||||
cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
|
||||
echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
|
||||
|
||||
- name: Run the gate entry point
|
||||
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
|
||||
# exit code IS the result: no `|| true`, no pipe that could swallow it.
|
||||
#
|
||||
# THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
|
||||
# container runtime" and exists for `.githooks/pre-push`, where a push must not fail
|
||||
# because Gitea blinked or because someone is on a train. CI is the opposite machine: it
|
||||
# has the network, it is not in anyone's way, and it is the half that emails. The
|
||||
# published-versions gate — the R-115 mechanism, which asks Gitea whether a released
|
||||
# version can actually be downloaded — is network-bound and therefore runs ONLY here.
|
||||
# Leaving `--fast` in place would have registered that gate and never run it, which is the
|
||||
# built-but-never-wired failure this project has shipped four times.
|
||||
env:
|
||||
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
|
||||
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
|
||||
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
|
||||
|
||||
- name: Alarm on failure
|
||||
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
|
||||
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
|
||||
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
|
||||
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
|
||||
# accepted id so "a message left the machine" is an observable, not an assumption.
|
||||
#
|
||||
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
|
||||
# else on purpose, and the first version of this step died on `curl: command not found`.
|
||||
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
|
||||
if: failure()
|
||||
env:
|
||||
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import json, os, sys, urllib.request, urllib.error
|
||||
|
||||
key = os.environ.get("RESEND_API_KEY", "")
|
||||
if not key:
|
||||
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
|
||||
"silent alarm is worse than none. Set the user-level Actions secret.")
|
||||
|
||||
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
||||
sha = os.environ.get("GITHUB_SHA", "?")
|
||||
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
||||
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
||||
|
||||
body = json.dumps({
|
||||
"from": "Felhom CI <monitoring@felhom.eu>",
|
||||
"to": ["admin@felhom.eu"],
|
||||
"subject": "[felhom CI] gates FAILED in %s" % repo,
|
||||
"text": (
|
||||
"The gate entry point exited non-zero.\n\n"
|
||||
"Repository : %s\n"
|
||||
"Commit : %s\n"
|
||||
"Run : %s/%s/actions/runs/%s\n\n"
|
||||
"The failing gate names itself in the run log.\n\n"
|
||||
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
|
||||
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
|
||||
"outranks whatever the push was for.\n"
|
||||
) % (repo, sha, srv, repo, run),
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request(
|
||||
"https://api.resend.com/emails", data=body, method="POST",
|
||||
headers={"Authorization": "Bearer %s" % key,
|
||||
"Content-Type": "application/json",
|
||||
# Cloudflare fronts api.resend.com and BLOCKS the default
|
||||
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
|
||||
# exactly like an auth failure and is not one. Measured 2026-08-02.
|
||||
"User-Agent": "felhom-ci/1.0"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
|
||||
except urllib.error.HTTPError as e:
|
||||
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
|
||||
PY
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
#!/bin/sh
|
||||
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
|
||||
#
|
||||
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
|
||||
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
|
||||
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
|
||||
# the bypass becomes the habit.
|
||||
#
|
||||
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
|
||||
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
|
||||
#
|
||||
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
|
||||
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
|
||||
# git config core.hooksPath .githooks
|
||||
# Any manual entry-point run WARNS when the clone is unarmed.
|
||||
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
|
||||
# hatch that cannot be reached is one that gets removed the first time it is
|
||||
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
|
||||
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
|
||||
#
|
||||
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
|
||||
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
|
||||
# explicit rev-parse below does not depend on that.
|
||||
set -u
|
||||
|
||||
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
|
||||
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
|
||||
exit 1
|
||||
}
|
||||
cd "$root" || exit 1
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
|
||||
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "pre-push [felhom-agent]: running scripts/agent_gates.py --fast ..."
|
||||
python3 "scripts/agent_gates.py" --fast
|
||||
rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "pre-push [felhom-agent]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
|
||||
echo " 'git push --no-verify' and state that you did in the session report." >&2
|
||||
else
|
||||
echo "pre-push [felhom-agent]: gates OK - push proceeding."
|
||||
fi
|
||||
exit $rc
|
||||
+1557
File diff suppressed because it is too large
Load Diff
@@ -63,13 +63,50 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
|
||||
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
|
||||
> elsewhere).
|
||||
|
||||
> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
|
||||
> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
|
||||
> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
|
||||
> and undownloadable, where a documented-path reinstall would have silently downgraded them while
|
||||
> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
|
||||
> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
|
||||
> verifies by an **independent download** rather than trusting the publish step's own output.
|
||||
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
|
||||
> than reimplementing it.
|
||||
>
|
||||
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
|
||||
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
|
||||
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
||||
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
|
||||
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
|
||||
> protected is asserted directly instead: the gate now also refuses a **published version with no
|
||||
> tag**. If the push fails after a successful publish the script says so loudly and prints the
|
||||
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
|
||||
>
|
||||
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
|
||||
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
|
||||
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
|
||||
> version yourself:
|
||||
>
|
||||
> ```bash
|
||||
> V=0.122.0
|
||||
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
|
||||
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
|
||||
> sha256sum /tmp/felhom-agent-check
|
||||
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
|
||||
> ```
|
||||
>
|
||||
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
|
||||
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
|
||||
> ever changes, change both or one version name means two binaries again.
|
||||
|
||||
| Step | Where | One-liner |
|
||||
|---|---|---|
|
||||
| Build | DooPlex (local) | `cd /mnt/5_hdd/felhom.eu/git/felhom-agent && git pull && go build -ldflags '-X main.version=<v>' -o /tmp/felhom-agent-<v> ./cmd/felhom-agent` |
|
||||
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
|
||||
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
||||
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
||||
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
||||
| Publish | DooPlex (local) | `scripts/publish-agent.sh <ver> <bin>` (REGISTRY_* creds); hub Day-0 manifest vouch = operator follow-up |
|
||||
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
|
||||
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
|
||||
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
||||
|
||||
## Proxmox model (the load-bearing rules)
|
||||
@@ -88,7 +125,14 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
|
||||
## Demo host (for live tests)
|
||||
|
||||
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
|
||||
available to CC as plain `ssh felhom-pve`. The agent pins the served leaf cert — verify the
|
||||
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
|
||||
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
|
||||
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
|
||||
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
|
||||
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
|
||||
**Which box is safe to break, and what may be done to each:
|
||||
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
|
||||
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
|
||||
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
|
||||
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
|
||||
|
||||
@@ -97,13 +141,17 @@ demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`;
|
||||
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
|
||||
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
|
||||
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
|
||||
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP** and currently holds
|
||||
> `192.168.0.147` — so the PVE API is at `https://192.168.0.147:8006` there, and **the agent does
|
||||
> not run at all**: `localapi` binds the literal `192.168.0.162` → `bind: cannot assign requested
|
||||
> address` → the service is `failed` and has never started at the remote site. Fixing it means
|
||||
> editing `listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest's bootstrap endpoint
|
||||
> (plus the leaf-cert SAN the controller pins) — Viktor GO required. Details + findings:
|
||||
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
|
||||
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
|
||||
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
|
||||
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
||||
>
|
||||
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
|
||||
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
|
||||
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
|
||||
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
|
||||
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
|
||||
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
|
||||
|
||||
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
|
||||
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
|
||||
@@ -127,7 +175,24 @@ All shippable work commits **directly to `main`**; `main` equals what is deploye
|
||||
|
||||
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
|
||||
guess** when you'd otherwise invent input/output.
|
||||
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
|
||||
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
|
||||
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
|
||||
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
|
||||
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
|
||||
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
|
||||
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
|
||||
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
|
||||
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
|
||||
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
|
||||
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
|
||||
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
|
||||
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
|
||||
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
|
||||
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
|
||||
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
|
||||
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
|
||||
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
|
||||
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
|
||||
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
|
||||
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
|
||||
|
||||
+135
@@ -5,6 +5,141 @@
|
||||
|
||||
## Current
|
||||
|
||||
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
|
||||
None touches data; all three cost attention, which every other signal depends on.
|
||||
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
|
||||
from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
|
||||
timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
|
||||
a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
|
||||
twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
|
||||
still read, and a record missing archive-or-tier is NOT reported), exposes
|
||||
`ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
|
||||
newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
|
||||
twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
|
||||
"host-report gauge" in its doc comment and had **no caller** for weeks.
|
||||
- **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
|
||||
future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
|
||||
tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
|
||||
- **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
|
||||
locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
|
||||
made the gate correctly fail a correct release ~half the time. The old order's invariant is now
|
||||
asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
|
||||
a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
|
||||
- **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
|
||||
bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
|
||||
`CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
|
||||
identical. The verification command is in `CLAUDE.md`.
|
||||
|
||||
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
|
||||
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
|
||||
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
|
||||
weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
|
||||
literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
|
||||
tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
|
||||
where it matters most. Red-proved at 0 runs over 5 simulated days.
|
||||
- **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
|
||||
keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
|
||||
upgrade, deliberately.
|
||||
- **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
|
||||
(verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is NOT carried into the evaluation interval.
|
||||
- **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
|
||||
both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
|
||||
evaluation interval is also its retry interval for a multi-GB restore.
|
||||
- The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
|
||||
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
|
||||
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
|
||||
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
|
||||
- **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
|
||||
DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
|
||||
a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
|
||||
evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
|
||||
with its error in that same line.
|
||||
- **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
|
||||
encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
|
||||
named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
|
||||
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
|
||||
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
|
||||
host tier has never been restore-testable there, and the due-check cannot distinguish that from
|
||||
a newborn tier.
|
||||
|
||||
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
|
||||
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
|
||||
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
|
||||
`RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
|
||||
lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
|
||||
down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
|
||||
invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
|
||||
guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
|
||||
instead of keeping a second private definition of "should be running". Retry bounded at 3
|
||||
(1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
|
||||
Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
|
||||
proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.
|
||||
|
||||
- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
|
||||
structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
|
||||
restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
|
||||
**(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
|
||||
VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
|
||||
**(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
|
||||
PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
|
||||
`/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
|
||||
counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
|
||||
`Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
|
||||
`pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
|
||||
(`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
|
||||
consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
|
||||
privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
|
||||
binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
|
||||
PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.
|
||||
|
||||
- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
|
||||
self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
|
||||
its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
|
||||
F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
|
||||
`sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
|
||||
ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.
|
||||
|
||||
- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
|
||||
backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
|
||||
the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
|
||||
deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
|
||||
measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
|
||||
**Size is the only tier-agnostic discriminator** — `verification` and `encrypted` are absent on
|
||||
every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
|
||||
either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
|
||||
smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
|
||||
test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
|
||||
demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
|
||||
observable. Deployed on both boxes. Detail: `REPORT.md`.
|
||||
**Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
|
||||
2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
|
||||
accumulate. Filed as R-99 (LOW).
|
||||
|
||||
- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
|
||||
drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
|
||||
`/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
|
||||
`/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
|
||||
**SMART-only** device from its containing filesystem (never touches backing/durable_id — the
|
||||
removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
|
||||
watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
|
||||
`felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
|
||||
- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
|
||||
`Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
|
||||
enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
|
||||
controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
|
||||
ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
|
||||
so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
|
||||
- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
|
||||
Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
|
||||
(`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
|
||||
clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
|
||||
ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
|
||||
**The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
|
||||
static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.
|
||||
|
||||
- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
|
||||
felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
|
||||
host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
|
||||
|
||||
@@ -1,286 +1,317 @@
|
||||
# REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1**
|
||||
# REPORT — R-86: a restore-test proves each BACKUP, not the clock
|
||||
|
||||
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`).
|
||||
**Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog
|
||||
prevented it (§6b).** Every claim below was observed.
|
||||
**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.120.0 → v0.121.0 → v0.121.1**
|
||||
(`4618169`, `4d82591`, `53d0c6b`) ·
|
||||
released, published, verified by independent download, deployed to demo-felhom and **proven live**.
|
||||
Sibling half: `felhom.eu` hub **v0.91.0 → v0.91.1** — the two ship together.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this closes
|
||||
## 1. Baselines, re-read on arrival
|
||||
|
||||
`INCIDENT-guest-dhclient-killed-2026-07-20.md` §5, "OPEN RISK": **the guest's DHCP client is
|
||||
unsupervised.** ifupdown starts it once at boot and nothing restarts it. When it was killed on
|
||||
2026-07-20 the guest kept working for another **~80 minutes** on its unexpired lease; only at expiry
|
||||
did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync
|
||||
and the controller→agent channel with them — a 1h15m outage in which every observable signal said
|
||||
healthy for the first 80 minutes.
|
||||
|
||||
**So the design consequence is the whole feature: liveness of the DHCP client is itself a probe.**
|
||||
The watchdog flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the address and route
|
||||
are still perfectly present. Waiting for the IP to disappear is waiting out exactly that silent
|
||||
window — and the red-proof reproduces it (§5).
|
||||
|
||||
Host tier is not a preference: a guest with no default route cannot repair its own default route.
|
||||
|
||||
---
|
||||
|
||||
## 2. Shipped
|
||||
|
||||
`internal/guestnet` (probe.go / watchdog.go / report.go), built on the wg-tunnel + storage watchdog
|
||||
loop shape, started with `go wd.Watch(ctx)` like `selfheal`.
|
||||
|
||||
- **Four fixed-shape `pct exec` probes**, all constant argv + the numeric vmid: address, default
|
||||
route, `/etc/network/interfaces` mode, `pgrep -x dhclient`. No shell anywhere; no guest-supplied
|
||||
data is ever interpolated into a command.
|
||||
- **Heal = the incident's restored invocation, verbatim**, logged at INFO before it runs:
|
||||
`pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0`
|
||||
A test pins that argv element by element.
|
||||
- **Dampers** (this runs a privileged command inside a customer's container, so it is built to
|
||||
under-act): two CONSECUTIVE bad probes before any heal, ≥10 min between heals per guest, ≤3
|
||||
heals/hour, and observe-only while the guest — or the agent itself — has been up under 3 minutes.
|
||||
- **Refuses to act** on: a static guest (dhclient must never fight a static config; a static guest
|
||||
missing its address is reported loudly and left to **R-50**), an unknown interface mode, a guest
|
||||
it cannot probe, and an ownership-unproven guest list. The guest source is the pool-verified
|
||||
`ListLXC` ∩ felhom-pool (audit A1) — never a bare `ListLXC`, which under a broad token would run
|
||||
dhclient inside a co-tenant's container.
|
||||
- **A failed PROBE is never a dead client.** `pgrep` exits 1 with EMPTY stderr on no-match; anything
|
||||
on stderr means the probe itself failed → `unknown`. Without that rule a missing `pgrep` would
|
||||
heal forever.
|
||||
- **Healthy cycles log a Debug line.** v0.91.2's lesson, one day old: if the quiet path is silent,
|
||||
"no alarms" and "never probed" are the same evidence.
|
||||
- **Not in the `errc` fan-out** — a watchdog over customer guests must never be able to terminate
|
||||
the agent. A test asserts that, because joining the fan-out would also make the shutdown drain
|
||||
bound off by one.
|
||||
- **Report block:** `GuestNetStatus` on `HostReport` (`guest_net`, omitempty), additive and stored
|
||||
opaquely hub-side like `pbs_dr` / `wireguard`. **No hub code was touched.**
|
||||
|
||||
**Two deliberate deviations from TASK-D, both stated up front:**
|
||||
|
||||
1. **`GuestNetStatus`, not `WireGuestNet`.** In this repo `Wire*` is the DOWN direction
|
||||
(`WireDesiredState` / `WirePBSDR` — what the hub sends the agent); UP-direction report stanzas
|
||||
are `*Status`. `WireGuestNet` on `HostReport` would have been the only report block named against
|
||||
the convention.
|
||||
2. **A sudoers change was required** — see §4. The brief said none was needed.
|
||||
|
||||
**Config `guest_net` is this repo's first default-ON gate.** Every other gate defaults to false
|
||||
because those features reach outward (an offsite endpoint, an OOB tunnel) and enrolling a box by an
|
||||
update would be wrong. This one looks only inward at guests the agent already owns, and the failure
|
||||
it prevents exists on every box today. A watchdog that must be remembered per box is a watchdog
|
||||
that is missing on the box that needed it. Opt-out is explicit: `"guest_net": {"disable": true}`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase-0 probes
|
||||
|
||||
**P3 (watchdog ground truth) — DONE 2026-07-21, live from guest 9201.** These exact bytes are the
|
||||
parser fixtures, including the literal backslash `ip -o` emits and the trailing space on the route:
|
||||
|
||||
```
|
||||
ip -4 -o addr show dev eth0 → 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft 4916sec preferred_lft 4916sec
|
||||
ip route show default → default via 192.168.0.1 dev eth0
|
||||
pgrep -x dhclient → 235839 (rc=0; rc=1 + EMPTY stderr when absent)
|
||||
ps -o args= -C dhclient → dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||||
/etc/network/interfaces → iface eth0 inet dhcp
|
||||
pct exec <bad vmid> → rc=2, stderr "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist"
|
||||
```
|
||||
|
||||
The live `ps` line confirms the running client's argv is byte-identical to the incident's restored
|
||||
invocation — i.e. the heal reproduces the guest's own boot-time command, not an approximation.
|
||||
A docker-bridge-only route table is also pinned as a negative (`172.17.0.0/16 dev docker0 …` must
|
||||
never read as a default route — those were the exact leftovers in the incident).
|
||||
|
||||
**P4 (config surface + report pattern + loop precedent) — DONE** by reading the tree; the wg loop's
|
||||
interval/damping/logging shape and the `pbs_dr` stanza's collector-seam pattern are what this copies.
|
||||
|
||||
---
|
||||
|
||||
## 4. The live finding: three of four probes had no sudoers grant
|
||||
|
||||
The agent runs non-root; `Privileged.Mode=sudo` fails closed with no prompt. The **first sweep after
|
||||
deploying v0.92.0** logged:
|
||||
|
||||
```
|
||||
level=WARN msg="guestnet: guest network not actionable — reporting only" vmid=9201
|
||||
state=unknown mode=unknown has_ip=true has_route=false
|
||||
detail="dhclient liveness probe failed: sudo: a password is required"
|
||||
```
|
||||
|
||||
The watchdog behaved exactly as designed — it reported `unknown` and healed nothing rather than
|
||||
acting blind — but it was blind. The existing allowlist granted only lanresolver's address read
|
||||
(`pct exec [0-9]* -- ip -4 -o addr show dev eth0`), which is why `has_ip=true` while route, mode and
|
||||
liveness all failed.
|
||||
|
||||
Fixed in **v0.92.1**: a `FELHOM_GUESTNET` alias with four fixed vectors (route, interfaces, pgrep,
|
||||
heal). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can
|
||||
widen the grant. The address read is **not** duplicated — it stays FELHOM_DNSMASQ's; one command,
|
||||
one grant.
|
||||
|
||||
Plus **four `guestnet-*` capability rows**, so a host that has not taken the new sudoers file is
|
||||
VISIBLE as degraded instead of silently watchdog-less. Deliberately **non-critical**: a missing
|
||||
grant must not page an operator for every box on rollout day (the R-50b amber-fleet lesson).
|
||||
|
||||
**v0.92.0 is superseded, not overwritten — do not vouch it.** It was published before this was
|
||||
found, so its binary lacks the capability rows and its release lacks the sudoers file. A published
|
||||
version stays immutable (the v0.91.0 → v0.91.1 precedent).
|
||||
|
||||
---
|
||||
|
||||
## 5. Tests and red-proofs
|
||||
|
||||
Green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok **except the known
|
||||
flake** `TestGenerateRecoveryCode_EntropyAndFormat` (`internal/escrow`), which fails when the
|
||||
wordlist yields a hyphenated word (`drop-down` → 11 tokens instead of 10). Confirmed pre-existing:
|
||||
`internal/escrow` has not been touched since v0.88.0 and this task changes nothing there; observed
|
||||
3/8 runs, consistent with the documented ~1/5.
|
||||
|
||||
New: `internal/guestnet/watchdog_test.go` (16 cases), `internal/hub/collect_guestnet_test.go` (2),
|
||||
`cmd/felhom-agent/guestnet_wiring_test.go` (2).
|
||||
|
||||
| # | Red-proof | Mutation | Result |
|
||||
| Repo | `main` @ commit | Version | Matched §1? |
|
||||
|---|---|---|---|
|
||||
| E | detect on process liveness, not address presence | `classify`'s DHCP arm reverted to IP-presence-only | **FAIL ×5.** The decisive one: `classify = "healthy", want "unhealthy"` for the July-20 fixture, `detail="address, default route and dhclient all present"`, and `heal ran 0 times`. That is the 80-minute silent window, reproduced exactly. Restored, green. |
|
||||
| W | the watchdog must be wired | `SetGuestNetReporter` and `go gnWatchdog.Watch(ctx)` both commented out | **FAIL** with both reasons named — *"the guest_net stanza would never reach the hub (the exact v0.91.0 inert-seam defect)"* and *"it would be constructed, reported on, and never probe anything"*. Restored, green. |
|
||||
| `felhom-agent` | `1b14cfd0b48b` | `v0.120.0` | **yes** |
|
||||
| `felhom.eu` | `e34b614e5b65` | CHANGELOG `v0.90.0`, deployed image `0.90.1` | **yes — the discrepancy was real and is fixed** (entry backfilled) |
|
||||
|
||||
**Every damper is asserted as an exact count, and the load-bearing assertions are the negatives** —
|
||||
a static guest, an unprobeable guest, a boot-race guest (young guest AND young agent), a failed
|
||||
probe tool and an ownership-unproven guest list must each record **zero** heal calls. The ceilings
|
||||
are driven by an injected clock over a scripted **10 hours** of permanent failure: ≤30 heals total,
|
||||
and never a second heal inside the 10-minute cool-off.
|
||||
Highest register ID in use was **R-184**; `R-185`–`R-187` established free by grep across all four
|
||||
repos and `documentation/`.
|
||||
|
||||
**Seam discipline (§9 rule 6)** — three production-path tests: the `guest_net` stanza is asserted
|
||||
through the real `Collect` (and asserted ABSENT from the wire when no reporter is wired, so "not
|
||||
wired" and "found nothing" can never look identical); and the `main.go` wiring is an AST walk for
|
||||
the construction, the reporter call and the started goroutine. The AST form is deliberate — a
|
||||
`strings.Contains` version of the twin test in felhom-controller **passed its own red-proof**,
|
||||
because a commented-out call still contains the string.
|
||||
## 2. The rule, in one sentence — and the trap it avoids
|
||||
|
||||
---
|
||||
> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
|
||||
> The tier is **DUE** when A exists and **A has not already been proven**.
|
||||
|
||||
## 6. Live validation (method: journald + capability self-check on felhom-pve)
|
||||
The literal reading of R-86's own wording — *"due when the newest archive is ≥ 24 h old"* — is
|
||||
**never true on a daily tier**, because a new archive lands each day and resets the newest-archive age
|
||||
to zero long before it reaches the lag. It would have switched restore-testing **off** for the tier
|
||||
that matters most, silently, while looking like the row was implemented.
|
||||
|
||||
**Evidence that a daily tier does become due**, at three levels:
|
||||
|
||||
1. **Unit, time-driven** — `TestDue_DailyTierIsProvedDailyOnItsOwnArchive`: five simulated days, one
|
||||
archive a day, evaluated hourly (120 evaluations) → **exactly 5 runs**, and run *i* tests day
|
||||
*i−1*'s archive, never the still-settling one.
|
||||
2. **The red-proof of the naive rule** — implemented and observed failing at **0 runs over 5 days**
|
||||
(§6), which is the trap made visible rather than argued about.
|
||||
3. **Live** — the offsite tier on demo-felhom became due on its own archive and ran (§7).
|
||||
|
||||
## 3. What changed
|
||||
|
||||
| Piece | File | Change |
|
||||
|---|---|---|
|
||||
| the due-check | `internal/backup/restoretest_due.go` (new) | `EvaluateDue` / `evaluateTier` — per-tier verdict + the reason, ordered oldest-proven first |
|
||||
| the trigger | `internal/backup/schedule.go` | the ticker is now the **evaluation interval**; `pickForThisRun` answers *"is anything due?"*, and "nothing" is a normal answer |
|
||||
| the state | `internal/backup/restoretest_state.go` | records **which archive** was proven, with migration |
|
||||
| the picker | `internal/backup/runner.go` | `PickSettledRestoreCandidateOn(ctx, target, notAfter)`; `PickRestoreCandidateOn` is a one-line call into it |
|
||||
| the knobs | `internal/config/config.go` | `restore_test_eval_interval_seconds` + `restore_test_settle_seconds`; the old key deprecated, not repurposed |
|
||||
| the wiring | `cmd/felhom-agent/main.go` | settle-aware picker + `Settle`; deprecation WARN; new `--selftest=restore-test-due` |
|
||||
| the observable (**v0.121.1**) | `internal/backup/schedule.go`, `restoretest_due.go` | a not-due evaluation logs one **INFO** line naming every tier's verdict — see §9 |
|
||||
|
||||
### The state records the archive (§8.2)
|
||||
|
||||
A timestamp cannot answer *"have we proven **this** archive"* — it is the same class as the workspace
|
||||
rule that a timestamp records an *attempt*, not a *result*: here it records a result, but not **which**
|
||||
result. `RestoreTestState` now holds `{archive, proven_at}` per tier.
|
||||
|
||||
**Migration:** a pre-R-86 file (`{"target": "<RFC3339>"}`) keeps its **time** — rotation ordering
|
||||
survives a deploy, which is why the file exists at all — and yields **no proven archive**, so each
|
||||
tier is due exactly once after the upgrade. One extra test per tier, once, is the safe direction;
|
||||
reading a legacy time as proof of whatever archive is current would invent a guarantee.
|
||||
|
||||
### The config (§8.3) — and a correction to the spec
|
||||
|
||||
The spec said *"`0` must keep meaning disabled"*. **In the code as it stands, `0` means *use the
|
||||
default* and NEGATIVE means disabled** (`RestoreTestCadence`, pre-existing). Making `0` disable would
|
||||
have switched restore-testing off on every box that leaves the key unset — the worst possible reading
|
||||
— so the actual semantics were preserved and this is flagged rather than silently followed.
|
||||
|
||||
- `restore_test_eval_interval_seconds` — how often due-ness is **asked**. Default **6 h**.
|
||||
- `restore_test_settle_seconds` — how long an archive must sit. Default **24 h**.
|
||||
- `restore_test_cadence_seconds` — **deprecated**. Negative still **disables**, verbatim. A positive
|
||||
value seeds the **settle lag** (the quantity a person setting it was expressing: how long may pass
|
||||
between a backup and confidence that it restores), and the daemon logs one start-up WARN naming
|
||||
both replacements. It is deliberately **not** carried into the evaluation interval: a box that set
|
||||
72 h to spare a weak endpoint would otherwise get a 72-hour-latency due-check, whereas what it
|
||||
wanted — fewer heavy restores — is what per-archive due-ness already gives it.
|
||||
|
||||
## 4. Part 1.4 — the measurement, and the interval chosen from it
|
||||
|
||||
Measured on demo-felhom, 2026-08-03, via `--selftest=restore-test-due` and by timing the underlying
|
||||
API call directly (3 runs each):
|
||||
|
||||
| tier | what it is | one due-check |
|
||||
|---|---|---|
|
||||
| `felhom-backup` (dir) | local, on-box | **18 ms** (18.7 / 18.3 / 18.5) |
|
||||
| `felhom-pbs` | offsite, **WAN to ep0** | **392 ms** (375 / 378 / 424) |
|
||||
| both together | one full evaluation | **430 ms** |
|
||||
|
||||
**Cost does not set the interval** — even at one evaluation a minute the offsite leg would be ~0.7 %
|
||||
of the link's time. What sets it is the other bound, and it is not in the brief: **under a per-archive
|
||||
due-check a FAILING tier stays due, so the evaluation interval is also its RETRY interval — and a
|
||||
retry is a multi-GB restore.** Every few minutes would be an incident of its own; the old timer
|
||||
retried a broken tier once a day.
|
||||
|
||||
**6 h chosen from both ends:** at most four heavy retries a day in the worst case, and at most 6 h of
|
||||
latency between an archive settling and its proof — negligible against a 24 h settle lag, so a daily
|
||||
tier is still proved daily. No second rate limiter was added (§8.4): the pacing remains one test per
|
||||
archive generation.
|
||||
|
||||
## 5. Two hazards the new frequency created, and their fixes
|
||||
|
||||
Both are consequences of evaluating often rather than daily, and neither is in the brief:
|
||||
|
||||
1. **The due-check now runs BEFORE the heavy-operation gate is taken.** Holding that gate for a read
|
||||
that answers "nothing to do" would open a window at *every* evaluation in which a starting backup
|
||||
cannot acquire — and a backup that cannot acquire does not merely wait, it **records a failure and
|
||||
pages the operator** (F-A1). Nothing heavy starts before the gate; due-ness does not expire while
|
||||
we check.
|
||||
2. **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete
|
||||
1-byte phantom (F-CRIT-2's artefact, which server-side prune does **not** collect) would be picked
|
||||
forever, fail forever, never earn proof, and leave the tier due at *every* evaluation — turning the
|
||||
evaluation interval into the retry rate for a multi-GB restore. `archivePlausiblyComplete` (the
|
||||
canonical helper, with its warn-once companion) is applied in the shared scan, so both callers
|
||||
agree. **This is a behaviour change to `PickRestoreCandidateOn`** and is recorded as such.
|
||||
|
||||
## 6. Tests and red-proofs
|
||||
|
||||
Green gate, both repos: `go build ./... && go vet ./... && go test ./...` — agent **29 packages ok,
|
||||
rc=0**; hub **rc=0**. The test run and the commit were always separate commands.
|
||||
|
||||
| # | Test | Asserts | Mutation | Observed |
|
||||
|---|---|---|---|---|
|
||||
| A | `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` | 5 runs over 5 days, each on the settled archive | the naive rule (`now-landed >= settle`, proven-archive check deleted, cutoff removed) | **FAIL** — `a daily tier must be proved once per day; got 0 run(s) over 5 days: []` |
|
||||
| B | `TestDue_WeeklyTierIsProvedOncePerArchive` | 84 evaluations over 3 weeks → exactly 3 runs, one per archive | — | pass |
|
||||
| C | `TestDue_RestartRunsNothing` | two restarts + 4 evaluations → **0 runs** | `ProvenArchive` reverted to per-tier time | **FAIL** — `2 restart(s) produced 4 run(s)` |
|
||||
| D | `TestDue_NewSettledArchiveMakesAProvedTierDueAgain` | a newly settled archive re-arms the tier, and the NEW archive is tested | — | pass |
|
||||
| E | `TestDue_FailingTierIsRetriedAndNeverProven` | 3 evaluations → 3 retries, no proof recorded | credit on failure (`rt.Pass &&` dropped) | **FAIL** — `got 1 run(s) over 3 evaluations` + `TestRotation_FailureEarnsNoCredit` also failed |
|
||||
| F | `TestDue_TwoDueTiersRunOneAtATime` | one evaluation → one run; the other is deferred and runs next | — | pass |
|
||||
| F′ | `TestDue_DeferredBehindABackupStaysDue` | the gate holds; a deferred tier stays DUE | — | pass |
|
||||
| H | `TestDue_NewbornTierIsNotDueAndNotAnError` | no archive → not due, no error, **and a reason** | — | pass |
|
||||
| — | `TestDue_UnsettledArchiveIsNotACandidate` | a 2 h-old archive is not a candidate under a 24 h lag | — | pass |
|
||||
| — | `TestDue_LookupFailureIsUnknownNotNotDue` | a tier that cannot be listed is UNKNOWN, the error travels, the other tier still runs | — | pass |
|
||||
| — | `TestRestoreTestState_LegacyFileMigratesToNothingProven` | legacy time kept, no archive claimed | — | pass |
|
||||
| — | `TestPickRestoreCandidate_SkipsImplausibleArchives` | the newest entry is not a candidate if it cannot be complete | guard removed | **FAIL** — `pick = "phantom" want the newest COMPLETE archive 'real'` |
|
||||
| I | `TestMainWiresTheSettleAwareTierPicker` | **AST** of `main.go`: settle picker wired, old picker gone, `Settle` set, eval-interval accessor called | the wiring line commented out | **FAIL** — `main.go never passes runner.PickSettledRestoreCandidateOn …` (a `strings.Contains` check would have PASSED — the string is still there, in a comment) |
|
||||
| I′ | `TestMainStillWiresTheHeavyOperationGateAndPerRunSpec` | R-85's gate + per-run spec survive | — | pass |
|
||||
|
||||
Hub-side (Scenario G) is in `felhom.eu/REPORT.md`, including **a hollow test caught by its own
|
||||
red-proof**: the first weekly fixture had no jitter, sat on exactly 168 h, and PASSED under the
|
||||
flat-window mutation.
|
||||
|
||||
### Tests deliberately changed, and why
|
||||
|
||||
`TestRotation_BothTiersExercisedAcrossCadences` asserted *4 ticks → 4 runs*. That was a faithful
|
||||
statement of the defect — every tick produced a heavy restore-test, because the ticker **was** the
|
||||
trigger. It is now `TestRotation_BothTiersExercisedOncePerArchive`: **2 runs across 4 evaluations**,
|
||||
one per tier, one per archive. Strictly stronger — it pins both the coverage R-85 won and the pacing
|
||||
R-86 adds. The old assertion is quoted in the test's comment so the change is legible.
|
||||
|
||||
## 7. The live run — triggered by due-ness, on real hardware
|
||||
|
||||
Deployed to **demo-felhom** (Tier 0). The deployed binary is the **published artifact downloaded from
|
||||
Gitea**, not a local rebuild — see R-186.
|
||||
|
||||
### 7.1 The due verdict, per tier, before anything ran
|
||||
|
||||
```
|
||||
eval_interval=6h0m0s settle=24h0m0s
|
||||
tier=felhom-backup due=false archive="" landed=- proven=""
|
||||
reason: no settled archive yet — nothing to prove (newborn or still settling)
|
||||
tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z"
|
||||
landed=2026-07-28T04:49:43Z proven=""
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) has not been proven; nothing proven yet
|
||||
```
|
||||
|
||||
`felhom-backup` reads "no settled archive" for a reason that is **not** the one it appears to be —
|
||||
see **R-185**: the agent cannot list that storage at all.
|
||||
|
||||
### 7.2 A real run, started by the due-check
|
||||
|
||||
Only the **evaluation interval** was shortened for the validation (a systemd drop-in, since removed):
|
||||
the due rule, the settle lag and the restore-test itself were untouched.
|
||||
|
||||
```
|
||||
15:14:38 backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)
|
||||
target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z
|
||||
landed=2026-07-28T04:49:43Z
|
||||
reason="newest settled archive … has not been proven; nothing proven on this tier yet"
|
||||
15:14:39 restore-test: full-fidelity restore params derived from the archive config scratch=990000
|
||||
… proxmox-backup-client restore --crypt-mode=encrypt … (felhom-agent@pve!agent)
|
||||
15:25:08 audit: gate decision class=guest_destroy guest=990000 source=one_shot_job allowed=true
|
||||
15:25:14 restore-test: scratch guest torn down vmid=990000
|
||||
15:25:14 backup: scheduled restore-test PASSED archive=felhom-pbs:… duration_s=635.1
|
||||
```
|
||||
|
||||
A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored into a scratch guest,
|
||||
booted, verified and destroyed — **635 s**, unattended, and started by *"this archive has not been
|
||||
proven"* rather than by a timer.
|
||||
|
||||
### 7.3 The state now names that archive, and a second evaluation runs nothing
|
||||
|
||||
```json
|
||||
{ "felhom-pbs": { "archive": "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z",
|
||||
"proven_at": "2026-08-03T13:25:14Z" } }
|
||||
```
|
||||
|
||||
```
|
||||
tier=felhom-pbs due=false proven="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z"
|
||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
||||
```
|
||||
|
||||
### 7.4 Teardown — all three layers
|
||||
|
||||
| Layer | Before | After |
|
||||
|---|---|---|
|
||||
| scratch guest 990000 | `stopped lock=create` during the run | **absent** from `pct list` |
|
||||
| its volumes | 5 thin LVs (32 G + 200 G + 50 G + 2×1 G) | **0** matches in `lvs` |
|
||||
| hub-side record | — | the run's **`restore_tests[]` entry is RETAINED deliberately** — it is the proof the hub's staleness check reads, and deleting it would delete the result. No event was created: the run passed, and `restore_test_failed`/`restore_test_stale` fire only on failure or staleness |
|
||||
|
||||
`pvesm status` before and after: `local-lvm` 1.95 % used before the run, and the thin volumes are gone
|
||||
after it — the restore reclaimed to the same shape it started in.
|
||||
|
||||
### 7.5 The restart, which is the defect a person would actually notice
|
||||
|
||||
Under the old scheduler every agent deploy restarted the ticker, so a restore-test ran one interval
|
||||
after each deploy regardless of what had been proven. The proof of the fix cannot be *"nothing
|
||||
appeared in the log"* — that is the absent-line trap this project has a standing rule about — so
|
||||
v0.121.1 makes a quiet evaluation say what it decided, and the evaluation interval was shortened to
|
||||
2 min for the validation so evaluations are **observable**, not assumed:
|
||||
|
||||
```
|
||||
15:32:10 felhom-agent daemon starting version=0.121.1 ← the restart
|
||||
15:32:11 backup: restore-test scheduler starting (per-archive due-check) eval_interval=2m0s settle=24h0m0s
|
||||
15:34:12 backup: restore-test evaluated — nothing due
|
||||
verdicts="felhom-backup: no settled archive yet — nothing to prove (newborn or still settling);
|
||||
felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven"
|
||||
15:36:12 backup: restore-test evaluated — nothing due (same verdicts)
|
||||
|
||||
runs since the restart: 0
|
||||
```
|
||||
|
||||
Evaluations demonstrably **happened** and demonstrably **decided**; nothing ran. The 6 h default was
|
||||
restored afterwards (§8).
|
||||
|
||||
|
||||
## 8. Release and deployment
|
||||
|
||||
| Step | Evidence |
|
||||
|---|---|
|
||||
| Publish 0.92.0 | `AGENT_SHA256=b1302790d412d22e969936ff52e3ee33e3edc111b8426364a21cdb1c5127ca6a`, round-trip GET verified — **superseded, do not vouch** |
|
||||
| Publish 0.92.1 | `AGENT_SHA256=7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`, round-trip GET verified |
|
||||
| Deploy | `visudo -c` parsed OK → sudoers installed 0440 root:root (backup `/root/felhom-agent.sudoers.bak-preR54`) + binary installed (backup `felhom-agent.bak-0.91.2-preR54`) → `felhom-agent --version` = **0.92.1**, service `active` |
|
||||
| Capability self-check | **`ok=68 total=68 degraded=0 inactive=0`** (was 64/64 before the four `guestnet-*` rows) — the sudoers grant is proven from the agent's own side, not assumed |
|
||||
| Watchdog start | `INFO guestnet: watchdog starting interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s` |
|
||||
| **Healthy cycle** | **`level=DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true`** (12:34:15 CEST) |
|
||||
| Default-ON proven | `/etc/felhom-agent/agent.json` has **no** `guest_net` key at all — the watchdog runs on defaults, which is the whole point of the inverted gate |
|
||||
| Released via `scripts/release-agent.sh 0.121.0` | tag `v0.121.0` at `4d82591`, package published |
|
||||
| Verified by **independent download** | sha256 `b2128f3cd4539225a2842f541f56ffaf5390b1d97f3f3a80076ec5f53dbc7d7a`, 14 081 336 B, round-trip GET matched |
|
||||
| Gate re-run after release | `2 released version(s) to verify: 0.120.0, 0.121.0` → both **installable** |
|
||||
| Deployed | `felhom-agent --version` → **0.121.0**, `systemctl is-active` → **active**, prior binary kept as `.bak-0.120.0` |
|
||||
| Startup | `capabilities self-check ok=68 total=68 degraded=0`, and `backup: restore-test scheduler starting (per-archive due-check) eval_interval=6h0m0s settle=24h0m0s` |
|
||||
| Second release, same path | **v0.121.1** — tag `v0.121.1`, sha256 `afaeeb509d1ed70d6e6bebac0393a3cd5be59d51e3db9ff96ef8524bd78546d7`, round-trip verified |
|
||||
| Deployed (published bytes again) | `felhom-agent --version` → **0.121.1**, `active`; prior kept as `.bak-0.121.0` |
|
||||
| Validation config removed | the 2-min drop-in deleted; the daemon back on **`eval_interval=6h0m0s settle=24h0m0s`** |
|
||||
| Fleet | demo-hp still runs **0.120.0** — deliberate: pointing machines at a version is what **vouching** does |
|
||||
| **Vouching** | **NOT done — deliberately the operator's act.** Hub UI → Configs → Day-0 artifacts: agent **`0.121.1`**, sha `afaeeb50…` (0.121.0 also published, sha `b2128f3c…`) |
|
||||
| Config compatibility checked on both boxes | `restore_test_cadence_seconds = 0` on demo-felhom **and** demo-hp, and the installer writes `0` — so no box is on the deprecation path, and a fresh install gets the new defaults with no installer change |
|
||||
|
||||
**Note for the operator:** `log_level` on felhom-pve was temporarily raised to `debug` to capture
|
||||
that Debug line (backup at `/root/agent.json.bak-debuglevel`). It is **still `debug`**, deliberately,
|
||||
so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-2.**
|
||||
## 9. Findings filed (none fixed blind)
|
||||
|
||||
---
|
||||
- **R-185 — the agent cannot see demo-felhom's host backup tier at all.** The PVE token has no ACL on
|
||||
`/storage/felhom-backup`, so the content listing returns `{"data":[]}` where root sees three
|
||||
archives (6.1–6.3 GB, 08-01/02/03). Verified three ways, including `local` — which *has* a grant —
|
||||
returning its archives through the same token. **Pre-existing and independent of R-86** (R-85's
|
||||
rotation had the same blindness). The part worth fixing is the **silence**: a permission-blinded
|
||||
tier is today indistinguishable from a newborn one, and the agent already records the backups it
|
||||
wrote to that target, so the contradiction is detectable.
|
||||
- **R-186 — a released binary's sha cannot be reproduced from its tag.** `release-agent.sh` builds
|
||||
before tagging, so Go stamps a pseudo-version into the published bytes: published `b2128f3c…`
|
||||
(14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source and toolchain. The
|
||||
build order is deliberate, so the fix is not to swap the steps blind. **Mitigated here** by
|
||||
deploying the published artifact.
|
||||
- **R-187 — R-115's one-command release had never run its publish leg** (`CLOSED`, fixed in the same
|
||||
session): `publish-agent.sh` has been mode `0644` since 2026-06-28 because every earlier caller used
|
||||
`bash …`, and `release-agent.sh` called it directly → `Permission denied` on the first real release.
|
||||
Fixed both ways: the mode bit restored **and** the call made mode-independent. The tag the failed
|
||||
run created was withdrawn (nothing had been published under it — verified 404) and recreated on the
|
||||
fix commit, so one version name still means one binary.
|
||||
|
||||
## 6b. STOP-2 — the incident replay (operator-present, 2026-07-21)
|
||||
- **R-188 — a correct agent release emails a CI failure about half the time.** `on: [push]` fires the
|
||||
gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag before publishing
|
||||
(deliberately). CI can therefore run the published-versions gate inside the window where the tag
|
||||
exists and the package does not, and correctly report *"every released agent version must be
|
||||
INSTALLABLE"* for a release that completes seconds later. **Measured across two releases in one
|
||||
session:** v0.121.0 → runs #12 success / #13 failure on the same sha; v0.121.1 → #17 failure / #18
|
||||
success on the same sha; and one pair both green — a race, not a rule. It matters because R-168
|
||||
made CI email on failure so a red gate cannot be missed; a signal that cries wolf on every second
|
||||
correct release is how that mail becomes something you archive unread.
|
||||
- **R-189 — a passing restore-test can be invisible to the hub, and R-86 widened that window.**
|
||||
Observed live: **the 15:25:14 PASS reached no host-report at all**. `restore_tests[]` comes from an
|
||||
**in-memory** store (*"lost on restart; the cadence re-populates"*) and the report interval is
|
||||
900 s; the agent was restarted 2 m 43 s after the run for the v0.121.1 deploy. That used to
|
||||
self-heal within 24 h because the next cadence re-tested the tier — **under per-archive due-ness
|
||||
the agent will not re-test a proven archive**, so the hub can stay ignorant until the next archive
|
||||
generation. The persisted proof already exists: `RestoreTestState.Snapshot()` is documented *"for
|
||||
the host-report gauge"* and has **no production caller** — a seam built and never wired, and an
|
||||
invariant asserted only in a comment, in one method. Bounded, not over-ranked: the hub scans its
|
||||
retained window and the offsite tier's window (12 d) is wider than its archive rhythm (7 d), so one
|
||||
lost report is tolerated. Filed, not fixed — it is a report-contract change.
|
||||
|
||||
The 2026-07-20 kill, repeated deliberately. **The `/proc/<pid>/cgroup` check the incident produced
|
||||
was applied before the kill** — the script refuses unless the pid's cgroup is guest 9201's, which is
|
||||
the rule that would have prevented the original outage:
|
||||
## 10. CI
|
||||
|
||||
```
|
||||
GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc)
|
||||
KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST
|
||||
after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT
|
||||
```
|
||||
|
||||
That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease
|
||||
left before any symptom would appear.
|
||||
|
||||
| Time (CEST) | Event |
|
||||
|---|---|
|
||||
| 12:43:15 | `DEBUG guest network healthy … dhclient_alive=true` — last good cycle |
|
||||
| 12:43:18 | **kill -9** |
|
||||
| **12:44:15** | **detected in 57 s, on process liveness alone** — `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`, detail *"dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"* |
|
||||
| 12:45:15 | second consecutive bad probe → `healing`, then `running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` |
|
||||
| **12:45:18** | **`guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1** |
|
||||
|
||||
**Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun.** The
|
||||
strongest evidence is therefore what did *not* happen, verified from inside the guest:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `cloudflared` | **`Up 29 hours`** — the tunnel never dropped, never reconnected |
|
||||
| DNS `gitea.dooplex.hu` | OK |
|
||||
| hub | **302 in 0.16 s** |
|
||||
| `https://felhom.demo-felhom.eu/` | **302 in 0.25 s** |
|
||||
| new dhclient | pid 2043719, `cgroup=0::/lxc/9201/ns/.lxc`, argv byte-identical to the original |
|
||||
|
||||
On 2026-07-20 every one of those was dead for 1h15m. This time the outage was **prevented**, not
|
||||
detected.
|
||||
|
||||
**The negative leg — 30 healthy minutes, run in full.** From the heal at 12:45:18 to 13:16:15:
|
||||
**30 healthy Debug cycles** (exactly one per minute — the cadence is precise), **0 heals**,
|
||||
**0 WARN**, **0 ERROR**. So the damper is not a mute and the quiet path is not silence: "no alarms"
|
||||
stayed continuously distinguishable from "not probing", which is the v0.91.2 lesson holding in
|
||||
production.
|
||||
|
||||
**An unplanned damper proof, against a REAL transient.** STOP-1's guest reboot at 12:53 caught the
|
||||
guest mid-boot:
|
||||
|
||||
```
|
||||
12:53:16 INFO guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1
|
||||
12:54:15 DEBUG guest network healthy …
|
||||
12:54:15 INFO guest network recovered previous_state=unhealthy
|
||||
```
|
||||
|
||||
One bad probe, **no action**, then recovery. The two-consecutive-probes rule and the boot-race guard
|
||||
did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a
|
||||
genuine transient, not a scripted one.
|
||||
|
||||
## 6c. The stanza ON THE WIRE — confirmed hub-side after STOP-3
|
||||
|
||||
Initially unverifiable here: `--selftest=hub` builds its own one-shot collector and never wires the
|
||||
guestnet reporter, so it would have printed a misleading absence — worse evidence than none. (That
|
||||
divergence between the selftest collector and the daemon's is worth a small fix; the same caveat is
|
||||
already commented in the code for the pbs reporter.)
|
||||
|
||||
Closed instead by a **read-only** query of the hub's own store (`kubectl cp` of `/data/hub.db`,
|
||||
opened `mode=ro`, copy deleted afterwards). The `guest_net` stanza is present, complete, and
|
||||
round-trips every field:
|
||||
|
||||
```json
|
||||
"guest_net": { "checked_at": "2026-07-21T11:32:10Z", "guests": [
|
||||
{ "vmid": 9201, "state": "healthy", "mode": "dhcp", "ip": "192.168.0.104",
|
||||
"has_route": true, "dhclient_alive": true, "checked_at": "2026-07-21T11:31:07Z",
|
||||
"message": "address, default route and dhclient all present" } ] }
|
||||
```
|
||||
|
||||
And the report history tells the whole story of this session from the hub's side:
|
||||
|
||||
| received (UTC) | agent | state | evidence |
|
||||
| Repo | Run | Commit | Result |
|
||||
|---|---|---|---|
|
||||
| 10:30:27 | **0.92.0** | **`unknown`** | the sudoers-blind window — reported honestly as unknown, never as a false healthy and never as a false dead. The fail-safe is visible fleet-side, and this is independent confirmation that 0.92.0 was genuinely blind (so superseding it was right) |
|
||||
| 10:48:12 | 0.92.1 | `healthy` | **`last_heal_at=2026-07-21T10:45:12Z`, `heals_last_hour=1`** — the STOP-2 heal, surfaced to the hub |
|
||||
| 10:49:14 / 11:04:14 | 0.92.1 | `healthy` | same heal stamp, counter still 1 |
|
||||
| 11:32:10 | 0.92.1 | `healthy` | heal history absent — see the limitation below |
|
||||
| `felhom-agent` | **#15** (id 83) | `4d82591` | **success** |
|
||||
| `felhom-agent` | #13 (id 81) | `4618169` | **failure — explained, and it is CI doing its job** |
|
||||
| `felhom.eu` | **#48** (id 86) | `ff2655c` | **success** |
|
||||
|
||||
**A limitation this surfaced, worth stating plainly: the damping state is in-memory only.** The
|
||||
counters vanished from the 11:32 report because the agent was restarted at 11:17Z (the `log_level`
|
||||
revert), which resets `heals`/`lastHealAt`. So the "≥10 min apart, ≤3 per hour" ceilings hold within
|
||||
one agent lifetime, not across restarts. In practice the exposure is small — a restart also re-arms
|
||||
the 3-minute settle window and the two-consecutive-bad-probes rule — but a crash-looping agent could
|
||||
heal more often than the ceiling implies. Not worth persisting state for today; worth knowing before
|
||||
anyone quotes the ceiling as a hard guarantee.
|
||||
Run #13 fired on the **tag push** from the *failed* first release: `v0.121.0` existed as a tag while
|
||||
nothing was published, and `check-published-versions.py` correctly refused — *"every released agent
|
||||
version must be INSTALLABLE"*. That is precisely the state R-115's gate exists to catch, caught within
|
||||
minutes and self-resolved by the corrected release. Confirmed locally afterwards: both 0.120.0 and
|
||||
0.121.0 verify. `--no-verify` was **not** used anywhere.
|
||||
|
||||
## 7. Deliverables
|
||||
## 11. Observations — noticed, recorded, not acted on
|
||||
|
||||
- `c0966d7` — v0.92.0: the watchdog, the report block, config, tests.
|
||||
- `0e8fd81` — the sudoers grant + capability rows (the live finding).
|
||||
- `98adb72` — v0.92.1: supersede + version bump.
|
||||
- Published: `felhom-agent` **0.92.1** / `7424bc1c3c533eff…` (0.92.0 superseded).
|
||||
- Live on felhom-pve: binary 0.92.1 + the new sudoers file.
|
||||
|
||||
## 8. Operator actions outstanding
|
||||
|
||||
1. ~~STOP-2~~ — **DONE 2026-07-21, passed** (§6b).
|
||||
2. ~~STOP-3~~ — **DONE 2026-07-21.** Manifest Agent → **0.92.1**, sha matches the published artifact
|
||||
byte for byte; MinAgent → 0.92.1; PBS wrapper sha unchanged (`104db0a4…`, correct — the wrapper
|
||||
was not touched); controller floor → 0.156.0. The host page shows all four `guestnet-*`
|
||||
capability rows **ok**, which is the fleet-visible proof of the sudoers grant.
|
||||
3. ~~Revert `log_level` to `info`~~ — **DONE** (13:17 CEST, after the quiet window closed; agent
|
||||
restarted clean, caps `68/68 ok, degraded=0`, watchdog back up). The temporary raise is recorded
|
||||
here only so the journald volume change is explainable; `/root/agent.json.bak-debuglevel` remains
|
||||
as the pre-change copy.
|
||||
- **`felhom.eu/CONTEXT.md` has duplicate standing-ruling IDs** — three `S-14`s and two `S-15`s already
|
||||
in the file before this session. New rulings were numbered **S-17/S-18** rather than adding to the
|
||||
collision; the existing duplicates are untouched.
|
||||
- **`agent_gates.py --fast` skips the published-versions gate**, so the pre-push hook cannot catch an
|
||||
unpublished release — only CI can. That is the intended split (no network in a hook), and it is why
|
||||
run #13 mattered.
|
||||
- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for this check. Not
|
||||
changed here (the read window is the same as before), but it is the cost centre if the fleet grows.
|
||||
|
||||
@@ -88,10 +88,12 @@
|
||||
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
|
||||
| `hub.Client.Report` | internal/hub/client.go | `Report(ctx, *HostReport) (*ControlEnvelope, error)` | the heartbeat | Typed `TransportError`/`HTTPError`, never contain the bearer token |
|
||||
| `hub.Loop` + `MultiObserver` | internal/hub/loop.go | `NewLoop(...)`; `MultiObserver(obs...)` | resilient report loop + envelope fan-out | Errors logged, loop continues; interval clamped 60–3600 s |
|
||||
| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned |
|
||||
| `provision.BackHalf.Provision` | internal/provision/backhalf.go | `Provision(ctx, Input) (Result, error)` | guest bootstrap back-half | mint→render→0600 write→chown 100000:100000→`pct set` ro bind→onboot; token NEVER logged/returned. Bootstrap `local_api.endpoint` = the caller's `cfg.LocalAPI.ListenAddr` (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template) |
|
||||
| `buildBringUpConfig` island NIC | internal/reconcile/bringup.go | (pure) `BringUpSpec{IslandBridge,IslandGuestAddr}` → `params["net1"]` | R-50 island control plane | When BOTH island fields are set (from `cfg.LocalAPI`), attaches a static `net1=name=eth1,bridge=<vmbr9>,ip=<.2/30>` (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in `LocalAPIConfig.Validate`. The guestnet healer is eth0-only (`parseMode` is dev-scoped) so it never touches the static island NIC |
|
||||
| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
|
||||
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
|
||||
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
|
||||
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
|
||||
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
|
||||
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
|
||||
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
|
||||
@@ -108,7 +110,10 @@
|
||||
| Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
|
||||
| Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
|
||||
| Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
|
||||
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests |
|
||||
| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
|
||||
| `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
|
||||
| `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
|
||||
| `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |
|
||||
| Detached IN-MEMORY verify job (single slot, deliberately unpersisted) | internal/localapi/netverifyjob.go | claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered) |
|
||||
| Optional dependency degradation | internal/localapi/server.go (`Options`) | nil dep ⇒ endpoint answers "not configured" (503), never a crash |
|
||||
| Version channel (v0.82.0) | internal/localapi/server.go (`Options.AgentVersion`; `Handler()` mux wrap) | sets `X-Felhom-Agent-Version` on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted |
|
||||
@@ -142,11 +147,17 @@
|
||||
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
|
||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
|
||||
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
|
||||
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
|
||||
| `hub.AddressEnumerator` (v0.119.0) | internal/hub/hostaddr.go | **defaults to the REAL `systemInterfaces`** when `Collector.addrEnum` is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record) | internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from `ip -o addr show` on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. `filterHostAddresses` keeps GLOBAL UNICAST only: one predicate that drops loopback, `fe80::/10`, and `169.254/16` — the last being the R-50 island literal, identical on every box and actively misleading if surfaced |
|
||||
| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
|
||||
| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
|
||||
| `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go |
|
||||
@@ -165,6 +176,14 @@
|
||||
- **New privileged host op**: validate args (internal/storage/validate.go style) → exec via `Runner` → add a `Cmnd_Alias` to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
|
||||
- **New reconcile action**: `ActionKind` + `classOfAction` (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
|
||||
- **Hub-report field**: extend `hub.HostReport` (internal/hub/report.go) + `Collector` — hub side must mirror + allowlist it (cross-repo).
|
||||
- **DR-recipe section (host-half)**: add the field to `DRRecipeHostHalf` (internal/hub/dr_recipe.go) **AND** to
|
||||
the hub's `hostHalfShape` + `AssembledRecipe` (felhom.eu `hub/internal/store/dr_recipe.go`). Those two
|
||||
hub structs are **ALLOW-LISTS**: a section only the agent knows about is stored intact and silently
|
||||
dropped before any operator sees it — that is R-122, which cost `offsite_restic` its entire existence.
|
||||
Then update BOTH copies of `testdata/host-report.golden.json` (byte-identical, cross-repo) and extend
|
||||
`TestAssembleDRRecipe_CarriesEveryEmittedSection`. **A recipe field that cannot be resolved records an
|
||||
explicit unknown with a reason — never a default, an empty string, or a placeholder** (`DRState*` /
|
||||
`DRReason*`); a recipe read during a rebuild must not present a guess as a fact.
|
||||
- **Envelope-driven behavior**: implement `hub.EnvelopeObserver`, add to the `MultiObserver` in cmd/felhom-agent/main.go.
|
||||
- **Selftest mode**: `selftestFlag` + `runSelftest*` in cmd/felhom-agent/main.go.
|
||||
- **Config**: internal/config/config.go (`Load` + `applyEnv` `FELHOM_AGENT_*` overlay; keep secrets out of `Redacted()` output).
|
||||
@@ -173,8 +192,9 @@
|
||||
|
||||
- Two lsblk `-J` parsers with near-identical structs: `parseLsblkDevice`/`lsblkDevice` (internal/storage/hostops.go) vs `parseLsblkNodes`/`lsblkDev` (internal/storage/claim.go).
|
||||
- Two smartctl `-a -j` paths: `SudoHostOps.SMART` (internal/storage/hostops.go, parsed `hub.SmartSummary`) vs `Privileged.SMART` (internal/proxmox/privileged.go, raw map).
|
||||
- **SMART device resolution (v0.95.0):** `smartDeviceFor` (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (`dmWholeDisk` in internal/storage/smartdev.go, via `/sys/block/<dm>/slaves`, `sysBlockRoot` test seam). `storage.SmartReader.SMARTForBacking` is the shared read the localapi `/disks` union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-`local` SMART device comes from `containingMountDevice` (SMART-only; never feeds backing/durable_id).
|
||||
- Atomic tmp+rename JSON store implemented 3×: `IntentStore.saveLocked` (internal/storage/intent.go), `FormatJobStore.save` (internal/localapi/formatjob.go), `GuestBindStore.saveLocked` (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
|
||||
- `run(ctx, name, args...) error` stderr-wrapping helper duplicated 4×: `SudoHostOps.run`, `Privileged.run`, `BackHalf.run` (internal/provision/backhalf.go), `GuestBinder.run` (internal/localapi/guestbind.go).
|
||||
- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go), `isHostMountpoint` + `countHostMounts` (internal/localapi/intermediary.go).
|
||||
- Several independent /proc mount-table readers: `SudoHostOps.mountedSet` (internal/storage/hostops.go), `ProcHostReader.Mounts` (internal/storage/hostread.go). **In localapi they were unified in v0.117.0**: `isHostMountpoint` and `countHostMounts` are now one-liners over `hostMountEntries`, the single parser that also yields devno/fstype/super-options for `bindLiveness`.
|
||||
- Deliberate mirror: `antiRetargetResolveExpect` (internal/localapi/wipe_reresolve.go) duplicates `WipeExecutor.Execute` steps 1–3 (internal/signedjobs/wipe.go) across packages.
|
||||
- `stableParentDir` literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); `trim` (internal/storage/hostops.go) vs `trimBody` (internal/proxmox/errors.go) output-truncation twins.
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
// R-82 live regression (2026-07-26): the restore-test derived its tier from the CONFIGURED default
|
||||
// target instead of the archive's own storage. Restoring a `felhom-pbs:` archive on a box whose
|
||||
// primary target is "local" was classified "local" → the 10-minute local wait instead of the
|
||||
// generous PBS one → the wait expired mid-restore at 600s against a 14.46 GB WAN restore, teardown
|
||||
// fired at a still-restoring guest, and the scratch leaked.
|
||||
func TestArchiveStorageID(t *testing.T) {
|
||||
cases := []struct{ in, want string }{
|
||||
{"felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z", "felhom-pbs"},
|
||||
{"local:backup/vzdump-lxc-9201-2026_07_26-09_03_19.tar.zst", "local"},
|
||||
{"", ""},
|
||||
{"no-prefix", ""},
|
||||
{":leading-colon", ""}, // i>0 guard: a leading colon is not a storage id
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := archiveStorageID(c.in); got != c.want {
|
||||
t.Fatalf("archiveStorageID(%q) = %q, want %q", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
+244
-19
@@ -165,7 +165,7 @@ func main() {
|
||||
showVersion bool
|
||||
)
|
||||
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-sysdata-grow/-cores/-memory; keeps the guest)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
||||
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
|
||||
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
|
||||
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
|
||||
@@ -175,8 +175,13 @@ func main() {
|
||||
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
|
||||
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
|
||||
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
|
||||
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)")
|
||||
flag.StringVar(&sysDataMount, "sysdata-mount", "", "for --selftest=bring-up|provision: the mpN slot of the user-data volume to grow (default mp1)")
|
||||
// R-165: the second volume is gone (build-golden.sh v3.0.0 ships ONE). These two flags are kept
|
||||
// ACCEPTED because felhom-host-install.sh passes -sysdata-grow and an installer and an agent do not
|
||||
// upgrade in the same instant — removing them would make every install fail on an unknown flag.
|
||||
// -sysdata-grow is NOT inert: its GiB are folded into the single volume's grow (bringup.go 4b), so
|
||||
// an old installer still produces the same total capacity. -sysdata-mount selects nothing.
|
||||
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "DEPRECATED (R-165): there is one data volume now; this value is ADDED to -datavol-grow rather than growing a second volume. Kept so an older felhom-host-install.sh keeps working")
|
||||
flag.StringVar(&sysDataMount, "sysdata-mount", "", "DEPRECATED (R-165): ignored — there is no second volume to select")
|
||||
flag.IntVar(&cores, "cores", 0, "for --selftest=bring-up|provision: cap the guest to N CPU cores (0 = keep golden default). Applied in the pre-start config PUT.")
|
||||
flag.IntVar(&memoryMB, "memory", 0, "for --selftest=bring-up|provision: cap the guest RAM to N MiB (0 = keep golden default). Applied pre-start.")
|
||||
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
|
||||
@@ -233,6 +238,8 @@ func main() {
|
||||
os.Exit(runSelftestBackup(context.Background(), cfg, logger, vmid))
|
||||
case "restore-test":
|
||||
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
|
||||
case "restore-test-due":
|
||||
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
|
||||
case "pbs-verify":
|
||||
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
|
||||
case "lanresolver":
|
||||
@@ -462,6 +469,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
|
||||
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
|
||||
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
|
||||
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
|
||||
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
|
||||
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
|
||||
// command), so it uses a DIRECT runner regardless of the agent's privileged mode. Probe once at
|
||||
@@ -649,7 +657,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// Self-restore-test scheduler (slice 6): the fourth daemon goroutine. Runs the restore-
|
||||
// test on the configured cadence (default 24h). Disabled cleanly when the cadence is off
|
||||
// OR the scratch band / restore storage is misconfigured — the daemon still runs.
|
||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, logger)
|
||||
// R-85: persisted per-tier restore-test state + the host-wide one-heavy-op gate, both shared
|
||||
// with the local API so a backup and a restore-test can never run together.
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
heavyOps := &backup.InFlight{}
|
||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
||||
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
||||
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
||||
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
||||
// for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB
|
||||
// offsite restore-test reached no host-report at all.
|
||||
collector.SetProvenRestoreTests(rtState)
|
||||
|
||||
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
|
||||
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
|
||||
@@ -757,7 +775,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
return false
|
||||
},
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localTokens != nil {
|
||||
defer localTokens.Close()
|
||||
}
|
||||
@@ -1053,6 +1071,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// the dangling snapshot → start iff onboot). Runs BEFORE the backup loop starts, so a present
|
||||
// backup lock is stale by definition (guarded by a no-vzdump-running check; fail-safe otherwise).
|
||||
localSrv.RecoverStaleLockedGuests(ctx)
|
||||
// F-REBOOT: the startup recovery above only covers a guest left LOCKED by an interrupted
|
||||
// backup. A guest that simply ends up stopped-and-unlocked (a `pct reboot` whose shutdown
|
||||
// half completed and whose start half never fired — Campaign 8 fault 11, 9m47s of total
|
||||
// appliance outage with nothing retrying) needs a PERIODIC check. onboot is the "should be
|
||||
// running" signal, so a deliberately stopped guest is never touched.
|
||||
go localSrv.WatchGuestPower(ctx)
|
||||
go func() { errc <- localSrv.Run(ctx) }()
|
||||
}
|
||||
if lanLoop != nil {
|
||||
@@ -1164,6 +1188,34 @@ func pbsTargetsFromPVE(cfg config.Config, px *proxmox.Client, logger *slog.Logge
|
||||
|
||||
// storageTier returns the restore-test source tier for a backup storage id: "pbs" when that
|
||||
// storage is a PBS datastore, else "local". Best-effort (a lookup failure → "local").
|
||||
// archiveStorageID returns the storage a volid lives on — "felhom-pbs" from
|
||||
// "felhom-pbs:backup/ct/9201/2026-07-26T12:21:48Z". Empty when there is no storage prefix.
|
||||
func archiveStorageID(volid string) string {
|
||||
if i := strings.Index(volid, ":"); i > 0 {
|
||||
return volid[:i]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// restoreTierForArchive derives the restore tier from THE ARCHIVE'S OWN STORAGE, falling back to
|
||||
// the configured default target only when the volid carries no storage prefix.
|
||||
//
|
||||
// R-82 (found live 2026-07-26): this used to read the tier from cfg.Backup.BackupTarget(), i.e. the
|
||||
// PRIMARY tier's target. Restoring a `felhom-pbs:` archive on a box whose primary is "local" was
|
||||
// therefore classified "local" and got the 10-MINUTE local wait instead of the generous PBS one —
|
||||
// the wait expired mid-restore at 600s, teardown fired against a still-restoring guest, and the
|
||||
// scratch leaked. Exactly the failure RestoreTestSpec.RestoreTaskTimeout's doc comment predicts.
|
||||
//
|
||||
// The tier-aware machinery was already correct; it was fed the wrong input. With more than one tier
|
||||
// configured, "the configured target" is no longer a proxy for "the tier this archive belongs to".
|
||||
func restoreTierForArchive(ctx context.Context, px *proxmox.Client, archive, fallbackTarget string) string {
|
||||
id := archiveStorageID(archive)
|
||||
if id == "" {
|
||||
id = fallbackTarget
|
||||
}
|
||||
return storageTier(ctx, px, id)
|
||||
}
|
||||
|
||||
func storageTier(ctx context.Context, px *proxmox.Client, storageID string) string {
|
||||
stores, err := px.ListStorage(ctx)
|
||||
if err != nil {
|
||||
@@ -1200,27 +1252,76 @@ func readTrimmed(path string) (string, error) {
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// primaryBackupTargetOf returns the resolver the DR recipe uses to name WHICH storage holds this box's
|
||||
// local whole-guest archives (R-109).
|
||||
//
|
||||
// It reads the PRIMARY tier out of cfg.Backup.BackupTiers() rather than calling BackupTarget() directly.
|
||||
// Both return the same string today — BackupTiers() builds tier 0 from BackupTarget() — but the tier
|
||||
// list is the function the scheduler itself consults, so if primary-tier derivation ever changes the
|
||||
// recipe follows it instead of quietly disagreeing with the backup. One state, one owner.
|
||||
//
|
||||
// cfg is captured BY VALUE on purpose: that is the daemon-start snapshot, which is the config actually
|
||||
// in effect. See SetBackupTargetResolver for why re-reading agent.json here would be wrong.
|
||||
func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget {
|
||||
return func() hub.ConfiguredBackupTarget {
|
||||
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
|
||||
for _, t := range tiers {
|
||||
if t.Primary {
|
||||
return hub.ConfiguredBackupTarget{StorageID: t.TargetID, Known: true}
|
||||
}
|
||||
}
|
||||
// Unreachable with today's BackupTiers (tier 0 is always primary), and if that ever stops being
|
||||
// true the recipe says "I could not tell" rather than picking a tier at random.
|
||||
return hub.ConfiguredBackupTarget{}
|
||||
}
|
||||
}
|
||||
|
||||
// buildRestoreTestScheduler constructs the restore-test cadence scheduler from config. It
|
||||
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
|
||||
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
|
||||
// machinery still works on-demand via --selftest=restore-test.
|
||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, logger *slog.Logger) *backup.Scheduler {
|
||||
cadence := cfg.Backup.RestoreTestCadence()
|
||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
|
||||
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
|
||||
// per-archive due-check in internal/backup/restoretest_due.go.
|
||||
cadence := cfg.Backup.RestoreTestEvalInterval()
|
||||
if cadence > 0 {
|
||||
if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
|
||||
logger.Warn("daemon: restore-test cadence disabled (config invalid)", "err", err)
|
||||
logger.Warn("daemon: restore-test disabled (config invalid)", "err", err)
|
||||
cadence = 0
|
||||
}
|
||||
}
|
||||
if cadence > 0 && cfg.Backup.RestoreTestLegacyCadenceInUse() {
|
||||
// Said ONCE, at start-up, naming both replacements: a key whose meaning changed under a box
|
||||
// without a word is the silent repurposing R-86 §8.3 forbids.
|
||||
logger.Warn("daemon: backup.restore_test_cadence_seconds is DEPRECATED — R-86 replaced the interval trigger with a per-archive due-check; this value now seeds the SETTLE lag only. Set backup.restore_test_settle_seconds and backup.restore_test_eval_interval_seconds explicitly",
|
||||
"settle", cfg.Backup.RestoreTestSettle(), "eval_interval", cadence)
|
||||
}
|
||||
min, max := cfg.Backup.ScratchBand()
|
||||
target := cfg.Backup.BackupTarget()
|
||||
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
|
||||
// Every configured tier is a rotation candidate, not just the primary.
|
||||
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
|
||||
tierIDs := make([]string, 0, len(cfgTiers))
|
||||
for _, t := range cfgTiers {
|
||||
tierIDs = append(tierIDs, t.TargetID)
|
||||
}
|
||||
return backup.NewScheduler(backup.SchedulerOptions{
|
||||
Runner: engine,
|
||||
Pick: runner.PickRestoreCandidate,
|
||||
Store: store,
|
||||
Spec: func() reconcile.RestoreTestSpec {
|
||||
tier := storageTier(context.Background(), px, target)
|
||||
// R-85 (1.1): the spec is built PER RUN, from the archive that was picked.
|
||||
//
|
||||
// This used to be an immediately-invoked function, so storageTier() and
|
||||
// restoreTaskTimeout() ran ONCE at daemon start and their result was reused for every run
|
||||
// forever. That froze the tier — and with it the timeout — making an offsite restore-test
|
||||
// impossible to schedule, and leaving any storage-type or config change stale until the
|
||||
// daemon restarted.
|
||||
//
|
||||
// The tier comes from the ARCHIVE (restoreTierForArchive, the v0.100.0 rule), never from
|
||||
// the configured target: config-derived was what classified a PBS archive as "local" and
|
||||
// killed a 14.46 GB WAN restore at the 10-minute local bound.
|
||||
Spec: func(ctx context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
tier := restoreTierForArchive(ctx, px, archive, target)
|
||||
return reconcile.RestoreTestSpec{
|
||||
RestoreStorage: cfg.Backup.RestoreStorage,
|
||||
ScratchMin: min,
|
||||
@@ -1228,9 +1329,22 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
SourceTier: tier,
|
||||
RestoreTaskTimeout: restoreTaskTimeout(cfg, tier),
|
||||
}
|
||||
}(),
|
||||
},
|
||||
Cadence: cadence,
|
||||
Logger: logger,
|
||||
// R-86: the settle lag — how long an archive must have sat before it is a candidate. With
|
||||
// the per-archive due-check, this plus the archive rhythm is the whole schedule.
|
||||
Settle: cfg.Backup.RestoreTestSettle(),
|
||||
Logger: logger,
|
||||
|
||||
// R-85: rotate across EVERY configured tier, oldest-proven first (operator ruling, Option 1).
|
||||
// Before this the scheduler only ever saw cfg.Backup.BackupTarget(), so the offsite tier's
|
||||
// archives were never candidates and the DR tier went unproven for its whole existence.
|
||||
// R-86 demoted that ordering to the tie-break BETWEEN DUE TIERS and widened this picker to
|
||||
// the settle-aware one, which is what makes due-ness per archive generation.
|
||||
Tiers: tierIDs,
|
||||
TierPick: runner.PickSettledRestoreCandidateOn,
|
||||
State: rtState,
|
||||
InFlight: inFlight,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1239,7 +1353,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
|
||||
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
|
||||
// fixed. The opened token store is returned via outTokens so the caller can Close it.
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
if !cfg.LocalAPI.Enabled() {
|
||||
return nil
|
||||
}
|
||||
@@ -1269,7 +1383,41 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
}
|
||||
// v0.48.0: ride the served leaf fp on every host report so the hub can detect a re-key fleet-wide.
|
||||
collector.SetLeafFingerprint(fp)
|
||||
runner := backup.NewBackupRunner(px, cfg.Backup.BackupTarget(), "", "felhom local-api", cfg.Backup.PruneBackupsSpec(), logger)
|
||||
// R-82: ONE RUNNER PER TIER. The runner holds its target, mode, notes and retention as
|
||||
// immutable construction state, and localPruneSpec reads that retention — so parameterising a
|
||||
// single runner by target would risk a call pairing tier A's target with tier B's retention.
|
||||
// One runner per tier keeps each tier's policy structurally inseparable from its target.
|
||||
backupTiers, tierWarnings := cfg.Backup.BackupTiers()
|
||||
for _, wmsg := range tierWarnings {
|
||||
// LOUD on purpose: a silently dropped backup tier is an "applied and empty" DR tier, which
|
||||
// is the exact fault R-82 exists to fix. Never downgrade this to DEBUG.
|
||||
logger.Error("backup tier REJECTED — this tier will never run", "detail", wmsg)
|
||||
}
|
||||
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
|
||||
var runner *backup.BackupRunner
|
||||
for _, t := range backupTiers {
|
||||
prune := ""
|
||||
if t.KeepLast > 0 {
|
||||
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
|
||||
}
|
||||
// Pruning a PBS target is allowed ONLY for an additional tier with an explicit keep_last
|
||||
// (the primary's target AND retention both default, so it could prune the DR by accident).
|
||||
allowPBSPrune := !t.Primary && t.KeepLast > 0
|
||||
r := backup.NewBackupRunnerFull(px, t.TargetID, "", "felhom local-api", prune, t.WaitTimeout, allowPBSPrune, logger)
|
||||
if t.Primary {
|
||||
runner = r
|
||||
}
|
||||
apiTiers = append(apiTiers, localapi.BackupTier{
|
||||
TargetID: t.TargetID,
|
||||
Cadence: t.Cadence,
|
||||
WaitTimeout: t.WaitTimeout,
|
||||
Primary: t.Primary,
|
||||
Service: r,
|
||||
})
|
||||
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
|
||||
"keep_last", t.KeepLast, "wait_timeout", t.WaitTimeout.String(),
|
||||
"prune_pbs_allowed", allowPBSPrune, "primary", t.Primary)
|
||||
}
|
||||
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
|
||||
// (same fenced ExecRunner the host-storage + provision back-half use).
|
||||
gaMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
@@ -1283,10 +1431,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||
Guests: px,
|
||||
Backups: runner,
|
||||
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
|
||||
InFlight: inFlight, // R-85: shared with the restore-test scheduler (Scenario F)
|
||||
Store: store,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
@@ -1296,7 +1447,12 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
GuestAttach: guestBinder, // slice 10 P2: bind enrolled data drives into the guest
|
||||
Memory: px, // v0.90.0 R-24: guest RAM resize (SetConfig live cgroup apply)
|
||||
// Network storage (NAS) — Part A1: the privileged host network-mount surface (NFS/SMB automount).
|
||||
NetStorage: hostOps,
|
||||
NetStorage: hostOps,
|
||||
// E-2a: the fenced root shim for the backup-target move. Same runner mode as every other
|
||||
// privileged call; the sudoers vector is what actually bounds it.
|
||||
Privileged: &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath},
|
||||
ConfigPath: cfg.SourcePath,
|
||||
StateDir: cfg.WGTunnel.WithDefaults().StateDir,
|
||||
SmbCredsDir: cfg.Privileged.SmbCredsDir,
|
||||
ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap
|
||||
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
|
||||
@@ -1455,6 +1611,10 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
|
||||
// the selftest reflects exactly what a freshly-restarted daemon's first collect emits.
|
||||
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargetsFromPVE(cfg, px, logger), pbs.NewSnapshotStore(), pbs.DefaultLiveSnapshotTimeout, logger)
|
||||
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
|
||||
// R-109: wire the backup-target resolver here TOO. Without it selftest=hub would print a recipe whose
|
||||
// backup_target reads unknown/agent_backup_config_unavailable while the daemon's is resolved — and
|
||||
// this one-shot exists precisely so "the report it would send" can be trusted to match.
|
||||
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg))
|
||||
|
||||
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
defer cancel()
|
||||
@@ -1612,6 +1772,67 @@ func runSelftestBackup(ctx context.Context, cfg config.Config, logger *slog.Logg
|
||||
// running → teardown) of -archive (or the newest backup on the local target) into a scratch
|
||||
// guest. Standalone (no hub). Runs engine.Recover first so a leaked scratch from a prior
|
||||
// crashed test is reaped before this run.
|
||||
// runSelftestRestoreTestDue prints the per-tier DUE verdict the scheduler would act on, and what
|
||||
// each evaluation COST — read-only, so it is safe on any box at any time.
|
||||
//
|
||||
// It exists for two reasons R-86 needed and could not get from a log line. First, the due-check's
|
||||
// verdict is the whole schedule now: "why did nothing run last night?" is answerable only by asking
|
||||
// the same question the scheduler asks, against the same storages, in the same order. Second, the
|
||||
// evaluation interval had to be chosen from a MEASURED cost rather than a guess — an offsite tier's
|
||||
// candidate lookup crosses the WAN, and a monitoring loop that costs more than it is worth is how a
|
||||
// check becomes the load. It reuses the daemon's own construction path (buildRestoreTestScheduler),
|
||||
// so what it prints is what the daemon would decide, not a re-derivation of it.
|
||||
func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *slog.Logger) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
return 1
|
||||
}
|
||||
px, err := newProxmoxClient(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err)
|
||||
return 1
|
||||
}
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
|
||||
|
||||
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
|
||||
start := time.Now()
|
||||
verdicts := sched.EvaluateDue(ctx)
|
||||
total := time.Since(start)
|
||||
if len(verdicts) == 0 {
|
||||
fmt.Println("no tiers configured for restore-testing (or rotation not wired)")
|
||||
return 0
|
||||
}
|
||||
rc := 0
|
||||
for _, v := range verdicts {
|
||||
proven, _ := rtState.ProvenArchive(v.Target)
|
||||
fmt.Printf("tier=%-16s due=%-5v archive=%q landed=%s proven=%q\n reason: %s\n",
|
||||
v.Target, v.Due, v.Archive, formatOrDash(v.Landed), proven, v.Reason)
|
||||
if v.Err != nil {
|
||||
// A tier we could not list is UNKNOWN, and it is a non-zero exit: an unreadable tier is
|
||||
// a real condition, not a quiet "nothing to do".
|
||||
fmt.Printf(" ERROR: %v\n", v.Err)
|
||||
rc = 3
|
||||
}
|
||||
}
|
||||
// Per-tier timing, measured one tier at a time so the WAN leg is attributable (R-86 Part 1.4).
|
||||
for _, v := range verdicts {
|
||||
t0 := time.Now()
|
||||
_ = sched.EvaluateDueTier(ctx, v.Target)
|
||||
fmt.Printf("cost tier=%-16s one_lookup=%s\n", v.Target, time.Since(t0).Round(time.Millisecond))
|
||||
}
|
||||
fmt.Printf("cost all_tiers=%s\n", total.Round(time.Millisecond))
|
||||
return rc
|
||||
}
|
||||
|
||||
// formatOrDash renders a time, or "-" when it is zero (no archive).
|
||||
func formatOrDash(t time.Time) string {
|
||||
if t.IsZero() {
|
||||
return "-"
|
||||
}
|
||||
return t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
@@ -1666,7 +1887,7 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
|
||||
}
|
||||
min, max := cfg.Backup.ScratchBand()
|
||||
fmt.Printf(" restoring %s into scratch band [%d,%d] on %s …\n", archive, min, max, cfg.Backup.RestoreStorage)
|
||||
rtTier := storageTier(ctx, px, target)
|
||||
rtTier := restoreTierForArchive(ctx, px, archive, target)
|
||||
res := engine.RunRestoreTest(ctx, reconcile.RestoreTestSpec{
|
||||
Archive: archive, RestoreStorage: cfg.Backup.RestoreStorage,
|
||||
ScratchMin: min, ScratchMax: max, SourceTier: rtTier,
|
||||
@@ -1771,6 +1992,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
Cores: sizing.Cores, MemoryMB: sizing.MemoryMB,
|
||||
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
|
||||
SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount,
|
||||
IslandBridge: cfg.LocalAPI.IslandBridge, IslandGuestAddr: cfg.LocalAPI.IslandGuestAddr, // R-50 island NIC (both empty = pre-R-50)
|
||||
}
|
||||
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
|
||||
res := engine.RunBringUp(ctx, spec)
|
||||
@@ -1934,6 +2156,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
|
||||
Cores: a.sizing.Cores, MemoryMB: a.sizing.MemoryMB,
|
||||
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
|
||||
SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount,
|
||||
IslandBridge: cfg.LocalAPI.IslandBridge, IslandGuestAddr: cfg.LocalAPI.IslandGuestAddr, // R-50 island NIC (both empty = pre-R-50)
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
|
||||
@@ -2799,6 +3022,8 @@ func (f *selftestFlag) Set(v string) error {
|
||||
f.mode = "backup"
|
||||
case "restore-test":
|
||||
f.mode = "restore-test"
|
||||
case "restore-test-due":
|
||||
f.mode = "restore-test-due"
|
||||
case "pbs-verify":
|
||||
f.mode = "pbs-verify"
|
||||
case "lanresolver":
|
||||
@@ -2816,7 +3041,7 @@ func (f *selftestFlag) Set(v string) error {
|
||||
case "controller-swap":
|
||||
f.mode = "controller-swap"
|
||||
default:
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-86 Scenario I — the seam-discipline test for the due-check.
|
||||
//
|
||||
// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
|
||||
// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
|
||||
// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
|
||||
// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
|
||||
// tested, never called — and this repo has shipped it four times.
|
||||
//
|
||||
// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
|
||||
// match, and a comment is not a caller.
|
||||
func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var settlePicker, oldPicker, settleWired, evalInterval bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.SelectorExpr:
|
||||
// runner.PickSettledRestoreCandidateOn passed as a value (not called).
|
||||
switch node.Sel.Name {
|
||||
case "PickSettledRestoreCandidateOn":
|
||||
settlePicker = true
|
||||
case "PickRestoreCandidateOn":
|
||||
oldPicker = true
|
||||
}
|
||||
case *ast.KeyValueExpr:
|
||||
key, ok := node.Key.(*ast.Ident)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if key.Name == "Settle" {
|
||||
settleWired = true
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
|
||||
evalInterval = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !settlePicker {
|
||||
t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
|
||||
"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
|
||||
}
|
||||
if oldPicker {
|
||||
t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
|
||||
"two pickers means the one under test is not the one running")
|
||||
}
|
||||
if !settleWired {
|
||||
t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
|
||||
"and every freshly-landed archive would be an immediate candidate")
|
||||
}
|
||||
if !evalInterval {
|
||||
t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
|
||||
"the retired cadence knob")
|
||||
}
|
||||
}
|
||||
|
||||
// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
|
||||
// RUN, and the shared heavy-operation gate is still handed to the scheduler.
|
||||
func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var inFlightWired, specIsAFunc bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
kv, ok := n.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
key, ok := kv.Key.(*ast.Ident)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch key.Name {
|
||||
case "InFlight":
|
||||
inFlightWired = true
|
||||
case "Spec":
|
||||
// A FuncLit means it is evaluated per run; anything else is a frozen value.
|
||||
if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
|
||||
specIsAFunc = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !inFlightWired {
|
||||
t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
|
||||
"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
|
||||
}
|
||||
if !specIsAFunc {
|
||||
t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
|
||||
"(the tier and its timeout evaluated once at daemon start, forever)")
|
||||
}
|
||||
}
|
||||
|
||||
func parseMainForWiring(t *testing.T) *ast.File {
|
||||
t.Helper()
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
|
||||
//
|
||||
// This test exists because the method it feeds is the project's own cautionary tale:
|
||||
// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
|
||||
// was written and **had no caller at all** — a seam built, documented and never connected, found
|
||||
// only when a live restore-test's PASS reached no host-report. The fix must not become the next
|
||||
// instance, so the wiring is asserted rather than trusted.
|
||||
//
|
||||
// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
|
||||
// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
|
||||
func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var wired, feedsState bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
|
||||
return true
|
||||
}
|
||||
wired = true
|
||||
// ...and it must be fed the PERSISTED state, not the in-memory store.
|
||||
if len(call.Args) == 1 {
|
||||
if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
|
||||
feedsState = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !wired {
|
||||
t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
|
||||
"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
|
||||
}
|
||||
if wired && !feedsState {
|
||||
t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
|
||||
"that does NOT survive a restart, so wiring it here would fix nothing")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
)
|
||||
|
||||
// COMPILE-TIME WITNESSES for OPTIONAL interfaces that are satisfied by a RUNTIME type assertion.
|
||||
//
|
||||
// WHY THIS FILE EXISTS. `localapi.BackupArchiveLister` is asserted at server.go's `newestArchiveOn`
|
||||
// via `tier.Service.(BackupArchiveLister)`. A failed assertion does not error — it degrades to
|
||||
// `archiveAbsent`, i.e. the pre-R-84 "ask the in-memory record only" behaviour. That degrade is
|
||||
// SILENT and it is behaviour-relevant: it is exactly the R-84 bug (a cold store after a restart
|
||||
// reading as "no backup ever") coming back, with nothing in any log to say so.
|
||||
//
|
||||
// The precedent is not hypothetical. During R-88 Part 2 the controller's `quiesceBackend` stopped
|
||||
// satisfying `quiesce.TieredBackend` when a signature changed, and `go build` AND `go vet` both
|
||||
// passed — because the interface is only ever asserted at runtime. Every box would have degraded to
|
||||
// the single-tier path, losing R-82's multi-tier backups, with no error anywhere. It was caught by
|
||||
// accident.
|
||||
//
|
||||
// A witness costs one line and converts that class of failure from a silent production degrade into
|
||||
// a compile error.
|
||||
//
|
||||
// THIS DOES NOT MAKE THE INTERFACE REQUIRED. The optionality is deliberate — it is what lets a
|
||||
// BackupService without a lister still work. The witness pins the IMPLEMENTATION (this concrete type
|
||||
// really does satisfy it), not the CONTRACT.
|
||||
var _ localapi.BackupArchiveLister = (*backup.BackupRunner)(nil)
|
||||
@@ -48,10 +48,16 @@
|
||||
},
|
||||
"local_api": {
|
||||
"enable": true,
|
||||
"listen_addr": "192.168.0.162:8443",
|
||||
"listen_addr": "169.254.253.1:8443",
|
||||
"cert_file": "/var/lib/felhom-agent/local-api.crt",
|
||||
"key_file": "/var/lib/felhom-agent/local-api.key",
|
||||
"token_store": "/var/lib/felhom-agent/local-tokens.log"
|
||||
"token_store": "/var/lib/felhom-agent/local-tokens.log",
|
||||
"island_bridge": "vmbr9",
|
||||
"island_guest_addr": "169.254.253.2/30"
|
||||
},
|
||||
"lan_resolver": {
|
||||
"enable": true,
|
||||
"host_ip": "192.168.0.162"
|
||||
},
|
||||
"log_level": "info"
|
||||
}
|
||||
|
||||
+110
-39
@@ -26,20 +26,42 @@
|
||||
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
|
||||
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
|
||||
#
|
||||
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a
|
||||
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller +
|
||||
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot
|
||||
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays
|
||||
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it
|
||||
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data
|
||||
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp:
|
||||
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3),
|
||||
# so the archive would carry NO images and provisioned guests would boot imageless.
|
||||
# OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
|
||||
#
|
||||
# The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
|
||||
# NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
|
||||
#
|
||||
# /var/lib/felhom/docker --bind--> /var/lib/docker (Docker's data-root)
|
||||
# /var/lib/felhom/sys_drive --bind--> /mnt/sys_drive (the controller's system_data_path)
|
||||
#
|
||||
# WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
|
||||
# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
|
||||
# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
|
||||
# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
|
||||
#
|
||||
# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
|
||||
# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
|
||||
# a different documented guarantee:
|
||||
# * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
|
||||
# stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
|
||||
# destroys every local recovery unit on the box;
|
||||
# * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
|
||||
# container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
|
||||
# /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
|
||||
# felhom-data-namespace mounts" would become false.
|
||||
# The neutral mount breaks neither, for one extra path and one extra fstab line.
|
||||
#
|
||||
# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
|
||||
# agent-recoverable if the data volume fills (the controller's prevention layer, and since
|
||||
# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
|
||||
# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
|
||||
# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
|
||||
# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
|
||||
set -euo pipefail
|
||||
|
||||
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
|
||||
# archive can always be traced to the script that produced it. Bump on any behavior change.
|
||||
GOLDEN_SCRIPT_VERSION="2.1.0"
|
||||
GOLDEN_SCRIPT_VERSION="3.0.0"
|
||||
|
||||
VMID="${1:-9100}"
|
||||
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
|
||||
@@ -59,25 +81,28 @@ if [ -z "$CONTROLLER_IMAGE" ]; then
|
||||
exit 1
|
||||
fi
|
||||
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
|
||||
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||
# OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
|
||||
#
|
||||
# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
|
||||
# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
|
||||
# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
|
||||
# per-customer target (bringup.go DataVolGrowGB).
|
||||
OS_SIZE_GB="${OS_SIZE_GB:-32}"
|
||||
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
|
||||
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's
|
||||
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once
|
||||
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like
|
||||
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 —
|
||||
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
|
||||
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
|
||||
# 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
|
||||
# did before the merge. It is deliberately NOT a per-customer size: provision grows it.
|
||||
GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
|
||||
# The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
|
||||
GOLDEN_VOLUME_MP="/var/lib/felhom"
|
||||
|
||||
echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
|
||||
# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
|
||||
# whole point of R-165 is that it stops existing rather than being made bigger.
|
||||
pct create "$VMID" "$TEMPLATE" \
|
||||
--hostname felhom-golden --unprivileged 1 \
|
||||
--features nesting=1,keyctl=1 \
|
||||
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
|
||||
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
|
||||
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
||||
|
||||
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
||||
@@ -106,8 +131,10 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
|
||||
# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
|
||||
# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
|
||||
# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
|
||||
# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty
|
||||
# before docker installs), so data-root needs no override. Log caps kill the most common runaway.
|
||||
# DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
|
||||
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
|
||||
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
|
||||
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
|
||||
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
|
||||
{
|
||||
"features": { "containerd-snapshotter": false },
|
||||
@@ -115,6 +142,32 @@ pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json
|
||||
"log-opts": { "max-size": "10m", "max-file": "3" }
|
||||
}
|
||||
JSON'
|
||||
echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
|
||||
# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
|
||||
# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
|
||||
# round would hide those files under the bind and silently ship a golden whose baked images are on the
|
||||
# rootfs — the exact failure class the assertions below exist to catch.
|
||||
#
|
||||
# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
|
||||
# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
|
||||
# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
|
||||
pct exec "$VMID" -- bash -c "
|
||||
set -e
|
||||
systemctl stop docker docker.socket containerd 2>/dev/null || true
|
||||
mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
|
||||
if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
|
||||
cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
|
||||
rm -rf /var/lib/docker/*
|
||||
fi
|
||||
mkdir -p /var/lib/docker /mnt/sys_drive
|
||||
printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker' >> /etc/fstab
|
||||
printf '%s /mnt/sys_drive none bind 0 0\n' '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
|
||||
systemctl daemon-reload
|
||||
mount /var/lib/docker
|
||||
mount /mnt/sys_drive
|
||||
systemctl start containerd
|
||||
"
|
||||
|
||||
echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
|
||||
# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
|
||||
# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
|
||||
@@ -122,12 +175,20 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
|
||||
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
|
||||
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
|
||||
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
|
||||
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
|
||||
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs
|
||||
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole
|
||||
# point of this volume is to clear that warning).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }'
|
||||
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
|
||||
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
|
||||
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
|
||||
# its baked images there.
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
|
||||
# ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
|
||||
# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
|
||||
# (clearing that warning is the whole point of the volume).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
|
||||
# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
|
||||
# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
|
||||
# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
|
||||
# there, and a shared pool neither `df` can see coming.
|
||||
pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo " both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
|
||||
|
||||
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
||||
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
||||
@@ -307,26 +368,36 @@ pct exec "$VMID" -- bash -c '
|
||||
|
||||
echo "[golden] stop + archive …"
|
||||
pct stop "$VMID"
|
||||
# --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the
|
||||
# /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0"
|
||||
# AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost
|
||||
# and the archive carries no images / no user-data volume (storage-split B3 trap).
|
||||
# --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
|
||||
# INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
|
||||
# exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
|
||||
# Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
|
||||
# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
|
||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
|
||||
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting."
|
||||
# ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
|
||||
# archive". After the merge there is no mp1 — so the same failure now looks like the volume being
|
||||
# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
|
||||
# inclusion line names the volume's actual mount path.
|
||||
if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
|
||||
grep -iE "mount point" /tmp/golden-vzdump.log || true
|
||||
exit 1
|
||||
fi
|
||||
# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
|
||||
# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
|
||||
if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
|
||||
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
|
||||
|
||||
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)"
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
|
||||
|
||||
#-------------------------------------------------------------------------------
|
||||
# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
|
||||
|
||||
@@ -133,6 +133,28 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
Cmnd_Alias FELHOM_STALELOCK = \
|
||||
/usr/sbin/pct unlock [0-9]*
|
||||
|
||||
# Restore-test scratch teardown (F-LEAK, Campaign 8, v0.110.0). A restore-test whose restore FAILS
|
||||
# leaves a scratch guest the API token CANNOT destroy: `FelhomAgentGuest` is granted at /pool/felhom and
|
||||
# a guest joins that pool only when its restore COMPLETES, so a failed restore leaves a pool-less guest
|
||||
# out of reach (403 VM.Allocate) holding its disks until a human removes it.
|
||||
#
|
||||
# TWO API-SIDE FIXES WERE TRIED AND BOTH REFUTED LIVE on 2026-07-28, which is why this grant exists:
|
||||
# 1. Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
|
||||
# VM.Allocate on the VM being added — pool membership cannot bootstrap its own authority.
|
||||
# 2. Grant FelhomAgentGuest per-path at /vms/990000..990009. Durable for exactly one use per slot:
|
||||
# PVE's own destroy path calls `AccessControl::remove_vm_access($vmid)` (LXC.pm:906), which DELETES
|
||||
# every ACL at /vms/<vmid> (AccessControl.pm:1898). The grant is consumed by the operation it
|
||||
# authorises, so after ten teardowns the band is ungranted and the defect returns.
|
||||
#
|
||||
# WHY THIS IS THE TIGHTEST AVAILABLE FENCE, not a widening: sudo matches the vmid LITERALLY, so
|
||||
# `99000[0-9]` is exactly the ten-slot scratch band the restore-test picks from — nothing else. There is
|
||||
# no `[0-9]*` coarse allowlist here on purpose: unlike `pct unlock`, this op DESTROYS, so the band must
|
||||
# be in the policy and not merely validated in the agent. Even a compromised agent asking for
|
||||
# `pct destroy 9201` is refused by sudo itself. Unlike an ACL, a sudoers rule is not consumed by use.
|
||||
# The agent re-checks the band in code before exec (defence in depth); this is the outer fence.
|
||||
Cmnd_Alias FELHOM_SCRATCH_TEARDOWN = \
|
||||
/usr/sbin/pct destroy 99000[0-9] --purge
|
||||
|
||||
# Network storage / NAS (Part A1, SPIKE-nas-storage-2026-06-29). The agent mounts a customer NAS share
|
||||
# HOST-SIDE under /mnt/felhom-drives/<name> via a systemd .automount (+ .mount) pair so it propagates
|
||||
# into the guest through the existing shared bind (an unprivileged LXC cannot mount NFS/CIFS itself).
|
||||
@@ -223,6 +245,15 @@ Cmnd_Alias FELHOM_SSHD = \
|
||||
# blind to an `applied`-but-401 tier. It is NOT a general file-read: the wrapper pins the directory
|
||||
# and prefix-asserts the resolved path, and the id grammar admits no slash. The secret goes to
|
||||
# STDOUT, never argv — sudo logs argv.
|
||||
# E-2a: the backup-target storage shim. Creating a PVE storage needs Datastore.Allocate at /storage
|
||||
# and the grant needs Permissions.Modify -- the agent holds NEITHER by design (blast-radius
|
||||
# containment; Permissions.Modify would let it rewrite its own authority). Both live behind this
|
||||
# fixed-vocabulary root shim instead, exactly like the mkfs and pbs-apply wrappers. The wrapper has
|
||||
# NO storage-removal path, enforces is_mountpoint 1, and refuses a target on the root device.
|
||||
Cmnd_Alias FELHOM_BACKUPTARGET = \
|
||||
/usr/local/sbin/felhom-backup-target-apply create *, \
|
||||
/usr/local/sbin/felhom-backup-target-apply grant *
|
||||
|
||||
Cmnd_Alias FELHOM_PBSDR = \
|
||||
/usr/local/sbin/felhom-pbs-apply create *, \
|
||||
/usr/local/sbin/felhom-pbs-apply reconcile *, \
|
||||
@@ -274,4 +305,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
|
||||
/usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
|
||||
/usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||||
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET
|
||||
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
|
||||
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
#!/bin/bash
|
||||
#===============================================================================
|
||||
# felhom-backup-target-apply — the ONLY path the felhom-agent sudoers permits for creating the
|
||||
# whole-guest backup TARGET storage and granting the agent access to it (E-2a).
|
||||
#
|
||||
# WHY A WRAPPER AT ALL. Creating a PVE storage needs `Datastore.Allocate` at `/storage`, and the ACL
|
||||
# grant needs `Permissions.Modify`. The agent holds NEITHER by design — its token is scoped per
|
||||
# storage path for blast-radius containment, and `Permissions.Modify` would let it rewrite its own
|
||||
# authority. Widening the PVE role to make the move possible would trade the entire containment model
|
||||
# for one feature. So the privileged half lives here: a minimal, auditable root shim with a fixed
|
||||
# vocabulary, exactly like felhom-mkfs-guarded and felhom-pbs-apply.
|
||||
#
|
||||
# THE NO-DELETE LAW (inherited from felhom-pbs-apply, same reasoning class). This wrapper contains NO
|
||||
# storage-removal path of any kind. `pvesm remove` on a dir storage does not delete the archives, but
|
||||
# it DOES silently orphan a configured backup tier, and a "cleanup" verb here would be reachable by
|
||||
# any bug in the agent. Retiring a target is a deliberate operator op, not this tool. Grep-assertable;
|
||||
# do not add one.
|
||||
#
|
||||
# THE TWO LAWS E-1 PAID FOR ON LIVE HARDWARE, both enforced here rather than trusted to the caller:
|
||||
#
|
||||
# F-1 the storage path must BE the drive's own mountpoint. A subdirectory fails the agent's
|
||||
# exactMount check, so the target reports `disconnected` FOREVER and its durable id degrades
|
||||
# off the filesystem UUID. Enforced: `mountpoint -q` must pass on the exact path given.
|
||||
#
|
||||
# F-2 --is_mountpoint 1 is not optional. Without it, an unplugged or late-mounting drive leaves a
|
||||
# bare directory on the ROOT filesystem and vzdump writes the whole-guest backup onto the
|
||||
# system drive — the exact device the whole change exists to escape — while PVE reports the
|
||||
# storage `active` and advertises the root filesystem's free space. Proven live: the unguarded
|
||||
# form had already created dump/ on pve-root. Hardcoded below; not a caller-supplied flag.
|
||||
#
|
||||
# Ops (all non-secret; nothing here touches a credential, so nothing arrives on stdin):
|
||||
# create <id> <mountpoint>
|
||||
# Create a `dir` storage with content=backup at <mountpoint>, is_mountpoint 1.
|
||||
# IDEMPOTENT: an existing entry with the SAME path is accepted (re-run safe, and the
|
||||
# installer re-run path depends on it). An existing entry with a DIFFERENT path is REFUSED
|
||||
# — silently repointing a live backup target is the failure this whole arc closes.
|
||||
# grant <id>
|
||||
# The dual grant: FelhomAgentStore on /storage/<id> to the agent user AND token (privsep
|
||||
# intersection — a token's rights are the intersection, so granting one is granting neither).
|
||||
# Without it every backup 403s on first run (E-1 finding F-3, found by the first real backup).
|
||||
#===============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
die() { echo "felhom-backup-target-apply: REFUSED: $*" >&2; exit 1; }
|
||||
|
||||
op="${1:-}"; id="${2:-}"
|
||||
[[ -n "$op" && -n "$id" ]] || die "usage: felhom-backup-target-apply <create|grant> <storage-id> [mountpoint]"
|
||||
|
||||
# Storage id: PVE grammar, conservative. Also the ACL path component — no slashes possible.
|
||||
[[ "$id" =~ ^[A-Za-z][A-Za-z0-9_.-]{0,27}$ ]] || die "bad storage id ($id)"
|
||||
|
||||
STORECFG=/etc/pve/storage.cfg
|
||||
|
||||
# current_path_of <id> — the configured `path` of dir storage <id>, or "" when absent/not-a-dir.
|
||||
current_path_of() {
|
||||
awk -v want="dir: $1" '
|
||||
$0 == want { found=1; next }
|
||||
found && /^[a-z]+: / { exit }
|
||||
found && $1 == "path" { print $2; exit }
|
||||
' "$STORECFG" 2>/dev/null || true
|
||||
}
|
||||
|
||||
case "$op" in
|
||||
create)
|
||||
[[ $# -eq 3 ]] || die "create takes <id> <mountpoint>"
|
||||
mp="$3"
|
||||
# Absolute, normalized, no traversal, no shell metacharacters. The value reaches pvesm and the
|
||||
# filesystem, so it is validated here rather than assumed well-formed.
|
||||
[[ "$mp" = /* ]] || die "mountpoint must be absolute ($mp)"
|
||||
[[ "$mp" != *".."* ]] || die "mountpoint must not contain .. ($mp)"
|
||||
[[ "$mp" =~ ^[A-Za-z0-9/_.-]+$ ]] || die "mountpoint has unexpected characters ($mp)"
|
||||
[[ "$mp" != "/" ]] || die "refusing / as a backup target"
|
||||
|
||||
# F-1 + F-2, checked as one: the path must BE a mountpoint right now. A bare directory here is
|
||||
# precisely the silent-retarget shape, and is_mountpoint would make PVE refuse it later anyway —
|
||||
# better to refuse now, with a reason, than to create a storage that can never activate.
|
||||
mountpoint -q "$mp" || die "$mp is not a mountpoint — the backup target must be the drive's OWN mountpoint (F-1), and an unmounted path would silently retarget onto the system drive (F-2)"
|
||||
|
||||
# Never the system disk: a target on the root filesystem is not drive-loss protection, it is the
|
||||
# thing we are escaping. The root device and the candidate's device are compared, not their paths.
|
||||
root_dev="$(findmnt -no SOURCE / 2>/dev/null || true)"
|
||||
mp_dev="$(findmnt -no SOURCE "$mp" 2>/dev/null || true)"
|
||||
[[ -n "$mp_dev" ]] || die "could not resolve the backing device of $mp"
|
||||
[[ "$mp_dev" != "$root_dev" ]] || die "$mp is backed by the ROOT device ($root_dev) — a backup target there protects against corruption only, never drive loss"
|
||||
|
||||
existing="$(current_path_of "$id")"
|
||||
if [[ -n "$existing" ]]; then
|
||||
if [[ "$existing" == "$mp" ]]; then
|
||||
echo "felhom-backup-target-apply: storage $id already exists at $mp — nothing to do (idempotent)" >&2
|
||||
exit 0
|
||||
fi
|
||||
die "storage $id already exists at $existing — refusing to repoint it at $mp (a live backup target is never silently moved)"
|
||||
fi
|
||||
|
||||
# is_mountpoint 1 is HARDCODED (F-2). content=backup only: this storage exists for vzdump archives
|
||||
# and must never become a place guests are allocated on.
|
||||
pvesm add dir "$id" --path "$mp" --content backup --is_mountpoint 1 >&2
|
||||
echo "felhom-backup-target-apply: created dir storage $id at $mp (content=backup, is_mountpoint 1)" >&2
|
||||
;;
|
||||
grant)
|
||||
[[ $# -eq 2 ]] || die "grant takes only <id>"
|
||||
# BOTH, always. A privsep token's rights are the intersection of the user's and the token's ACLs,
|
||||
# so granting one of the two grants nothing usable.
|
||||
pveum acl modify "/storage/$id" --users felhom-agent@pve --roles FelhomAgentStore >&2
|
||||
pveum acl modify "/storage/$id" --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore >&2
|
||||
echo "felhom-backup-target-apply: granted FelhomAgentStore on /storage/$id (user + token)" >&2
|
||||
;;
|
||||
*)
|
||||
die "unknown op ($op)"
|
||||
;;
|
||||
esac
|
||||
@@ -1,20 +1,30 @@
|
||||
# felhom-agent local API — host firewall narrowing (doc 03 §6, slice 8A)
|
||||
# felhom-agent local API — host firewall narrowing (doc 03 §6; R-50 island update 2026-07-25)
|
||||
#
|
||||
# Defense-in-depth for the per-guest local API (the controller→agent channel on the host
|
||||
# bridge). The PER-GUEST BEARER TOKEN is the authorization gate; this firewall rule is an
|
||||
# ADDITIONAL layer that limits who can even reach the port. The slice-8A spike found no rule
|
||||
# was needed for reachability on the demo (PVE firewall off) — this narrows exposure so that
|
||||
# only guests on the bridge subnet (not arbitrary LAN hosts) can open a connection.
|
||||
# Defense-in-depth for the per-guest local API (the controller→agent channel). The PER-GUEST BEARER
|
||||
# TOKEN + the served-leaf pin are the authorization gate; a firewall rule is only an ADDITIONAL layer
|
||||
# limiting who can even open the port.
|
||||
#
|
||||
# The agent already binds the listener to the host BRIDGE IP (local_api.listen_addr), not
|
||||
# 0.0.0.0. This file adds the subnet restriction. Apply it at HOST SETUP (it is a host-level
|
||||
# packet-filter change, intentionally OUTSIDE the agent's 3-exception privileged fence — the
|
||||
# agent never mutates the host firewall at runtime).
|
||||
# === R-50 ISLAND INSTALL (the default on a fresh appliance) =================================
|
||||
# The agent binds local_api.listen_addr on the HOST-INTERNAL island bridge — 169.254.253.1:8443 on
|
||||
# vmbr9, a bridge with NO physical port (bridge-ports none). That bind is the security win:
|
||||
# * Nothing listens on the LAN IP at all, so no LAN host (or off-site attacker on the LAN) can
|
||||
# reach the local API — the LAN:8443 surface is CLOSED by the bind, not by a rule.
|
||||
# * vmbr9 has no uplink, so 169.254.253.1:8443 is reachable ONLY from the one guest wired to the
|
||||
# /30 (169.254.253.2) — the controller. The portless bridge is the isolation.
|
||||
# So on an island install NO firewall rule is required for exposure; the topology provides it. If you
|
||||
# want belt-and-suspenders, restrict the port to the island bridge (it changes nothing, since nothing
|
||||
# off-bridge can route to a portless bridge anyway):
|
||||
#
|
||||
# Replace the bridge IP (192.168.0.162), port (8443), and the guest bridge subnet
|
||||
# (192.168.0.0/24) with this host's values.
|
||||
# nft add rule inet filter input iifname != "vmbr9" ip daddr 169.254.253.1 tcp dport 8443 drop
|
||||
#
|
||||
# Verify: from the guest, a TLS connect to 169.254.253.1:8443 succeeds; there is no LAN listener to
|
||||
# probe (`ss -lnt 'sport = :8443'` shows only the island IP).
|
||||
#
|
||||
# === LEGACY LAN BIND (byo, --no-island, or an explicit --bridge-ip) =========================
|
||||
# When the agent still binds a LAN bridge IP (e.g. 192.168.0.162:8443), the port is exposed to the
|
||||
# whole LAN and the subnet-narrowing rule below is worth applying. Replace the bridge IP, port, and
|
||||
# the guest bridge subnet with this host's values.
|
||||
#
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# Option A — nftables (recommended on PVE 8/9; inet filter table). Insert ABOVE any accept:
|
||||
#
|
||||
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
|
||||
@@ -22,13 +32,11 @@
|
||||
# nft add rule inet filter input ip daddr 192.168.0.162 tcp dport 8443 \
|
||||
# ip saddr 192.168.0.0/24 accept
|
||||
#
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# Option B — iptables:
|
||||
#
|
||||
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -s 192.168.0.0/24 -j ACCEPT
|
||||
# iptables -A INPUT -d 192.168.0.162 -p tcp --dport 8443 -j DROP
|
||||
#
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# Option C — PVE host firewall (/etc/pve/nodes/<node>/host.fw), if the PVE firewall is enabled.
|
||||
# Add under [RULES] (and ensure the firewall is enabled in cluster.fw / host.fw):
|
||||
#
|
||||
@@ -36,5 +44,6 @@
|
||||
# IN ACCEPT -source 192.168.0.0/24 -dport 8443 -proto tcp -log nolog
|
||||
# IN DROP -dport 8443 -proto tcp -log nolog
|
||||
#
|
||||
# Verify after applying: from a guest ON the bridge, a TLS connect to <bridge-ip>:8443 succeeds;
|
||||
# from an OFF-bridge host it is refused/dropped. (The token + leaf-pin still gate the request.)
|
||||
# Apply at HOST SETUP — a host-level packet-filter change, intentionally OUTSIDE the agent's
|
||||
# 3-exception privileged fence (the agent never mutates the host firewall at runtime). The token +
|
||||
# leaf-pin still gate the request regardless of which bind is in force.
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// F-CRIT-2 (Campaign 8): a failed backup must not look like a fresh one.
|
||||
//
|
||||
// Every fixture below is a VERBATIM shape captured from the live PVE API on 2026-07-28
|
||||
// (`pvesh get /nodes/<node>/storage/<store>/content`), not a hand-invented struct. That matters:
|
||||
// the `unparseable` path in this package went untested for months behind a JSON shape that did not
|
||||
// match production, and the whole point of this fix is that presence != validity.
|
||||
|
||||
// phantomEntry is the artefact a PBS daemon killed mid-upload leaves behind: listed as a restorable
|
||||
// backup, 1 byte, NEWEST, and carrying no `verification`/`encrypted`/`notes` at all because it has
|
||||
// no manifest (`index.json.blob` is absent on disk).
|
||||
func phantomEntry() proxmox.StorageContent {
|
||||
return proxmox.StorageContent{
|
||||
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z",
|
||||
Content: "backup",
|
||||
Format: "pbs-ct",
|
||||
Size: 1,
|
||||
CTime: 1785216674,
|
||||
VMID: 9201,
|
||||
}
|
||||
}
|
||||
|
||||
// goodPBSEntry is a real, complete offsite snapshot (demo-hp, 2026-07-28T03:40:42Z).
|
||||
func goodPBSEntry() proxmox.StorageContent {
|
||||
return proxmox.StorageContent{
|
||||
VolID: "felhom-pbs:backup/ct/9201/2026-07-28T03:40:42Z",
|
||||
Content: "backup",
|
||||
Format: "pbs-ct",
|
||||
Size: 4353457559,
|
||||
CTime: 1785210042,
|
||||
VMID: 9201,
|
||||
}
|
||||
}
|
||||
|
||||
// goodLocalEntry is a real, complete LOCAL vzdump (demo-hp). Note it legitimately has no
|
||||
// `verification` and no `encrypted` on the wire — a dir storage has no such concept — which is
|
||||
// exactly why those fields must never be used as completeness discriminators.
|
||||
func goodLocalEntry() proxmox.StorageContent {
|
||||
return proxmox.StorageContent{
|
||||
VolID: "local:backup/vzdump-lxc-9201-2026_07_28-07_29_54.tar.zst",
|
||||
Content: "backup",
|
||||
Format: "tar.zst",
|
||||
Size: 1590431865,
|
||||
CTime: 1785216594,
|
||||
VMID: 9201,
|
||||
}
|
||||
}
|
||||
|
||||
func runnerWithContent(t *testing.T, buf *bytes.Buffer, content []proxmox.StorageContent) *BackupRunner {
|
||||
t.Helper()
|
||||
lg := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
|
||||
return NewBackupRunner(&fakeBackupAPI{content: content}, "felhom-pbs", proxmox.ModeSnapshot, "", "", lg)
|
||||
}
|
||||
|
||||
// Group A — the phantom must NOT set tier freshness, even though it is the newest entry.
|
||||
//
|
||||
// RED-PROOF: restore the old predicate in NewestArchiveTime
|
||||
// (`if e.Content == "backup" && e.VMID == vmid && e.CTime > best`) → the phantom's ctime
|
||||
// (1785216674) wins over the good snapshot's (1785210042) and this test fails with
|
||||
// "got 1785216674, want 1785210042" — i.e. the exact F-CRIT-2 defect.
|
||||
func TestNewestArchiveTime_PhantomIsNotCounted(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
// phantom deliberately listed FIRST and is also the newest by ctime.
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
|
||||
|
||||
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatalf("NewestArchiveTime: %v", err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("found=false — the GOOD snapshot must still be counted; rejecting everything is the thrash path")
|
||||
}
|
||||
if got.Unix() != goodPBSEntry().CTime {
|
||||
t.Errorf("freshness came from the wrong entry: got ctime %d, want %d (the good snapshot)", got.Unix(), goodPBSEntry().CTime)
|
||||
}
|
||||
if got.Unix() == phantomEntry().CTime {
|
||||
t.Error("the 1-byte manifest-less phantom set tier freshness — this is F-CRIT-2")
|
||||
}
|
||||
}
|
||||
|
||||
// Group A — with ONLY a phantom present the tier must report "no backup", not a fresh one.
|
||||
// That is what lets the controller see age_state=absent and fire its first-backup valve.
|
||||
func TestNewestArchiveTime_OnlyPhantomReportsNotFound(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry()})
|
||||
|
||||
_, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatalf("NewestArchiveTime: %v", err)
|
||||
}
|
||||
if found {
|
||||
t.Error("found=true with only a phantom present — the tier would report fresh and go silent for a full cadence")
|
||||
}
|
||||
}
|
||||
|
||||
// Group B — THE SCENARIO-D GUARD. A valid snapshot on EITHER tier must still be counted.
|
||||
//
|
||||
// This is what makes Group A safe. A filter that is too aggressive does not merely lose safety
|
||||
// margin: the tier reports absent on every poll, backs up every cycle, and the R-88 breaker cannot
|
||||
// save it because those backups SUCCEED. That is a continuous multi-GB write loop across the fleet.
|
||||
//
|
||||
// RED-PROOF: make archivePlausiblyComplete return `false, "reject everything"` unconditionally →
|
||||
// both subtests fail with found=false.
|
||||
func TestNewestArchiveTime_ValidSnapshotsAreStillCounted(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
entry proxmox.StorageContent
|
||||
}{
|
||||
{"pbs offsite (has verification+encrypted on the wire)", goodPBSEntry()},
|
||||
{"local dir vzdump (has NEITHER verification NOR encrypted — and must still count)", goodLocalEntry()},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{tc.entry})
|
||||
|
||||
got, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatalf("NewestArchiveTime: %v", err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("a REAL %s backup was rejected — this is the backup-thrash path, not extra safety", tc.name)
|
||||
}
|
||||
if got.Unix() != tc.entry.CTime {
|
||||
t.Errorf("got ctime %d, want %d", got.Unix(), tc.entry.CTime)
|
||||
}
|
||||
if strings.Contains(buf.String(), "INCOMPLETE archive") {
|
||||
t.Errorf("a valid archive was announced as incomplete:\n%s", buf.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Group B — the smallest REAL backup measured anywhere on the fleet (612,397,450 B, a guest-9100
|
||||
// vzdump) must clear the floor with room to spare. If someone ever raises
|
||||
// minPlausibleArchiveBytes past this, that is the fleet-thrash bug and this test is the tripwire.
|
||||
func TestMinPlausibleArchiveBytes_LeavesHeadroomBelowTheSmallestRealBackup(t *testing.T) {
|
||||
const smallestObservedRealBackup int64 = 612397450 // fleet survey 2026-07-28
|
||||
if minPlausibleArchiveBytes >= smallestObservedRealBackup {
|
||||
t.Fatalf("floor %d B is not below the smallest real backup ever observed (%d B) — this WILL reject real archives",
|
||||
minPlausibleArchiveBytes, smallestObservedRealBackup)
|
||||
}
|
||||
if ratio := smallestObservedRealBackup / minPlausibleArchiveBytes; ratio < 100 {
|
||||
t.Errorf("floor %d B leaves only %dx headroom below the smallest real backup (%d B) — too tight",
|
||||
minPlausibleArchiveBytes, ratio, smallestObservedRealBackup)
|
||||
}
|
||||
}
|
||||
|
||||
// Group C — UNDECIDABLE ⇒ NOT COUNTED (the fail-safe direction).
|
||||
//
|
||||
// A zero/absent size is not evidence of a good backup; it is absence of evidence. Erring toward
|
||||
// "not fresh" costs one extra backup. Erring the other way is F-CRIT-2.
|
||||
//
|
||||
// RED-PROOF: flip the comparison in archivePlausiblyComplete to `e.Size > minPlausibleArchiveBytes
|
||||
// || e.Size == 0` (i.e. treat unknown as complete) → the size-0 case reports ok=true and this fails.
|
||||
func TestArchivePlausiblyComplete_UndecidableIsNotCounted(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
size int64
|
||||
}{
|
||||
{"the observed phantom", 1},
|
||||
{"absent size field (unmarshals to 0)", 0},
|
||||
{"just under the floor", minPlausibleArchiveBytes - 1},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
e := phantomEntry()
|
||||
e.Size = tc.size
|
||||
ok, why := archivePlausiblyComplete(e)
|
||||
if ok {
|
||||
t.Errorf("size %d counted as a complete backup — undecidable must fail safe", tc.size)
|
||||
}
|
||||
if why == "" {
|
||||
t.Error("rejection carried no reason — a silent rejection is a new quiet path")
|
||||
}
|
||||
})
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(goodPBSEntry()); !ok {
|
||||
t.Errorf("a real snapshot was rejected: %s", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Group D — the rejection is announced ONCE per snapshot, not once per due-check.
|
||||
//
|
||||
// The due-check runs every 5 minutes and a phantom persists indefinitely (server-side prune does
|
||||
// not collect it), so per-poll logging would emit ~288 identical lines a day and bury the signal.
|
||||
//
|
||||
// RED-PROOF: delete the `if seen { return }` guard in warnRejectedArchiveOnce → this test reports
|
||||
// "logged 5 times, want 1".
|
||||
func TestNewestArchiveTime_RejectionLoggedOncePerSnapshot(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), goodPBSEntry()})
|
||||
|
||||
const polls = 5
|
||||
for i := 0; i < polls; i++ {
|
||||
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
|
||||
t.Fatalf("poll %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
n := strings.Count(buf.String(), "INCOMPLETE archive")
|
||||
if n != 1 {
|
||||
t.Errorf("rejection logged %d times across %d polls, want exactly 1:\n%s", n, polls, buf.String())
|
||||
}
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, phantomEntry().VolID) {
|
||||
t.Errorf("the log line does not NAME the rejected snapshot:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "below the") {
|
||||
t.Errorf("the log line does not say WHY it was rejected:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "level=WARN") {
|
||||
t.Errorf("rejection was not logged at WARN:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// Group D — a SECOND, distinct phantom is announced separately. The dedupe must be per snapshot,
|
||||
// not a one-shot latch that hides every later phantom.
|
||||
func TestNewestArchiveTime_DistinctPhantomsEachAnnounced(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
second := phantomEntry()
|
||||
second.VolID = "felhom-pbs:backup/ct/9201/2026-07-29T05:31:14Z"
|
||||
second.CTime = phantomEntry().CTime + 86400
|
||||
r := runnerWithContent(t, &buf, []proxmox.StorageContent{phantomEntry(), second, goodPBSEntry()})
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
if _, _, err := r.NewestArchiveTime(context.Background(), 9201); err != nil {
|
||||
t.Fatalf("poll %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(buf.String(), "INCOMPLETE archive"); n != 2 {
|
||||
t.Errorf("got %d rejection lines for 2 distinct phantoms across 3 polls, want 2:\n%s", n, buf.String())
|
||||
}
|
||||
}
|
||||
@@ -135,10 +135,11 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
const big = 4 << 30 // a plausible whole-guest archive
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "a", Content: "backup", CTime: 10},
|
||||
{VolID: "b", Content: "backup", CTime: 99},
|
||||
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
|
||||
{VolID: "a", Content: "backup", CTime: 10, Size: big},
|
||||
{VolID: "b", Content: "backup", CTime: 99, Size: big},
|
||||
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
@@ -152,6 +153,26 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
|
||||
// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
|
||||
// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
|
||||
// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
|
||||
// PickSettledRestoreCandidateOn and this fails with
|
||||
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
|
||||
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
|
||||
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
if err != nil || vol != "real" {
|
||||
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- scheduler ---
|
||||
|
||||
type fakeRTRunner struct {
|
||||
@@ -168,9 +189,12 @@ func TestScheduler_TickRunsAndRecords(t *testing.T) {
|
||||
store := NewStore()
|
||||
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
|
||||
s := NewScheduler(SchedulerOptions{
|
||||
Runner: rt,
|
||||
Pick: func(context.Context) (string, error) { return "vol", nil },
|
||||
Store: store,
|
||||
Runner: rt,
|
||||
Pick: func(context.Context) (string, error) { return "vol", nil },
|
||||
Store: store,
|
||||
Spec: func(context.Context, string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
|
||||
},
|
||||
Cadence: time.Hour,
|
||||
Logger: quiet(),
|
||||
})
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package backup
|
||||
|
||||
import "sync"
|
||||
|
||||
// InFlight is the host-wide "one heavy guest operation at a time" gate.
|
||||
//
|
||||
// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
|
||||
// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
|
||||
// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
|
||||
// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
|
||||
// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
|
||||
// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
|
||||
// up recorded as failed.
|
||||
//
|
||||
// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
|
||||
// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
|
||||
//
|
||||
// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
|
||||
// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
|
||||
// coverage; cancelling a running backup costs the backup.
|
||||
//
|
||||
// CORRECTED 2026-07-28 (F-A1). That "DEFERS" was true of the restore-test caller and NOT of the
|
||||
// backup caller, and the comment did not say so. The controller's start path had no 409 branch, so
|
||||
// a refusal here was recorded as a tier FAILURE: the R-88 breaker armed and the operator was
|
||||
// emailed "Whole-guest backup FAILED" about a backup that was merely waiting its turn. Campaign 8
|
||||
// observed it on both demo boxes in the same minute.
|
||||
//
|
||||
// Fixed on the CONTROLLER side (v0.179.0), which is where the misreading lived — this gate's
|
||||
// behaviour was correct throughout and is unchanged. The controller now maps HTTP 409 to a
|
||||
// contention path: it defers the tier, keeps it DUE, and alarms only if contention outlives the
|
||||
// agent's own restore-test ceiling. Nothing here needs to change; the claim above is simply now
|
||||
// true of both callers.
|
||||
type InFlight struct {
|
||||
mu sync.Mutex
|
||||
what string // "" = idle
|
||||
}
|
||||
|
||||
// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
|
||||
// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
|
||||
func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
|
||||
if g == nil {
|
||||
// Not wired (older call sites, tests) → no gating, previous behaviour.
|
||||
return func() {}, "", true
|
||||
}
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
if g.what != "" {
|
||||
return nil, g.what, false
|
||||
}
|
||||
g.what = what
|
||||
var once sync.Once
|
||||
return func() {
|
||||
once.Do(func() {
|
||||
g.mu.Lock()
|
||||
g.what = ""
|
||||
g.mu.Unlock()
|
||||
})
|
||||
}, "", true
|
||||
}
|
||||
|
||||
// Busy reports what currently holds the gate ("" = idle).
|
||||
func (g *InFlight) Busy() string {
|
||||
if g == nil {
|
||||
return ""
|
||||
}
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
return g.what
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-86 — a restore-test follows the BACKUP, not the clock.
|
||||
//
|
||||
// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
|
||||
// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
|
||||
// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
|
||||
// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
|
||||
// sometimes twice on the same archive.
|
||||
//
|
||||
// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
|
||||
//
|
||||
// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
|
||||
// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
|
||||
// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
|
||||
// The naive rule silently switches restore-testing off for the tier that matters most, and it is
|
||||
// the version a reasonable person would write. It has a red-proof of its own
|
||||
// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
|
||||
//
|
||||
// The rule implemented here:
|
||||
//
|
||||
// Let A = the newest archive on this tier that is at least `settle` old.
|
||||
// The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
|
||||
//
|
||||
// daily tier → A is yesterday's archive; a new one settles each day → proved once per day
|
||||
// weekly tier → A is last week's until the next settles → proved once per week
|
||||
// newborn tier → A does not exist → UNKNOWN, never a fault
|
||||
//
|
||||
// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
|
||||
// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
|
||||
// produce a cadence nobody can predict from either.
|
||||
//
|
||||
// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
|
||||
// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
|
||||
// the trigger changed.
|
||||
|
||||
// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
|
||||
// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
|
||||
type DueVerdict struct {
|
||||
Target string // the tier's storage target id
|
||||
|
||||
// Due is true only when Archive is set and has not been proven.
|
||||
Due bool
|
||||
// Archive is the settled candidate A ("" when the tier holds none).
|
||||
Archive string
|
||||
// Landed is when A landed on the tier (zero when Archive is "").
|
||||
Landed time.Time
|
||||
// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
|
||||
ProvenArchive string
|
||||
// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
|
||||
// NEVER reported as "not due", which would silently retire a tier the moment its storage
|
||||
// stopped answering. Due stays false (we have no archive to test) and the error travels.
|
||||
Err error
|
||||
// Reason is the one-line human account of this verdict.
|
||||
Reason string
|
||||
}
|
||||
|
||||
// String renders a verdict for the operator log / selftest output.
|
||||
func (v DueVerdict) String() string {
|
||||
return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
|
||||
}
|
||||
|
||||
// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
|
||||
//
|
||||
// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
|
||||
// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
|
||||
// still never be starved — a tier that has waited longest is served first — and keeping it as the
|
||||
// order rather than as the trigger is the whole of this change.
|
||||
func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
|
||||
if !s.rotating() {
|
||||
return nil
|
||||
}
|
||||
order := s.tiers
|
||||
if s.rtState != nil {
|
||||
order = s.rtState.OldestFirst(s.tiers)
|
||||
}
|
||||
cutoff := s.settleCutoff()
|
||||
out := make([]DueVerdict, 0, len(order))
|
||||
for _, target := range order {
|
||||
out = append(out, s.evaluateTier(ctx, target, cutoff))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// settleCutoff is the newest landing time an archive may have and still count as settled.
|
||||
func (s *Scheduler) settleCutoff() time.Time {
|
||||
if s.settle <= 0 {
|
||||
return time.Time{} // no settle requirement configured → any archive is a candidate
|
||||
}
|
||||
return s.now().Add(-s.settle)
|
||||
}
|
||||
|
||||
// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
|
||||
// unit-tested directly rather than inferred from whether a fake runner happened to be called.
|
||||
func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
|
||||
v := DueVerdict{Target: target}
|
||||
archive, landed, err := s.tierPick(ctx, target, cutoff)
|
||||
if err != nil {
|
||||
// UNKNOWN, never "not due", and never silent.
|
||||
v.Err = err
|
||||
v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
|
||||
return v
|
||||
}
|
||||
v.Archive, v.Landed = archive, landed
|
||||
if archive == "" {
|
||||
v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
|
||||
return v
|
||||
}
|
||||
proven, ok := "", false
|
||||
if s.rtState != nil {
|
||||
proven, ok = s.rtState.ProvenArchive(target)
|
||||
}
|
||||
v.ProvenArchive = proven
|
||||
if ok && proven == archive {
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
|
||||
return v
|
||||
}
|
||||
v.Due = true
|
||||
switch {
|
||||
case !ok && proven == "":
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
|
||||
default:
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
|
||||
// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
|
||||
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
|
||||
return s.evaluateTier(ctx, target, s.settleCutoff())
|
||||
}
|
||||
|
||||
// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
|
||||
//
|
||||
// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
|
||||
// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
|
||||
// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
|
||||
// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
|
||||
// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
|
||||
func (s *Scheduler) verdictSummary(ctx context.Context) string {
|
||||
out := ""
|
||||
for _, v := range s.EvaluateDue(ctx) {
|
||||
if out != "" {
|
||||
out += "; "
|
||||
}
|
||||
switch {
|
||||
case v.Err != nil:
|
||||
out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
|
||||
default:
|
||||
out += v.Target + ": " + v.Reason
|
||||
}
|
||||
}
|
||||
if out == "" {
|
||||
return "no tiers configured"
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,593 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-86 — the restore-test follows the BACKUP, not the clock.
|
||||
//
|
||||
// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
|
||||
// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
|
||||
// asserted "no error" would pass against a scheduler that never ran anything at all — which is
|
||||
// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
|
||||
// archive, and did a second evaluation correctly run NOTHING.
|
||||
|
||||
// ── the fake tier storage ────────────────────────────────────────────────────────────────────
|
||||
|
||||
// archiveStub is one archive on a tier: its volid and when it landed.
|
||||
type archiveStub struct {
|
||||
volid string
|
||||
landed time.Time
|
||||
}
|
||||
|
||||
// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
|
||||
// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
|
||||
// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
|
||||
// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
|
||||
// picker; that the real picker obeys the same rule is asserted there, not here.
|
||||
type tierStorage struct {
|
||||
archives map[string][]archiveStub
|
||||
err map[string]error // target → lookup failure
|
||||
}
|
||||
|
||||
func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
|
||||
if e, ok := ts.err[target]; ok && e != nil {
|
||||
return "", time.Time{}, e
|
||||
}
|
||||
var best archiveStub
|
||||
for _, a := range ts.archives[target] {
|
||||
if !notAfter.IsZero() && a.landed.After(notAfter) {
|
||||
continue // not settled yet
|
||||
}
|
||||
if best.volid == "" || a.landed.After(best.landed) {
|
||||
best = a
|
||||
}
|
||||
}
|
||||
return best.volid, best.landed, nil
|
||||
}
|
||||
|
||||
// dueHarness is a scheduler with a driven clock over a fake tier storage.
|
||||
type dueHarness struct {
|
||||
s *Scheduler
|
||||
rr *rotRunner
|
||||
st *RestoreTestState
|
||||
ts *tierStorage
|
||||
clock time.Time
|
||||
path string
|
||||
}
|
||||
|
||||
func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
|
||||
t.Helper()
|
||||
return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
|
||||
}
|
||||
|
||||
func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
|
||||
t.Helper()
|
||||
h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
|
||||
h.st = NewRestoreTestState(statePath)
|
||||
h.s = NewScheduler(SchedulerOptions{
|
||||
Runner: h.rr,
|
||||
Store: NewStore(),
|
||||
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
|
||||
},
|
||||
Cadence: time.Hour,
|
||||
Settle: settle,
|
||||
Logger: quiet(),
|
||||
Tiers: tiers,
|
||||
TierPick: ts.pick,
|
||||
State: h.st,
|
||||
InFlight: &InFlight{},
|
||||
})
|
||||
h.s.now = func() time.Time { return h.clock }
|
||||
return h
|
||||
}
|
||||
|
||||
// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
|
||||
func (h *dueHarness) advance(step, total time.Duration) {
|
||||
for elapsed := time.Duration(0); elapsed < total; elapsed += step {
|
||||
h.clock = h.clock.Add(step)
|
||||
h.s.tick(context.Background())
|
||||
}
|
||||
}
|
||||
|
||||
var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
|
||||
|
||||
// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
|
||||
func dailyArchives(tier string, n int) []archiveStub {
|
||||
out := make([]archiveStub, 0, n)
|
||||
for d := 0; d < n; d++ {
|
||||
out = append(out, archiveStub{
|
||||
volid: fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
|
||||
landed: day0.AddDate(0, 0, d),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
|
||||
//
|
||||
// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
|
||||
// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
|
||||
// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
|
||||
// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
|
||||
// the tier that matters most.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
|
||||
// was replaced by the naive age rule:
|
||||
//
|
||||
// - if ok && proven == archive { … not due … }
|
||||
// - if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
|
||||
//
|
||||
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
|
||||
// old enough". Result:
|
||||
//
|
||||
// --- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
|
||||
// restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
|
||||
//
|
||||
// Zero runs — restore-testing off. Restored immediately afterwards.
|
||||
func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
|
||||
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
// Five days, evaluated hourly.
|
||||
h.advance(time.Hour, 5*24*time.Hour)
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 5 {
|
||||
t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
|
||||
}
|
||||
// And each run must be on the archive that settled that day — day0's on day 1, and so on.
|
||||
for i, a := range got {
|
||||
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
|
||||
if a != want {
|
||||
t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
|
||||
}
|
||||
}
|
||||
// The newest archive is NEVER the one tested: it has not settled.
|
||||
if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
|
||||
t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
|
||||
func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
|
||||
{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
|
||||
{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
|
||||
{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
|
||||
}}}
|
||||
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
|
||||
// Three weeks, evaluated every 6 hours — 84 evaluations.
|
||||
h.advance(6*time.Hour, 21*24*time.Hour)
|
||||
|
||||
got := h.rr.seen()
|
||||
want := []string{
|
||||
"felhom-pbs:backup/ct/9201/w0",
|
||||
"felhom-pbs:backup/ct/9201/w1",
|
||||
"felhom-pbs:backup/ct/9201/w2",
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
|
||||
//
|
||||
// This is the defect a person actually notices: today every deploy restarts the ticker, so a
|
||||
// restore-test runs one interval after each deploy regardless of what has already been proven.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
|
||||
// ProvenArchive ignore the stored archive —
|
||||
//
|
||||
// - if !ok || p.Archive == "" { return "", false }
|
||||
// - return "", false // per-tier time only, the pre-R-86 state
|
||||
//
|
||||
// → --- FAIL: TestDue_RestartRunsNothing
|
||||
//
|
||||
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
|
||||
// produced 4 run(s)
|
||||
//
|
||||
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
|
||||
// today's behaviour with the ticker's phase reset by the deploy. Restored.
|
||||
func TestDue_RestartRunsNothing(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "rt.json")
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
|
||||
start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
|
||||
|
||||
h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
|
||||
}
|
||||
|
||||
// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
|
||||
total := 0
|
||||
for i := 0; i < 2; i++ {
|
||||
h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
h2.s.tick(context.Background())
|
||||
h2.s.tick(context.Background())
|
||||
total += len(h2.rr.seen())
|
||||
}
|
||||
if total != 0 {
|
||||
t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
|
||||
func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
h.s.tick(context.Background()) // proves day0's archive
|
||||
h.s.tick(context.Background()) // nothing new has settled → nothing
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
|
||||
}
|
||||
|
||||
// Day 2, 03:00 — day1's archive has now settled.
|
||||
h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
|
||||
h.s.tick(context.Background())
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
|
||||
}
|
||||
if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
|
||||
t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
|
||||
//
|
||||
// - if rt.Pass && s.rtState != nil && target != "" {
|
||||
// - if s.rtState != nil && target != "" {
|
||||
//
|
||||
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
|
||||
//
|
||||
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
|
||||
// evaluations
|
||||
//
|
||||
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
|
||||
// freshly verified, which is the loudest signal this system produces going silent. Restored.
|
||||
func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
h.s.tick(context.Background())
|
||||
}
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
|
||||
}
|
||||
if _, ok := h.st.ProvenArchive("local"); ok {
|
||||
t.Fatal("a FAILED restore-test must not record the archive as proven")
|
||||
}
|
||||
if _, ok := h.st.LastSuccess("local"); ok {
|
||||
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
|
||||
func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{
|
||||
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
|
||||
"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
|
||||
}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
|
||||
// Both tiers are due at this instant.
|
||||
due := h.s.EvaluateDue(context.Background())
|
||||
if len(due) != 2 || !due[0].Due || !due[1].Due {
|
||||
t.Fatalf("precondition: both tiers should be due; got %v", due)
|
||||
}
|
||||
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
|
||||
}
|
||||
|
||||
// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
|
||||
h.s.tick(context.Background())
|
||||
got := h.rr.seen()
|
||||
if len(got) != 2 || got[0] == got[1] {
|
||||
t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
|
||||
func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
gate := &InFlight{}
|
||||
h.s.inFlight = gate
|
||||
release, _, _ := gate.TryAcquire("backup:felhom-pbs")
|
||||
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
|
||||
}
|
||||
if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
|
||||
t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
|
||||
}
|
||||
release()
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("must resume once the gate frees; got %d run(s)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
|
||||
func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
|
||||
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
|
||||
due := h.s.EvaluateDue(context.Background())
|
||||
if len(due) != 1 {
|
||||
t.Fatalf("want one verdict, got %v", due)
|
||||
}
|
||||
v := due[0]
|
||||
if v.Due || v.Err != nil || v.Archive != "" {
|
||||
t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
|
||||
}
|
||||
if v.Reason == "" {
|
||||
t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
|
||||
func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
|
||||
t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("nothing settled → no run; got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
|
||||
// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
|
||||
// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
|
||||
func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
|
||||
boom := errors.New("storage unreachable")
|
||||
ts := &tierStorage{
|
||||
archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
|
||||
err: map[string]error{"felhom-pbs": boom},
|
||||
}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
|
||||
var pbs DueVerdict
|
||||
for _, v := range h.s.EvaluateDue(context.Background()) {
|
||||
if v.Target == "felhom-pbs" {
|
||||
pbs = v
|
||||
}
|
||||
}
|
||||
if pbs.Err == nil {
|
||||
t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
|
||||
}
|
||||
if pbs.Due {
|
||||
t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
|
||||
}
|
||||
if pbs.Reason == "" {
|
||||
t.Fatal("the failure must be explained, not merely flagged")
|
||||
}
|
||||
|
||||
// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
|
||||
// tier its proof.
|
||||
h.s.tick(context.Background())
|
||||
if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
|
||||
t.Fatalf("the readable tier must still be proved; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
|
||||
|
||||
// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
|
||||
// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
|
||||
// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
|
||||
// invented by a migration.
|
||||
func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
|
||||
if err := writeFileForTest(path, legacy); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
st := NewRestoreTestState(path)
|
||||
if _, ok := st.ProvenArchive("local"); ok {
|
||||
t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
|
||||
}
|
||||
at, ok := st.LastSuccess("local")
|
||||
if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
|
||||
}
|
||||
// Ordering still works off the legacy times.
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||
t.Fatalf("oldest-first must still order legacy records; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The new shape round-trips, archive and all.
|
||||
func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
st := NewRestoreTestState(path)
|
||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
re := NewRestoreTestState(path)
|
||||
got, ok := re.ProvenArchive("felhom-pbs")
|
||||
if !ok || got != "felhom-pbs:backup/ct/9201/x" {
|
||||
t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
|
||||
}
|
||||
at, ok := re.LastSuccess("felhom-pbs")
|
||||
if !ok || !at.Equal(now) {
|
||||
t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
|
||||
func writeFileForTest(path, content string) error {
|
||||
return os.WriteFile(path, []byte(content), 0o600)
|
||||
}
|
||||
|
||||
// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
|
||||
// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
|
||||
// quiet journal is equally consistent with a healthy loop and a dead goroutine.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
|
||||
// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
|
||||
// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
|
||||
func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{
|
||||
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
|
||||
"felhom-pbs": nil, // no archive at all
|
||||
}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
// Prove the local tier so NOTHING is due.
|
||||
if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
|
||||
// test would pass against a tick that never calls it.
|
||||
var logbuf strings.Builder
|
||||
h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
h.s.tick(context.Background())
|
||||
got := logbuf.String()
|
||||
for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
|
||||
// reads as "nothing due" is the silence this rule exists to prevent.
|
||||
func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
|
||||
ts := &tierStorage{
|
||||
archives: map[string][]archiveStub{"local": nil},
|
||||
err: map[string]error{"felhom-pbs": errors.New("storage unreachable")},
|
||||
}
|
||||
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
got := h.s.verdictSummary(context.Background())
|
||||
if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
|
||||
t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
|
||||
//
|
||||
// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
|
||||
// agent will not repeat the work. So the persisted record has to be able to become a host-report
|
||||
// entry — without inventing anything it does not know.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
|
||||
// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
|
||||
//
|
||||
// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
|
||||
// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
|
||||
// per-tier proof on it); got [{... SourceTier: ...}]
|
||||
//
|
||||
// Restored.
|
||||
func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
|
||||
// file has ever had, which is what a real box carries after two upgrades.
|
||||
legacy := `{
|
||||
"old-v1": "2026-07-30T02:11:07Z",
|
||||
"old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
|
||||
"felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
|
||||
}`
|
||||
if err := writeFileForTest(path, legacy); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
|
||||
}
|
||||
e := got[0]
|
||||
if e.SourceTier != "pbs" {
|
||||
t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
|
||||
}
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
|
||||
t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
|
||||
}
|
||||
if e.TestedAt != "2026-08-03T13:25:14Z" {
|
||||
t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
|
||||
}
|
||||
if e.Verified != "boot+running" {
|
||||
t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
|
||||
}
|
||||
// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
|
||||
if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
|
||||
t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
|
||||
e.DurationSeconds, e.ScratchVMID)
|
||||
}
|
||||
// The legacy records still serve the DUE-check, which is a separate question from reporting.
|
||||
if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
|
||||
t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
|
||||
}
|
||||
}
|
||||
|
||||
// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
|
||||
// the production path, not a hand-built fixture.
|
||||
func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
|
||||
h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
|
||||
got := h.st.ProvenRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
|
||||
}
|
||||
if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
|
||||
t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
|
||||
func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
|
||||
h.s.tick(context.Background())
|
||||
if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
|
||||
"would outlive the fault; got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
|
||||
//
|
||||
// R-85 (1.4). This one genuinely needs PERSISTENCE, unlike R-84 — and the difference is worth
|
||||
// stating, because the two look like the same problem and are not:
|
||||
//
|
||||
// - R-84 (backup freshness) had a GROUND TRUTH to consult: the archive is still on the storage,
|
||||
// so the agent could ask "when did a backup last land?" and never persist anything. That is
|
||||
// strictly better, because a pruned archive correctly stops counting.
|
||||
// - A restore-test leaves NO artifact — the scratch guest is destroyed as its final act. There is
|
||||
// nothing to query. "Did we prove this tier restores?" exists only as remembered state, so it
|
||||
// must be written down or it is lost.
|
||||
//
|
||||
// Why it must survive a restart: rotation is oldest-first (the operator ruling), so an in-memory map
|
||||
// would reset every tier to "never tested" on each restart. Ordering would then depend on map
|
||||
// iteration order, and one tier could be starved indefinitely while the other is re-tested — with
|
||||
// agent deploys as routine as they are, that is not a corner case.
|
||||
//
|
||||
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
|
||||
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
|
||||
// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
|
||||
// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
|
||||
// ever becoming due again. The rule earns its keep twice now.
|
||||
//
|
||||
// R-86 (1.2) — WHICH ARCHIVE, not just when.
|
||||
//
|
||||
// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
|
||||
// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
|
||||
// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
|
||||
// a tier is due once it holds a settled archive that has not been proven — so the identity of the
|
||||
// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
|
||||
//
|
||||
// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
|
||||
// here it records a result, but not WHICH result, and that is just as unable to answer the question
|
||||
// being asked of it.
|
||||
type RestoreTestState struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
last map[string]provenTier // target id → what was last PROVEN on that tier
|
||||
}
|
||||
|
||||
// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
|
||||
// and when.
|
||||
//
|
||||
// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
|
||||
// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
|
||||
// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
|
||||
// hub can stay ignorant of a real success until the next archive generation.
|
||||
//
|
||||
// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
|
||||
// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
|
||||
// at report-building time — a network call that can fail, on a path where failing means mis-labelling
|
||||
// a proof. Store what you knew when you knew it.
|
||||
type provenTier struct {
|
||||
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
|
||||
Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
|
||||
Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
|
||||
At time.Time // when that run passed (UTC)
|
||||
}
|
||||
|
||||
// reportable reports whether this record can be re-reported to the hub as a restore-test result.
|
||||
//
|
||||
// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
|
||||
// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
|
||||
// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
|
||||
// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
|
||||
func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
|
||||
|
||||
// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
|
||||
//
|
||||
// v1 (pre-R-86) "<target>": "<RFC3339>" — a time, no archive
|
||||
// v2 (R-86) "<target>": {archive, proven_at} — due-check usable, not reportable
|
||||
// v3 (R-189) "<target>": {archive, tier, verified, …} — both
|
||||
//
|
||||
// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
|
||||
// readers above test for — the migration needs no version number because the absence IS the answer.
|
||||
type provenTierJSON struct {
|
||||
Archive string `json:"archive"`
|
||||
Tier string `json:"tier,omitempty"`
|
||||
Verified string `json:"verified,omitempty"`
|
||||
ProvenAt string `json:"proven_at"`
|
||||
}
|
||||
|
||||
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
|
||||
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
|
||||
// corrupt file from wedging the daemon.
|
||||
//
|
||||
// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
|
||||
// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
|
||||
// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
|
||||
// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
|
||||
// read a legacy time as proof of whatever archive happens to be current, which would mark an
|
||||
// unproven archive proven — inventing a guarantee out of a migration.
|
||||
func NewRestoreTestState(path string) *RestoreTestState {
|
||||
s := &RestoreTestState{path: path, last: map[string]provenTier{}}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
var raw map[string]json.RawMessage
|
||||
if json.Unmarshal(data, &raw) != nil {
|
||||
return s
|
||||
}
|
||||
for target, msg := range raw {
|
||||
// Legacy shape: a bare RFC3339 string.
|
||||
var legacy string
|
||||
if json.Unmarshal(msg, &legacy) == nil {
|
||||
if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
|
||||
s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
|
||||
}
|
||||
continue
|
||||
}
|
||||
var cur provenTierJSON
|
||||
if json.Unmarshal(msg, &cur) != nil {
|
||||
continue // one unreadable entry must not lose the others
|
||||
}
|
||||
t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
|
||||
if perr != nil {
|
||||
continue
|
||||
}
|
||||
s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
|
||||
// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
|
||||
// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
|
||||
//
|
||||
// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
|
||||
// the next reader will notice failures are absent and try to "fix" it:
|
||||
//
|
||||
// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
|
||||
// the system quietly less tested than it believes. It must survive a restart.
|
||||
//
|
||||
// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
|
||||
// so a lost failure heals itself within one interval. Persisting it would do the opposite of
|
||||
// helping: a healed tier would keep reporting a failure that is no longer true.
|
||||
func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
|
||||
if target == "" {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
|
||||
return s.saveLocked()
|
||||
}
|
||||
|
||||
// LastSuccess returns when this tier was last proven (ok=false = never).
|
||||
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
p, ok := s.last[target]
|
||||
return p.At, ok
|
||||
}
|
||||
|
||||
// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
|
||||
// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
|
||||
// not this one has not been proven.
|
||||
func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
p, ok := s.last[target]
|
||||
if !ok || p.Archive == "" {
|
||||
return "", false
|
||||
}
|
||||
return p.Archive, true
|
||||
}
|
||||
|
||||
// Snapshot returns a copy of the last-proven TIMES.
|
||||
//
|
||||
// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
|
||||
// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
|
||||
// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
|
||||
// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
|
||||
// only the times; if it acquires none, delete it rather than let it claim a purpose again.
|
||||
func (s *RestoreTestState) Snapshot() map[string]time.Time {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make(map[string]time.Time, len(s.last))
|
||||
for k, v := range s.last {
|
||||
out[k] = v.At
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
|
||||
//
|
||||
// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
|
||||
// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
|
||||
//
|
||||
// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
|
||||
// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
|
||||
// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
|
||||
// is not a claim; a fabricated one would be.
|
||||
//
|
||||
// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
|
||||
// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
|
||||
// worse defect than the one this fixes.
|
||||
func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]hub.RestoreTest, 0, len(s.last))
|
||||
for _, p := range s.last {
|
||||
if !p.reportable() {
|
||||
continue
|
||||
}
|
||||
out = append(out, hub.RestoreTest{
|
||||
SourceArchive: p.Archive,
|
||||
SourceTier: p.Tier,
|
||||
Pass: true,
|
||||
Verified: p.Verified,
|
||||
TestedAt: p.At.UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
|
||||
// iteration is randomised.
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
|
||||
return out
|
||||
}
|
||||
|
||||
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
|
||||
//
|
||||
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
|
||||
// naturally prioritises a tier that has never been restore-tested at all — which on this fleet was
|
||||
// the offsite tier, unproven for its entire existence.
|
||||
//
|
||||
// Ties break on target id so the order is deterministic; without that, two tiers proven in the same
|
||||
// second would rotate by map iteration order, which is randomised in Go and would make the
|
||||
// behaviour untestable and occasionally starving.
|
||||
func (s *RestoreTestState) OldestFirst(targets []string) []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := append([]string(nil), targets...)
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
pi, oki := s.last[out[i]]
|
||||
pj, okj := s.last[out[j]]
|
||||
ti, tj := pi.At, pj.At
|
||||
switch {
|
||||
case !oki && !okj:
|
||||
return out[i] < out[j] // both never proven → deterministic
|
||||
case !oki:
|
||||
return true // never proven wins
|
||||
case !okj:
|
||||
return false
|
||||
case !ti.Equal(tj):
|
||||
return ti.Before(tj)
|
||||
default:
|
||||
return out[i] < out[j]
|
||||
}
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *RestoreTestState) saveLocked() error {
|
||||
raw := make(map[string]provenTierJSON, len(s.last))
|
||||
for target, p := range s.last {
|
||||
raw[target] = provenTierJSON{
|
||||
Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
|
||||
ProvenAt: p.At.UTC().Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
data, err := json.MarshalIndent(raw, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := s.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, s.path)
|
||||
}
|
||||
@@ -0,0 +1,354 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-85 Phase 2 — tier rotation, persisted state, and the one-heavy-operation gate.
|
||||
//
|
||||
// The failure this prevents is not hypothetical: demo-hp's DR tier reported `applied` with ZERO
|
||||
// snapshots for five days and nobody noticed, because the scheduler could only ever see the primary
|
||||
// tier. Rotation is what makes the offsite tier testable at all.
|
||||
|
||||
// rotRunner records which archives it was asked to restore.
|
||||
type rotRunner struct {
|
||||
mu sync.Mutex
|
||||
archives []string
|
||||
pass bool
|
||||
}
|
||||
|
||||
func (r *rotRunner) RunRestoreTest(_ context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.archives = append(r.archives, spec.Archive)
|
||||
return reconcile.RestoreTestResult{
|
||||
Archive: spec.Archive, SourceTier: spec.SourceTier,
|
||||
Pass: r.pass, Verified: "boot+running",
|
||||
}
|
||||
}
|
||||
func (r *rotRunner) seen() []string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]string(nil), r.archives...)
|
||||
}
|
||||
|
||||
// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
|
||||
// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
|
||||
// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
|
||||
var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
|
||||
func archiveFor(m map[string]string) TierPicker {
|
||||
return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
|
||||
a := m[target]
|
||||
if a == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
return a, testLanded, nil
|
||||
}
|
||||
}
|
||||
|
||||
func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPicker, gate *InFlight) *Scheduler {
|
||||
t.Helper()
|
||||
return NewScheduler(SchedulerOptions{
|
||||
Runner: rr,
|
||||
Store: NewStore(),
|
||||
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
|
||||
},
|
||||
Cadence: time.Hour,
|
||||
Logger: quiet(),
|
||||
Tiers: []string{"local", "felhom-pbs"},
|
||||
TierPick: pick,
|
||||
State: st,
|
||||
InFlight: gate,
|
||||
})
|
||||
}
|
||||
|
||||
// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
|
||||
//
|
||||
// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
|
||||
// it was a faithful statement of the defect. It read:
|
||||
//
|
||||
// 4 ticks → 4 runs, and consecutive runs must hit different tiers
|
||||
//
|
||||
// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
|
||||
// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
|
||||
// newest settled archive is already proven is not re-tested just because time passed. So the
|
||||
// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
|
||||
// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
|
||||
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
|
||||
// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
|
||||
// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
|
||||
func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
|
||||
rr := &rotRunner{pass: true}
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
|
||||
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
|
||||
"felhom-pbs": "felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
|
||||
}), &InFlight{})
|
||||
s.now = func() time.Time { return time.Now().UTC() }
|
||||
|
||||
for i := 0; i < 4; i++ {
|
||||
s.tick(context.Background())
|
||||
}
|
||||
|
||||
got := rr.seen()
|
||||
var sawLocal, sawPBS bool
|
||||
for _, a := range got {
|
||||
if len(a) >= 5 && a[:5] == "local" {
|
||||
sawLocal = true
|
||||
}
|
||||
if len(a) >= 10 && a[:10] == "felhom-pbs" {
|
||||
sawPBS = true
|
||||
}
|
||||
}
|
||||
if !sawLocal || !sawPBS {
|
||||
t.Fatalf("both tiers must be exercised; got %v", got)
|
||||
}
|
||||
// Exactly one run per tier: the archives never changed, so nothing became due a second time.
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
|
||||
}
|
||||
if got[0] == got[1] {
|
||||
t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A tier with NO archive is skipped, not failed, and the other tier still runs. A brand-new offsite
|
||||
// tier legitimately has nothing to restore; turning that into a failure would make every fresh box
|
||||
// look broken for its first week.
|
||||
func TestRotation_EmptyTierSkippedNotFailed(t *testing.T) {
|
||||
rr := &rotRunner{pass: true}
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
|
||||
"local": "local:backup/vzdump-lxc-9201-x.tar.zst",
|
||||
"felhom-pbs": "", // provisioned but empty
|
||||
}), &InFlight{})
|
||||
|
||||
s.tick(context.Background())
|
||||
got := rr.seen()
|
||||
if len(got) != 1 || got[0][:5] != "local" {
|
||||
t.Fatalf("an empty tier must be skipped and the testable one still run; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Nothing testable anywhere → a clean no-op, not an error and not a run.
|
||||
func TestRotation_NoArchivesAnywhereIsANoOp(t *testing.T) {
|
||||
rr := &rotRunner{pass: true}
|
||||
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
|
||||
archiveFor(map[string]string{}), &InFlight{})
|
||||
s.tick(context.Background())
|
||||
if got := rr.seen(); len(got) != 0 {
|
||||
t.Fatalf("no archives anywhere → no run; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED restore-test must NOT earn rotation credit, or a tier that fails every time would look
|
||||
// freshly proven and quietly stop being retried.
|
||||
func TestRotation_FailureEarnsNoCredit(t *testing.T) {
|
||||
rr := &rotRunner{pass: false}
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
|
||||
"local": "local:backup/x.tar.zst",
|
||||
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
|
||||
}), &InFlight{})
|
||||
s.tick(context.Background())
|
||||
if _, ok := st.LastSuccess("local"); ok {
|
||||
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
|
||||
}
|
||||
if _, ok := st.LastSuccess("felhom-pbs"); ok {
|
||||
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO E — rotation survives a restart ─────────────────────────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make the state in-memory (construct a fresh
|
||||
// `NewRestoreTestState` on a DIFFERENT path for the second scheduler, i.e. lose the file) and this
|
||||
// fails with "after a restart the OTHER tier must be next; got felhom-pbs" — the same tier repeats
|
||||
// and the other is starved indefinitely, which with agent deploys as routine as they are is not a
|
||||
// corner case.
|
||||
func TestRotation_SurvivesRestart(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "rt.json")
|
||||
picks := archiveFor(map[string]string{
|
||||
"local": "local:backup/x.tar.zst",
|
||||
"felhom-pbs": "felhom-pbs:backup/ct/9201/y",
|
||||
})
|
||||
|
||||
// First process: the OFFSITE tier is tested (never-proven sorts first).
|
||||
rr1 := &rotRunner{pass: true}
|
||||
st1 := NewRestoreTestState(path)
|
||||
s1 := rotScheduler(t, rr1, st1, picks, &InFlight{})
|
||||
s1.tick(context.Background())
|
||||
first := rr1.seen()
|
||||
if len(first) != 1 {
|
||||
t.Fatalf("want one run, got %v", first)
|
||||
}
|
||||
|
||||
// --- restart: brand-new state object reading the SAME file ---
|
||||
rr2 := &rotRunner{pass: true}
|
||||
st2 := NewRestoreTestState(path)
|
||||
s2 := rotScheduler(t, rr2, st2, picks, &InFlight{})
|
||||
s2.tick(context.Background())
|
||||
second := rr2.seen()
|
||||
if len(second) != 1 {
|
||||
t.Fatalf("want one run after restart, got %v", second)
|
||||
}
|
||||
|
||||
if second[0] == first[0] {
|
||||
t.Fatalf("after a restart the OTHER tier must be next; got %s twice (rotation state was lost)", second[0])
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — no collision with a backup ──────────────────────────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): drop the TryAcquire guard from `tick` and this fails with
|
||||
// "the restore-test must DEFER while a backup holds the gate; concurrent operations = 2" — the
|
||||
// count is the assertion, since "both completed" would pass against a fully concurrent
|
||||
// implementation.
|
||||
func TestRotation_DefersWhileABackupHoldsTheGate(t *testing.T) {
|
||||
gate := &InFlight{}
|
||||
release, _, ok := gate.TryAcquire("backup:felhom-pbs")
|
||||
if !ok {
|
||||
t.Fatal("precondition: the gate should have been free")
|
||||
}
|
||||
defer release()
|
||||
|
||||
rr := &rotRunner{pass: true}
|
||||
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
|
||||
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
|
||||
|
||||
s.tick(context.Background())
|
||||
|
||||
concurrent := 1 + len(rr.seen()) // the backup holding the gate, plus anything the tick started
|
||||
if concurrent != 1 {
|
||||
t.Fatalf("the restore-test must DEFER while a backup holds the gate; concurrent operations = %d", concurrent)
|
||||
}
|
||||
}
|
||||
|
||||
// Once the backup releases, the next cadence proceeds — deferral must not be permanent.
|
||||
func TestRotation_ResumesAfterTheGateFrees(t *testing.T) {
|
||||
gate := &InFlight{}
|
||||
release, _, _ := gate.TryAcquire("backup:local")
|
||||
|
||||
rr := &rotRunner{pass: true}
|
||||
s := rotScheduler(t, rr, NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")),
|
||||
archiveFor(map[string]string{"local": "local:backup/x.tar.zst"}), gate)
|
||||
|
||||
s.tick(context.Background())
|
||||
if len(rr.seen()) != 0 {
|
||||
t.Fatal("should have deferred while held")
|
||||
}
|
||||
release()
|
||||
s.tick(context.Background())
|
||||
if len(rr.seen()) != 1 {
|
||||
t.Fatalf("must resume once the gate frees; got %v", rr.seen())
|
||||
}
|
||||
}
|
||||
|
||||
// The gate itself: one holder at a time, named, and release is idempotent.
|
||||
func TestInFlight_Semantics(t *testing.T) {
|
||||
g := &InFlight{}
|
||||
rel, busy, ok := g.TryAcquire("backup:local")
|
||||
if !ok || busy != "" {
|
||||
t.Fatalf("first acquire must succeed; ok=%v busy=%q", ok, busy)
|
||||
}
|
||||
if _, busy2, ok2 := g.TryAcquire("restore-test"); ok2 || busy2 != "backup:local" {
|
||||
t.Fatalf("second acquire must fail and NAME the holder; ok=%v busy=%q", ok2, busy2)
|
||||
}
|
||||
rel()
|
||||
rel() // idempotent — a double release must not free someone else's later claim
|
||||
if g.Busy() != "" {
|
||||
t.Fatalf("gate should be idle after release; busy=%q", g.Busy())
|
||||
}
|
||||
if _, _, ok3 := g.TryAcquire("restore-test"); !ok3 {
|
||||
t.Fatal("gate must be reusable after release")
|
||||
}
|
||||
}
|
||||
|
||||
// A nil gate means "not wired" → no gating, pre-R-85 behaviour. Keeps every existing caller working.
|
||||
func TestInFlight_NilIsUngated(t *testing.T) {
|
||||
var g *InFlight
|
||||
rel, _, ok := g.TryAcquire("x")
|
||||
if !ok {
|
||||
t.Fatal("a nil gate must not block")
|
||||
}
|
||||
rel()
|
||||
if g.Busy() != "" {
|
||||
t.Fatal("a nil gate is never busy")
|
||||
}
|
||||
}
|
||||
|
||||
// ── oldest-first ordering ────────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestOldestFirst_Ordering(t *testing.T) {
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
now := time.Now().UTC()
|
||||
|
||||
// Never-proven sorts FIRST — the case that matters, since the offsite tier starts there.
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||
// both never proven → deterministic tie-break by id
|
||||
if got[0] != "felhom-pbs" && got[0] != "local" {
|
||||
t.Fatalf("unexpected: %v", got)
|
||||
}
|
||||
}
|
||||
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
|
||||
}
|
||||
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
|
||||
t.Fatalf("the least recently proven must sort first; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Ordering must be DETERMINISTIC for equal timestamps, or two tiers proven in the same second would
|
||||
// rotate by Go's randomised map iteration — untestable, and occasionally starving.
|
||||
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
now := time.Now().UTC()
|
||||
_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
|
||||
_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
|
||||
for i := 0; i < 20; i++ {
|
||||
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
|
||||
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The state file round-trips, and a corrupt file degrades to "nothing proven" rather than wedging.
|
||||
func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "rt.json")
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
|
||||
st := NewRestoreTestState(path)
|
||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reopened := NewRestoreTestState(path)
|
||||
got, ok := reopened.LastSuccess("felhom-pbs")
|
||||
if !ok || !got.Equal(now) {
|
||||
t.Fatalf("state must round-trip; got %v ok=%v want %v", got, ok, now)
|
||||
}
|
||||
|
||||
bad := filepath.Join(dir, "corrupt.json")
|
||||
if err := os.WriteFile(bad, []byte("{{{not json"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := NewRestoreTestState(bad)
|
||||
if _, ok := c.LastSuccess("felhom-pbs"); ok {
|
||||
t.Fatal("a corrupt state file must degrade to 'nothing proven', not invent a timestamp")
|
||||
}
|
||||
}
|
||||
+216
-10
@@ -6,6 +6,7 @@ import (
|
||||
"log/slog"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
@@ -38,21 +39,57 @@ type BackupRunner struct {
|
||||
// each successful backup, so the agent's own backups can't pile up and refill root. Empty → no prune
|
||||
// (the legacy behaviour; restore-test/selftest runners pass ""). NEVER applied to a PBS target.
|
||||
retention string
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// waitTimeout bounds the WaitTask poll on this runner's vzdump. Per-TIER since R-82: 30m is
|
||||
// right for a local vzdump and badly wrong for an offsite PBS upload (see the 2026-07-26 live
|
||||
// failure recorded on config.BackupTargetConfig.WaitTimeoutSeconds). 0 → 30m (legacy).
|
||||
waitTimeout time.Duration
|
||||
// allowPBSPrune permits `--prune-backups` on a PBS-type target. OFF by default and ON only for
|
||||
// an ADDITIONAL tier whose keep_last was set explicitly (operator ruling 2026-07-26: keep two
|
||||
// weeks of weekly offsite backups).
|
||||
//
|
||||
// The blanket PBS refusal it replaces existed for a real reason and still applies to the
|
||||
// PRIMARY tier: BackupTarget() DEFAULTS to "felhom-pbs" and KeepLast() DEFAULTS to 3, so a box
|
||||
// with neither key set would silently prune its offsite DR to 3 restore points. An additional
|
||||
// tier cannot have that accident — its keep_last defaults to 0 (never prune), so any value
|
||||
// there is a deliberate act.
|
||||
allowPBSPrune bool
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
|
||||
// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
|
||||
// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
|
||||
// due-check is served from the local-API handler goroutines.
|
||||
rejectedMu sync.Mutex
|
||||
rejected map[string]struct{}
|
||||
}
|
||||
|
||||
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
|
||||
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
|
||||
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
|
||||
func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, logger *slog.Logger) *BackupRunner {
|
||||
return NewBackupRunnerWithWait(api, target, mode, notes, retention, 0, logger)
|
||||
}
|
||||
|
||||
// NewBackupRunnerWithWait is NewBackupRunner plus an explicit vzdump wait bound (0 → 30m).
|
||||
func NewBackupRunnerWithWait(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, logger *slog.Logger) *BackupRunner {
|
||||
return NewBackupRunnerFull(api, target, mode, notes, retention, waitTimeout, false, logger)
|
||||
}
|
||||
|
||||
// NewBackupRunnerFull is the full constructor. allowPBSPrune must be true ONLY for an additional
|
||||
// tier with an explicitly configured keep_last — see BackupRunner.allowPBSPrune.
|
||||
func NewBackupRunnerFull(api BackupAPI, target string, mode proxmox.BackupMode, notes, retention string, waitTimeout time.Duration, allowPBSPrune bool, logger *slog.Logger) *BackupRunner {
|
||||
if mode == "" {
|
||||
mode = proxmox.ModeSnapshot
|
||||
}
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention, logger: logger, now: func() time.Time { return time.Now().UTC() }}
|
||||
if waitTimeout <= 0 {
|
||||
waitTimeout = 30 * time.Minute
|
||||
}
|
||||
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, retention: retention,
|
||||
waitTimeout: waitTimeout, allowPBSPrune: allowPBSPrune, logger: logger,
|
||||
now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
// localPruneSpec returns the `--prune-backups` spec to apply to THIS backup, or "" to skip pruning. It
|
||||
@@ -71,8 +108,11 @@ func (r *BackupRunner) localPruneSpec(ctx context.Context) string {
|
||||
}
|
||||
for _, s := range stores {
|
||||
if s.Storage == r.target {
|
||||
if s.Type == "pbs" {
|
||||
return "" // PBS retention is out of scope — never prune the offsite DR
|
||||
if s.Type == "pbs" && !r.allowPBSPrune {
|
||||
// Not opted in → never prune the offsite DR (the pre-R-82 rule, and still the rule
|
||||
// for the primary tier, whose target+retention both DEFAULT and could prune by
|
||||
// accident).
|
||||
return ""
|
||||
}
|
||||
return r.retention
|
||||
}
|
||||
@@ -147,7 +187,7 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
defer stopWatch()
|
||||
go r.watchForSnapshot(watchCtx, upid, onSnapshot)
|
||||
}
|
||||
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: 30 * time.Minute}); err != nil {
|
||||
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: r.waitTimeout}); err != nil {
|
||||
rec.Error = err.Error()
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
|
||||
@@ -210,18 +250,69 @@ func (r *BackupRunner) watchForSnapshot(ctx context.Context, upid string, onSnap
|
||||
// PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
|
||||
// when there is none — the restore-test then no-ops cleanly.
|
||||
func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
|
||||
contents, err := r.api.StorageContent(ctx, r.target)
|
||||
return r.PickRestoreCandidateOn(ctx, r.target)
|
||||
}
|
||||
|
||||
// PickRestoreCandidateOn is PickRestoreCandidate for an ARBITRARY tier's storage (R-85 1.2), so the
|
||||
// scheduler can rotate across tiers instead of only ever seeing this runner's own target.
|
||||
//
|
||||
// Contract preserved: "" + nil error when the storage holds no archive. **A tier with nothing to
|
||||
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
|
||||
// that into a failure would make every fresh box look broken for its first week.
|
||||
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
|
||||
archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
|
||||
return archive, err
|
||||
}
|
||||
|
||||
// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
|
||||
// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
|
||||
// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
|
||||
// one-line call into this and its contract is untouched (one scan, one owner).
|
||||
//
|
||||
// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
|
||||
// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
|
||||
// restore-testing the archive a backup is still writing proves nothing about the backup that
|
||||
// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
|
||||
// on that: the tier is due when a settled archive exists that has not been proven.
|
||||
//
|
||||
// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
|
||||
// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
|
||||
// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
|
||||
// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
|
||||
// the archive generation rather than by the evaluation interval.
|
||||
//
|
||||
// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
|
||||
// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
|
||||
// turning that into a failure would make every fresh box look broken for its first week.
|
||||
func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
|
||||
if target == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
contents, err := r.api.StorageContent(ctx, target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return "", time.Time{}, err
|
||||
}
|
||||
var best string
|
||||
var bestCTime int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content == "backup" && e.CTime > bestCTime {
|
||||
if e.Content != "backup" {
|
||||
continue
|
||||
}
|
||||
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
|
||||
continue // not settled yet — a newer archive is not a reason to re-prove an older one
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
}
|
||||
if e.CTime > bestCTime {
|
||||
bestCTime, best = e.CTime, e.VolID
|
||||
}
|
||||
}
|
||||
return best, nil
|
||||
if best == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
return best, time.Unix(bestCTime, 0).UTC(), nil
|
||||
}
|
||||
|
||||
// latestArchive finds the newest backup archive volid + size for vmid on the target.
|
||||
@@ -243,6 +334,121 @@ func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int
|
||||
return vol, size, nil
|
||||
}
|
||||
|
||||
// NewestArchiveTime reports when this guest's newest backup archive LANDED ON THIS TARGET, from the
|
||||
// storage itself. ok=false means the target genuinely holds no archive for this guest.
|
||||
//
|
||||
// R-84: this is the cure for the redundant-backup-after-restart problem. The agent's backup Store is
|
||||
// in-memory ("lost on restart; the cadence re-populates"), so after every restart /backup/due
|
||||
// reported "no successful backup recorded yet" and the controller dutifully took another one. On the
|
||||
// local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
|
||||
// every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
|
||||
// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
|
||||
// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
|
||||
//
|
||||
// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
|
||||
//
|
||||
// smallest REAL backup anywhere on the fleet ... 612,397,450 B (~584 MiB, a guest-9100 vzdump)
|
||||
// demo-hp local / PBS ..................... 1.59 GB / 4.35-4.37 GB
|
||||
// demo-felhom local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
|
||||
// the phantom left by a PBS daemon killed mid-upload ....... 1 B
|
||||
//
|
||||
// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
|
||||
// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
|
||||
// archive — which is the property that matters, because a floor set too HIGH does not merely lose
|
||||
// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
|
||||
const minPlausibleArchiveBytes int64 = 1 << 20
|
||||
|
||||
// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
|
||||
// why. Pure, so the contract is unit-testable without a storage.
|
||||
//
|
||||
// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
|
||||
// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
|
||||
// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
|
||||
//
|
||||
// - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
|
||||
// storage has no verification concept — and absent on a good PBS snapshot until verify-new
|
||||
// catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
|
||||
// one: continuous re-backup across the fleet.
|
||||
// - `encrypted` fails the same way, and for the same reason.
|
||||
// - `notes` happens to be present on both good tiers today only because the agent sets it; an
|
||||
// archive written by any other path lacks it. Too fragile to gate freshness on.
|
||||
//
|
||||
// Size is the only signal that means the same thing on every tier.
|
||||
//
|
||||
// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
|
||||
// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
|
||||
// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
|
||||
// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
|
||||
func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
|
||||
if e.Size < minPlausibleArchiveBytes {
|
||||
return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
|
||||
e.Size, minPlausibleArchiveBytes)
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
|
||||
//
|
||||
// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
|
||||
// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
|
||||
// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
|
||||
// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
|
||||
// logging per poll would emit ~288 identical lines a day and bury the one that matters.
|
||||
func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
|
||||
r.rejectedMu.Lock()
|
||||
if r.rejected == nil {
|
||||
r.rejected = map[string]struct{}{}
|
||||
}
|
||||
_, seen := r.rejected[e.VolID]
|
||||
if !seen {
|
||||
r.rejected[e.VolID] = struct{}{}
|
||||
}
|
||||
r.rejectedMu.Unlock()
|
||||
if seen {
|
||||
return
|
||||
}
|
||||
r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
|
||||
"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
|
||||
}
|
||||
|
||||
// demo-felhom in a single afternoon of deploys (2026-07-26).
|
||||
//
|
||||
// Asking the STORAGE rather than persisting the store is deliberate:
|
||||
// - it is ground truth, not remembered state — if an archive was pruned or deleted it correctly
|
||||
// stops counting, whereas a persisted record would keep claiming a backup that no longer exists;
|
||||
// - it needs no new on-disk state and no migration;
|
||||
// - it is the same source `latestArchive` already trusts to build the post-backup record.
|
||||
//
|
||||
// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
|
||||
// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
|
||||
// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
|
||||
// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
|
||||
// richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
|
||||
// synthesized record would put invented numbers into the host-report.
|
||||
func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
|
||||
contents, err := r.api.StorageContent(ctx, r.target)
|
||||
if err != nil {
|
||||
return time.Time{}, false, err
|
||||
}
|
||||
var best int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content != "backup" || e.VMID != vmid {
|
||||
continue
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
}
|
||||
if e.CTime > best {
|
||||
best = e.CTime
|
||||
}
|
||||
}
|
||||
if best < 0 {
|
||||
return time.Time{}, false, nil
|
||||
}
|
||||
return time.Unix(best, 0).UTC(), true, nil
|
||||
}
|
||||
|
||||
// parseBackupMode extracts the actual mode from a vzdump task log line `… backup mode: <x>`
|
||||
// (e.g. "INFO: backup mode: stop"). Returns "" if not found.
|
||||
func parseBackupMode(lines []string) string {
|
||||
|
||||
+206
-29
@@ -18,27 +18,77 @@ type RestoreTestRunner interface {
|
||||
// there is none yet (the tick then no-ops).
|
||||
type CandidatePicker func(ctx context.Context) (string, error)
|
||||
|
||||
// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
|
||||
//
|
||||
// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
|
||||
// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
|
||||
// the resulting value reused for every run for the lifetime of the process. Two consequences:
|
||||
// - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
|
||||
// - it was a latent staleness bug in its own right — a storage-type or config change did not take
|
||||
// effect until the daemon restarted.
|
||||
//
|
||||
// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
|
||||
// the configured target: deriving it from config is what produced the 600 s false failure when a
|
||||
// PBS archive was classified "local" and got the 10-minute local wait.
|
||||
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
|
||||
|
||||
// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
|
||||
// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
|
||||
// satisfies it. A zero notAfter means "no settle requirement".
|
||||
//
|
||||
// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
|
||||
// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
|
||||
// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
|
||||
// brand-new offsite tier legitimately has nothing to restore yet.
|
||||
type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
|
||||
|
||||
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
|
||||
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
|
||||
// AND a valid scratch band is configured (validated by the caller before construction).
|
||||
type Scheduler struct {
|
||||
runner RestoreTestRunner
|
||||
pick CandidatePicker
|
||||
store *Store
|
||||
spec reconcile.RestoreTestSpec // archive is filled per-tick
|
||||
runner RestoreTestRunner
|
||||
pick CandidatePicker
|
||||
store *Store
|
||||
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
|
||||
// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
|
||||
// longer the thing that decides a test happens; see restoretest_due.go.
|
||||
cadence time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// settle is how long an archive must have sat before it is a candidate (R-86).
|
||||
settle time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
|
||||
// (single tier via `pick`), which keeps every existing caller and test working untouched.
|
||||
tiers []string // configured tier target ids, primary first
|
||||
tierPick TierPicker // newest archive on a named tier
|
||||
rtState *RestoreTestState // persisted last-successful-per-tier (drives oldest-first)
|
||||
inFlight *InFlight // shared with the backup path — Scenario F
|
||||
}
|
||||
|
||||
// SchedulerOptions configures a Scheduler.
|
||||
type SchedulerOptions struct {
|
||||
Runner RestoreTestRunner
|
||||
Pick CandidatePicker
|
||||
Store *Store
|
||||
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout
|
||||
Cadence time.Duration // 0 → disabled
|
||||
Logger *slog.Logger
|
||||
Runner RestoreTestRunner
|
||||
Pick CandidatePicker
|
||||
Store *Store
|
||||
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
|
||||
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
|
||||
Spec SpecBuilder
|
||||
// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
|
||||
// (R-86). 0 → disabled.
|
||||
Cadence time.Duration
|
||||
// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
|
||||
// 0 → no settle requirement (any archive is a candidate).
|
||||
Settle time.Duration
|
||||
Logger *slog.Logger
|
||||
|
||||
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
|
||||
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
|
||||
// named tier; State persists last-successful-per-tier; InFlight is the shared one-heavy-op gate.
|
||||
Tiers []string
|
||||
TierPick TierPicker
|
||||
State *RestoreTestState
|
||||
InFlight *InFlight
|
||||
}
|
||||
|
||||
// NewScheduler builds a Scheduler.
|
||||
@@ -48,27 +98,43 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Scheduler{
|
||||
runner: opts.Runner,
|
||||
pick: opts.Pick,
|
||||
store: opts.Store,
|
||||
spec: opts.Spec,
|
||||
cadence: opts.Cadence,
|
||||
logger: logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
runner: opts.Runner,
|
||||
pick: opts.Pick,
|
||||
store: opts.Store,
|
||||
spec: opts.Spec,
|
||||
cadence: opts.Cadence,
|
||||
settle: opts.Settle,
|
||||
logger: logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
tiers: append([]string(nil), opts.Tiers...),
|
||||
tierPick: opts.TierPick,
|
||||
rtState: opts.State,
|
||||
inFlight: opts.InFlight,
|
||||
}
|
||||
}
|
||||
|
||||
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
|
||||
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
|
||||
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
|
||||
// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
|
||||
// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
|
||||
//
|
||||
// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
|
||||
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
|
||||
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
|
||||
//
|
||||
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
|
||||
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
|
||||
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
|
||||
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
|
||||
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
|
||||
//
|
||||
// Returns nil on ctx cancellation.
|
||||
func (s *Scheduler) Run(ctx context.Context) error {
|
||||
if s.cadence <= 0 || s.runner == nil || s.pick == nil {
|
||||
if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
|
||||
s.logger.Info("backup: restore-test cadence disabled")
|
||||
<-ctx.Done()
|
||||
return nil
|
||||
}
|
||||
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
|
||||
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
|
||||
"eval_interval", s.cadence, "settle", s.settle)
|
||||
t := time.NewTicker(s.cadence)
|
||||
defer t.Stop()
|
||||
for {
|
||||
@@ -82,19 +148,64 @@ func (s *Scheduler) Run(ctx context.Context) error {
|
||||
}
|
||||
}
|
||||
|
||||
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
|
||||
// no backup exists yet. Deterministic given s.now — tests call it directly.
|
||||
// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
|
||||
// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
|
||||
// is already in flight. Deterministic given s.now — tests call it directly.
|
||||
//
|
||||
// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
|
||||
// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
|
||||
func (s *Scheduler) tick(ctx context.Context) {
|
||||
archive, err := s.pick(ctx)
|
||||
if s.spec == nil {
|
||||
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
|
||||
// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
|
||||
// spec must cost a restore-test, never the agent.
|
||||
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
|
||||
return
|
||||
}
|
||||
// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
|
||||
//
|
||||
// It used to be the other way round, and correctly so: the gate was held for one heavy run a
|
||||
// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
|
||||
// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
|
||||
// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
|
||||
// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
|
||||
// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
|
||||
// not be able to manufacture that.
|
||||
//
|
||||
// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
|
||||
// before anything heavy begins.
|
||||
archive, target, err := s.pickForThisRun(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
|
||||
return
|
||||
}
|
||||
if archive == "" {
|
||||
s.logger.Info("backup: restore-test skipped; no backup available yet")
|
||||
// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
|
||||
//
|
||||
// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
|
||||
// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
|
||||
// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
|
||||
// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
|
||||
// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
|
||||
// nothing run last night?" is in the log rather than in a re-derivation.
|
||||
s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
|
||||
return
|
||||
}
|
||||
spec := s.spec
|
||||
|
||||
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
|
||||
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
|
||||
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
|
||||
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
|
||||
// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
|
||||
release, busy, ok := s.inFlight.TryAcquire("restore-test")
|
||||
if !ok {
|
||||
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
|
||||
"busy", busy, "target", target, "archive", archive)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
|
||||
spec := s.spec(ctx, archive)
|
||||
spec.Archive = archive
|
||||
res := s.runner.RunRestoreTest(ctx, spec)
|
||||
if res.Skipped {
|
||||
@@ -102,6 +213,19 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
}
|
||||
rt := ToHubRestoreTest(res, s.now())
|
||||
s.store.RecordRestoreTest(rt)
|
||||
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
|
||||
// that fails every time would look freshly proven and quietly stop being retried.
|
||||
if rt.Pass && s.rtState != nil && target != "" {
|
||||
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
|
||||
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
|
||||
// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
|
||||
// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
|
||||
// this run was actually judged as, and deriving it later would need a storage lookup that
|
||||
// can fail on the one path where failing means mislabelling a proof.
|
||||
if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
|
||||
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case !rt.Pass:
|
||||
// A failing restore-test is the loudest DR signal there is.
|
||||
@@ -118,3 +242,56 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
"archive", rt.SourceArchive, "duration_s", rt.DurationSeconds, "warnings", res.StartWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
// rotating reports whether multi-tier rotation is wired.
|
||||
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
|
||||
|
||||
// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
|
||||
// order.
|
||||
//
|
||||
// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
|
||||
// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
|
||||
// due?", and "nothing" is a normal, frequent and correct answer.
|
||||
//
|
||||
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
|
||||
// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
|
||||
// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
|
||||
// that a test happens at all.
|
||||
//
|
||||
// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
|
||||
// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
|
||||
// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
|
||||
// storage being unreadable must not cost the other tier its proof, and must not be silent either.
|
||||
//
|
||||
// Returns ("", "", nil) when nothing anywhere is due.
|
||||
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
|
||||
if !s.rotating() {
|
||||
// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
|
||||
// identity and no persisted proof here, so there is nothing to compare an archive against
|
||||
// and no due-check is possible. It runs on every evaluation, exactly as it always did.
|
||||
a, perr := s.pick(ctx)
|
||||
return a, "", perr
|
||||
}
|
||||
var firstErr error
|
||||
for _, v := range s.EvaluateDue(ctx) {
|
||||
if v.Err != nil {
|
||||
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
|
||||
"target", v.Target, "err", v.Err)
|
||||
if firstErr == nil {
|
||||
firstErr = v.Err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !v.Due {
|
||||
s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
|
||||
continue
|
||||
}
|
||||
s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
|
||||
"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
|
||||
return v.Archive, v.Target, nil
|
||||
}
|
||||
if firstErr != nil {
|
||||
return "", "", firstErr
|
||||
}
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-85 (1.1) — the spec is built PER RUN, never frozen at construction.
|
||||
//
|
||||
// It used to be an immediately-invoked function at daemon start, so storageTier() and
|
||||
// restoreTaskTimeout() were evaluated ONCE and the value reused for every run for the process
|
||||
// lifetime. That is what made an offsite restore-test impossible to schedule at all, and it was a
|
||||
// latent staleness bug besides: a storage-type or config change did not take effect until restart.
|
||||
|
||||
type specSpy struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
archives []string
|
||||
tiers []string // what the builder decided, per call
|
||||
}
|
||||
|
||||
func (sp *specSpy) build(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
sp.mu.Lock()
|
||||
defer sp.mu.Unlock()
|
||||
sp.calls++
|
||||
sp.archives = append(sp.archives, archive)
|
||||
// Decide the tier from the ARCHIVE, exactly as main.go does (the v0.100.0 rule).
|
||||
tier := "local"
|
||||
if len(archive) > 10 && archive[:10] == "felhom-pbs" {
|
||||
tier = "pbs"
|
||||
}
|
||||
sp.tiers = append(sp.tiers, tier)
|
||||
return reconcile.RestoreTestSpec{
|
||||
RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: tier,
|
||||
}
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF (observed): change Scheduler.spec back to a frozen
|
||||
// `reconcile.RestoreTestSpec` value captured at construction → this fails with
|
||||
// "the spec builder must run ONCE PER RUN, got 1 call(s) across 3 ticks", because a frozen value is
|
||||
// evaluated exactly once no matter how many ticks fire. Restored.
|
||||
func TestScheduler_SpecIsBuiltPerRun(t *testing.T) {
|
||||
sp := &specSpy{}
|
||||
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
|
||||
n := 0
|
||||
s := NewScheduler(SchedulerOptions{
|
||||
Runner: rt,
|
||||
Pick: func(context.Context) (string, error) {
|
||||
n++
|
||||
return fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", n), nil
|
||||
},
|
||||
Store: NewStore(),
|
||||
Spec: sp.build,
|
||||
Cadence: time.Hour,
|
||||
Logger: quiet(),
|
||||
})
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
s.tick(context.Background())
|
||||
}
|
||||
|
||||
sp.mu.Lock()
|
||||
defer sp.mu.Unlock()
|
||||
if sp.calls != 3 {
|
||||
t.Fatalf("the spec builder must run ONCE PER RUN, got %d call(s) across 3 ticks", sp.calls)
|
||||
}
|
||||
// And it must see the archive THIS run picked — not a stale one.
|
||||
for i, a := range sp.archives {
|
||||
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-%d.tar.zst", i+1)
|
||||
if a != want {
|
||||
t.Fatalf("run %d: builder saw archive %q, want %q — the spec is not tracking the picked archive", i+1, a, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The tier must follow the ARCHIVE across runs. A builder that saw only the configured target would
|
||||
// return the same tier every time — which is exactly the v0.100.0 defect that killed a 14.46 GB WAN
|
||||
// restore at the 10-minute local bound.
|
||||
func TestScheduler_SpecTierFollowsTheArchive(t *testing.T) {
|
||||
sp := &specSpy{}
|
||||
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Pass: true, Verified: "boot+running"}}
|
||||
archives := []string{
|
||||
"local:backup/vzdump-lxc-9201-x.tar.zst",
|
||||
"felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z",
|
||||
}
|
||||
i := 0
|
||||
s := NewScheduler(SchedulerOptions{
|
||||
Runner: rt,
|
||||
Pick: func(context.Context) (string, error) {
|
||||
a := archives[i%len(archives)]
|
||||
i++
|
||||
return a, nil
|
||||
},
|
||||
Store: NewStore(), Spec: sp.build, Cadence: time.Hour, Logger: quiet(),
|
||||
})
|
||||
s.tick(context.Background())
|
||||
s.tick(context.Background())
|
||||
|
||||
sp.mu.Lock()
|
||||
defer sp.mu.Unlock()
|
||||
if len(sp.tiers) != 2 || sp.tiers[0] != "local" || sp.tiers[1] != "pbs" {
|
||||
t.Fatalf("the tier must follow the archive per run; got %v", sp.tiers)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil spec builder must SKIP loudly, not panic — a wiring bug costs a restore-test, never the
|
||||
// daemon goroutine.
|
||||
func TestScheduler_NilSpecSkipsInsteadOfPanicking(t *testing.T) {
|
||||
rt := &fakeRTRunner{}
|
||||
s := NewScheduler(SchedulerOptions{
|
||||
Runner: rt,
|
||||
Pick: func(context.Context) (string, error) { return "vol", nil },
|
||||
Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
|
||||
})
|
||||
s.tick(context.Background()) // must not panic
|
||||
if rt.runs != 0 {
|
||||
t.Fatalf("a nil spec must not run a restore-test; got %d run(s)", rt.runs)
|
||||
}
|
||||
}
|
||||
@@ -10,8 +10,22 @@ import (
|
||||
// Store holds the agent's LATEST backup result per target and the latest restore-test
|
||||
// result — the point-in-time state the host-report surfaces. It is updated by the backup
|
||||
// runner + the restore-test scheduler/selftest and read by the collector via the hub
|
||||
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence
|
||||
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access.
|
||||
// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
|
||||
// collector vs scheduler access.
|
||||
//
|
||||
// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
|
||||
// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
|
||||
// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
|
||||
// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
|
||||
// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
|
||||
// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
|
||||
// consecutive host-reports carried `0 restore-tests`.
|
||||
//
|
||||
// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
|
||||
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
|
||||
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
|
||||
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
|
||||
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
|
||||
type Store struct {
|
||||
mu sync.Mutex
|
||||
byTarget map[string]hub.Backup // latest backup per target id
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-82 Slice A.1 — per-target cadence + retention resolution.
|
||||
//
|
||||
// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
|
||||
// resolve to exactly one tier that behaves as it does now. The second property is that a
|
||||
// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
|
||||
// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
|
||||
// R-82 exists to fix.
|
||||
|
||||
func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
|
||||
// Exactly the shape live on demo-felhom today.
|
||||
var b BackupConfig
|
||||
raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
|
||||
if err := json.Unmarshal([]byte(raw), &b); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tiers, warnings := b.BackupTiers()
|
||||
if len(warnings) != 0 {
|
||||
t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
|
||||
}
|
||||
if len(tiers) != 1 {
|
||||
t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
|
||||
}
|
||||
got := tiers[0]
|
||||
if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
|
||||
t.Fatalf("legacy tier changed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
|
||||
// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
|
||||
func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
|
||||
tiers, warnings := BackupConfig{}.BackupTiers()
|
||||
if len(warnings) != 0 || len(tiers) != 1 {
|
||||
t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
|
||||
}
|
||||
if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
|
||||
t.Fatalf("zero-config defaults changed: %+v", tiers[0])
|
||||
}
|
||||
}
|
||||
|
||||
// The whole point: local daily + PBS weekly, each with its OWN retention.
|
||||
func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
|
||||
var b BackupConfig
|
||||
raw := `{
|
||||
"local_backup_target":"local",
|
||||
"local_backup_retention":3,
|
||||
"backup_cadence_seconds":86400,
|
||||
"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
|
||||
}`
|
||||
if err := json.Unmarshal([]byte(raw), &b); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tiers, warnings := b.BackupTiers()
|
||||
if len(warnings) != 0 {
|
||||
t.Fatalf("unexpected warnings: %v", warnings)
|
||||
}
|
||||
if len(tiers) != 2 {
|
||||
t.Fatalf("want 2 tiers, got %+v", tiers)
|
||||
}
|
||||
if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
|
||||
t.Fatalf("primary tier wrong: %+v", tiers[0])
|
||||
}
|
||||
if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
|
||||
t.Fatalf("PBS tier wrong: %+v", tiers[1])
|
||||
}
|
||||
// THE knob-sharing check: the two retentions are independent values, not one shared number.
|
||||
if tiers[0].KeepLast == tiers[1].KeepLast {
|
||||
t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
|
||||
}
|
||||
}
|
||||
|
||||
// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
|
||||
// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
|
||||
func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
|
||||
b := BackupConfig{
|
||||
LocalBackupTarget: "local",
|
||||
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
|
||||
}
|
||||
tiers, warnings := b.BackupTiers()
|
||||
if len(tiers) != 1 {
|
||||
t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
|
||||
}
|
||||
if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
|
||||
t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
|
||||
}
|
||||
if !strings.Contains(warnings[0], "felhom-pbs") {
|
||||
t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
|
||||
b := BackupConfig{
|
||||
LocalBackupTarget: "local",
|
||||
ExtraTargets: []BackupTargetConfig{
|
||||
{TargetID: "", CadenceSeconds: 3600},
|
||||
{TargetID: "local", CadenceSeconds: 3600}, // repeats the primary
|
||||
{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
|
||||
{TargetID: "felhom-pbs", CadenceSeconds: 99}, // duplicate
|
||||
},
|
||||
}
|
||||
tiers, warnings := b.BackupTiers()
|
||||
if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
|
||||
t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
|
||||
}
|
||||
if len(warnings) != 3 {
|
||||
t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
|
||||
}
|
||||
}
|
||||
|
||||
// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
|
||||
// itself because someone forgot a field.
|
||||
func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
|
||||
b := BackupConfig{
|
||||
LocalBackupTarget: "local",
|
||||
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
|
||||
}
|
||||
tiers, _ := b.BackupTiers()
|
||||
if len(tiers) != 2 {
|
||||
t.Fatalf("got %+v", tiers)
|
||||
}
|
||||
if tiers[1].KeepLast != 0 {
|
||||
t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
|
||||
}
|
||||
// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
|
||||
b.ExtraTargets[0].KeepLast = -5
|
||||
tiers, _ = b.BackupTiers()
|
||||
if tiers[1].KeepLast != 0 {
|
||||
t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
|
||||
}
|
||||
}
|
||||
|
||||
// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
|
||||
func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
|
||||
for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
|
||||
b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
|
||||
tiers, _ := b.BackupTiers()
|
||||
if tiers[0].KeepLast != tc.want {
|
||||
t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
|
||||
// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
|
||||
// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
|
||||
// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
|
||||
// makes the next attempt collide with the guest lock vzdump still holds.
|
||||
func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
|
||||
b := BackupConfig{
|
||||
LocalBackupTarget: "local",
|
||||
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
|
||||
}
|
||||
tiers, _ := b.BackupTiers()
|
||||
if len(tiers) != 2 {
|
||||
t.Fatalf("got %+v", tiers)
|
||||
}
|
||||
if tiers[0].WaitTimeout != 30*time.Minute {
|
||||
t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
|
||||
}
|
||||
if tiers[1].WaitTimeout != 12*time.Hour {
|
||||
t.Fatalf("an offsite tier must default to a GENEROUS wait (operator ruling: let the first backup run as long as needed) — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
|
||||
}
|
||||
// And it must be overridable per tier.
|
||||
b.ExtraTargets[0].WaitTimeoutSeconds = 3600
|
||||
tiers, _ = b.BackupTiers()
|
||||
if tiers[1].WaitTimeout != time.Hour {
|
||||
t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
|
||||
}
|
||||
// The two tiers must NOT share one bound.
|
||||
if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
|
||||
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
|
||||
}
|
||||
}
|
||||
+248
-13
@@ -235,6 +235,16 @@ type LocalAPIConfig struct {
|
||||
// TokenStore is the durable, hashed token→guest map (only a HASH of each token is
|
||||
// persisted; the plaintext exists transiently at mint→write-to-mount, then is discarded).
|
||||
TokenStore string `json:"token_store"` // default /var/lib/felhom-agent/local-tokens.log
|
||||
// IslandBridge + IslandGuestAddr configure the R-50 host-internal control-plane bridge. When
|
||||
// BOTH are set, the provisioner attaches each guest a static net1 on IslandBridge with
|
||||
// IslandGuestAddr, so the controller reaches the agent over a fixed private address that no
|
||||
// LAN/DHCP/site move can invalidate (the F1 fix — AUDIT-vacation-remote-ops-2026-07-20). Empty
|
||||
// (the default) = LAN-only, byte-for-byte the pre-R-50 behaviour. On an island install ListenAddr
|
||||
// is the host side (169.254.253.1:8443); IslandGuestAddr is the guest side (169.254.253.2/30 — a
|
||||
// /30 is exactly host + one guest). Additive-only: it never removes a NIC, so a guest restored on
|
||||
// a non-island host (both empty) is unaffected.
|
||||
IslandBridge string `json:"island_bridge"` // e.g. "vmbr9" (portless host-internal bridge)
|
||||
IslandGuestAddr string `json:"island_guest_addr"` // guest net1 CIDR, e.g. "169.254.253.2/30"
|
||||
}
|
||||
|
||||
// Default local-API file locations (under the agent's state dir).
|
||||
@@ -249,6 +259,12 @@ func (l LocalAPIConfig) Enabled() bool {
|
||||
return l.Enable && strings.TrimSpace(l.ListenAddr) != ""
|
||||
}
|
||||
|
||||
// IslandEnabled reports whether the provisioner should attach a guest island NIC (net1). True only
|
||||
// when BOTH the bridge and the guest CIDR are set (R-50); empty = pre-R-50 LAN-only behaviour.
|
||||
func (l LocalAPIConfig) IslandEnabled() bool {
|
||||
return strings.TrimSpace(l.IslandBridge) != "" && strings.TrimSpace(l.IslandGuestAddr) != ""
|
||||
}
|
||||
|
||||
// TokenStorePath returns the configured token-store path (default applied).
|
||||
func (l LocalAPIConfig) TokenStorePath() string {
|
||||
if l.TokenStore != "" {
|
||||
@@ -283,6 +299,17 @@ func (l LocalAPIConfig) Validate() error {
|
||||
if _, _, err := net.SplitHostPort(l.ListenAddr); err != nil {
|
||||
return fmt.Errorf("config: local_api.listen_addr %q is not host:port: %w", l.ListenAddr, err)
|
||||
}
|
||||
// R-50: island fields are all-or-nothing, and the guest addr must be a CIDR (the net1 ip= value).
|
||||
// A half-set island (bridge without guest addr, or vice versa) is a provisioning mistake, not a
|
||||
// silent LAN fallback — fail loudly so a botched install config is caught at load, not at day-0.
|
||||
if (strings.TrimSpace(l.IslandBridge) != "") != (strings.TrimSpace(l.IslandGuestAddr) != "") {
|
||||
return fmt.Errorf("config: local_api.island_bridge and local_api.island_guest_addr must be set together (got bridge=%q guest_addr=%q)", l.IslandBridge, l.IslandGuestAddr)
|
||||
}
|
||||
if l.IslandEnabled() {
|
||||
if _, _, err := net.ParseCIDR(strings.TrimSpace(l.IslandGuestAddr)); err != nil {
|
||||
return fmt.Errorf("config: local_api.island_guest_addr %q is not a CIDR (want e.g. 169.254.253.2/30): %w", l.IslandGuestAddr, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -306,9 +333,23 @@ type BackupConfig struct {
|
||||
LocalBackupTarget string `json:"local_backup_target"`
|
||||
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
|
||||
RestoreStorage string `json:"restore_storage"`
|
||||
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
|
||||
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
|
||||
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
|
||||
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
|
||||
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
|
||||
// the interval trigger with a per-archive due-check — so a positive value now seeds
|
||||
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
|
||||
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
|
||||
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
|
||||
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
|
||||
// generation no matter how often it is asked. This interval sets two things — the latency
|
||||
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
|
||||
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
|
||||
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
|
||||
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
|
||||
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
|
||||
// Restore-testing an archive a backup is still writing proves nothing about the backup that
|
||||
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
|
||||
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
|
||||
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
|
||||
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
|
||||
// always excluded. Defaults to 990000–990009.
|
||||
@@ -339,6 +380,132 @@ type BackupConfig struct {
|
||||
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
|
||||
// NEVER applied to a PBS target (offsite retention is a separate lifecycle).
|
||||
LocalBackupRetention int `json:"local_backup_retention"`
|
||||
|
||||
// ExtraTargets (R-82) are ADDITIONAL backup tiers beyond the primary one above — the shape that
|
||||
// makes "local daily + PBS weekly" expressible at all. Each carries its OWN cadence and its OWN
|
||||
// retention, because those are semantically different per tier: keep-last=3 on a daily tier is
|
||||
// three DAYS of restore points; on a weekly tier it is three WEEKS. Sharing one knob between
|
||||
// tiers silently means one of them is wrong.
|
||||
//
|
||||
// ADDITIVE BY CONSTRUCTION: an existing config with no `backup_targets` key resolves to exactly
|
||||
// one tier — the primary — and behaves byte-identically to pre-R-82. Nothing here changes the
|
||||
// local tier.
|
||||
ExtraTargets []BackupTargetConfig `json:"backup_targets"`
|
||||
}
|
||||
|
||||
// BackupTargetConfig is ONE additional backup tier: a vzdump storage plus its own cadence and
|
||||
// retention. A tier with no cadence is not a tier — see BackupTiers for why that is rejected loudly
|
||||
// rather than defaulted.
|
||||
type BackupTargetConfig struct {
|
||||
// TargetID is the Proxmox storage id (content=backup), e.g. "felhom-pbs".
|
||||
TargetID string `json:"target_id"`
|
||||
// CadenceSeconds is THIS tier's /backup/due window. REQUIRED (>0) — see BackupTiers.
|
||||
CadenceSeconds int `json:"cadence_seconds"`
|
||||
// KeepLast is THIS tier's per-run `--prune-backups` keep-last. 0/unset → NEVER prune this tier
|
||||
// (the fail-safe default, and the current behaviour for every PBS target). A PBS tier is never
|
||||
// pruned by the per-run flag regardless — see BackupRunner.localPruneSpec.
|
||||
KeepLast int `json:"keep_last"`
|
||||
// WaitTimeoutSeconds bounds how long the agent WAITS for this tier's vzdump task. 0/unset →
|
||||
// defaultExtraTierWaitTimeout.
|
||||
//
|
||||
// THIS FIELD EXISTS BECAUSE OF A LIVE FAILURE (2026-07-26, R-82 Slice A validation). The runner
|
||||
// hard-coded a 30-minute wait, which is right for a local vzdump (minutes) and badly wrong for
|
||||
// an offsite PBS backup over a home uplink: the first full ~10 GB snapshot ran past 30 min, the
|
||||
// agent gave up waiting and recorded success=false — WHILE THE BACKUP WAS STILL RUNNING. That
|
||||
// false failure is worse than a slow pass: the tier stays "due", a retry collides with the
|
||||
// guest lock vzdump still holds, and the hub sees a DR tier that never succeeds.
|
||||
//
|
||||
// Same reasoning as RestoreTestPBSRestoreTimeoutSeconds on the restore side, and the same
|
||||
// direction: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a
|
||||
// corrupt status plus lock contention.
|
||||
WaitTimeoutSeconds int `json:"wait_timeout_seconds"`
|
||||
}
|
||||
|
||||
// Per-tier vzdump wait bounds.
|
||||
//
|
||||
// The PRIMARY keeps the historical 30 minutes: it is the local tier, a local vzdump takes minutes,
|
||||
// and one hanging 30 minutes is a genuine fault worth surfacing. Unchanged behaviour.
|
||||
//
|
||||
// An ADDITIONAL tier is by construction the offsite/WAN one in this design, where the binding
|
||||
// constraint is uplink speed, not health. Measured on demo-felhom: ~33 MB/min over the wg link to
|
||||
// Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. Operator ruling 2026-07-26: "let the
|
||||
// first backup run as long as needed" — 12h gives that real margin on a slower link while still
|
||||
// being BOUNDED, so a genuinely hung task eventually surfaces instead of hanging forever.
|
||||
const (
|
||||
defaultPrimaryTierWaitTimeout = 30 * time.Minute
|
||||
defaultExtraTierWaitTimeout = 12 * time.Hour
|
||||
)
|
||||
|
||||
// BackupTier is a RESOLVED backup tier: one target, its own cadence, its own retention. The agent
|
||||
// builds one runner per tier from these.
|
||||
type BackupTier struct {
|
||||
TargetID string
|
||||
Cadence time.Duration
|
||||
// WaitTimeout bounds the wait on this tier's vzdump task (see WaitTimeoutSeconds).
|
||||
WaitTimeout time.Duration
|
||||
// KeepLast is the per-run prune keep-last; 0 means DO NOT PRUNE this tier.
|
||||
KeepLast int
|
||||
// Primary marks the tier that the UNTARGETED local-API endpoints act on — the pre-R-82 tier.
|
||||
// Exactly one tier is primary, and it is always first.
|
||||
Primary bool
|
||||
}
|
||||
|
||||
// BackupTiers resolves the effective tier list, primary first, plus any warnings the caller MUST
|
||||
// log (they describe tiers that were REJECTED, and a silently-dropped backup tier is precisely the
|
||||
// "applied and empty" fault R-82 exists to fix).
|
||||
//
|
||||
// Rules:
|
||||
// - Tier 0 is always the primary, built from BackupTarget()/BackupCadence()/KeepLast() — so a
|
||||
// config with no `backup_targets` is byte-identical to pre-R-82.
|
||||
// - An extra with an empty target_id is rejected.
|
||||
// - An extra with cadence_seconds <= 0 is REJECTED, not defaulted. Defaulting a PBS tier to the
|
||||
// 24h local default would quietly turn a weekly tier into a daily one and fill the DR datastore;
|
||||
// a tier whose cadence you did not state is not a tier.
|
||||
// - An extra repeating the primary's target is rejected (one policy per target, or the two
|
||||
// cadences race and neither is the truth).
|
||||
// - Duplicate extras are rejected after the first.
|
||||
func (b BackupConfig) BackupTiers() ([]BackupTier, []string) {
|
||||
primary := BackupTier{
|
||||
TargetID: b.BackupTarget(),
|
||||
Cadence: b.BackupCadence(),
|
||||
KeepLast: b.KeepLast(),
|
||||
WaitTimeout: defaultPrimaryTierWaitTimeout,
|
||||
Primary: true,
|
||||
}
|
||||
tiers := []BackupTier{primary}
|
||||
var warnings []string
|
||||
|
||||
seen := map[string]bool{primary.TargetID: true}
|
||||
for i, t := range b.ExtraTargets {
|
||||
id := strings.TrimSpace(t.TargetID)
|
||||
switch {
|
||||
case id == "":
|
||||
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: empty target_id — tier ignored", i))
|
||||
continue
|
||||
case seen[id]:
|
||||
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: target %q already configured — duplicate tier ignored", i, id))
|
||||
continue
|
||||
case t.CadenceSeconds <= 0:
|
||||
warnings = append(warnings, fmt.Sprintf("backup_targets[%d] (%s): cadence_seconds must be > 0 — tier ignored (a cadence is NOT defaulted: a weekly tier silently running daily would fill the DR datastore)", i, id))
|
||||
continue
|
||||
}
|
||||
seen[id] = true
|
||||
keep := t.KeepLast
|
||||
if keep < 0 {
|
||||
keep = 0
|
||||
}
|
||||
wait := defaultExtraTierWaitTimeout
|
||||
if t.WaitTimeoutSeconds > 0 {
|
||||
wait = time.Duration(t.WaitTimeoutSeconds) * time.Second
|
||||
}
|
||||
tiers = append(tiers, BackupTier{
|
||||
TargetID: id,
|
||||
Cadence: time.Duration(t.CadenceSeconds) * time.Second,
|
||||
KeepLast: keep,
|
||||
WaitTimeout: wait,
|
||||
})
|
||||
}
|
||||
return tiers, warnings
|
||||
}
|
||||
|
||||
// defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept).
|
||||
@@ -392,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
|
||||
return defaultBackupTarget
|
||||
}
|
||||
|
||||
// Default scratch VMID band + restore-test cadence.
|
||||
// Default scratch VMID band + the two R-86 restore-test knobs.
|
||||
const (
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
defaultRestoreTestCadence = 24 * time.Hour
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
|
||||
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
|
||||
// and neither bound alone would have picked it:
|
||||
//
|
||||
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
|
||||
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
|
||||
// is one storage-content listing per tier:
|
||||
//
|
||||
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
|
||||
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
|
||||
// both tiers together .................. 430 ms
|
||||
//
|
||||
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
|
||||
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
|
||||
// number that would have forbidden a frequent poll is the one nobody measures.
|
||||
//
|
||||
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
|
||||
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
|
||||
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
|
||||
// timer retried a broken tier once a day.
|
||||
//
|
||||
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
|
||||
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
|
||||
// second rate limiter anywhere — the pacing remains one test per archive generation.
|
||||
defaultRestoreTestEvalInterval = 6 * time.Hour
|
||||
|
||||
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
|
||||
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
|
||||
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
|
||||
defaultRestoreTestSettle = 24 * time.Hour
|
||||
)
|
||||
|
||||
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
|
||||
// 0 → 24h default, negative → 0 (disabled).
|
||||
func (b BackupConfig) RestoreTestCadence() time.Duration {
|
||||
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
|
||||
// value as-is, 0 → the measured default, negative → 0 (disabled).
|
||||
//
|
||||
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
|
||||
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
|
||||
// sets the interval, because the interval no longer decides that a test happens.
|
||||
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
|
||||
if b.RestoreTestCadenceSeconds < 0 {
|
||||
return 0 // legacy DISABLE — preserved verbatim
|
||||
}
|
||||
switch {
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
|
||||
case b.RestoreTestCadenceSeconds < 0:
|
||||
case b.RestoreTestEvalIntervalSeconds > 0:
|
||||
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
|
||||
case b.RestoreTestEvalIntervalSeconds < 0:
|
||||
return 0 // disabled
|
||||
default:
|
||||
return defaultRestoreTestCadence
|
||||
return defaultRestoreTestEvalInterval
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
|
||||
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
|
||||
//
|
||||
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
|
||||
// was expressing "how long may pass between a backup and the confidence that it restores". That
|
||||
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
|
||||
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
|
||||
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
|
||||
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
|
||||
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
|
||||
func (b BackupConfig) RestoreTestSettle() time.Duration {
|
||||
switch {
|
||||
case b.RestoreTestSettleSeconds > 0:
|
||||
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
|
||||
case b.RestoreTestSettleSeconds < 0:
|
||||
return 0 // explicitly no settle requirement
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
|
||||
default:
|
||||
return defaultRestoreTestSettle
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
|
||||
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
|
||||
// without saying so is exactly the silent repurposing §8.3 forbids.
|
||||
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
|
||||
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
|
||||
}
|
||||
|
||||
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
|
||||
// negative → 0 (disabled).
|
||||
func (b BackupConfig) PBSVerifyCadence() time.Duration {
|
||||
@@ -657,6 +890,8 @@ func applyEnv(cfg *Config) {
|
||||
cfg.Backup.RestoreStorage = v
|
||||
}
|
||||
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
|
||||
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
|
||||
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
|
||||
}
|
||||
|
||||
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
|
||||
|
||||
@@ -171,3 +171,46 @@ func TestDeploymentModeEnvOverlay(t *testing.T) {
|
||||
t.Errorf("env overlay did not set deployment_mode: %q", cfg.DeploymentMode)
|
||||
}
|
||||
}
|
||||
|
||||
// R-50: the island NIC fields are all-or-nothing and the guest addr must be a CIDR. A half-set or
|
||||
// malformed island must fail at config load (a botched install) rather than silently fall back to
|
||||
// LAN-only, which would leave a guest with an island bind and no island NIC — the exact silent break
|
||||
// R-50 exists to kill. Covers LocalAPIConfig.Validate + IslandEnabled.
|
||||
func TestLocalAPIConfig_IslandValidation(t *testing.T) {
|
||||
base := LocalAPIConfig{Enable: true, ListenAddr: "169.254.253.1:8443"}
|
||||
|
||||
// both empty → fine (pre-R-50 default), IslandEnabled false
|
||||
if err := base.Validate(); err != nil {
|
||||
t.Errorf("no island config must validate: %v", err)
|
||||
}
|
||||
if base.IslandEnabled() {
|
||||
t.Errorf("IslandEnabled must be false when unset")
|
||||
}
|
||||
// both set, valid CIDR → fine, IslandEnabled true
|
||||
ok := base
|
||||
ok.IslandBridge, ok.IslandGuestAddr = "vmbr9", "169.254.253.2/30"
|
||||
if err := ok.Validate(); err != nil {
|
||||
t.Errorf("valid island config must validate: %v", err)
|
||||
}
|
||||
if !ok.IslandEnabled() {
|
||||
t.Errorf("IslandEnabled must be true when both set")
|
||||
}
|
||||
// bridge only → rejected (all-or-nothing)
|
||||
half := base
|
||||
half.IslandBridge = "vmbr9"
|
||||
if err := half.Validate(); err == nil {
|
||||
t.Errorf("half-set island (bridge only) must be rejected")
|
||||
}
|
||||
// guest addr only → rejected
|
||||
half2 := base
|
||||
half2.IslandGuestAddr = "169.254.253.2/30"
|
||||
if err := half2.Validate(); err == nil {
|
||||
t.Errorf("half-set island (guest addr only) must be rejected")
|
||||
}
|
||||
// both set but guest addr is not a CIDR → rejected
|
||||
bad := base
|
||||
bad.IslandBridge, bad.IslandGuestAddr = "vmbr9", "169.254.253.2" // missing /30
|
||||
if err := bad.Validate(); err == nil {
|
||||
t.Errorf("island guest addr without a CIDR mask must be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,8 +11,16 @@ import (
|
||||
)
|
||||
|
||||
func TestWordlistLoaded(t *testing.T) {
|
||||
if WordlistSize() != 7776 {
|
||||
t.Fatalf("EFF large wordlist should be 7776 words, got %d", WordlistSize())
|
||||
// The EFF large list is 7776 entries; joinSafe removes the 4 that contain RecoveryCodeSep
|
||||
// (drop-down, felt-tip, t-shirt, yo-yo), leaving 7772 as the effective draw space.
|
||||
if got := WordlistSize(); got != 7772 {
|
||||
t.Fatalf("effective wordlist should be 7772 words (7776 EFF - 4 hyphenated), got %d", got)
|
||||
}
|
||||
if got := WordlistFilteredOut(); got != 4 {
|
||||
t.Fatalf("joinSafe should have removed exactly 4 hyphenated entries, removed %d", got)
|
||||
}
|
||||
if got := WordlistSize() + WordlistFilteredOut(); got != 7776 {
|
||||
t.Fatalf("filtered + removed should reconstitute the 7776-word EFF list, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,19 +33,27 @@ func TestGenerateRecoveryCode_EntropyAndFormat(t *testing.T) {
|
||||
inList[w] = true
|
||||
}
|
||||
for i := 0; i < 50; i++ {
|
||||
r, err := GenerateRecoveryCode()
|
||||
// Count words by GENERATION count, not by re-splitting the joined string: the two agree
|
||||
// only because joinSafe holds, and conflating them is what made this test flake ~1/5.
|
||||
words, err := generateWords(wordlist)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateRecoveryCode: %v", err)
|
||||
t.Fatalf("generateWords: %v", err)
|
||||
}
|
||||
words := strings.Split(r, "-")
|
||||
if len(words) != RecoveryCodeWords {
|
||||
t.Fatalf("recovery code must be %d words, got %d (%q)", RecoveryCodeWords, len(words), r)
|
||||
t.Fatalf("generator must draw %d words, drew %d", RecoveryCodeWords, len(words))
|
||||
}
|
||||
for _, w := range words {
|
||||
if !inList[w] {
|
||||
t.Errorf("recovery-code word %q is not from the EFF wordlist", w)
|
||||
}
|
||||
}
|
||||
// Separately assert the property joinSafe buys: the joined code segments back to the same
|
||||
// count. Never print r — it is a live-shaped secret.
|
||||
r := strings.Join(words, RecoveryCodeSep)
|
||||
if got := len(strings.Split(r, RecoveryCodeSep)); got != RecoveryCodeWords {
|
||||
t.Fatalf("joined code must segment into %d words, got %d (a drawn word contained %q)",
|
||||
RecoveryCodeWords, got, RecoveryCodeSep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+58
-17
@@ -29,9 +29,20 @@ import (
|
||||
//go:embed eff_large_wordlist.txt
|
||||
var wordlistRaw []byte
|
||||
|
||||
// wordlist is the EFF large wordlist (7776 words, 12.92 bits/word) — the diceware standard for
|
||||
// human-transcribed passphrases. Parsed once at init.
|
||||
var wordlist = parseWordlist(wordlistRaw)
|
||||
// RecoveryCodeSep joins the words of a recovery code R. It is ALSO the reason for the
|
||||
// joinSafe filter below: a word that itself contains the separator makes the joined code
|
||||
// ambiguous to segment by eye, which is unaffordable in the one situation R exists for — a
|
||||
// customer transcribing it during a disaster. Do not change it: R is consumed as a whole
|
||||
// passphrase (see Wrap/Unwrap), so the separator is a transcription aid, not a parsed delimiter.
|
||||
const RecoveryCodeSep = "-"
|
||||
|
||||
// wordlist is the EFF large wordlist (the diceware standard for human-transcribed passphrases),
|
||||
// minus the handful of entries that contain RecoveryCodeSep. Parsed and filtered once at init.
|
||||
// Sizes are asserted in wordlist_test.go so a wordlist swap cannot silently move the entropy floor.
|
||||
var wordlist = joinSafe(parseWordlist(wordlistRaw))
|
||||
|
||||
// wordlistRawSize is the unfiltered parse length, kept for audit (see WordlistFilteredOut).
|
||||
var wordlistRawSize = len(parseWordlist(wordlistRaw))
|
||||
|
||||
func parseWordlist(raw []byte) []string {
|
||||
var w []string
|
||||
@@ -44,28 +55,55 @@ func parseWordlist(raw []byte) []string {
|
||||
return w
|
||||
}
|
||||
|
||||
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the 7776-word EFF
|
||||
// list ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
|
||||
// joinSafe drops every word containing RecoveryCodeSep, so that a generated code always segments
|
||||
// back into exactly RecoveryCodeWords words. In the EFF large list this removes exactly 4 entries
|
||||
// (drop-down, felt-tip, t-shirt, yo-yo) of 7776, costing ~0.0007 bits/word — the floor still holds
|
||||
// (asserted in the tests). Generation-time only: codes already issued remain valid, because R is
|
||||
// verified as a whole passphrase and is never re-split.
|
||||
func joinSafe(words []string) []string {
|
||||
out := make([]string, 0, len(words))
|
||||
for _, w := range words {
|
||||
if strings.Contains(w, RecoveryCodeSep) {
|
||||
continue
|
||||
}
|
||||
out = append(out, w)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the filtered EFF
|
||||
// list (7772 words) ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
|
||||
const RecoveryCodeWords = 10
|
||||
|
||||
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
|
||||
// (crypto/rand via big.Int — no modulo bias) from the EFF large wordlist, hyphen-joined.
|
||||
//
|
||||
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
|
||||
func GenerateRecoveryCode() (string, error) {
|
||||
if len(wordlist) < 2 {
|
||||
return "", fmt.Errorf("escrow: wordlist not loaded (%d words)", len(wordlist))
|
||||
// generateWords draws RecoveryCodeWords words uniformly (crypto/rand via big.Int — no modulo bias)
|
||||
// from list. Split out from GenerateRecoveryCode so tests can drive an unfiltered list and prove
|
||||
// the filter is what keeps a code segmentable.
|
||||
func generateWords(list []string) ([]string, error) {
|
||||
if len(list) < 2 {
|
||||
return nil, fmt.Errorf("escrow: wordlist not loaded (%d words)", len(list))
|
||||
}
|
||||
n := big.NewInt(int64(len(wordlist)))
|
||||
n := big.NewInt(int64(len(list)))
|
||||
words := make([]string, RecoveryCodeWords)
|
||||
for i := range words {
|
||||
idx, err := rand.Int(rand.Reader, n)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("escrow: recovery-code rng: %w", err)
|
||||
return nil, fmt.Errorf("escrow: recovery-code rng: %w", err)
|
||||
}
|
||||
words[i] = wordlist[idx.Int64()]
|
||||
words[i] = list[idx.Int64()]
|
||||
}
|
||||
return strings.Join(words, "-"), nil
|
||||
return words, nil
|
||||
}
|
||||
|
||||
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
|
||||
// from the filtered EFF large wordlist, joined with RecoveryCodeSep.
|
||||
//
|
||||
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
|
||||
func GenerateRecoveryCode() (string, error) {
|
||||
words, err := generateWords(wordlist)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return strings.Join(words, RecoveryCodeSep), nil
|
||||
}
|
||||
|
||||
// RecoveryCodeEntropyBits is the approximate entropy of a generated code, for display/audit only
|
||||
@@ -77,5 +115,8 @@ func RecoveryCodeEntropyBits() float64 {
|
||||
return float64(RecoveryCodeWords) * math.Log2(float64(len(wordlist)))
|
||||
}
|
||||
|
||||
// WordlistSize is the loaded wordlist length (for audit/tests).
|
||||
// WordlistSize is the effective (filtered) wordlist length — the draw space. For audit/tests.
|
||||
func WordlistSize() int { return len(wordlist) }
|
||||
|
||||
// WordlistFilteredOut is how many parsed entries joinSafe removed. For audit/tests.
|
||||
func WordlistFilteredOut() int { return wordlistRawSize - len(wordlist) }
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The four EFF large-list entries that contain RecoveryCodeSep. Named here so a wordlist swap that
|
||||
// changes the set fails loudly rather than silently re-opening the ambiguity.
|
||||
var hyphenatedEFFWords = []string{"drop-down", "felt-tip", "t-shirt", "yo-yo"}
|
||||
|
||||
func TestJoinSafe_RemovesExactlyTheHyphenatedEFFWords(t *testing.T) {
|
||||
raw := parseWordlist(wordlistRaw)
|
||||
rawSet := make(map[string]bool, len(raw))
|
||||
for _, w := range raw {
|
||||
rawSet[w] = true
|
||||
}
|
||||
for _, w := range hyphenatedEFFWords {
|
||||
if !rawSet[w] {
|
||||
t.Fatalf("fixture drift: %q is no longer in the embedded EFF list", w)
|
||||
}
|
||||
}
|
||||
|
||||
filtered := joinSafe(raw)
|
||||
if len(raw)-len(filtered) != len(hyphenatedEFFWords) {
|
||||
t.Fatalf("joinSafe removed %d entries, expected exactly %d",
|
||||
len(raw)-len(filtered), len(hyphenatedEFFWords))
|
||||
}
|
||||
got := make(map[string]bool, len(filtered))
|
||||
for _, w := range filtered {
|
||||
if strings.Contains(w, RecoveryCodeSep) {
|
||||
t.Errorf("filtered wordlist still contains a separator-bearing word %q", w)
|
||||
}
|
||||
got[w] = true
|
||||
}
|
||||
for _, w := range hyphenatedEFFWords {
|
||||
if got[w] {
|
||||
t.Errorf("joinSafe kept %q, which contains %q", w, RecoveryCodeSep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEntropyFloorSurvivesFiltering states the numbers explicitly: dropping 4 of 7776 words costs
|
||||
// ~0.0007 bits/word, so the 10-word code stays above the 128-bit floor with room to spare.
|
||||
func TestEntropyFloorSurvivesFiltering(t *testing.T) {
|
||||
const floorBits = 128.0
|
||||
before := float64(RecoveryCodeWords) * math.Log2(7776)
|
||||
after := RecoveryCodeEntropyBits()
|
||||
|
||||
if after < floorBits {
|
||||
t.Fatalf("filtered entropy %.3f bits is below the %.0f-bit floor", after, floorBits)
|
||||
}
|
||||
if want := float64(RecoveryCodeWords) * math.Log2(float64(WordlistSize())); math.Abs(after-want) > 1e-9 {
|
||||
t.Fatalf("RecoveryCodeEntropyBits() = %.6f, want %.6f (10 * log2(%d))", after, want, WordlistSize())
|
||||
}
|
||||
// Concrete expectations, so a wordlist change that quietly erodes the margin is visible:
|
||||
// 10*log2(7776) = 129.248 bits before, 10*log2(7772) = 129.241 bits after — a 0.007-bit cost.
|
||||
if math.Abs(before-129.248) > 0.001 {
|
||||
t.Fatalf("unfiltered entropy baseline moved: %.3f, expected 129.248", before)
|
||||
}
|
||||
if math.Abs(after-129.241) > 0.001 {
|
||||
t.Fatalf("filtered entropy moved: %.3f, expected 129.241", after)
|
||||
}
|
||||
if cost := before - after; cost > 0.01 {
|
||||
t.Fatalf("filtering cost %.4f bits, expected well under 0.01", cost)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratedCodeSegments_FilteredVsUnfiltered is the deterministic red-proof companion.
|
||||
//
|
||||
// Against a list where EVERY word contains the separator, a 10-word draw MUST segment into more
|
||||
// than 10 parts — that is the pre-fix behaviour, reproduced with probability 1 instead of the ~1/5
|
||||
// flake the real list produced. Against the same list run through joinSafe, generation must refuse
|
||||
// (nothing is left to draw from), proving joinSafe — not luck — is what makes a code segmentable.
|
||||
func TestGeneratedCodeSegments_FilteredVsUnfiltered(t *testing.T) {
|
||||
unfiltered := hyphenatedEFFWords
|
||||
|
||||
words, err := generateWords(unfiltered)
|
||||
if err != nil {
|
||||
t.Fatalf("generateWords(unfiltered): %v", err)
|
||||
}
|
||||
if len(words) != RecoveryCodeWords {
|
||||
t.Fatalf("generator drew %d words, want %d", len(words), RecoveryCodeWords)
|
||||
}
|
||||
joined := strings.Join(words, RecoveryCodeSep)
|
||||
segs := len(strings.Split(joined, RecoveryCodeSep))
|
||||
if segs <= RecoveryCodeWords {
|
||||
t.Fatalf("unfiltered draw segmented into %d parts; the pre-fix defect should yield more than %d",
|
||||
segs, RecoveryCodeWords)
|
||||
}
|
||||
if segs != 2*RecoveryCodeWords {
|
||||
t.Fatalf("every fixture word has exactly one separator, so 10 words must segment into 20 parts, got %d", segs)
|
||||
}
|
||||
|
||||
// Same fixture, filtered: the draw space is empty, so generation must error rather than
|
||||
// silently fall back to something ambiguous.
|
||||
if _, err := generateWords(joinSafe(unfiltered)); err == nil {
|
||||
t.Fatal("generateWords on a fully-filtered list must fail, not return a code")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateRecoveryCode_NeverContainsAmbiguousWord is the production-wiring test: it asserts the
|
||||
// exported entry point (not just the helper) draws from the filtered list.
|
||||
func TestGenerateRecoveryCode_NeverContainsAmbiguousWord(t *testing.T) {
|
||||
inFiltered := make(map[string]bool, len(wordlist))
|
||||
for _, w := range wordlist {
|
||||
inFiltered[w] = true
|
||||
}
|
||||
for i := 0; i < 500; i++ {
|
||||
r, err := GenerateRecoveryCode()
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateRecoveryCode: %v", err)
|
||||
}
|
||||
parts := strings.Split(r, RecoveryCodeSep)
|
||||
if len(parts) != RecoveryCodeWords {
|
||||
// Do not print r: it is a live-shaped secret.
|
||||
t.Fatalf("code %d segmented into %d parts, want %d", i, len(parts), RecoveryCodeWords)
|
||||
}
|
||||
for _, p := range parts {
|
||||
if !inFiltered[p] {
|
||||
t.Fatalf("segment %q is not a filtered-wordlist word", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -548,3 +548,21 @@ func TestClassify_Table(t *testing.T) {
|
||||
}
|
||||
|
||||
var _ io.Writer = (*bytes.Buffer)(nil)
|
||||
|
||||
// A3 (R-50): the guestnet healer is eth0-only and MUST stay blind to the island NIC. A guest on an
|
||||
// island host presents eth0 DHCP (the LAN leg the healer owns) PLUS eth1 static (the island). Because
|
||||
// parseMode is interface-scoped, adding eth1 static cannot flip eth0's detected mode — so the healer
|
||||
// keeps treating eth0 as DHCP and never runs dhclient against the static island NIC (which would
|
||||
// sabotage it). This is the verify-only guarantee that let R-50 ship the island NIC without a healer
|
||||
// change. Red-proof: make parseMode scan globally instead of per-dev and the eth0 assertion fails.
|
||||
func TestParseMode_IslandStaticNICDoesNotConfuseEth0(t *testing.T) {
|
||||
interfaces := "auto lo\niface lo inet loopback\n\n" +
|
||||
"auto eth0\niface eth0 inet dhcp\n\n" +
|
||||
"auto eth1\niface eth1 inet static\n address 169.254.253.2/30\n"
|
||||
if got := parseMode(interfaces, "eth0"); got != ModeDHCP {
|
||||
t.Errorf("eth0 must classify DHCP even with an island eth1 static present, got %q", got)
|
||||
}
|
||||
if got := parseMode(interfaces, "eth1"); got != ModeStatic {
|
||||
t.Errorf("eth1 (island) must classify static when asked directly (dev-scoped), got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
+124
-6
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
|
||||
RestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
|
||||
//
|
||||
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
|
||||
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
|
||||
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
|
||||
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
|
||||
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
|
||||
//
|
||||
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
|
||||
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
|
||||
//
|
||||
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
|
||||
type ProvenRestoreTestReporter interface {
|
||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||
type PBSReporter interface {
|
||||
@@ -79,16 +95,19 @@ type Collector struct {
|
||||
storage StorageObserver
|
||||
backups BackupReporter
|
||||
restoreTests RestoreTestReporter
|
||||
provenTests ProvenRestoreTestReporter
|
||||
pbs PBSReporter
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
|
||||
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
||||
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
|
||||
hostID string
|
||||
agentVersion string
|
||||
logger *slog.Logger
|
||||
@@ -123,6 +142,30 @@ func (c *Collector) SetTempReader(t TempReader) *Collector {
|
||||
return c
|
||||
}
|
||||
|
||||
// SetBackupTargetResolver wires the DR recipe to the agent's own backup config (R-109), so the recipe
|
||||
// can name WHICH storage holds the local whole-guest archives. Returns the collector for chaining.
|
||||
//
|
||||
// The resolver MUST report the tier that is IN EFFECT, which is the daemon-start snapshot — NOT the
|
||||
// current contents of agent.json. A backup-target move rewrites that file and deliberately does not
|
||||
// restart the agent (the E-1 lesson: restarting mid-backup records a spurious failure for a run that
|
||||
// succeeded), so between the write and the restart the file names a target no backup is writing to yet.
|
||||
// Re-reading the file here — the live-reload shape used for escrow.pbs_storage_id — would make the
|
||||
// recipe point at the new storage while every archive still landed on the old one. One state, one
|
||||
// owner: the recipe follows what performs the backup.
|
||||
func (c *Collector) SetBackupTargetResolver(f func() ConfiguredBackupTarget) *Collector {
|
||||
c.backupTarget = f
|
||||
return c
|
||||
}
|
||||
|
||||
// configuredBackupTarget consults the resolver. An unwired seam is reported as NOT KNOWN — never as a
|
||||
// guess — so the recipe records an explicit unknown instead of a target the agent never verified.
|
||||
func (c *Collector) configuredBackupTarget() ConfiguredBackupTarget {
|
||||
if c.backupTarget == nil {
|
||||
return ConfiguredBackupTarget{}
|
||||
}
|
||||
return c.backupTarget()
|
||||
}
|
||||
|
||||
// SetCapabilityProber wires the privileged-capability self-check (v0.44.0): each collect runs it
|
||||
// and attaches the snapshot. nil → the report carries an empty []. Returns the collector for chaining.
|
||||
func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capability.Status) *Collector {
|
||||
@@ -228,10 +271,11 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
|
||||
Capabilities: c.capabilities(ctx),
|
||||
LeafFingerprint: c.leafFP,
|
||||
Addresses: c.collectAddresses(),
|
||||
}
|
||||
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
|
||||
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
|
||||
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots)
|
||||
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots, c.configuredBackupTarget())
|
||||
// S3: offsite-tunnel status stanza (nil reporter = feature disabled → omitted; the pubkey in
|
||||
// it is the operator's revocation-recovery handle).
|
||||
if c.wg != nil {
|
||||
@@ -400,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||
return []Backup{}
|
||||
}
|
||||
|
||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||
//
|
||||
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
|
||||
// than from a preference between them:
|
||||
//
|
||||
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
|
||||
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
|
||||
// record is short-lived by design;
|
||||
// - the persisted state holds the last SUCCESS per tier and survives a restart.
|
||||
//
|
||||
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
|
||||
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
|
||||
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
|
||||
// read at the hub as two tests.
|
||||
//
|
||||
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
|
||||
// would be a worse defect than the one this closes.
|
||||
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
|
||||
if c.restoreTests == nil {
|
||||
return []RestoreTest{}
|
||||
out := []RestoreTest{}
|
||||
if c.restoreTests != nil {
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
out = append(out, r...)
|
||||
}
|
||||
}
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
return r
|
||||
if c.provenTests == nil {
|
||||
return out
|
||||
}
|
||||
return []RestoreTest{}
|
||||
|
||||
// Index what we already have by tier, keeping the newest per tier.
|
||||
best := map[string]int{} // tier → index into out
|
||||
for i, rt := range out {
|
||||
if rt.SourceTier == "" {
|
||||
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
|
||||
}
|
||||
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
|
||||
best[rt.SourceTier] = i
|
||||
}
|
||||
}
|
||||
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
|
||||
if p.SourceTier == "" {
|
||||
continue // not usable as a per-tier proof; the state layer already filters these
|
||||
}
|
||||
i, seen := best[p.SourceTier]
|
||||
if !seen {
|
||||
out = append(out, p)
|
||||
best[p.SourceTier] = len(out) - 1
|
||||
continue
|
||||
}
|
||||
if newerRestoreTest(p, out[i]) {
|
||||
out[i] = p
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
|
||||
// treated as OLDER, so a malformed entry can never displace a good one.
|
||||
func newerRestoreTest(a, b RestoreTest) bool {
|
||||
ta, aok := parseRestoreTestedAt(a.TestedAt)
|
||||
tb, bok := parseRestoreTestedAt(b.TestedAt)
|
||||
if !aok {
|
||||
return false
|
||||
}
|
||||
if !bok {
|
||||
return true
|
||||
}
|
||||
return ta.After(tb)
|
||||
}
|
||||
|
||||
func parseRestoreTestedAt(s string) (time.Time, bool) {
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return t.UTC(), true
|
||||
}
|
||||
|
||||
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
|
||||
// argument because the persisted state is opened later in main() than the collector is built; the
|
||||
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
|
||||
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
|
||||
// all, and this fix must not become the next instance of that.
|
||||
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
|
||||
|
||||
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
|
||||
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
|
||||
if c.pbs == nil {
|
||||
|
||||
@@ -45,6 +45,14 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
State: StorageStateAttached, Reachable: true, MountPath: "/mnt/usb-backup", TotalBytes: 2000000000000,
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
},
|
||||
// A pbs target so the recipe's pbs coord has a storage.cfg row to resolve its namespace
|
||||
// from (R-106). Without one the fixture produces namespace_state=unknown while the golden
|
||||
// pins a resolved coord — key-set-equal but semantically a fiction.
|
||||
{
|
||||
Name: "felhom-pbs", Type: StorageTypePBS, DurableID: "repo+fp", Content: "backup",
|
||||
State: StorageStateAttached, Reachable: true, PBSNamespace: "felhom-spike",
|
||||
Smart: SmartSummary{Health: SmartUnknown},
|
||||
},
|
||||
},
|
||||
Backups: []Backup{
|
||||
{
|
||||
@@ -72,9 +80,13 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
},
|
||||
AuditTail: []AuditEntry{},
|
||||
Cloudflared: Cloudflared{Status: "active"},
|
||||
// v0.119.0: host addresses. Populated so the bidirectional key-set guard exercises the new
|
||||
// element keys, not just the presence of the array.
|
||||
Addresses: []HostAddress{{Iface: "vmbr0", CIDR: "192.168.0.162/24"}},
|
||||
}
|
||||
// dr_recipe host-half: built from the same guest/storage/pbs facts (the production path).
|
||||
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots)
|
||||
report.DRRecipe = BuildDRRecipeHostHalf(report.Guests, report.StorageTargets, report.PBSSnapshots,
|
||||
ConfiguredBackupTarget{StorageID: "usb-backup", Known: true})
|
||||
b, _ := json.Marshal(report)
|
||||
var got map[string]any
|
||||
json.Unmarshal(b, &got)
|
||||
@@ -97,6 +109,9 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"]))
|
||||
// slice-6-Phase-B addition — pbs_snapshots[0] key set.
|
||||
assertSameKeys(t, "pbs_snapshots[0]", firstElem(golden["pbs_snapshots"]), firstElem(got["pbs_snapshots"]))
|
||||
// v0.119.0 addition — addresses[0] key set (iface/cidr), the cross-repo wire for the hub's
|
||||
// Network card.
|
||||
assertSameKeys(t, "addresses[0]", firstElem(golden["addresses"]), firstElem(got["addresses"]))
|
||||
|
||||
// DR-recipe host-half — the agent's secret-free reconstruction-scaffolding section. Assert the
|
||||
// dr_recipe key set + each sub-array's element key set (the cross-repo wire pinned in the golden).
|
||||
@@ -106,6 +121,7 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
|
||||
assertSameKeys(t, "dr_recipe.guests[0]", firstElem(field(grec, "guests")), firstElem(field(srec, "guests")))
|
||||
assertSameKeys(t, "dr_recipe.drives[0]", firstElem(field(grec, "drives")), firstElem(field(srec, "drives")))
|
||||
assertSameKeys(t, "dr_recipe.pve_storage[0]", firstElem(field(grec, "pve_storage")), firstElem(field(srec, "pve_storage")))
|
||||
assertSameKeys(t, "dr_recipe.backup_target", field(grec, "backup_target"), field(srec, "backup_target"))
|
||||
}
|
||||
|
||||
// field extracts a nested object value from a decoded JSON map (nil if absent/not a map).
|
||||
|
||||
+138
-10
@@ -30,6 +30,36 @@ import "sort"
|
||||
// ignore-unknown (encoding/json default) for forward-compat, mirroring storage_manifest.
|
||||
const DRRecipeVersion = 1
|
||||
|
||||
// Recipe field states (R-106/R-109). A recipe is read at the worst possible moment — by an operator
|
||||
// rebuilding a machine that is gone — so a field the agent cannot resolve must SAY SO rather than emit
|
||||
// a default, an empty string, or a plausible-looking placeholder. A guess read as fact costs more than
|
||||
// an admitted gap: it sends the restore at the wrong archive and nothing contradicts it. This is the
|
||||
// same cannot-tell-must-not-lie rule R-117 needed a third state for.
|
||||
const (
|
||||
DRStateResolved = "resolved"
|
||||
DRStateUnknown = "unknown"
|
||||
)
|
||||
|
||||
// Reasons a resolved-value field is unknown. Enum-shaped, never free text, so the wire stays pinnable
|
||||
// and TestDRRecipeHostHalf_NoSecrets has a fixed vocabulary to walk.
|
||||
const (
|
||||
// DRReasonNoBackupConfig: the collector was built without a backup-config seam, so the agent could
|
||||
// not consult the very config its own scheduler reads. Nothing is guessed.
|
||||
DRReasonNoBackupConfig = "agent_backup_config_unavailable"
|
||||
// DRReasonNoSuchStorage: the configured target id matches no storage this host observes. The id is
|
||||
// still recorded (it IS what the config says) and the state says it could not be corroborated.
|
||||
DRReasonNoSuchStorage = "not_a_known_storage"
|
||||
// DRReasonNoPBSStorage: snapshots exist but no pbs storage was observed, so there is no storage.cfg
|
||||
// row to read the namespace from.
|
||||
DRReasonNoPBSStorage = "no_pbs_storage_observed"
|
||||
)
|
||||
|
||||
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
|
||||
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
|
||||
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
|
||||
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
|
||||
const PBSRootNamespace = "root"
|
||||
|
||||
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
|
||||
// from facts the report already collects — no new privileged reads.
|
||||
type DRRecipeHostHalf struct {
|
||||
@@ -38,6 +68,45 @@ type DRRecipeHostHalf struct {
|
||||
PBS *DRPBSCoord `json:"pbs,omitempty"`
|
||||
Drives []DRDrive `json:"drives"`
|
||||
PVEStorage []DRPVEStorage `json:"pve_storage"`
|
||||
// BackupTarget names WHICH storage holds the local whole-guest archives (R-109). Always present —
|
||||
// its own State field carries "I could not tell", so the section is never simply absent.
|
||||
BackupTarget *DRBackupTarget `json:"backup_target"`
|
||||
}
|
||||
|
||||
// DRBackupTarget answers the one question pve_storage cannot: of every storage listed there, WHICH one
|
||||
// does this box's primary backup tier actually write its whole-guest archives to?
|
||||
//
|
||||
// Before R-109 the recipe listed each storage's name/type/content and said nothing about the target.
|
||||
// That was harmless while the target was the well-known `local`; the 2026-07-28 vzdump-target move
|
||||
// ended that. Every box now carries TWO content=backup dir storages — `felhom-backup` (live) and
|
||||
// `local` (archives frozen at the move, never refreshed since) — and they are indistinguishable by
|
||||
// name, type and content alone. A restorer picking the frozen one gets a guest that restores cleanly
|
||||
// and is silently months out of date, which is the worst shape a backup defect can take.
|
||||
type DRBackupTarget struct {
|
||||
// State is DRStateResolved | DRStateUnknown. A reader MUST consult it before trusting StorageID:
|
||||
// the id is also recorded in one unknown case (see DRReasonNoSuchStorage).
|
||||
State string `json:"state"`
|
||||
// StorageID is the PVE storage id of the PRIMARY backup tier. Empty only when the config could not
|
||||
// be consulted at all.
|
||||
StorageID string `json:"storage_id,omitempty"`
|
||||
// MountPath is where that storage's archives land on the host — the disambiguation a restorer
|
||||
// actually needs, since it is what separates felhom-backup's /mnt/hdd_1 from local's /var/lib/vz.
|
||||
// "" for a pbs target (no host mount) and when unresolved.
|
||||
MountPath string `json:"mount_path,omitempty"`
|
||||
// Reason is why State is unknown (one of the DRReason* constants); "" when resolved.
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// ConfiguredBackupTarget is what the agent's own backup config says the PRIMARY tier writes to.
|
||||
//
|
||||
// Known=false is a REAL state, not a nil-guard: it means the collector was constructed without the
|
||||
// backup-config seam (the --selftest one-shots did exactly this before v0.118.0), and the recipe then
|
||||
// records unknown instead of inventing a target. Deliberately a struct rather than a `(string, bool)`
|
||||
// return — the (value, ok) shape is what made "errors degrade to unknown, never to no-backup"
|
||||
// unimplementable in newestArchiveOn, and this field has the same three-way reading.
|
||||
type ConfiguredBackupTarget struct {
|
||||
StorageID string
|
||||
Known bool
|
||||
}
|
||||
|
||||
// DRGuest is the sizing needed to recreate the LXC at the right size (GuestSpec, already on the wire).
|
||||
@@ -51,8 +120,21 @@ type DRGuest struct {
|
||||
// DRPBSCoord is WHERE the whole-CT snapshot lives — COORDINATES ONLY. The encryption key is escrow-only;
|
||||
// the access token is identity-escrow-only. Neither is here.
|
||||
type DRPBSCoord struct {
|
||||
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
|
||||
Namespace string `json:"namespace"` // PBS namespace the restore targets
|
||||
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
|
||||
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
|
||||
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
|
||||
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
|
||||
// configured; "" when NamespaceState is unknown.
|
||||
//
|
||||
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
|
||||
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
|
||||
// therefore always empty, ToHub normalised empty → "root", and every per-customer box reported the
|
||||
// root namespace while its backups were really in `demo-hp` / `demo-felhom`.
|
||||
Namespace string `json:"namespace"`
|
||||
// NamespaceState is DRStateResolved | DRStateUnknown — consult it before trusting Namespace.
|
||||
NamespaceState string `json:"namespace_state"`
|
||||
// NamespaceReason is why NamespaceState is unknown; "" when resolved.
|
||||
NamespaceReason string `json:"namespace_reason,omitempty"`
|
||||
LatestSnapshotID string `json:"latest_snapshot_id"` // most-recent snapshot's backup_id (a coordinate)
|
||||
}
|
||||
|
||||
@@ -83,7 +165,7 @@ const driveIntentEnrolled = "enrolled"
|
||||
// it is unit-tested directly (no live reads). drives[] = the user-data external drives (usb/local-dir
|
||||
// with a durable-id); pve_storage[] = every storage target (the storage.cfg scaffolding); pbs = the
|
||||
// latest PBS snapshot's coordinates; guests[] = each guest's sizing (skip guests with no spec).
|
||||
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot) *DRRecipeHostHalf {
|
||||
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot, backupTarget ConfiguredBackupTarget) *DRRecipeHostHalf {
|
||||
h := &DRRecipeHostHalf{
|
||||
RecipeVersion: DRRecipeVersion,
|
||||
Guests: []DRGuest{},
|
||||
@@ -103,11 +185,14 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
|
||||
})
|
||||
}
|
||||
|
||||
var pbsRepoID string
|
||||
var pbsRepoID, pbsNamespace string
|
||||
var pbsStorageFound bool
|
||||
for _, t := range targets {
|
||||
h.PVEStorage = append(h.PVEStorage, DRPVEStorage{Name: t.Name, Type: t.Type, Content: t.Content})
|
||||
if t.Type == StorageTypePBS && pbsRepoID == "" {
|
||||
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
|
||||
if t.Type == StorageTypePBS && !pbsStorageFound {
|
||||
pbsStorageFound = true
|
||||
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
|
||||
pbsNamespace = t.PBSNamespace // storage.cfg's namespace — "" here means the ROOT namespace
|
||||
}
|
||||
if isUserDataDrive(t) {
|
||||
h.Drives = append(h.Drives, DRDrive{
|
||||
@@ -119,12 +204,41 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
|
||||
}
|
||||
}
|
||||
|
||||
if c := latestPBSCoord(pbs, pbsRepoID); c != nil {
|
||||
h.BackupTarget = resolveBackupTarget(targets, backupTarget)
|
||||
|
||||
if c := latestPBSCoord(pbs, pbsRepoID, pbsNamespace, pbsStorageFound); c != nil {
|
||||
h.PBS = c
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// resolveBackupTarget records WHICH storage the primary backup tier writes to (R-109), or records
|
||||
// explicitly that it could not tell. Three outcomes, and the two unknowns are deliberately distinct —
|
||||
// "I could not read my own config" and "my config names a storage that is not here" send an operator
|
||||
// to different places.
|
||||
//
|
||||
// MountPath prefers the live mount and falls back to the CONFIGURED path: during a rebuild the drive is
|
||||
// frequently absent, and when it is, MountPath empties out while ConfigPath is the only thing left that
|
||||
// still says which drive the row was about (the R-116 lesson). The storage's absence from the host is a
|
||||
// separate signal (E-2's backup_target_absent); it does not make the recipe's answer unknown, because
|
||||
// the question here is which storage.cfg row to restore FROM, and that is still known.
|
||||
func resolveBackupTarget(targets []StorageTarget, cfg ConfiguredBackupTarget) *DRBackupTarget {
|
||||
if !cfg.Known || cfg.StorageID == "" {
|
||||
return &DRBackupTarget{State: DRStateUnknown, Reason: DRReasonNoBackupConfig}
|
||||
}
|
||||
for _, t := range targets {
|
||||
if t.Name != cfg.StorageID {
|
||||
continue
|
||||
}
|
||||
mount := t.MountPath
|
||||
if mount == "" {
|
||||
mount = t.ConfigPath
|
||||
}
|
||||
return &DRBackupTarget{State: DRStateResolved, StorageID: cfg.StorageID, MountPath: mount}
|
||||
}
|
||||
return &DRBackupTarget{State: DRStateUnknown, StorageID: cfg.StorageID, Reason: DRReasonNoSuchStorage}
|
||||
}
|
||||
|
||||
// isUserDataDrive selects the external user-data drives the recipe enumerates (felhom-usb / felhom-flash
|
||||
// class): a usb or local-dir storage with a filesystem-UUID durable id and a host mount path. local /
|
||||
// lvmthin / pbs / nfs / cifs are scaffolding (they land in pve_storage) but are not user-data drives.
|
||||
@@ -137,16 +251,30 @@ func isUserDataDrive(t StorageTarget) bool {
|
||||
|
||||
// latestPBSCoord picks the most-recent snapshot (lexical max of the RFC3339 backup_time) and returns
|
||||
// its coordinates. Returns nil when there is no snapshot to target.
|
||||
func latestPBSCoord(snaps []PBSSnapshot, repoID string) *DRPBSCoord {
|
||||
//
|
||||
// The namespace comes from the pbs STORAGE (storage.cfg), never from the snapshot — see DRPBSCoord's
|
||||
// Namespace comment for why the snapshot's own field cannot answer it (R-106). storageFound=false with
|
||||
// snapshots present is a genuine unknown: something listed snapshots, but there is no storage row to
|
||||
// read a namespace from, so the recipe says so rather than defaulting to root.
|
||||
func latestPBSCoord(snaps []PBSSnapshot, repoID, namespace string, storageFound bool) *DRPBSCoord {
|
||||
if len(snaps) == 0 {
|
||||
return nil
|
||||
}
|
||||
sorted := append([]PBSSnapshot(nil), snaps...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].BackupTime > sorted[j].BackupTime })
|
||||
latest := sorted[0]
|
||||
return &DRPBSCoord{
|
||||
c := &DRPBSCoord{
|
||||
RepoID: repoID,
|
||||
Namespace: latest.Namespace,
|
||||
LatestSnapshotID: latest.BackupID,
|
||||
NamespaceState: DRStateUnknown,
|
||||
NamespaceReason: DRReasonNoPBSStorage,
|
||||
}
|
||||
if storageFound {
|
||||
c.NamespaceState, c.NamespaceReason = DRStateResolved, ""
|
||||
// An empty configured namespace is not a missing answer — it IS the root namespace.
|
||||
if c.Namespace = namespace; c.Namespace == "" {
|
||||
c.Namespace = PBSRootNamespace
|
||||
}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -30,7 +32,7 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
|
||||
{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}, // latest
|
||||
}
|
||||
|
||||
h := BuildDRRecipeHostHalf(guests, targets, pbs)
|
||||
h := BuildDRRecipeHostHalf(guests, targets, pbs, ConfiguredBackupTarget{StorageID: "felhom-flash", Known: true})
|
||||
|
||||
if h.RecipeVersion != 1 {
|
||||
t.Errorf("recipe_version=%d, want 1", h.RecipeVersion)
|
||||
@@ -68,7 +70,8 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
|
||||
|
||||
// TestBuildDRRecipeHostHalf_NoPBS: no snapshots → pbs omitted (nil), no panic.
|
||||
func TestBuildDRRecipeHostHalf_NoPBS(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil, []StorageTarget{{Name: "local", Type: StorageTypeLocal}}, nil)
|
||||
h := BuildDRRecipeHostHalf(nil, []StorageTarget{{Name: "local", Type: StorageTypeLocal}}, nil,
|
||||
ConfiguredBackupTarget{StorageID: "local", Known: true})
|
||||
if h.PBS != nil {
|
||||
t.Errorf("pbs should be nil with no snapshots, got %+v", h.PBS)
|
||||
}
|
||||
@@ -89,6 +92,7 @@ func TestDRRecipeHostHalf_V1DriveShape(t *testing.T) {
|
||||
MountPath: "/mnt/felhom-usb", TotalBytes: 931 << 30},
|
||||
},
|
||||
nil,
|
||||
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
|
||||
)
|
||||
if len(h.Drives) != 1 {
|
||||
t.Fatalf("want 1 drive, got %d", len(h.Drives))
|
||||
@@ -124,6 +128,7 @@ func TestDRRecipeHostHalf_NoSecrets(t *testing.T) {
|
||||
{Name: "felhom-usb", Type: StorageTypeUSB, DurableID: "uuid:da9e7089", Role: "bulk-data", MountPath: "/mnt/felhom-usb", TotalBytes: 1},
|
||||
},
|
||||
[]PBSSnapshot{{Namespace: "root", BackupID: "9201", BackupTime: "2026-06-16T08:00:00Z"}},
|
||||
ConfiguredBackupTarget{StorageID: "felhom-usb", Known: true},
|
||||
)
|
||||
b, err := json.Marshal(h)
|
||||
if err != nil {
|
||||
@@ -132,6 +137,294 @@ func TestDRRecipeHostHalf_NoSecrets(t *testing.T) {
|
||||
assertNoSecretKeys(t, b)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// R-106 / R-109 — the recipe records the RESOLVED backup target and the REAL PBS namespace.
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
// capturedDemoFelhomTargets is the storage set demo-felhom really had on 2026-07-30, not an invented
|
||||
// one. PROVENANCE — every field was captured, none composed:
|
||||
//
|
||||
// - names/types/contents: the pve_storage block of the box's own PRE-FIX recipe, downloaded from the
|
||||
// hub at GET /customers/demo-felhom/dr-recipe.json (agent v0.115.0).
|
||||
// - paths + is_mountpoint + the pbs namespace: `cat /etc/pve/storage.cfg` on felhom-pve, same day —
|
||||
// `dir: local path /var/lib/vz`, `dir: felhom-backup path /mnt/hdd_1 is_mountpoint 1`,
|
||||
// `pbs: felhom-pbs ... namespace demo-felhom`.
|
||||
//
|
||||
// THE AMBIGUITY THIS PINS IS REAL, and assertBackupCandidateAmbiguity below refuses to let the fixture
|
||||
// quietly lose it: `local` and `felhom-backup` BOTH carry content=backup, and since the 2026-07-28
|
||||
// vzdump-target move `local` holds archives frozen at that date. Naming the wrong one restores a guest
|
||||
// that is silently months stale.
|
||||
func capturedDemoFelhomTargets() []StorageTarget {
|
||||
return []StorageTarget{
|
||||
{Name: "local-lvm", Type: StorageTypeLVMThin, DurableID: "pve/data", Content: "images,rootdir"},
|
||||
{
|
||||
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
|
||||
DurableID: "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
|
||||
MountPath: "/mnt/hdd_1", ConfigPath: "/mnt/hdd_1", TotalBytes: 983351140352,
|
||||
},
|
||||
{
|
||||
Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup",
|
||||
DurableID: "repo+fp", PBSNamespace: "demo-felhom",
|
||||
},
|
||||
// The decoy: same content, plausible name, historically THE vzdump target. ConfigPath only —
|
||||
// `local` lives on the LVM root and is not its own mount, so the observer leaves MountPath empty.
|
||||
{Name: "local", Type: StorageTypeLocal, Content: "backup,import,vztmpl,iso", ConfigPath: "/var/lib/vz"},
|
||||
}
|
||||
}
|
||||
|
||||
// capturedDemoFelhomSnapshots mirrors what the box's pre-fix recipe carried: latest_snapshot_id "9201".
|
||||
// Namespace is deliberately EMPTY on every element — that is exactly what the PBS API returns once the
|
||||
// list is namespace-scoped via `?ns=`, and it is the input that used to become the bogus "root".
|
||||
func capturedDemoFelhomSnapshots() []PBSSnapshot {
|
||||
return []PBSSnapshot{
|
||||
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-29T22:00:00Z"},
|
||||
{Namespace: "", BackupID: "9201", BackupTime: "2026-07-30T22:00:00Z"}, // latest
|
||||
}
|
||||
}
|
||||
|
||||
// assertBackupCandidateAmbiguity fails if the fixture stopped containing TWO plausible content=backup
|
||||
// storages. Without this the consequence test below could pass on a fixture with only one candidate —
|
||||
// which is precisely the hollow shape that let two defects ship green earlier in this arc.
|
||||
func assertBackupCandidateAmbiguity(t *testing.T, h *DRRecipeHostHalf) {
|
||||
t.Helper()
|
||||
var candidates []string
|
||||
for _, s := range h.PVEStorage {
|
||||
if strings.Contains(s.Content, "backup") && (s.Type == StorageTypeLocalDir || s.Type == StorageTypeLocal) {
|
||||
candidates = append(candidates, s.Name)
|
||||
}
|
||||
}
|
||||
if len(candidates) < 2 {
|
||||
t.Fatalf("fixture no longer poses the R-109 problem: want >=2 content=backup dir storages, got %v", candidates)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_BackupTargetNamesTheLiveStorage is THE consequence assertion for R-109: given a box that
|
||||
// really carries two content=backup dir storages, the generated recipe names the LIVE one, gives its
|
||||
// mountpoint, and does not name the frozen one. Not "the function returned a non-empty string".
|
||||
func TestDRRecipe_BackupTargetNamesTheLiveStorage(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
|
||||
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
|
||||
|
||||
assertBackupCandidateAmbiguity(t, h)
|
||||
|
||||
bt := h.BackupTarget
|
||||
if bt == nil {
|
||||
t.Fatal("backup_target is absent — the recipe still cannot say where the local archives are (R-109)")
|
||||
}
|
||||
if bt.State != DRStateResolved {
|
||||
t.Errorf("state=%q want %q (reason=%q)", bt.State, DRStateResolved, bt.Reason)
|
||||
}
|
||||
if bt.StorageID != "felhom-backup" {
|
||||
t.Errorf("storage_id=%q — the recipe must name the LIVE target, not %q", bt.StorageID, "felhom-backup")
|
||||
}
|
||||
if bt.MountPath != "/mnt/hdd_1" {
|
||||
t.Errorf("mount_path=%q want /mnt/hdd_1 — the mountpoint is what separates it from local's /var/lib/vz", bt.MountPath)
|
||||
}
|
||||
// Unambiguous: the frozen decoy must not be what the field names.
|
||||
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
|
||||
t.Errorf("recipe names the FROZEN target (%q at %q) — a restore from it is silently stale", bt.StorageID, bt.MountPath)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_PBSNamespaceIsThePerCustomerOne is the consequence assertion for R-106: the recipe carries
|
||||
// the namespace the box's backups actually live in, resolved from storage.cfg, and specifically NOT the
|
||||
// "root" that every box used to report.
|
||||
func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), capturedDemoFelhomSnapshots(),
|
||||
ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
|
||||
|
||||
if h.PBS == nil {
|
||||
t.Fatal("pbs coord absent with snapshots present")
|
||||
}
|
||||
if h.PBS.Namespace == PBSRootNamespace {
|
||||
t.Errorf("namespace=%q — this is the R-106 symptom: the snapshot's empty ns normalised to root "+
|
||||
"while the box's backups are in demo-felhom", h.PBS.Namespace)
|
||||
}
|
||||
if h.PBS.Namespace != "demo-felhom" {
|
||||
t.Errorf("namespace=%q want demo-felhom (storage.cfg's `namespace` on the pbs storage)", h.PBS.Namespace)
|
||||
}
|
||||
if h.PBS.NamespaceState != DRStateResolved {
|
||||
t.Errorf("namespace_state=%q want %q (reason=%q)", h.PBS.NamespaceState, DRStateResolved, h.PBS.NamespaceReason)
|
||||
}
|
||||
if h.PBS.RepoID != "felhom-pbs" || h.PBS.LatestSnapshotID != "9201" {
|
||||
t.Errorf("coord drifted: repo=%q snapshot=%q", h.PBS.RepoID, h.PBS.LatestSnapshotID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
|
||||
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
|
||||
// honest root case is never confused with "I could not tell".
|
||||
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil,
|
||||
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
|
||||
capturedDemoFelhomSnapshots(),
|
||||
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
|
||||
|
||||
if h.PBS.NamespaceState != DRStateResolved {
|
||||
t.Errorf("namespace_state=%q — an unconfigured namespace IS the root namespace, not an unknown", h.PBS.NamespaceState)
|
||||
}
|
||||
if h.PBS.Namespace != PBSRootNamespace {
|
||||
t.Errorf("namespace=%q want %q", h.PBS.Namespace, PBSRootNamespace)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable is the WRONG case: the agent could not consult
|
||||
// its own backup config. The recipe must say so explicitly and emit NO storage_id key at all — an
|
||||
// absent value must not be representable as a plausible-looking answer.
|
||||
func TestDRRecipe_BackupTargetUnknownWhenConfigUnavailable(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil, capturedDemoFelhomTargets(), nil, ConfiguredBackupTarget{})
|
||||
|
||||
bt := h.BackupTarget
|
||||
if bt == nil {
|
||||
t.Fatal("backup_target must be PRESENT and say unknown, not vanish")
|
||||
}
|
||||
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
|
||||
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoBackupConfig)
|
||||
}
|
||||
// Absence recorded as absence: no id, and no id KEY on the wire.
|
||||
if bt.StorageID != "" {
|
||||
t.Errorf("storage_id=%q — an unresolvable target must not be filled in", bt.StorageID)
|
||||
}
|
||||
b, err := json.Marshal(bt)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var keys map[string]json.RawMessage
|
||||
if err := json.Unmarshal(b, &keys); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, banned := range []string{"storage_id", "mount_path"} {
|
||||
if _, ok := keys[banned]; ok {
|
||||
t.Errorf("unknown backup_target must not carry a %q key; got %s", banned, b)
|
||||
}
|
||||
}
|
||||
// And nothing in it may read as one of the real candidates.
|
||||
for _, decoy := range []string{"felhom-backup", "local", "/var/lib/vz", "/mnt/hdd_1"} {
|
||||
if strings.Contains(string(b), decoy) {
|
||||
t.Errorf("unknown backup_target leaked a plausible value %q: %s", decoy, b)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_BackupTargetUnknownWhenStorageMissing: the config names a storage this host does not
|
||||
// have. That is unknown for a DIFFERENT reason — and the configured id IS still recorded, because
|
||||
// "config says felhom-backup, no such storage here" sends an operator somewhere useful while silence
|
||||
// does not.
|
||||
func TestDRRecipe_BackupTargetUnknownWhenStorageMissing(t *testing.T) {
|
||||
targets := []StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup", ConfigPath: "/var/lib/vz"}}
|
||||
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
|
||||
|
||||
bt := h.BackupTarget
|
||||
if bt.State != DRStateUnknown || bt.Reason != DRReasonNoSuchStorage {
|
||||
t.Errorf("state=%q reason=%q want %q/%q", bt.State, bt.Reason, DRStateUnknown, DRReasonNoSuchStorage)
|
||||
}
|
||||
if bt.StorageID != "felhom-backup" {
|
||||
t.Errorf("storage_id=%q want the CONFIGURED id recorded even though it matched nothing", bt.StorageID)
|
||||
}
|
||||
// It must NOT silently fall back to the only content=backup storage present.
|
||||
if bt.StorageID == "local" || bt.MountPath == "/var/lib/vz" {
|
||||
t.Error("resolution fell back to the wrong storage instead of reporting unknown")
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage: snapshots exist but no pbs storage was observed, so
|
||||
// there is no storage.cfg row to read a namespace from. The recipe must NOT default to root — that
|
||||
// default is the entire R-106 defect.
|
||||
func TestDRRecipe_PBSNamespaceUnknownWithoutPBSStorage(t *testing.T) {
|
||||
h := BuildDRRecipeHostHalf(nil,
|
||||
[]StorageTarget{{Name: "local", Type: StorageTypeLocal, Content: "backup"}},
|
||||
capturedDemoFelhomSnapshots(),
|
||||
ConfiguredBackupTarget{StorageID: "local", Known: true})
|
||||
|
||||
if h.PBS == nil {
|
||||
t.Fatal("pbs coord should still be emitted (the snapshot id is a real coordinate)")
|
||||
}
|
||||
if h.PBS.NamespaceState != DRStateUnknown || h.PBS.NamespaceReason != DRReasonNoPBSStorage {
|
||||
t.Errorf("namespace_state=%q reason=%q want %q/%q",
|
||||
h.PBS.NamespaceState, h.PBS.NamespaceReason, DRStateUnknown, DRReasonNoPBSStorage)
|
||||
}
|
||||
if h.PBS.Namespace != "" {
|
||||
t.Errorf("namespace=%q — with no storage row to read, the field must be empty, never %q",
|
||||
h.PBS.Namespace, PBSRootNamespace)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone is the DR-shaped case: the recipe is read while
|
||||
// the target drive is absent, so MountPath has emptied out. ConfigPath is then the only thing that still
|
||||
// says where the archives live (the R-116 lesson) — and the target is still RESOLVED, because which
|
||||
// storage.cfg row to restore from is known regardless of whether its device is currently present.
|
||||
func TestDRRecipe_BackupTargetUsesConfigPathWhenDeviceGone(t *testing.T) {
|
||||
targets := []StorageTarget{{
|
||||
Name: "felhom-backup", Type: StorageTypeLocalDir, Content: "backup",
|
||||
MountPath: "", ConfigPath: "/mnt/hdd_1", // device gone: observer empties MountPath, keeps ConfigPath
|
||||
}}
|
||||
h := BuildDRRecipeHostHalf(nil, targets, nil, ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true})
|
||||
|
||||
bt := h.BackupTarget
|
||||
if bt.State != DRStateResolved {
|
||||
t.Errorf("state=%q — an absent device does not make the TARGET unknown", bt.State)
|
||||
}
|
||||
if bt.MountPath != "/mnt/hdd_1" {
|
||||
t.Errorf("mount_path=%q want the configured path /mnt/hdd_1", bt.MountPath)
|
||||
}
|
||||
}
|
||||
|
||||
// fakePBSReporter is a PBSReporter returning fixed snapshots (the verify loop's seam).
|
||||
type fakePBSReporter struct{ snaps []PBSSnapshot }
|
||||
|
||||
func (f fakePBSReporter) PBSSnapshots(context.Context) []PBSSnapshot { return f.snaps }
|
||||
|
||||
// TestCollectDRRecipe_ProductionPath runs the REAL generation path — Collector.Collect(), the method the
|
||||
// daemon calls every cycle — rather than BuildDRRecipeHostHalf directly. It is here because both defects
|
||||
// this file fixes were invisible to a direct-call test: the namespace one lived in what the observer put
|
||||
// on StorageTarget, and the target one lived in whether anything wired the config seam at all. A seam
|
||||
// that is correct and never wired is the failure mode this repo has hit four times.
|
||||
func TestCollectDRRecipe_ProductionPath(t *testing.T) {
|
||||
px := &fakePx{node: "n", ns: newTestNodeStatus()}
|
||||
obs := fakeObserver{targets: capturedDemoFelhomTargets()}
|
||||
pbsRep := fakePBSReporter{snaps: capturedDemoFelhomSnapshots()}
|
||||
|
||||
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
|
||||
c.SetBackupTargetResolver(func() ConfiguredBackupTarget {
|
||||
return ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true}
|
||||
})
|
||||
|
||||
r, err := c.Collect(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Collect: %v", err)
|
||||
}
|
||||
if r.DRRecipe == nil {
|
||||
t.Fatal("collect produced no dr_recipe")
|
||||
}
|
||||
if bt := r.DRRecipe.BackupTarget; bt == nil || bt.State != DRStateResolved || bt.StorageID != "felhom-backup" {
|
||||
t.Errorf("backup_target through Collect = %+v, want resolved/felhom-backup", bt)
|
||||
}
|
||||
if p := r.DRRecipe.PBS; p == nil || p.Namespace != "demo-felhom" || p.NamespaceState != DRStateResolved {
|
||||
t.Errorf("pbs namespace through Collect = %+v, want demo-felhom/resolved", p)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectDRRecipe_UnwiredSeamReportsUnknown: a Collector built WITHOUT the resolver (every
|
||||
// --selftest one-shot did exactly this before v0.118.0) must produce an explicit unknown. This is the
|
||||
// test that would have caught shipping the seam without wiring it.
|
||||
func TestCollectDRRecipe_UnwiredSeamReportsUnknown(t *testing.T) {
|
||||
px := &fakePx{node: "n", ns: newTestNodeStatus()}
|
||||
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{targets: capturedDemoFelhomTargets()},
|
||||
nil, nil, nil, "h", "0.118.0", quietLogger())
|
||||
|
||||
r, err := c.Collect(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Collect: %v", err)
|
||||
}
|
||||
bt := r.DRRecipe.BackupTarget
|
||||
if bt == nil || bt.State != DRStateUnknown || bt.Reason != DRReasonNoBackupConfig {
|
||||
t.Fatalf("unwired resolver must yield unknown/%s, got %+v", DRReasonNoBackupConfig, bt)
|
||||
}
|
||||
if bt.StorageID != "" {
|
||||
t.Errorf("unwired resolver invented a target %q", bt.StorageID)
|
||||
}
|
||||
}
|
||||
|
||||
// assertNoSecretKeys walks decoded JSON and fails on any object key matching secretNameRe. Shared by
|
||||
// the agent boundary assertions. (durable_id/repo_id/latest_snapshot_id are identifiers/coordinates —
|
||||
// none match the credential regex.)
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"net"
|
||||
"net/netip"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// Host addresses (v0.119.0) — "which addresses does this box actually hold?"
|
||||
//
|
||||
// The hub could not answer that at all: HostMetrics carried node/cpu/mem/disk/load/uptime/temp and
|
||||
// no address of any kind, so the LAN IP of a managed host was invisible in every operator surface.
|
||||
// Two sources looked like answers and are not: `lan_resolver.host_ip` is an OPTIONAL config value
|
||||
// (absent unless that feature is configured), and DeriveHostIP(local_api.listen_addr) yields the
|
||||
// R-50 island literal 169.254.253.1 — a link-local address that is the same on every box. Reporting
|
||||
// either would have produced a confident wrong answer, which is worse than the blank it replaces.
|
||||
//
|
||||
// This reads the kernel's own view instead, and it issues NO block I/O (the CLAUDE.md health-check
|
||||
// rule): net.Interfaces() is a netlink/procfs read, needs no privilege, and touches no filesystem.
|
||||
|
||||
// HostAddress is one routable address the host holds, tagged with the interface carrying it.
|
||||
//
|
||||
// Deliberately iface+cidr rather than a single `lan_ip`: a Proxmox host legitimately holds several
|
||||
// (a management bridge, a tailnet, the WG tunnel), and picking one of them to call "the" LAN IP is a
|
||||
// guess the agent is not entitled to make — on a box whose management bridge is not vmbr0 that guess
|
||||
// is silently wrong. The agent reports what exists; the hub does the labelling.
|
||||
type HostAddress struct {
|
||||
Iface string `json:"iface"` // e.g. "vmbr0", "wg-felhom", "tailscale0"
|
||||
CIDR string `json:"cidr"` // e.g. "192.168.0.162/24" — prefix length kept, it is operator-relevant
|
||||
}
|
||||
|
||||
// ifaceAddrs is one enumerated interface: the ONLY facts the filter needs. Keeping the seam this
|
||||
// narrow is what lets the filter be tested against real measured shapes without a network stack.
|
||||
type ifaceAddrs struct {
|
||||
Name string
|
||||
Up bool
|
||||
Loopback bool
|
||||
CIDRs []string
|
||||
}
|
||||
|
||||
// AddressEnumerator returns the host's interfaces. Injectable so the filter can be driven with the
|
||||
// shapes measured on real hardware (see hostaddr_test.go) instead of whatever the test box happens
|
||||
// to have.
|
||||
type AddressEnumerator func() ([]ifaceAddrs, error)
|
||||
|
||||
// systemInterfaces is the production enumerator: the kernel's interface table.
|
||||
func systemInterfaces() ([]ifaceAddrs, error) {
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]ifaceAddrs, 0, len(ifaces))
|
||||
for _, i := range ifaces {
|
||||
e := ifaceAddrs{
|
||||
Name: i.Name,
|
||||
Up: i.Flags&net.FlagUp != 0,
|
||||
Loopback: i.Flags&net.FlagLoopback != 0,
|
||||
}
|
||||
// A per-interface error is not fatal: one unreadable interface must not cost the report
|
||||
// every other address (serve-degraded, as everywhere else in the collector).
|
||||
addrs, aerr := i.Addrs()
|
||||
if aerr != nil {
|
||||
out = append(out, e)
|
||||
continue
|
||||
}
|
||||
for _, a := range addrs {
|
||||
e.CIDRs = append(e.CIDRs, a.String())
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// filterHostAddresses keeps every GLOBAL UNICAST address on an up, non-loopback interface.
|
||||
//
|
||||
// IsGlobalUnicast() is the whole rule, and it was chosen by measuring both demo hosts rather than by
|
||||
// listing interface names to exclude. It drops, in one predicate:
|
||||
// - loopback (127.0.0.1, ::1)
|
||||
// - IPv6 link-local (fe80::/10) — every bridge carries one, pure noise
|
||||
// - IPv4 link-local (169.254.0.0/16) — which is exactly the R-50 island address on vmbr9, an
|
||||
// identical constant on every box and therefore actively misleading if surfaced
|
||||
//
|
||||
// It needs NO veth/fwbr/tap denylist: on a Proxmox host that per-guest plumbing carries no IP at
|
||||
// all, so it self-excludes by having nothing to report. Verified on demo-felhom and demo-hp —
|
||||
// veth9201i0/i1, fwbr*, and the unused NICs all appear in `ip link` and in no `ip addr` output.
|
||||
//
|
||||
// What survives on a real box: vmbr0's LAN address, wg-felhom's tunnel address, and tailscale0's
|
||||
// tailnet addresses. All three are true and useful; none is labelled here.
|
||||
func filterHostAddresses(in []ifaceAddrs) []HostAddress {
|
||||
out := []HostAddress{}
|
||||
for _, i := range in {
|
||||
if i.Loopback || !i.Up {
|
||||
continue
|
||||
}
|
||||
for _, c := range i.CIDRs {
|
||||
p, err := netip.ParsePrefix(c)
|
||||
if err != nil {
|
||||
continue // not a CIDR we understand — skip it, never fail the report
|
||||
}
|
||||
if !p.Addr().IsGlobalUnicast() {
|
||||
continue
|
||||
}
|
||||
out = append(out, HostAddress{Iface: i.Name, CIDR: p.String()})
|
||||
}
|
||||
}
|
||||
// Deterministic order so a report diff reflects a real change, not interface-table ordering.
|
||||
sort.Slice(out, func(a, b int) bool {
|
||||
if out[a].Iface != out[b].Iface {
|
||||
return out[a].Iface < out[b].Iface
|
||||
}
|
||||
return out[a].CIDR < out[b].CIDR
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// collectAddresses is the collector's entry point. It returns a non-nil slice so the field always
|
||||
// marshals as [] — an absent key and "this box has no routable address" must not look alike to the
|
||||
// hub, and [] is the honest encoding of the latter.
|
||||
func (c *Collector) collectAddresses() []HostAddress {
|
||||
enum := c.addrEnum
|
||||
if enum == nil {
|
||||
// Default to the REAL enumerator, deliberately inverting the nil-reporter-means-off
|
||||
// convention used by the optional stanzas above. Those gate on a config feature; this has
|
||||
// no dependency and no feature flag, so a forgotten wiring call in main.go would produce a
|
||||
// silently empty field — the inert-seam failure this repo has shipped four times.
|
||||
enum = systemInterfaces
|
||||
}
|
||||
ifaces, err := enum()
|
||||
if err != nil {
|
||||
c.logger.Warn("host addresses: interface enumeration failed", "err", err)
|
||||
return []HostAddress{}
|
||||
}
|
||||
return filterHostAddresses(ifaces)
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The fixtures below are MEASURED, not invented: `ip -o addr show` on demo-felhom (N100) and
|
||||
// demo-hp (HP t740) on 2026-07-31, transcribed verbatim including the interfaces that carry no
|
||||
// address. That matters — the filter's claim that it needs no veth/fwbr denylist rests on those
|
||||
// interfaces genuinely having nothing to report, and a hand-written fixture that omitted them would
|
||||
// have proved the claim by assuming it.
|
||||
|
||||
// demoFelhomIfaces is demo-felhom's real interface table.
|
||||
func demoFelhomIfaces() []ifaceAddrs {
|
||||
return []ifaceAddrs{
|
||||
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
|
||||
{Name: "enp1s0", Up: false}, // physical NIC, no address
|
||||
{Name: "wlp2s0", Up: false}, // wifi, no address
|
||||
{Name: "tailscale0", Up: true, CIDRs: []string{
|
||||
"100.70.170.35/32", "fd7a:115c:a1e0::5236:aa24/128", "fe80::4197:26fc:ccba:b0d9/64"}},
|
||||
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24", "fe80::6a1d:efff:fe5d:a664/64"}},
|
||||
{Name: "veth9201i0", Up: true}, // per-guest plumbing — no address
|
||||
{Name: "veth9201i1", Up: true}, // per-guest plumbing — no address
|
||||
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::48d4:f6ff:fe05:2f98/64"}},
|
||||
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.2/32"}},
|
||||
}
|
||||
}
|
||||
|
||||
func hasAddr(got []HostAddress, iface, cidr string) bool {
|
||||
for _, a := range got {
|
||||
if a.Iface == iface && a.CIDR == cidr {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func flatten(got []HostAddress) string {
|
||||
var b strings.Builder
|
||||
for _, a := range got {
|
||||
b.WriteString(a.Iface + "=" + a.CIDR + " ")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// The LAN address is the whole point of the feature — it must survive the filter.
|
||||
// RED-PROOF 1: drop the `!p.Addr().IsGlobalUnicast()` continue → the vmbr9 + fe80 assertions below
|
||||
// go red (the LAN one still passes, which is exactly why the negatives are asserted too).
|
||||
func TestFilterHostAddresses_RealHost(t *testing.T) {
|
||||
got := filterHostAddresses(demoFelhomIfaces())
|
||||
|
||||
// --- what MUST be there ---
|
||||
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
|
||||
t.Fatalf("the LAN address was filtered away — the feature reports nothing: %s", flatten(got))
|
||||
}
|
||||
if !hasAddr(got, "wg-felhom", "10.77.0.2/32") {
|
||||
t.Errorf("the WireGuard address was filtered away: %s", flatten(got))
|
||||
}
|
||||
if !hasAddr(got, "tailscale0", "100.70.170.35/32") {
|
||||
t.Errorf("the tailnet address was filtered away: %s", flatten(got))
|
||||
}
|
||||
|
||||
// --- what MUST NOT be there, each for its own reason ---
|
||||
for _, bad := range []struct{ iface, cidr, why string }{
|
||||
{"lo", "127.0.0.1/8", "loopback is not an address of the host on any network"},
|
||||
{"lo", "::1/128", "IPv6 loopback"},
|
||||
{"vmbr9", "169.254.253.1/30", "the R-50 island literal — IDENTICAL on every box, so surfacing it is actively misleading"},
|
||||
{"vmbr0", "fe80::6a1d:efff:fe5d:a664/64", "IPv6 link-local, one per bridge, pure noise"},
|
||||
{"tailscale0", "fe80::4197:26fc:ccba:b0d9/64", "IPv6 link-local"},
|
||||
} {
|
||||
if hasAddr(got, bad.iface, bad.cidr) {
|
||||
t.Errorf("%s %s must be filtered (%s); got: %s", bad.iface, bad.cidr, bad.why, flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// The no-denylist claim: not one veth/physical interface contributed a row.
|
||||
for _, a := range got {
|
||||
if strings.HasPrefix(a.Iface, "veth") || a.Iface == "enp1s0" || a.Iface == "wlp2s0" {
|
||||
t.Errorf("%s produced a row — the fixture says it has no address, so the filter invented one", a.Iface)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// demo-hp is different hardware (4 unused NICs, different ordering) and must filter identically —
|
||||
// the rule is about address CLASS, not about one box's interface names.
|
||||
func TestFilterHostAddresses_SecondHostFiltersIdentically(t *testing.T) {
|
||||
got := filterHostAddresses([]ifaceAddrs{
|
||||
{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8", "::1/128"}},
|
||||
{Name: "enp2s0f0", Up: false}, {Name: "enp1s0f0", Up: false},
|
||||
{Name: "enp1s0f1", Up: false}, {Name: "enp1s0f2", Up: false},
|
||||
{Name: "enp1s0f3", Up: false}, {Name: "wlo1", Up: false},
|
||||
{Name: "tailscale0", Up: true, CIDRs: []string{
|
||||
"100.76.96.79/32", "fd7a:115c:a1e0::ce36:6051/128", "fe80::e06a:ce64:80e1:7821/64"}},
|
||||
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.87/24", "fe80::7ed3:aff:fe77:d976/64"}},
|
||||
{Name: "wg-felhom", Up: true, CIDRs: []string{"10.77.0.3/32"}},
|
||||
{Name: "vmbr9", Up: true, CIDRs: []string{"169.254.253.1/30", "fe80::2484:92ff:fe7d:52a5/64"}},
|
||||
{Name: "veth9201i0", Up: true}, {Name: "veth9201i1", Up: true},
|
||||
})
|
||||
if !hasAddr(got, "vmbr0", "192.168.0.87/24") {
|
||||
t.Fatalf("demo-hp's LAN address was filtered away: %s", flatten(got))
|
||||
}
|
||||
if hasAddr(got, "vmbr9", "169.254.253.1/30") {
|
||||
t.Errorf("demo-hp's island address leaked through: %s", flatten(got))
|
||||
}
|
||||
// The island address is byte-identical on both boxes — the strongest argument for excluding it.
|
||||
if strings.Contains(flatten(got), "169.254.") {
|
||||
t.Errorf("a link-local IPv4 survived: %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// A DOWN interface holding a stale address must not be reported as if the box were reachable there.
|
||||
// RED-PROOF 2: drop `|| !i.Up` → this goes red.
|
||||
func TestFilterHostAddresses_DownInterfaceExcluded(t *testing.T) {
|
||||
got := filterHostAddresses([]ifaceAddrs{
|
||||
{Name: "vmbr0", Up: true, CIDRs: []string{"192.168.0.162/24"}},
|
||||
{Name: "vmbr1", Up: false, CIDRs: []string{"10.9.9.9/24"}},
|
||||
})
|
||||
if hasAddr(got, "vmbr1", "10.9.9.9/24") {
|
||||
t.Errorf("a DOWN interface's address was reported: %s", flatten(got))
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Errorf("want exactly the one up interface, got: %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// Order must be deterministic, or every report diff shows phantom churn.
|
||||
func TestFilterHostAddresses_DeterministicOrder(t *testing.T) {
|
||||
a := filterHostAddresses(demoFelhomIfaces())
|
||||
// Same facts, opposite enumeration order.
|
||||
rev := demoFelhomIfaces()
|
||||
for i, j := 0, len(rev)-1; i < j; i, j = i+1, j-1 {
|
||||
rev[i], rev[j] = rev[j], rev[i]
|
||||
}
|
||||
b := filterHostAddresses(rev)
|
||||
if flatten(a) != flatten(b) {
|
||||
t.Errorf("interface-table order changed the report:\n a=%s\n b=%s", flatten(a), flatten(b))
|
||||
}
|
||||
}
|
||||
|
||||
// A host with nothing routable yields [] and never nil — an absent key and "no addresses" must not
|
||||
// look alike on the wire.
|
||||
func TestFilterHostAddresses_EmptyIsNonNil(t *testing.T) {
|
||||
got := filterHostAddresses([]ifaceAddrs{{Name: "lo", Up: true, Loopback: true, CIDRs: []string{"127.0.0.1/8"}}})
|
||||
if got == nil {
|
||||
t.Fatal("filter returned nil — it would marshal as null, not []")
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Errorf("want no addresses, got %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// A malformed entry is skipped, never fatal — one bad address must not cost the report the others.
|
||||
func TestFilterHostAddresses_MalformedSkipped(t *testing.T) {
|
||||
got := filterHostAddresses([]ifaceAddrs{
|
||||
{Name: "vmbr0", Up: true, CIDRs: []string{"not-an-address", "192.168.0.162/24"}},
|
||||
})
|
||||
if len(got) != 1 || !hasAddr(got, "vmbr0", "192.168.0.162/24") {
|
||||
t.Errorf("a malformed sibling address broke the good one: %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// --- the WIRING half: the collector must actually call the filter ---
|
||||
|
||||
// The seam defaults to the REAL enumerator, so a forgotten wiring call cannot make this inert.
|
||||
// RED-PROOF 3: replace the collectAddresses body with `return []HostAddress{}` → red.
|
||||
func TestCollectAddresses_UsesTheInjectedEnumerator(t *testing.T) {
|
||||
c := &Collector{logger: slog.Default()}
|
||||
c.addrEnum = func() ([]ifaceAddrs, error) { return demoFelhomIfaces(), nil }
|
||||
got := c.collectAddresses()
|
||||
if !hasAddr(got, "vmbr0", "192.168.0.162/24") {
|
||||
t.Fatalf("the collector did not run the filter over the enumerator's output: %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// An enumeration failure degrades to [] and a WARN — never a failed report.
|
||||
func TestCollectAddresses_EnumerationErrorDegrades(t *testing.T) {
|
||||
c := &Collector{logger: slog.Default()}
|
||||
c.addrEnum = func() ([]ifaceAddrs, error) { return nil, errors.New("netlink is unhappy") }
|
||||
got := c.collectAddresses()
|
||||
if got == nil {
|
||||
t.Fatal("an enumeration error produced nil, which marshals as null")
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Errorf("want [] on error, got %s", flatten(got))
|
||||
}
|
||||
}
|
||||
|
||||
// The production enumerator must return SOMETHING on the machine running the tests, and must not
|
||||
// panic. This is the only test that touches the real network stack; it asserts the contract
|
||||
// (non-nil, no error, loopback correctly flagged) rather than any specific address, because the
|
||||
// test host's addresses are not ours to predict.
|
||||
func TestSystemInterfaces_ProductionEnumeratorWorks(t *testing.T) {
|
||||
ifaces, err := systemInterfaces()
|
||||
if err != nil {
|
||||
t.Fatalf("systemInterfaces: %v", err)
|
||||
}
|
||||
if len(ifaces) == 0 {
|
||||
t.Fatal("no interfaces at all — even a container has lo")
|
||||
}
|
||||
var sawLoopback bool
|
||||
for _, i := range ifaces {
|
||||
if i.Loopback {
|
||||
sawLoopback = true
|
||||
}
|
||||
}
|
||||
if !sawLoopback {
|
||||
t.Error("no interface reported the loopback flag — the flag mapping is wrong")
|
||||
}
|
||||
// And the filter must survive real input without panicking.
|
||||
_ = filterHostAddresses(ifaces)
|
||||
}
|
||||
@@ -43,6 +43,12 @@ type HostReport struct {
|
||||
// alert. Not a secret (the fp is public; the token is never reported).
|
||||
LeafFingerprint string `json:"leaf_fingerprint"`
|
||||
|
||||
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
|
||||
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
|
||||
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
|
||||
// see hostaddr.go for why it is iface+cidr rather than a single lan_ip.
|
||||
Addresses []HostAddress `json:"addresses"`
|
||||
|
||||
// DR recipe — the agent (storage/guest/PBS) half of the secret-free reconstruction recipe
|
||||
// (SPIKE-dr-recipe-2026-06-16). Derived from the facts above; carries ONLY identifiers/intents/
|
||||
// sizes/coordinates, never a secret. The hub assembles it with the controller's app half.
|
||||
@@ -280,6 +286,30 @@ type StorageTarget struct {
|
||||
|
||||
MountPath string `json:"mount_path"` // host mountpoint (dir/usb); "" for network/lvm
|
||||
BackingDevice string `json:"backing_device"` // resolved block device (e.g. /dev/sdb1); "" for network
|
||||
// ConfigPath is the storage's CONFIGURED path from storage.cfg (proxmox.Storage.Path) — not a
|
||||
// resolved mount. It is the only identity a dir storage keeps when its device is gone: MountPath
|
||||
// and BackingDevice both empty out (observe.go's exactMount block) and DurableID degrades off the
|
||||
// fs-UUID, so the configured path is what still says WHICH drive this row is about (R-116).
|
||||
//
|
||||
// `json:"-"` DELIBERATELY. This struct is a cross-repo contract duplicated in felhom.eu/hub and
|
||||
// pinned by testdata/host-report.golden.json + contract_test.go's key-set comparison; a wire-visible
|
||||
// field here would need a matching change in the other repo to stay non-drifting. Nothing off-box
|
||||
// needs this value — its only consumer is the agent's own /disks construction, in-process.
|
||||
ConfigPath string `json:"-"`
|
||||
// PBSNamespace is the storage's CONFIGURED PBS namespace from storage.cfg (proxmox.Storage.Namespace)
|
||||
// — "" for the root namespace, set for S4 per-customer tenancy. Present ONLY on pbs targets.
|
||||
//
|
||||
// It exists because storage.cfg is the ONE authority on which namespace this box's backups use:
|
||||
// `vzdump --storage <pbs>` makes PVE read this exact field, and the agent's own verify client is
|
||||
// built from it (`cmd/felhom-agent/main.go` → `pbs.Config{Namespace: s.Namespace}`). The DR recipe
|
||||
// therefore resolves the namespace from HERE and not from a listed snapshot — a namespace-scoped
|
||||
// PBS list does not echo `ns` per item, so the snapshot's own field is empty and normalising that
|
||||
// empty to "root" is what made the recipe claim "root" on every per-customer box (R-106).
|
||||
//
|
||||
// `json:"-"` for the SAME reason as ConfigPath above: this struct is a cross-repo contract pinned by
|
||||
// testdata/host-report.golden.json, and nothing off-box reads this value — its only consumer is the
|
||||
// agent's own dr_recipe construction, in-process.
|
||||
PBSNamespace string `json:"-"`
|
||||
// ClassHint is a fast|slow HINT derived from the backing disk's rotational flag — a
|
||||
// hint only; the authoritative class is hub-owned (locked decision). "" when not
|
||||
// derivable (network targets have no local rotational flag).
|
||||
@@ -316,6 +346,11 @@ type ThinPoolFill struct {
|
||||
type SmartSummary struct {
|
||||
Health string `json:"health"`
|
||||
|
||||
// ModelName is smartctl's own device model (v0.95.0), captured from the JSON already parsed, so
|
||||
// the UI can show a human label ("TOSHIBA MQ04ABF100") instead of a raw UUID. omitempty +
|
||||
// pointer: absent on an old agent or a device that reports no model.
|
||||
ModelName *string `json:"model_name,omitempty"`
|
||||
|
||||
TemperatureC *int `json:"temperature_c"`
|
||||
PowerOnHours *int `json:"power_on_hours"`
|
||||
|
||||
|
||||
@@ -32,10 +32,11 @@ func TestHostReport_FieldNamesAndEmptyCollections(t *testing.T) {
|
||||
Cloudflared: Cloudflared{Status: "active"},
|
||||
Capabilities: []capability.Status{},
|
||||
LeafFingerprint: "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245",
|
||||
Addresses: []HostAddress{},
|
||||
}
|
||||
// dr_recipe is always set on the real path (Collect); set it here too so the "no null" invariant
|
||||
// covers it (empty pbs is omitempty → omitted, never null).
|
||||
r.DRRecipe = BuildDRRecipeHostHalf(r.Guests, r.StorageTargets, r.PBSSnapshots)
|
||||
r.DRRecipe = BuildDRRecipeHostHalf(r.Guests, r.StorageTargets, r.PBSSnapshots, ConfiguredBackupTarget{})
|
||||
b, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -51,6 +52,7 @@ func TestHostReport_FieldNamesAndEmptyCollections(t *testing.T) {
|
||||
// empty collections must be [] not null
|
||||
`"storage_targets":[]`, `"backups":[]`, `"restore_tests":[]`, `"pbs_snapshots":[]`, `"audit_tail":[]`,
|
||||
`"capabilities":[]`,
|
||||
`"addresses":[]`,
|
||||
`"leaf_fingerprint":"60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"`,
|
||||
} {
|
||||
if !strings.Contains(got, field) {
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-189 — a passing restore-test must survive an agent restart and reach the hub.
|
||||
//
|
||||
// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test
|
||||
// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged
|
||||
// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on
|
||||
// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86
|
||||
// made the agent refuse to re-test an archive it has already proven.
|
||||
//
|
||||
// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was
|
||||
// hollow because a perfectly regular series landed exactly on a threshold and passed under the
|
||||
// mutation it was meant to catch.
|
||||
|
||||
type fakeLatest struct{ tests []RestoreTest }
|
||||
|
||||
func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||
|
||||
type fakeProven struct{ tests []RestoreTest }
|
||||
|
||||
func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||
|
||||
func rt(tier, archive string, pass bool, at time.Time) RestoreTest {
|
||||
return RestoreTest{
|
||||
SourceArchive: archive, SourceTier: tier, Pass: pass,
|
||||
Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
|
||||
// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what
|
||||
// is under test, not the rest of the collection.
|
||||
func mergeCollector(latest, proven []RestoreTest) *Collector {
|
||||
c := &Collector{}
|
||||
if latest != nil {
|
||||
c.restoreTests = &fakeLatest{tests: latest}
|
||||
}
|
||||
if proven != nil {
|
||||
c.provenTests = &fakeProven{tests: proven}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func findTier(got []RestoreTest, tier string) (RestoreTest, int) {
|
||||
var hit RestoreTest
|
||||
n := 0
|
||||
for _, e := range got {
|
||||
if e.SourceTier == tier {
|
||||
hit, n = e, n+1
|
||||
}
|
||||
}
|
||||
return hit, n
|
||||
}
|
||||
|
||||
// ── SCENARIO A — a proof survives a restart and reaches the hub ──────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from
|
||||
// collectRestoreTests (return the in-memory slice as it used to) →
|
||||
//
|
||||
// --- FAIL: TestMerge_ProofSurvivesARestart
|
||||
// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies)
|
||||
//
|
||||
// which is exactly the live observation: `0 restore-tests`. Restored.
|
||||
func TestMerge_ProofSurvivesARestart(t *testing.T) {
|
||||
provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp
|
||||
// After a restart the in-memory store is EMPTY — this is the whole point.
|
||||
c := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||
rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt),
|
||||
})
|
||||
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got)
|
||||
}
|
||||
e := got[0]
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" {
|
||||
t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive)
|
||||
}
|
||||
if e.SourceTier != "pbs" || !e.Pass {
|
||||
t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass)
|
||||
}
|
||||
if e.TestedAt != provenAt.Format(time.RFC3339) {
|
||||
t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B — the report does not invent a pass ───────────────────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the
|
||||
// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the
|
||||
// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which
|
||||
// this test injects directly and the assertion below rejects.
|
||||
func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) {
|
||||
// Nothing proven anywhere: no in-memory result, no persisted proof.
|
||||
c := mergeCollector([]RestoreTest{}, []RestoreTest{})
|
||||
if got := c.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+
|
||||
"worse than the defect being fixed; got %+v", got)
|
||||
}
|
||||
|
||||
// And an entry the state layer could not describe (no tier) is never promoted into a proof.
|
||||
c2 := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||
{SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true,
|
||||
TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)},
|
||||
})
|
||||
if got := c2.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ─────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry
|
||||
// unconditionally) →
|
||||
//
|
||||
// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier
|
||||
// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests
|
||||
//
|
||||
// Restored.
|
||||
func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) {
|
||||
lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box
|
||||
fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)
|
||||
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||
)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
e, n := findTier(got, "pbs")
|
||||
if n != 1 {
|
||||
t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got)
|
||||
}
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||
t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt)
|
||||
}
|
||||
|
||||
// ...and the older-in-memory / newer-persisted direction, which is the post-restart case.
|
||||
c2 := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||
)
|
||||
e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs")
|
||||
if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||
t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D — a failure still reaches the hub ─────────────────────────────────────────────
|
||||
//
|
||||
// The merge must not mask a failure with an older stored success. A failing tier is retried at the
|
||||
// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest
|
||||
// DR signal this system produces.
|
||||
func TestMerge_AFailureIsStillReported(t *testing.T) {
|
||||
provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC)
|
||||
failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC)
|
||||
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)},
|
||||
)
|
||||
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||
if n != 1 {
|
||||
t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background()))
|
||||
}
|
||||
if e.Pass {
|
||||
t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+
|
||||
"silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// Two different tiers are both reported — the merge is per tier, not a single slot.
|
||||
func TestMerge_BothTiersSurvive(t *testing.T) {
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true,
|
||||
time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true,
|
||||
time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))},
|
||||
)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if _, n := findTier(got, "local"); n != 1 {
|
||||
t.Fatalf("the in-memory tier must survive the merge; got %+v", got)
|
||||
}
|
||||
if _, n := findTier(got, "pbs"); n != 1 {
|
||||
t.Fatalf("the persisted tier must survive the merge; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A malformed timestamp must never displace a good entry — "unparseable" is not "newest".
|
||||
func TestMerge_MalformedTimestampNeverWins(t *testing.T) {
|
||||
good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC))
|
||||
bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"}
|
||||
|
||||
c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad})
|
||||
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||
if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" {
|
||||
t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil proven-source leaves the pre-R-189 behaviour exactly as it was.
|
||||
func TestMerge_NilProvenSourceIsANoOp(t *testing.T) {
|
||||
only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))
|
||||
c := mergeCollector([]RestoreTest{only}, nil)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if len(got) != 1 || got[0].SourceArchive != only.SourceArchive {
|
||||
t.Fatalf("a nil durable source must not change anything; got %+v", got)
|
||||
}
|
||||
}
|
||||
+13
-3
@@ -134,6 +134,9 @@
|
||||
"audit_tail": [],
|
||||
"capabilities": [],
|
||||
"leaf_fingerprint": "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245",
|
||||
"addresses": [
|
||||
{ "iface": "vmbr0", "cidr": "192.168.0.162/24" }
|
||||
],
|
||||
"dr_recipe": {
|
||||
"recipe_version": 1,
|
||||
"guests": [
|
||||
@@ -141,7 +144,8 @@
|
||||
],
|
||||
"pbs": {
|
||||
"repo_id": "felhom-pbs",
|
||||
"namespace": "root",
|
||||
"namespace": "felhom-spike",
|
||||
"namespace_state": "resolved",
|
||||
"latest_snapshot_id": "9001"
|
||||
},
|
||||
"drives": [
|
||||
@@ -154,7 +158,13 @@
|
||||
],
|
||||
"pve_storage": [
|
||||
{ "name": "local-lvm", "type": "lvmthin", "content": "rootdir,images" },
|
||||
{ "name": "usb-backup", "type": "usb", "content": "backup" }
|
||||
]
|
||||
{ "name": "usb-backup", "type": "usb", "content": "backup" },
|
||||
{ "name": "felhom-pbs", "type": "pbs", "content": "backup" }
|
||||
],
|
||||
"backup_target": {
|
||||
"state": "resolved",
|
||||
"storage_id": "usb-backup",
|
||||
"mount_path": "/mnt/usb-backup"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-88 Part 2 — the agent gains a third state.
|
||||
//
|
||||
// `newestArchiveOn` promised in its own doc comment that "errors degrade to unknown, never to
|
||||
// no-backup", while its `(time.Time, bool)` return made that impossible: an error and a genuine
|
||||
// not-found both produced `(zero, false)`, so `/backup/due` answered a POSITIVE
|
||||
// "no successful backup recorded yet" with a nil age. The controller read that as "never backed up"
|
||||
// and fired its window-gate safety valve, quiescing customer app stacks outside the backup window.
|
||||
//
|
||||
// These tests assert the WIRE, because the wire is the contract another component reads.
|
||||
|
||||
// listerBackups is a fakeBackups that also implements BackupArchiveLister, with a controllable outcome.
|
||||
type listerBackups struct {
|
||||
fakeBackups
|
||||
t time.Time
|
||||
found bool
|
||||
err error
|
||||
}
|
||||
|
||||
func (l *listerBackups) NewestArchiveTime(context.Context, int) (time.Time, bool, error) {
|
||||
return l.t, l.found, l.err
|
||||
}
|
||||
|
||||
// dueWithLister builds a server whose single tier's service is the given lister, and returns the
|
||||
// /backup/due response.
|
||||
// NOTE: the server must receive `lb` ITSELF, not its embedded fakeBackups — the tier's Service is
|
||||
// type-asserted to BackupArchiveLister, and the embedded value does not satisfy it. Passing the
|
||||
// inner struct silently routes every case to archiveUnknown, which looks like a code bug and is not.
|
||||
func dueWithLister(t *testing.T, lb *listerBackups, store *fakeStore) BackupDueResponse {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{},
|
||||
Backups: lb,
|
||||
Store: store,
|
||||
Storage: fakeStorage{},
|
||||
Tokens: staticTokens{"A": 8200, "B": 9300},
|
||||
BackupCadence: 24 * time.Hour,
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewServer: %v", err)
|
||||
}
|
||||
srv.now = func() time.Time { return testNow }
|
||||
return dueOf(t, srv.Handler())
|
||||
}
|
||||
|
||||
// ── SCENARIO A (agent half) — an unreadable storage is UNKNOWN, not "never" ──────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make newestArchiveOn return archiveAbsent on the error path (the
|
||||
// pre-fix collapse) and this fails with
|
||||
//
|
||||
// "an unreadable storage must report age_state=unknown, got \"absent\" — that is a POSITIVE claim
|
||||
// of 'never backed up' built out of two absences"
|
||||
//
|
||||
// Restored.
|
||||
func TestAgeState_UnreadableStorageIsUnknown(t *testing.T) {
|
||||
lb := &listerBackups{err: errors.New("proxmox: GET storage content: 500 connection refused")}
|
||||
got := dueWithLister(t, lb, &fakeStore{}) // empty store = cold in-memory record
|
||||
|
||||
if got.AgeState != AgeStateUnknown {
|
||||
t.Fatalf("an unreadable storage must report age_state=%q, got %q — that is a POSITIVE claim of "+
|
||||
"'never backed up' built out of two absences", AgeStateUnknown, got.AgeState)
|
||||
}
|
||||
if !got.Due {
|
||||
t.Fatal("FAIL-SAFE DIRECTION: unknown must still be DUE — an unreadable storage must never suppress a backup")
|
||||
}
|
||||
if got.AgeSecs != nil {
|
||||
t.Fatalf("an unknown age must not invent a number; got %d", *got.AgeSecs)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B (agent half) — a genuine first-ever backup is ABSENT ──────────────────────────
|
||||
//
|
||||
// B is what makes A safe: an implementation that reported everything as "unknown" would pass A and
|
||||
// silently starve a brand-new box, because the controller only fires the first-backup valve on ABSENT.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make newestArchiveOn return archiveUnknown when !found and this
|
||||
// fails with
|
||||
//
|
||||
// "a genuine never-backed-up tier must report age_state=\"absent\", got \"unknown\" — the
|
||||
// controller only licenses a first backup outside the window on ABSENT"
|
||||
//
|
||||
// Restored.
|
||||
func TestAgeState_GenuinelyNeverIsAbsent(t *testing.T) {
|
||||
lb := &listerBackups{found: false} // read SUCCEEDED, nothing there
|
||||
got := dueWithLister(t, lb, &fakeStore{})
|
||||
|
||||
if got.AgeState != AgeStateAbsent {
|
||||
t.Fatalf("a genuine never-backed-up tier must report age_state=%q, got %q — the controller only "+
|
||||
"licenses a first backup outside the window on ABSENT", AgeStateAbsent, got.AgeState)
|
||||
}
|
||||
if !got.Due {
|
||||
t.Fatal("a never-backed-up tier must be due")
|
||||
}
|
||||
}
|
||||
|
||||
// A real archive → known, with a real age.
|
||||
func TestAgeState_FoundIsKnown(t *testing.T) {
|
||||
lb := &listerBackups{t: testNow.Add(-2 * time.Hour), found: true}
|
||||
got := dueWithLister(t, lb, &fakeStore{})
|
||||
|
||||
if got.AgeState != AgeStateKnown {
|
||||
t.Fatalf("a readable archive must report age_state=%q, got %q", AgeStateKnown, got.AgeState)
|
||||
}
|
||||
if got.AgeSecs == nil || *got.AgeSecs < 7100 || *got.AgeSecs > 7300 {
|
||||
t.Fatalf("expected ~7200s age, got %v", got.AgeSecs)
|
||||
}
|
||||
if got.Due {
|
||||
t.Fatal("2h old against a 24h cadence is not due")
|
||||
}
|
||||
}
|
||||
|
||||
// An unparseable in-memory timestamp is UNKNOWN too — a backup DID happen, we just cannot date it.
|
||||
// Reporting "absent" there would be the same false-positive claim in a different costume.
|
||||
func TestAgeState_UnparseableTimestampIsUnknown(t *testing.T) {
|
||||
st := &fakeStore{backups: []hub.Backup{{VMID: 8200, Success: true, StartedAt: "not-a-timestamp"}}}
|
||||
lb := &listerBackups{err: errors.New("storage unreadable")}
|
||||
got := dueWithLister(t, lb, st)
|
||||
|
||||
if got.AgeState != AgeStateUnknown {
|
||||
t.Fatalf("an unparseable backup time means we cannot DATE a backup that exists — want %q, got %q",
|
||||
AgeStateUnknown, got.AgeState)
|
||||
}
|
||||
if !got.Due {
|
||||
t.Fatal("still due — fail safe toward taking a backup")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D (agent half) — additive on the wire ───────────────────────────────────────────
|
||||
//
|
||||
// An OLD controller decodes into a struct without `age_state` and ignores it. What it MUST still see
|
||||
// unchanged is every pre-existing field.
|
||||
func TestAgeState_IsAdditive_PreExistingFieldsUnchanged(t *testing.T) {
|
||||
lb := &listerBackups{t: testNow.Add(-48 * time.Hour), found: true}
|
||||
got := dueWithLister(t, lb, &fakeStore{})
|
||||
|
||||
if !got.Due || got.Reason != "older than cadence" {
|
||||
t.Fatalf("pre-existing due/reason semantics changed: due=%v reason=%q", got.Due, got.Reason)
|
||||
}
|
||||
if got.AgeSecs == nil {
|
||||
t.Fatal("age_seconds must still be present for a known age")
|
||||
}
|
||||
// And the state rides alongside rather than replacing anything.
|
||||
if got.AgeState != AgeStateKnown {
|
||||
t.Fatalf("age_state should be %q, got %q", AgeStateKnown, got.AgeState)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// BackupTargetWrapperPath is the pinned sudoers vector (configs/felhom-backup-target-apply). The
|
||||
// agent cannot create a PVE storage or grant an ACL itself — Datastore.Allocate at /storage and
|
||||
// Permissions.Modify are deliberately outside its role — so the privileged half runs here.
|
||||
const BackupTargetWrapperPath = "/usr/local/sbin/felhom-backup-target-apply"
|
||||
|
||||
// backupTargetRequest is POST /backup/target: move the PRIMARY whole-guest backup tier onto the
|
||||
// drive mounted at Where, creating the storage if needed.
|
||||
type backupTargetRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
Where string `json:"where"` // the drive's OWN host mountpoint (F-1)
|
||||
ID string `json:"id,omitempty"` // storage id; default backupTargetStorageID
|
||||
}
|
||||
|
||||
// backupTargetStorageID is the conventional id, matching what E-1 created by hand on both demo boxes.
|
||||
// Keeping the name identical is what makes this endpoint IDEMPOTENT on an already-migrated box: the
|
||||
// wrapper accepts an existing entry with the same path and changes nothing.
|
||||
const backupTargetStorageID = "felhom-backup"
|
||||
|
||||
// handleSetBackupTarget performs the whole move as one ordered operation: create the storage, grant
|
||||
// the agent access, repoint the primary tier in agent.json, and hand back what the caller must do to
|
||||
// make it take effect.
|
||||
//
|
||||
// THE ORDER IS THE DESIGN, and each step is a precondition for the next:
|
||||
//
|
||||
// create → grant → config
|
||||
//
|
||||
// Reversed, a config pointing at a storage that does not exist would make the tier DEFER (harmless
|
||||
// but silent), and a config pointing at an ungranted storage would make every backup 403 on its
|
||||
// first run — which is exactly what E-1 hit when the grant was forgotten (finding F-3). Creating and
|
||||
// granting BEFORE the config means the worst interruption leaves an unused storage, never a broken
|
||||
// tier.
|
||||
//
|
||||
// IT DOES NOT RESTART THE AGENT. That is deliberate and it is the E-1 lesson encoded: the backup
|
||||
// tiers are built once at daemon start, so the move needs a restart to take effect — but restarting
|
||||
// while a backup or restore-test is in flight cancels the wait and records a SPURIOUS tier failure
|
||||
// for a backup that actually succeeded (E-1 did exactly this to a felhom-pbs run). A restart that
|
||||
// this handler fires itself could never be re-checked against in-flight work by the caller, so the
|
||||
// response reports `restart_required` and the caller performs it behind its own immediate
|
||||
// in-flight check.
|
||||
func (s *Server) handleSetBackupTarget(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if s.privileged == nil {
|
||||
writeErr(w, http.StatusServiceUnavailable, "privileged runner not configured on this host")
|
||||
return
|
||||
}
|
||||
var req backupTargetRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
where := strings.TrimSpace(req.Where)
|
||||
if where == "" {
|
||||
writeErr(w, http.StatusBadRequest, "where (the drive's own mountpoint) is required")
|
||||
return
|
||||
}
|
||||
id := strings.TrimSpace(req.ID)
|
||||
if id == "" {
|
||||
id = backupTargetStorageID
|
||||
}
|
||||
|
||||
// AGENT-SIDE VALIDATION FIRST, from the agent's own storage view — never the caller's claim.
|
||||
// The wrapper re-checks everything as root (it is the security boundary), but refusing here gives
|
||||
// the customer a reason instead of a shell error, and keeps a bad request from reaching sudo at all.
|
||||
if err := s.validateBackupTargetMount(r.Context(), where); err != nil {
|
||||
s.logger.Warn("local-api: backup-target move refused", "vmid", vmid, "where", where, "err", err)
|
||||
writeErr(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if _, errOut, err := s.privileged.Run(r.Context(), BackupTargetWrapperPath, "create", id, where); err != nil {
|
||||
s.logger.Error("local-api: backup-target create failed", "id", id, "where", where, "err", err, "stderr", string(errOut))
|
||||
writeErr(w, http.StatusBadGateway, "could not create the backup storage: "+wrapperReason(errOut, err))
|
||||
return
|
||||
}
|
||||
if _, errOut, err := s.privileged.Run(r.Context(), BackupTargetWrapperPath, "grant", id); err != nil {
|
||||
// The storage exists but the agent cannot write to it. Say so precisely: this is the exact
|
||||
// state that produced E-1's "403 permission denied at /storage/felhom-backup" on first backup.
|
||||
s.logger.Error("local-api: backup-target grant failed", "id", id, "err", err, "stderr", string(errOut))
|
||||
writeErr(w, http.StatusBadGateway, "storage created but the access grant failed — backups would 403: "+wrapperReason(errOut, err))
|
||||
return
|
||||
}
|
||||
if err := s.setConfiguredBackupTarget(id); err != nil {
|
||||
s.logger.Error("local-api: backup-target config write failed", "id", id, "err", err)
|
||||
writeErr(w, http.StatusInternalServerError, "storage is ready but the config could not be updated: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
s.logger.Info("local-api: backup target moved — RESTART REQUIRED for it to take effect",
|
||||
"vmid", vmid, "target", id, "where", where)
|
||||
writeOK(w, map[string]any{
|
||||
"vmid": vmid, "target": id, "where": where,
|
||||
// The caller must restart the agent BEHIND ITS OWN in-flight check — see the doc comment.
|
||||
"restart_required": true,
|
||||
})
|
||||
}
|
||||
|
||||
// validateBackupTargetMount refuses a mount that cannot be a real backup target, from the agent's own
|
||||
// storage view + mount table. Mirrors the wrapper's laws so the customer gets a reason, not a shell error.
|
||||
func (s *Server) validateBackupTargetMount(ctx context.Context, where string) error {
|
||||
if s.storage == nil {
|
||||
return fmt.Errorf("storage view unavailable")
|
||||
}
|
||||
// It must currently BE a mountpoint (F-1/F-2). Resolved from the mount table, which is the same
|
||||
// source the wrapper's `mountpoint -q` consults.
|
||||
mounts, err := s.hostReader().Mounts()
|
||||
if err != nil {
|
||||
return fmt.Errorf("could not read the mount table")
|
||||
}
|
||||
var dev string
|
||||
for _, m := range mounts {
|
||||
if m.MountPoint == where {
|
||||
dev = m.Device
|
||||
break
|
||||
}
|
||||
}
|
||||
if dev == "" {
|
||||
return fmt.Errorf("%s is not a mountpoint — the backup target must be the drive's own mountpoint", where)
|
||||
}
|
||||
// Never the system disk: a target there protects against corruption only, never drive loss.
|
||||
for _, m := range mounts {
|
||||
if m.MountPoint == "/" && m.Device == dev {
|
||||
return fmt.Errorf("%s is on the system disk — a backup target there cannot survive a drive failure", where)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// setConfiguredBackupTarget rewrites backup.local_backup_target in agent.json.
|
||||
//
|
||||
// Read-modify-write over map[string]json.RawMessage so UNKNOWN KEYS ARE PRESERVED VERBATIM — the
|
||||
// same discipline as pbsdr.seedEscrowStorageID, and the property that made E-1's hand edit safe to
|
||||
// begin with. A typed round-trip would silently drop any key this build does not know about.
|
||||
//
|
||||
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
|
||||
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
|
||||
// parked first, so a torn write is recoverable.
|
||||
func (s *Server) setConfiguredBackupTarget(id string) error {
|
||||
path := s.configPath
|
||||
if path == "" {
|
||||
return fmt.Errorf("no config path known to this agent (env-only config)")
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
var bk map[string]json.RawMessage
|
||||
if cur, ok := doc["backup"]; ok {
|
||||
if err := json.Unmarshal(cur, &bk); err != nil {
|
||||
return fmt.Errorf("parse backup section: %w", err)
|
||||
}
|
||||
} else {
|
||||
bk = map[string]json.RawMessage{}
|
||||
}
|
||||
idJSON, _ := json.Marshal(id)
|
||||
bk["local_backup_target"] = idJSON
|
||||
bkJSON, err := json.Marshal(bk)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
doc["backup"] = bkJSON
|
||||
out, err := json.MarshalIndent(doc, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.stateDir != "" {
|
||||
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
|
||||
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
|
||||
}
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(out); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
|
||||
// terms the customer can act on ("not a mountpoint", "already exists at …") — falling back to the
|
||||
// exec error only when stderr said nothing useful.
|
||||
func wrapperReason(errOut []byte, err error) string {
|
||||
for _, line := range strings.Split(string(errOut), "\n") {
|
||||
if strings.Contains(line, "REFUSED:") {
|
||||
return strings.TrimSpace(line)
|
||||
}
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// backupTargetServer builds a server whose PRIMARY tier is `felhom-backup`, mounted at /mnt/nvme-1tb
|
||||
// on its own non-system device — i.e. the exact live shape E-1 created on demo-hp and demo-felhom:
|
||||
// the drive is simultaneously the enrolled user-data drive AND the whole-guest vzdump target.
|
||||
func backupTargetServer(t *testing.T) http.Handler {
|
||||
t.Helper()
|
||||
sv := fakeStorage{targets: []hub.StorageTarget{
|
||||
{
|
||||
Name: "felhom-backup", Type: hub.StorageTypeLocalDir, State: hub.StorageStateAttached,
|
||||
Reachable: true, MountPath: "/mnt/nvme-1tb", BackingDevice: "/dev/nvme0n1",
|
||||
Content: "backup",
|
||||
},
|
||||
{
|
||||
Name: "spare-drive", Type: hub.StorageTypeLocalDir, State: hub.StorageStateAttached,
|
||||
Reachable: true, MountPath: "/mnt/spare", BackingDevice: "/dev/sdz1",
|
||||
Content: "backup",
|
||||
},
|
||||
}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{}, Storage: sv,
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
Disks: &fakeDiskOps{}, DiskGate: &fakeGate{}, HostReader: sysOnSDA(),
|
||||
// Service is load-bearing: normalizeBackupTiers DROPS any tier with a nil Service and falls
|
||||
// back to the legacy single tier with an empty TargetID — which silently made an earlier
|
||||
// version of this test exercise nothing.
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "felhom-backup", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
|
||||
{TargetID: "felhom-pbs", Cadence: 168 * time.Hour, Service: &fakeBackups{}},
|
||||
},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
// E-2c — ejecting the drive that holds the only local whole-guest backup must be refused.
|
||||
//
|
||||
// This is a REGRESSION GUARD on a live configuration, not a hypothetical. E-1 (2026-07-28) moved the
|
||||
// vzdump target onto each demo box's secondary drive, and `RoleForStorage` types a local-dir on a
|
||||
// non-system device as user-data — so the pre-existing role gate PASSES it and the customer could
|
||||
// self-serve eject the drive holding their backups. It would have succeeded silently.
|
||||
//
|
||||
// The assertion is on the CONSEQUENCE (the request is refused) plus the remedy being named, because a
|
||||
// refusal the customer cannot act on just moves the failure.
|
||||
func TestEjectRefusedOnTheBackupTargetDrive(t *testing.T) {
|
||||
h := backupTargetServer(t)
|
||||
rr := do(t, h, http.MethodPost, "/disks/eject", "A", `{"vmid":8200,"where":"/mnt/nvme-1tb"}`)
|
||||
if rr.Code == http.StatusOK {
|
||||
t.Fatalf("eject of the backup-target drive SUCCEEDED (%d) — the box would silently lose its "+
|
||||
"local drive-loss protection with nothing alarming", rr.Code)
|
||||
}
|
||||
body := rr.Body.String()
|
||||
if !strings.Contains(body, "felhom-backup") {
|
||||
t.Errorf("refusal must NAME the backup target so the customer knows which role blocks it; got: %s", body)
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(body), "reassign") {
|
||||
t.Errorf("refusal must name the REMEDY (reassign the target first), else it is a dead end; got: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// Decommission strands the target just as thoroughly as eject — it migrates data off and retires the
|
||||
// drive. Same gate, asserted separately because it is a different handler and a different caller.
|
||||
func TestDecommissionRefusedOnTheBackupTargetDrive(t *testing.T) {
|
||||
h := backupTargetServer(t)
|
||||
rr := do(t, h, http.MethodPost, "/disks/decommission", "A", `{"vmid":8200,"where":"/mnt/nvme-1tb"}`)
|
||||
if rr.Code == http.StatusOK {
|
||||
t.Fatalf("decommission of the backup-target drive SUCCEEDED (%d)", rr.Code)
|
||||
}
|
||||
if !strings.Contains(rr.Body.String(), "felhom-backup") {
|
||||
t.Errorf("refusal must name the backup target; got: %s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// THE OVER-CORRECTION GUARD, and the reason this is a narrow gate instead of a role reclassification.
|
||||
//
|
||||
// The tempting fix — make RoleForStorage return RoleBackup for the target — would also refuse every
|
||||
// OTHER user-data drive op on a box, and on the demo boxes it would refuse the customer's own data
|
||||
// drive, because that drive IS the target. This pins that a non-target drive stays ejectable: the new
|
||||
// gate must block exactly one drive, not harden the whole eject path.
|
||||
//
|
||||
// It asserts "not blocked BY THIS GATE" rather than "succeeds", because eject has other legitimate
|
||||
// failure modes in a fake harness; what must never appear is this gate's message.
|
||||
func TestEjectStillAllowedOnANonTargetDrive(t *testing.T) {
|
||||
h := backupTargetServer(t)
|
||||
rr := do(t, h, http.MethodPost, "/disks/eject", "A", `{"vmid":8200,"where":"/mnt/spare"}`)
|
||||
if strings.Contains(rr.Body.String(), "whole-guest backup target") {
|
||||
t.Fatalf("the backup-target gate blocked a NON-target drive (/mnt/spare) — over-correction: "+
|
||||
"it must block exactly the target, not harden the whole eject path; got: %s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// E-2 — GET /disks must FLAG the backup-target drive, because the controller cannot work it out.
|
||||
//
|
||||
// The controller's own StoragePath.BackupTarget is customer INTENT, and on a box migrated by hand
|
||||
// (E-1, both demo boxes) nobody ever assigned it — intent is empty while the drive really is the
|
||||
// target. Without this flag the absent-target alarm could not name the drive on exactly the boxes
|
||||
// that have one, which is the only place it currently matters.
|
||||
func TestDisksFlagsTheBackupTargetDrive(t *testing.T) {
|
||||
h := backupTargetServer(t)
|
||||
rr := do(t, h, http.MethodGet, "/disks", "A", "")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("GET /disks = %d, body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
body := rr.Body.String()
|
||||
// The target must be flagged and the non-target must not be — asserted as a pair, since a
|
||||
// blanket true would satisfy a naive "is it flagged?" check.
|
||||
var got struct {
|
||||
Data struct {
|
||||
Disks []struct {
|
||||
Name string `json:"name"`
|
||||
BackupTarget bool `json:"backup_target"`
|
||||
} `json:"disks"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(body), &got); err != nil {
|
||||
t.Fatalf("decode: %v (body %s)", err, body)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, d := range got.Data.Disks {
|
||||
seen[d.Name] = d.BackupTarget
|
||||
}
|
||||
if !seen["felhom-backup"] {
|
||||
t.Errorf("felhom-backup is the primary tier's storage but backup_target is false; body: %s", body)
|
||||
}
|
||||
if seen["spare-drive"] {
|
||||
t.Errorf("spare-drive is NOT the target but was flagged — a blanket true is not a signal; body: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// wrapperRunner captures the wrapper vector without ever running sudo. The wrapper IS the security
|
||||
// boundary, so tests substitute it rather than bypassing it — what is asserted here is the ORDER and
|
||||
// the ARGUMENTS the agent sends, which is the agent's half of the contract.
|
||||
type wrapperRunner struct {
|
||||
calls [][]string
|
||||
failOn string // verb to fail, "" = all succeed
|
||||
}
|
||||
|
||||
func (r *wrapperRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
if len(args) > 0 && args[0] == r.failOn {
|
||||
return nil, []byte("felhom-backup-target-apply: REFUSED: synthetic " + r.failOn + " failure\n"),
|
||||
io.ErrUnexpectedEOF
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r *wrapperRunner) verbs() []string {
|
||||
var out []string
|
||||
for _, c := range r.calls {
|
||||
if len(c) > 1 {
|
||||
out = append(out, c[1])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// moveServer builds a server with /mnt/data mounted on its own device and / on another, plus a
|
||||
// throwaway agent.json the move can rewrite.
|
||||
func moveServer(t *testing.T, run *wrapperRunner) (http.Handler, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
cfgPath := filepath.Join(dir, "agent.json")
|
||||
// An UNKNOWN key is deliberately present: the rewrite must preserve it verbatim.
|
||||
seed := `{"backup":{"local_backup_target":"local","local_backup_retention":3},"some_future_key":{"keep":"me"}}`
|
||||
if err := os.WriteFile(cfgPath, []byte(seed), 0o600); err != nil {
|
||||
t.Fatalf("seed config: %v", err)
|
||||
}
|
||||
hr := fakeHostReader{mounts: []storage.Mount{
|
||||
{Device: "/dev/sda1", MountPoint: "/"},
|
||||
{Device: "/dev/sdb1", MountPoint: "/mnt/data"},
|
||||
}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{}, Storage: fakeStorage{},
|
||||
Tokens: staticTokens{"A": 8200}, HostReader: hr,
|
||||
Privileged: run, ConfigPath: cfgPath, StateDir: dir,
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
return srv.Handler(), cfgPath
|
||||
}
|
||||
|
||||
// THE ORDER IS THE CONTRACT: create → grant → config. Reversed, a config pointing at an ungranted
|
||||
// storage makes every backup 403 on first run, which is precisely what E-1 hit (finding F-3).
|
||||
func TestBackupTargetMoveOrdersCreateThenGrantThenConfig(t *testing.T) {
|
||||
run := &wrapperRunner{}
|
||||
h, cfgPath := moveServer(t, run)
|
||||
|
||||
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("move = %d, body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
got := strings.Join(run.verbs(), ",")
|
||||
if got != "create,grant" {
|
||||
t.Fatalf("wrapper verbs = %q, want create,grant (in that order)", got)
|
||||
}
|
||||
// The config must have been written only AFTER both wrapper calls succeeded.
|
||||
raw, _ := os.ReadFile(cfgPath)
|
||||
var doc map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
t.Fatalf("config unreadable after move: %v", err)
|
||||
}
|
||||
var bk map[string]any
|
||||
_ = json.Unmarshal(doc["backup"], &bk)
|
||||
if bk["local_backup_target"] != "felhom-backup" {
|
||||
t.Errorf("local_backup_target = %v, want felhom-backup", bk["local_backup_target"])
|
||||
}
|
||||
// Unknown keys preserved verbatim — the property that made E-1's hand edit safe.
|
||||
if _, ok := doc["some_future_key"]; !ok {
|
||||
t.Error("the rewrite DROPPED an unknown top-level key — a typed round-trip would do this " +
|
||||
"and silently discard config this build does not know about")
|
||||
}
|
||||
// Sibling keys inside `backup` survive too.
|
||||
if bk["local_backup_retention"] == nil {
|
||||
t.Error("the rewrite dropped local_backup_retention from the backup section")
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED GRANT MUST NOT LEAVE THE CONFIG POINTING AT THE NEW STORAGE. That state is exactly E-1's
|
||||
// 403-on-every-backup: the tier looks configured and cannot write.
|
||||
func TestBackupTargetMoveDoesNotRepointWhenTheGrantFails(t *testing.T) {
|
||||
run := &wrapperRunner{failOn: "grant"}
|
||||
h, cfgPath := moveServer(t, run)
|
||||
|
||||
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
|
||||
if rr.Code == http.StatusOK {
|
||||
t.Fatalf("move SUCCEEDED despite a failed grant (%d)", rr.Code)
|
||||
}
|
||||
if !strings.Contains(rr.Body.String(), "403") {
|
||||
t.Errorf("the error should name the consequence (backups would 403); got %s", rr.Body.String())
|
||||
}
|
||||
raw, _ := os.ReadFile(cfgPath)
|
||||
if strings.Contains(string(raw), "felhom-backup") {
|
||||
t.Fatal("the config was repointed at a storage the agent cannot write to — every backup " +
|
||||
"would 403 while the tier reported as configured")
|
||||
}
|
||||
}
|
||||
|
||||
// It must NOT restart the agent itself. Restarting with a backup in flight cancels the wait and
|
||||
// records a spurious tier failure for a backup that actually succeeded — E-1 did exactly that to a
|
||||
// felhom-pbs run. Only the caller can re-check in-flight work immediately before restarting.
|
||||
func TestBackupTargetMoveReportsRestartRequiredRatherThanRestarting(t *testing.T) {
|
||||
run := &wrapperRunner{}
|
||||
h, _ := moveServer(t, run)
|
||||
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data"}`)
|
||||
if !strings.Contains(rr.Body.String(), `"restart_required":true`) {
|
||||
t.Errorf("response must tell the caller a restart is required; got %s", rr.Body.String())
|
||||
}
|
||||
for _, c := range run.calls {
|
||||
joined := strings.Join(c, " ")
|
||||
if strings.Contains(joined, "systemctl") || strings.Contains(joined, "restart") {
|
||||
t.Fatalf("the handler restarted the agent itself: %q — the caller must do it behind its "+
|
||||
"own in-flight check", joined)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A path that is not a mountpoint is refused BEFORE sudo is reached (F-1): a subdirectory target
|
||||
// reports disconnected forever, and an unmounted path silently retargets onto the system drive.
|
||||
func TestBackupTargetMoveRefusesANonMountpoint(t *testing.T) {
|
||||
run := &wrapperRunner{}
|
||||
h, _ := moveServer(t, run)
|
||||
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/mnt/data/sub"}`)
|
||||
if rr.Code == http.StatusOK {
|
||||
t.Fatal("a non-mountpoint was accepted as the backup target")
|
||||
}
|
||||
if len(run.calls) != 0 {
|
||||
t.Errorf("a refused request still reached the privileged wrapper: %v", run.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// The system disk is refused: a target there protects against corruption only, never drive loss —
|
||||
// which is the entire point of the move.
|
||||
func TestBackupTargetMoveRefusesTheSystemDisk(t *testing.T) {
|
||||
run := &wrapperRunner{}
|
||||
h, _ := moveServer(t, run)
|
||||
rr := do(t, h, http.MethodPost, "/backup/target", "A", `{"vmid":8200,"where":"/"}`)
|
||||
if rr.Code == http.StatusOK {
|
||||
t.Fatal("the system disk was accepted as the backup target")
|
||||
}
|
||||
if len(run.calls) != 0 {
|
||||
t.Errorf("a refused request still reached the privileged wrapper: %v", run.calls)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package localapi
|
||||
|
||||
import "time"
|
||||
|
||||
// normalizeBackupTiers resolves the tier list the Server serves.
|
||||
//
|
||||
// Contract (R-82), and the reason this is a named function rather than inline setup: the UNTARGETED
|
||||
// local-API endpoints must keep behaving exactly as they did before multi-tier existed, forever.
|
||||
// That property lives here.
|
||||
//
|
||||
// - tiers == nil → synthesize ONE tier from the legacy (Backups, BackupCadence) pair and mark it
|
||||
// primary. This is the pre-R-82 shape; every existing caller and test hits this path.
|
||||
// - tiers supplied → keep order but hoist the primary to the front; if none is marked primary,
|
||||
// the FIRST becomes primary (a tier list with no primary would leave untargeted requests with
|
||||
// nothing to act on, which would silently stop backups).
|
||||
// - tiers with a nil Service are dropped: a tier with no runner cannot back anything up, and
|
||||
// advertising it would be an "applied and empty" tier — the exact fault R-82 exists to fix.
|
||||
func normalizeBackupTiers(tiers []BackupTier, legacy BackupService, cadence time.Duration) []BackupTier {
|
||||
usable := make([]BackupTier, 0, len(tiers))
|
||||
for _, t := range tiers {
|
||||
if t.Service == nil || t.TargetID == "" {
|
||||
continue
|
||||
}
|
||||
if t.Cadence <= 0 {
|
||||
t.Cadence = cadence
|
||||
}
|
||||
if t.WaitTimeout <= 0 {
|
||||
t.WaitTimeout = 2 * time.Hour
|
||||
}
|
||||
usable = append(usable, t)
|
||||
}
|
||||
if len(usable) == 0 {
|
||||
if legacy == nil {
|
||||
return nil
|
||||
}
|
||||
return []BackupTier{{TargetID: "", Cadence: cadence, WaitTimeout: 2 * time.Hour, Primary: true, Service: legacy}}
|
||||
}
|
||||
primary := -1
|
||||
for i, t := range usable {
|
||||
if t.Primary {
|
||||
primary = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if primary < 0 {
|
||||
primary = 0
|
||||
}
|
||||
out := make([]BackupTier, 0, len(usable))
|
||||
usable[primary].Primary = true
|
||||
out = append(out, usable[primary])
|
||||
for i, t := range usable {
|
||||
if i == primary {
|
||||
continue
|
||||
}
|
||||
t.Primary = false
|
||||
out = append(out, t)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,671 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-82 Slice A — per-target cadence, due and runner.
|
||||
//
|
||||
// The agent and the controller deploy INDEPENDENTLY. A new agent will serve old controllers for as
|
||||
// long as it takes the fleet to catch up, so the untargeted contract is frozen, not merely
|
||||
// "probably fine". These tests pin that freeze; the multi-tier behaviour is additive on top.
|
||||
|
||||
// tieredServer builds a two-tier server: primary "local" (24h) + "felhom-pbs" (7d), each with its
|
||||
// own runner, exactly as main.go wires it.
|
||||
func tieredServer(t *testing.T, st *fakeStore, localSvc, pbsSvc *fakeBackups) *Server {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{},
|
||||
Backups: localSvc,
|
||||
Store: st,
|
||||
// Both tier targets must be PRESENT in the storage view: since v0.102.0 a tier whose target
|
||||
// storage is absent DEFERS. A real box has both; a fake with no targets would silently
|
||||
// defer every tier and make these assertions vacuous.
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{
|
||||
{Name: "local", Type: "local"},
|
||||
{Name: "felhom-pbs", Type: "pbs"},
|
||||
}},
|
||||
Tokens: staticTokens{"A": 8200, "B": 9300},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: localSvc},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
|
||||
},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
return srv
|
||||
}
|
||||
|
||||
// seen returns the vmids this fake runner was invoked for (mutex-guarded — the backup runs on a
|
||||
// goroutine).
|
||||
func (f *fakeBackups) seen() []int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]int(nil), f.vmids...)
|
||||
}
|
||||
|
||||
// waitFor polls cond for up to 2s. POST /backup is fire-and-forget, so the assertion has to wait
|
||||
// for the goroutine rather than assume it has run.
|
||||
func waitFor(t *testing.T, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if cond() {
|
||||
return
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("condition not met within 2s")
|
||||
}
|
||||
|
||||
func backupAt(target string, vmid int, ago time.Duration, ok bool) hub.Backup {
|
||||
return hub.Backup{
|
||||
TargetID: target,
|
||||
VMID: vmid,
|
||||
Success: ok,
|
||||
StartedAt: testNow.Add(-ago).Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
|
||||
// ── RED-PROOF 1 — old controller ↔ new agent ────────────────────────────────────────────────
|
||||
//
|
||||
// An old controller sends `GET /backup/due` with no query string and parses the pre-R-82 response.
|
||||
// The response must be BYTE-IDENTICAL — not merely semantically similar. A stray `"target":"local"`
|
||||
// key is harmless to a tolerant JSON decoder and fatal to a strict one, and we do not get to choose
|
||||
// which the deployed fleet has.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): drop the `omitempty` from BackupDueResponse.Target and have
|
||||
// tierFromRequest echo the primary's id for an untargeted request → this test fails with the
|
||||
// observed body carrying `"target":"local"`. Restored.
|
||||
func TestBackupDue_Untargeted_ResponseBytesUnchanged(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
rr := do(t, h, "GET", "/backup/due", "A", "")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rr.Code)
|
||||
}
|
||||
var got map[string]any
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("unmarshal: %v (body %s)", err, rr.Body.String())
|
||||
}
|
||||
data, _ := got["data"].(map[string]any)
|
||||
if data == nil {
|
||||
t.Fatalf("no data object in %s", rr.Body.String())
|
||||
}
|
||||
if _, present := data["target"]; present {
|
||||
t.Fatalf("UNTARGETED response MUST NOT carry a target key — an old controller sees a changed contract; body: %s", rr.Body.String())
|
||||
}
|
||||
if data["due"] != false {
|
||||
t.Fatalf("2h-old local backup under a 24h cadence must not be due; body: %s", rr.Body.String())
|
||||
}
|
||||
if data["reason"] != "within cadence window" {
|
||||
t.Fatalf("reason string changed: %v", data["reason"])
|
||||
}
|
||||
}
|
||||
|
||||
// The untargeted verdict must come from the PRIMARY tier's cadence, not from whichever tier
|
||||
// happens to be freshest. With a stale local and a fresh PBS backup, untargeted must say DUE.
|
||||
func TestBackupDue_Untargeted_UsesPrimaryTierOnly(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("local", 8200, 30*time.Hour, true)) // stale for 24h cadence
|
||||
st.RecordBackup(backupAt("felhom-pbs", 8200, 1*time.Hour, true)) // fresh, different tier
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
var resp struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
rr := do(t, h, "GET", "/backup/due", "A", "")
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !resp.Data.Due {
|
||||
t.Fatalf("a fresh backup on ANOTHER tier must not satisfy the primary's cadence; got %+v", resp.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Per-tier due-ness ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// THE POINT OF THE WHOLE SLICE: a fresh daily local backup must NOT satisfy the weekly PBS tier.
|
||||
// Without the per-target filter in latestSuccessfulBackupForTarget the DR tier would never run —
|
||||
// which is exactly today's "applied and empty" state, re-created in code.
|
||||
func TestBackupDue_PerTier_LocalFreshDoesNotSatisfyPBS(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true)) // fresh daily
|
||||
st.RecordBackup(backupAt("felhom-pbs", 8200, 8*24*time.Hour, true)) // 8d — past the 7d weekly
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
var local, pbs struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=local", "A", "").Body.Bytes(), &local); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if local.Data.Due {
|
||||
t.Fatalf("local tier: 2h old under 24h cadence must NOT be due; got %+v", local.Data)
|
||||
}
|
||||
if !pbs.Data.Due {
|
||||
t.Fatalf("PBS tier: 8d old under a 7d cadence MUST be due; got %+v", pbs.Data)
|
||||
}
|
||||
if pbs.Data.Target != "felhom-pbs" || local.Data.Target != "local" {
|
||||
t.Fatalf("a targeted response must echo its tier; got local=%q pbs=%q", local.Data.Target, pbs.Data.Target)
|
||||
}
|
||||
}
|
||||
|
||||
// A 6-day-old PBS snapshot is INSIDE the weekly window — it must not be due. The mirror of the
|
||||
// hub-side threshold test in Slice C, asserted here at the source of truth.
|
||||
func TestBackupDue_PerTier_PBSWithinWeeklyWindow(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("felhom-pbs", 8200, 6*24*time.Hour, true))
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
var pbs struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pbs.Data.Due {
|
||||
t.Fatalf("6d old under a 7d cadence must NOT be due; got %+v", pbs.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// A failed backup must not satisfy any tier's cadence (pre-existing rule, re-asserted per-tier).
|
||||
func TestBackupDue_PerTier_FailedBackupDoesNotCount(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("felhom-pbs", 8200, time.Hour, false))
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
var pbs struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !pbs.Data.Due {
|
||||
t.Fatalf("a FAILED backup must not satisfy the cadence; got %+v", pbs.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// The fail-safe-toward-due rule survives per-tier: an unparseable timestamp yields DUE.
|
||||
// A spurious backup is cheap; a skipped one is not.
|
||||
func TestBackupDue_PerTier_UnparseableTimestampIsDue(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(hub.Backup{TargetID: "felhom-pbs", VMID: 8200, Success: true, StartedAt: "not-a-time"})
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
var pbs struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !pbs.Data.Due {
|
||||
t.Fatalf("unparseable timestamp must fail SAFE toward due; got %+v", pbs.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// An unknown target is a 400 — never a silent fallback to the primary. A controller asking about a
|
||||
// tier this agent does not serve must find out, not be handed a different tier's freshness and act
|
||||
// on it.
|
||||
func TestBackupDue_UnknownTarget_IsAnErrorNotAFallback(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
|
||||
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
|
||||
rr := do(t, h, "GET", "/backup/due?target=does-not-exist", "A", "")
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("unknown target must be 400 (got %d, body %s)", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// ── Tier advertisement ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
// GET /backup/tiers is the controller's capability probe. Primary must be first and flagged, so a
|
||||
// controller can tell which tier the untargeted endpoints act on.
|
||||
func TestBackupTiers_AdvertisesPrimaryFirst(t *testing.T) {
|
||||
h := tieredServer(t, &fakeStore{}, &fakeBackups{}, &fakeBackups{}).Handler()
|
||||
var resp struct {
|
||||
Data BackupTiersResponse `json:"data"`
|
||||
}
|
||||
rr := do(t, h, "GET", "/backup/tiers", "A", "")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d", rr.Code)
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(resp.Data.Tiers) != 2 {
|
||||
t.Fatalf("want 2 tiers, got %+v", resp.Data.Tiers)
|
||||
}
|
||||
if !resp.Data.Tiers[0].Primary || resp.Data.Tiers[0].Target != "local" {
|
||||
t.Fatalf("primary must be first and flagged; got %+v", resp.Data.Tiers)
|
||||
}
|
||||
if resp.Data.Tiers[1].Target != "felhom-pbs" || resp.Data.Tiers[1].CadenceSeconds != int64((7*24*time.Hour).Seconds()) {
|
||||
t.Fatalf("PBS tier mis-advertised: %+v", resp.Data.Tiers[1])
|
||||
}
|
||||
}
|
||||
|
||||
// ── POST /backup routing + per-tier single-flight ────────────────────────────────────────────
|
||||
|
||||
// A targeted POST must run THAT tier's runner. Routing both tiers to one runner would silently
|
||||
// write every "PBS" backup to local — a DR tier that reports success and stores nothing.
|
||||
func TestBackupPost_RoutesToTheTargetsOwnRunner(t *testing.T) {
|
||||
local, pbs := &fakeBackups{}, &fakeBackups{}
|
||||
srv := tieredServer(t, &fakeStore{}, local, pbs)
|
||||
h := srv.Handler()
|
||||
|
||||
if rr := do(t, h, "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
|
||||
|
||||
if got := len(local.seen()); got != 0 {
|
||||
t.Fatalf("the LOCAL runner must not have run for a PBS-targeted request (ran %d times)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Untargeted POST routes to the primary — the old controller's path.
|
||||
func TestBackupPost_UntargetedRoutesToPrimary(t *testing.T) {
|
||||
local, pbs := &fakeBackups{}, &fakeBackups{}
|
||||
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
|
||||
|
||||
if rr := do(t, h, "POST", "/backup", "A", ""); rr.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d", rr.Code)
|
||||
}
|
||||
waitFor(t, func() bool { return len(local.seen()) == 1 })
|
||||
if got := len(pbs.seen()); got != 0 {
|
||||
t.Fatalf("untargeted POST must not touch a non-primary tier (ran %d times)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ONE BACKUP AT A TIME PER GUEST (operator ruling 2026-07-26). A second tier's POST while another
|
||||
// tier is still in flight must be REFUSED — vzdump holds the guest lock, so it could not succeed
|
||||
// anyway, and attempting it records a spurious failure that leaves the tier permanently due.
|
||||
//
|
||||
// Crucially it must NOT be handed the busy tier's job id: that is exactly how a caller comes to
|
||||
// believe its own backup ran.
|
||||
func TestBackupPost_SecondTierRefusedWhileAnotherInFlight(t *testing.T) {
|
||||
localGate := make(chan struct{})
|
||||
local := &fakeBackups{gate: localGate}
|
||||
pbs := &fakeBackups{}
|
||||
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
|
||||
|
||||
var first BackupResponse
|
||||
rr1 := do(t, h, "POST", "/backup", "A", "") // local; blocks on the gate
|
||||
if err := json.Unmarshal(rr1.Body.Bytes(), &struct {
|
||||
Data *BackupResponse `json:"data"`
|
||||
}{Data: &first}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
waitFor(t, func() bool { return len(local.seen()) == 1 })
|
||||
|
||||
rr2 := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
|
||||
if rr2.Code != http.StatusConflict {
|
||||
t.Fatalf("a second tier must be REFUSED while another is in flight; got %d body %s", rr2.Code, rr2.Body.String())
|
||||
}
|
||||
// The STRUCTURAL requirement: the refusal must not return a job the caller could mistake for
|
||||
// its own. It is a 409 with ok=false and NO data object, so nothing is parseable as "my job".
|
||||
// (Naming the busy job in the human-readable message is deliberate and useful for diagnosis —
|
||||
// what must never happen is handing it back as BackupResponse.JobID on a 202.)
|
||||
var envelope struct {
|
||||
OK bool `json:"ok"`
|
||||
Data *BackupResponse `json:"data"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(rr2.Body.Bytes(), &envelope); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if envelope.OK {
|
||||
t.Fatalf("a refusal must not be ok=true; body %s", rr2.Body.String())
|
||||
}
|
||||
if envelope.Data != nil && envelope.Data.JobID != "" {
|
||||
t.Fatalf("the refusal must NOT hand back a job id as the caller's own (got %q); body %s",
|
||||
envelope.Data.JobID, rr2.Body.String())
|
||||
}
|
||||
if !strings.Contains(rr2.Body.String(), "local") {
|
||||
t.Fatalf("the refusal must NAME the busy tier so the caller can diagnose; body %s", rr2.Body.String())
|
||||
}
|
||||
if got := len(pbs.seen()); got != 0 {
|
||||
t.Fatalf("the refused tier must NOT have started a backup (ran %d times)", got)
|
||||
}
|
||||
close(localGate)
|
||||
}
|
||||
|
||||
// Once the busy tier finishes, the other tier may start — and gets its OWN tier-scoped job id.
|
||||
func TestBackupPost_SecondTierAllowedAfterFirstFinishes(t *testing.T) {
|
||||
local, pbs := &fakeBackups{}, &fakeBackups{}
|
||||
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
|
||||
|
||||
var first, second BackupResponse
|
||||
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
|
||||
Data *BackupResponse `json:"data"`
|
||||
}{Data: &first}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
waitFor(t, func() bool { return len(local.seen()) == 1 })
|
||||
// Wait for the local job to leave the in-flight phases.
|
||||
waitFor(t, func() bool {
|
||||
rr := do(t, h, "GET", "/backup/status", "A", "")
|
||||
return strings.Contains(rr.Body.String(), `"phase":"done"`)
|
||||
})
|
||||
|
||||
rr2 := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
|
||||
if rr2.Code != http.StatusAccepted {
|
||||
t.Fatalf("after the first tier finished the second must be allowed; got %d body %s", rr2.Code, rr2.Body.String())
|
||||
}
|
||||
if err := json.Unmarshal(rr2.Body.Bytes(), &struct {
|
||||
Data *BackupResponse `json:"data"`
|
||||
}{Data: &second}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if second.JobID == first.JobID {
|
||||
t.Fatalf("job ids must stay tier-scoped: %q vs %q", first.JobID, second.JobID)
|
||||
}
|
||||
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
|
||||
}
|
||||
|
||||
// `snapshotted` still counts as in flight — the vzdump is uploading and still holds the guest lock.
|
||||
// Checking only `running` (the pre-R-82 code) left a window where a second POST started a real
|
||||
// second vzdump.
|
||||
func TestBackupPost_SnapshottedCountsAsInFlight(t *testing.T) {
|
||||
gate := make(chan struct{})
|
||||
local := &fakeBackups{gate: gate, fireSnapshot: true} // fires onSnapshot, then blocks
|
||||
pbs := &fakeBackups{}
|
||||
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
|
||||
|
||||
do(t, h, "POST", "/backup", "A", "")
|
||||
waitFor(t, func() bool {
|
||||
rr := do(t, h, "GET", "/backup/status", "A", "")
|
||||
return strings.Contains(rr.Body.String(), `"phase":"snapshotted"`)
|
||||
})
|
||||
|
||||
rr := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "")
|
||||
if rr.Code != http.StatusConflict {
|
||||
t.Fatalf("a SNAPSHOTTED backup still holds the guest — a second tier must be refused; got %d body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
close(gate)
|
||||
}
|
||||
|
||||
// Same tier, still single-flight: a second POST to a running tier returns the SAME job.
|
||||
func TestBackupPost_SameTierStillSingleFlight(t *testing.T) {
|
||||
gate := make(chan struct{})
|
||||
local := &fakeBackups{gate: gate}
|
||||
h := tieredServer(t, &fakeStore{}, local, &fakeBackups{}).Handler()
|
||||
|
||||
var a, b BackupResponse
|
||||
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
|
||||
Data *BackupResponse `json:"data"`
|
||||
}{Data: &a}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
|
||||
Data *BackupResponse `json:"data"`
|
||||
}{Data: &b}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a.JobID != b.JobID {
|
||||
t.Fatalf("same tier must single-flight: %q vs %q", a.JobID, b.JobID)
|
||||
}
|
||||
close(gate)
|
||||
}
|
||||
|
||||
// ── normalizeBackupTiers — the compatibility core ────────────────────────────────────────────
|
||||
|
||||
func TestNormalizeBackupTiers(t *testing.T) {
|
||||
svc := &fakeBackups{}
|
||||
|
||||
t.Run("nil tiers synthesize the legacy single tier", func(t *testing.T) {
|
||||
got := normalizeBackupTiers(nil, svc, 24*time.Hour)
|
||||
if len(got) != 1 || !got[0].Primary || got[0].Cadence != 24*time.Hour {
|
||||
t.Fatalf("legacy synthesis broken: %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("primary is hoisted to the front", func(t *testing.T) {
|
||||
got := normalizeBackupTiers([]BackupTier{
|
||||
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: svc},
|
||||
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
|
||||
}, svc, 24*time.Hour)
|
||||
if len(got) != 2 || got[0].TargetID != "local" || !got[0].Primary || got[1].Primary {
|
||||
t.Fatalf("primary not hoisted / uniqueness broken: %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no primary marked → first becomes primary", func(t *testing.T) {
|
||||
got := normalizeBackupTiers([]BackupTier{
|
||||
{TargetID: "a", Cadence: time.Hour, Service: svc},
|
||||
{TargetID: "b", Cadence: time.Hour, Service: svc},
|
||||
}, svc, 24*time.Hour)
|
||||
if len(got) != 2 || got[0].TargetID != "a" || !got[0].Primary {
|
||||
t.Fatalf("want first-as-primary, got %+v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a tier with no runner is DROPPED, not advertised", func(t *testing.T) {
|
||||
got := normalizeBackupTiers([]BackupTier{
|
||||
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
|
||||
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: nil},
|
||||
}, svc, 24*time.Hour)
|
||||
if len(got) != 1 || got[0].TargetID != "local" {
|
||||
t.Fatalf("a serviceless tier must not be advertised (it could never run): %+v", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// R-82 Slice D: a tier whose TARGET STORAGE does not exist yet is DEFERRED, not due.
|
||||
//
|
||||
// A fresh box carries the offsite tier in its installer defaults, but `felhom-pbs` only appears when
|
||||
// the hub provisions the DR tier. Reporting "due" in that window would have the controller quiesce
|
||||
// the apps and fire a vzdump at a non-existent storage every cadence until provisioning happens.
|
||||
func TestBackupDue_TargetStorageMissing_Defers(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
// Storage view knows only "local" — the PBS tier's target is not provisioned yet.
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: st,
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local", Type: "local"}}},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: &fakeBackups{}},
|
||||
},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
h := srv.Handler()
|
||||
|
||||
var pbs, local struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pbs.Data.Due {
|
||||
t.Fatalf("an unprovisioned tier must DEFER, not fire a vzdump at a storage that does not exist; got %+v", pbs.Data)
|
||||
}
|
||||
if !strings.Contains(pbs.Data.Reason, "not present") {
|
||||
t.Fatalf("the deferral must say WHY, or it is indistinguishable from a healthy tier; got %q", pbs.Data.Reason)
|
||||
}
|
||||
// The provisioned tier is unaffected — no evidence yet, so due.
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=local", "A", "").Body.Bytes(), &local); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !local.Data.Due {
|
||||
t.Fatalf("a PROVISIONED tier with no backup yet must still be due; got %+v", local.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// A storage-view ERROR must NOT defer. "I could not check" is not "not there" — reading it that way
|
||||
// would silently suppress backups, the absence-is-not-failure rule this project keeps relearning.
|
||||
func TestBackupDue_StorageViewError_DoesNotSuppress(t *testing.T) {
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: errStorage{}, // reused from f2_role_fallback_test.go — Observe always fails
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: &fakeBackups{}},
|
||||
},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
|
||||
var pbs struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, srv.Handler(), "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !pbs.Data.Due {
|
||||
t.Fatalf("a storage-view error must not suppress the backup (fail toward due); got %+v", pbs.Data)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-84: the cold in-memory store must not cause a redundant backup ─────────────────────────
|
||||
|
||||
// archiveLister is a fakeBackups that ALSO knows when a backup last landed on its storage.
|
||||
type archiveLister struct {
|
||||
*fakeBackups
|
||||
at time.Time
|
||||
found bool
|
||||
err error
|
||||
}
|
||||
|
||||
func (a archiveLister) NewestArchiveTime(context.Context, int) (time.Time, bool, error) {
|
||||
return a.at, a.found, a.err
|
||||
}
|
||||
|
||||
func listerServer(t *testing.T, st *fakeStore, pbsSvc BackupService) http.Handler {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: st,
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: &fakeBackups{}},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
|
||||
},
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = func() time.Time { return testNow }
|
||||
return srv.Handler()
|
||||
}
|
||||
|
||||
func dueFor(t *testing.T, h http.Handler, target string) BackupDueResponse {
|
||||
t.Helper()
|
||||
var out struct {
|
||||
Data BackupDueResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target="+target, "A", "").Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out.Data
|
||||
}
|
||||
|
||||
// THE R-84 CASE. The in-memory store is EMPTY (the agent just restarted), but the storage holds a
|
||||
// snapshot from 2 hours ago. The tier must NOT be due — otherwise every agent deploy costs a fresh
|
||||
// multi-hour offsite upload. Three redundant local backups were observed on demo-felhom in one
|
||||
// afternoon of deploys before this.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the newestArchiveOn fold-in from handleBackupDue (the
|
||||
// pre-R-84 shape, in-memory only) → this fails with
|
||||
// "a restart must NOT make the tier due when the storage holds a 2h-old backup;
|
||||
//
|
||||
// got {... Due:true Reason:no successful backup recorded yet ...}". Restored.
|
||||
func TestBackupDue_ColdStore_UsesStorageGroundTruth(t *testing.T) {
|
||||
h := listerServer(t, &fakeStore{}, archiveLister{
|
||||
fakeBackups: &fakeBackups{}, at: testNow.Add(-2 * time.Hour), found: true,
|
||||
})
|
||||
got := dueFor(t, h, "felhom-pbs")
|
||||
if got.Due {
|
||||
t.Fatalf("a restart must NOT make the tier due when the storage holds a 2h-old backup; got %+v", got)
|
||||
}
|
||||
if got.AgeSecs == nil || *got.AgeSecs != int64((2*time.Hour).Seconds()) {
|
||||
t.Fatalf("the age must come from the storage; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Ground truth that is genuinely OLD still makes the tier due — this must not become a blanket
|
||||
// suppressor.
|
||||
func TestBackupDue_ColdStore_OldArchiveIsStillDue(t *testing.T) {
|
||||
h := listerServer(t, &fakeStore{}, archiveLister{
|
||||
fakeBackups: &fakeBackups{}, at: testNow.Add(-9 * 24 * time.Hour), found: true,
|
||||
})
|
||||
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
|
||||
t.Fatalf("a 9-day-old archive under a 7-day cadence MUST still be due; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A storage that genuinely holds nothing → due. The fix must not invent a backup.
|
||||
func TestBackupDue_ColdStore_NoArchiveIsDue(t *testing.T) {
|
||||
h := listerServer(t, &fakeStore{}, archiveLister{fakeBackups: &fakeBackups{}, found: false})
|
||||
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
|
||||
t.Fatalf("no archive anywhere → due; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A storage-read ERROR must fall back to the in-memory record, NOT be read as "a backup exists".
|
||||
// An unreadable storage must never make a tier look freshly backed up.
|
||||
func TestBackupDue_StorageReadError_DoesNotFakeFreshness(t *testing.T) {
|
||||
h := listerServer(t, &fakeStore{}, archiveLister{
|
||||
fakeBackups: &fakeBackups{}, at: testNow, found: true, err: errStorageRead,
|
||||
})
|
||||
if got := dueFor(t, h, "felhom-pbs"); !got.Due {
|
||||
t.Fatalf("a storage-read error must not fake freshness — the in-memory record is empty, so DUE; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
var errStorageRead = errors.New("simulated storage read failure")
|
||||
|
||||
// The in-memory record WINS when it is newer than the storage listing — a backup that just finished
|
||||
// this process lifetime is more current than a listing that may lag.
|
||||
func TestBackupDue_InMemoryRecordWinsWhenNewer(t *testing.T) {
|
||||
st := &fakeStore{}
|
||||
st.RecordBackup(backupAt("felhom-pbs", 8200, time.Hour, true)) // 1h ago, in memory
|
||||
h := listerServer(t, st, archiveLister{
|
||||
fakeBackups: &fakeBackups{}, at: testNow.Add(-9 * 24 * time.Hour), found: true, // stale listing
|
||||
})
|
||||
got := dueFor(t, h, "felhom-pbs")
|
||||
if got.Due {
|
||||
t.Fatalf("the fresher in-memory record must win over a stale listing; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A service WITHOUT the optional lister degrades to the pre-R-84 behaviour, unchanged.
|
||||
func TestBackupDue_ServiceWithoutLister_UnchangedBehaviour(t *testing.T) {
|
||||
h := listerServer(t, &fakeStore{}, &fakeBackups{}) // plain BackupService
|
||||
if got := dueFor(t, h, "felhom-pbs"); !got.Due || got.Reason != "no successful backup recorded yet" {
|
||||
t.Fatalf("a plain BackupService must behave exactly as before; got %+v", got)
|
||||
}
|
||||
}
|
||||
+270
-10
@@ -8,6 +8,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
@@ -78,8 +79,10 @@ type GuestAttacher interface {
|
||||
DetachDrive(ctx context.Context, where string) error
|
||||
// EnsureSharedParent makes the host stable parent shared + installs the boot-persistence unit.
|
||||
EnsureSharedParent(ctx context.Context) error
|
||||
// GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace (the
|
||||
// guest-usable signal — distinct from the host having the bind). Backs BoundUnderParent.
|
||||
// GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace — the
|
||||
// guest-VISIBILITY signal, distinct from the host having the bind. One of the three terms behind
|
||||
// BoundUnderParent. It is a path-presence test and NOT a liveness signal (R-117): a stale bind over a
|
||||
// dead device is still "seen". Liveness is bindLiveness's job.
|
||||
GuestSeesMount(ctx context.Context, vmid int, path string) bool
|
||||
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
|
||||
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
|
||||
@@ -142,15 +145,49 @@ type DiskInfo struct {
|
||||
// HDD look available when it wasn't bound. Only meaningful for user-data drives. LEGACY (per-drive
|
||||
// `pct set -mpN` model) — the intermediary model uses BoundUnderParent.
|
||||
GuestAttached bool `json:"guest_attached"`
|
||||
// BackupTarget (E-2) reports that this drive backs the PRIMARY whole-guest backup tier. Additive:
|
||||
// an older controller ignores it. It is the agent's answer, not the controller's intent flag —
|
||||
// on a hand-migrated box (E-1) intent is unset while the drive really is the target.
|
||||
BackupTarget bool `json:"backup_target,omitempty"`
|
||||
// GuestPath is the drive's STABLE in-guest path in the intermediary-mount model
|
||||
// (/mnt/felhom-drives/<name>). This is what the controller repoints HDD_PATH to and registers as the
|
||||
// storage path. Set for /mnt/<name> drives; "" otherwise. Distinct from MountPath (the RAW host PVE
|
||||
// mount the agent ops on).
|
||||
GuestPath string `json:"guest_path,omitempty"`
|
||||
// BoundUnderParent reports whether the drive's felhom-data is currently bound under the shared parent
|
||||
// at GuestPath (a host mount-table check) — i.e. live + usable in the guest in the intermediary model.
|
||||
// BoundUnderParent reports whether the drive is live + usable in the guest in the intermediary model.
|
||||
// The controller's drive-absent gate + auto-restart key on this (and State).
|
||||
//
|
||||
// It is a CONJUNCTION of THREE facts, and all three are load-bearing:
|
||||
// 1. felhom-data is bound under the shared parent at GuestPath (the guest-visible mount check), and
|
||||
// 2. the drive's RAW host mount is still mounted — i.e. the DEVICE is still there (R-113), and
|
||||
// 3. the bind actually WORKS: it names the same device as the raw mount, and that filesystem has
|
||||
// not aborted (R-117, v0.117.0 — see bindLiveness).
|
||||
// Half 1 alone was the bug R-113 fixed: the raw mount is device-bound and dies with its device, but
|
||||
// the agent's own bind is not, so half 1 stays true over a stale shell after the device is pulled. The
|
||||
// controller read that survivor as "present" and the drive-absent alarm could never fire — measured
|
||||
// live in E-2d (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Half 2 alone would regress boot
|
||||
// ordering, where the raw mounts early and the bind lands ~18s later; the conjunction keeps that
|
||||
// window reading absent.
|
||||
//
|
||||
// Terms 1 and 2 TOGETHER were still not liveness, which is R-117: both are path-presence tests
|
||||
// comparing only field 5 of a mountinfo line, so both stay true over a bind that names the drive that
|
||||
// went away while the raw mount healed onto the returning one via its fs-UUID-keyed unit. Measured
|
||||
// live: raw on 8:32 /dev/sdc, bind on 8:16 /dev/sdb with `shutdown`, this field TRUE, EIO on every
|
||||
// read and write, and the gate restarting the customer's apps onto it with no alarm on any channel
|
||||
// (felhom.eu audits/SPIKE-r117-bind-liveness-2026-07-30.md §5.2).
|
||||
//
|
||||
// THE TESTS THAT PIN THIS COMMENT, because for three releases it promised a property nothing tested
|
||||
// (spike §5.3): disks_bind_liveness_test.go — TestDisks_BindLiveness_StaleBindReadsAbsent (term 3,
|
||||
// case a), _AbortedFilesystemReadsAbsent (term 3, case b, the steady-state case that emits nothing
|
||||
// today), _UnknownIsTreatedAsPresent (the cannot-tell rule) and _HealthyReadsPresent (no false
|
||||
// negative). Each asserts the CONSEQUENCE — what this field reads — not the mechanism.
|
||||
BoundUnderParent bool `json:"bound_under_parent"`
|
||||
// Smart is the already-computed per-disk SMART health summary (v0.94.0), serialized here so the
|
||||
// controller can render a disk-health card + degradation alert WITHOUT any new smartctl load — the
|
||||
// value is copied straight from the target's Observe-time enrichment. omitempty + a pointer so a
|
||||
// device that exposes no SMART (USB bridge, unread) is ABSENT, not a misleading zero-value UNKNOWN;
|
||||
// the controller feature-detects presence and renders "Nincs adat" when nil (never alarms).
|
||||
Smart *hub.SmartSummary `json:"smart,omitempty"`
|
||||
}
|
||||
|
||||
// handleDisks lists the host's drives + data-bearing flags (read-only/benign).
|
||||
@@ -170,6 +207,9 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// F9: which host mount paths are actually BOUND into THIS guest's config (guest-usable, not just
|
||||
// host-present). A bind's mp= equals the guest path, which is the drive's host mount path (`where`).
|
||||
boundPaths := s.guestBoundPaths(r.Context(), vmid)
|
||||
// E-2: the PRIMARY tier's storage id — the whole-guest vzdump destination. "" when no tier
|
||||
// carries a target (the legacy single-tier shape), which correctly flags nothing.
|
||||
primaryTargetID := s.primaryTier().TargetID
|
||||
out := make([]DiskInfo, 0, len(targets))
|
||||
for _, t := range targets {
|
||||
di := DiskInfo{
|
||||
@@ -181,13 +221,79 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
UsedBytes: t.UsedBytes,
|
||||
UsedFraction: t.UsedFraction,
|
||||
GuestAttached: t.MountPath != "" && boundPaths[t.MountPath],
|
||||
// E-2: is THIS drive the whole-guest backup target? The agent is the only component that
|
||||
// can answer — the controller's own StoragePath.BackupTarget is customer INTENT, and on a
|
||||
// box migrated by hand (E-1) nobody ever assigned it, so intent is empty while the drive
|
||||
// really is the target. Reported here so the controller can name the drive in an
|
||||
// absent-target alarm and hide the destructive controls the agent would refuse anyway.
|
||||
BackupTarget: t.Name == primaryTargetID,
|
||||
}
|
||||
// Intermediary model: the stable in-guest path + whether felhom-data is bound under the parent.
|
||||
// Only user-data /mnt/<name> drives have a guest path (system/backup mounts never cross in).
|
||||
if di.Role == string(storage.RoleUserData) {
|
||||
if gp := StablePathForRaw(t.MountPath); gp != "" {
|
||||
di.GuestPath = gp
|
||||
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp)
|
||||
// R-113: AND in device presence. A conjunction, deliberately — it leaves the
|
||||
// boot-ordering behaviour the controller's gate depends on exactly as it was
|
||||
// (raw mounted early, bind not yet ⇒ still absent) while closing the case the
|
||||
// gate could never see (bind outlived the device ⇒ now absent).
|
||||
// R-117: AND in bind LIVENESS. The two terms above are both path-presence tests, so
|
||||
// both stay true over a bind that names the drive that went away while the raw mount
|
||||
// healed onto the returning one — EIO on every call, payload healthy.
|
||||
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
|
||||
s.devicePresent(t.MountPath) &&
|
||||
s.bindUsable(gp, t.MountPath)
|
||||
}
|
||||
}
|
||||
// R-116: carry the GUEST PATH on the backup-target row even when its role has flipped to
|
||||
// system — but ONLY when that flip was caused by the device vanishing.
|
||||
//
|
||||
// WHY. The controller keys the drive-absent alarm on the registered StoragePath, which for an
|
||||
// external drive is the GUEST path. When the device goes, Observe's exactMountDevice fails, so
|
||||
// t.BackingDevice becomes "" and RoleForStorage returns RoleSystem (role.go:180-181) — the block
|
||||
// above is skipped and this row loses its guest path. It keeps its MountPath, so the union loop
|
||||
// below DEDUPES the registry row away (`seen[d.MountPath]`), and /disks ends up carrying NO row
|
||||
// with that guest path at all. driveTargetByPath then has no entry, isTarget[guestPath] is a
|
||||
// missing key, and the specific backup_target_absent alarm cannot fire — the generic one goes
|
||||
// out instead, while the RETURN (rows rejoined) fires the specific recovery. An unmatchable
|
||||
// pair. Measured live: felhom.eu audits/SESSION-C-2026-07-29.md §5.
|
||||
//
|
||||
// THE GATES, each load-bearing:
|
||||
// di.GuestPath == "" — never touch the user-data path above; this is a fallback, not a rule.
|
||||
// di.BackupTarget — only the target row. No other system/backup mount gains a guest path,
|
||||
// so the boundary at :213-214 stands: this is not "system mounts now
|
||||
// cross into the guest", it is "the drive the alarm is about keeps its
|
||||
// identity while it is missing".
|
||||
// t.BackingDevice == "" — ONLY the vanished-device flip. A storage that is RoleSystem because
|
||||
// it is genuinely system-BACKED has a non-empty BackingDevice and is
|
||||
// excluded. Without this gate a dir storage at /mnt/<name> living on the
|
||||
// root disk would acquire a guest path.
|
||||
//
|
||||
// Case B (the COMMON fresh-box shape) is safe twice over: the target is the builtin `local` on
|
||||
// /var/lib/vz, and StablePathForRaw returns "" for anything that is not exactly /mnt/<name>
|
||||
// (DriveNameFromRaw, intermediary.go:79-88), so nothing is set even before the gates apply.
|
||||
//
|
||||
// This cannot make the gate read an absent drive as PRESENT: BoundUnderParent is assigned only
|
||||
// inside the two guest-path blocks a system-role row never enters, so it stays false, and
|
||||
// planDriveGates computes present[gp] = present[gp] || d.BoundUnderParent. Inert by construction
|
||||
// — pinned by TestAbsentTargetRowDoesNotRegisterPresence.
|
||||
//
|
||||
// v0.116.0 — WHY v0.115.0 (the MountPath-only form) WAS INERT, measured not reasoned. In the
|
||||
// absent state t.MountPath is ALSO "" — the same exactMount failure that emptied BackingDevice
|
||||
// empties it — so StablePathForRaw("") returned "" and this assigned nothing. Captured payload:
|
||||
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
|
||||
//
|
||||
// t.ConfigPath is the fix: the storage's CONFIGURED path from storage.cfg, which is configuration
|
||||
// and therefore survives the device. MountPath is tried FIRST so the present-state path and
|
||||
// v0.115.0's tested behaviour are byte-identical; ConfigPath is consulted only when the mount is
|
||||
// genuinely gone. MountPath is deliberately NOT back-filled from ConfigPath — see the union-dedup
|
||||
// note below for the consumer that would break, and because a path that is not mounted is not a
|
||||
// "host mountpoint" (this field's own contract, :152-153).
|
||||
if di.GuestPath == "" && di.BackupTarget && t.BackingDevice == "" {
|
||||
if gp := StablePathForRaw(t.MountPath); gp != "" {
|
||||
di.GuestPath = gp
|
||||
} else {
|
||||
di.GuestPath = StablePathForRaw(t.ConfigPath)
|
||||
}
|
||||
}
|
||||
// Inspect the backing device for the UI's data-bearing hint (the authoritative check
|
||||
@@ -206,6 +312,13 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
di.WipeDurableID = wid
|
||||
}
|
||||
}
|
||||
// v0.94.0: surface the already-computed SMART only when it was actually read (Health set).
|
||||
// A zero-value summary (enrich skipped / no smartctl device) has Health "" → stays omitted, so
|
||||
// the controller sees "absent" and renders "Nincs adat" rather than a false UNKNOWN.
|
||||
if t.Smart.Health != "" {
|
||||
sm := t.Smart
|
||||
di.Smart = &sm
|
||||
}
|
||||
out = append(out, di)
|
||||
}
|
||||
// Impl-2a: union in registry+units drives that Observe() does NOT surface (a drive with no PVE
|
||||
@@ -213,16 +326,43 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// NOT duplicated, and no Observe row is dropped (so this can never regress the current view).
|
||||
if s.driveTargets != nil {
|
||||
seen := make(map[string]bool, len(out))
|
||||
// R-116: dedup ALSO by guest path. `seen` keys on MountPath, the one field the absent state
|
||||
// empties, so with the device gone /mnt/<name> is absent from `seen` and the registry row was NOT
|
||||
// skipped — /disks carried the drive TWICE, the Observe row holding BackupTarget with no key and
|
||||
// the registry row holding both keys with BackupTarget defaulted false. driveTargetByPath
|
||||
// (controller intermediary.go:602-618) assigns rather than ORs, and the registry row is appended
|
||||
// LAST, so its false won on both keys. Measured, 4 rows vs 3:
|
||||
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
|
||||
//
|
||||
// THE JOIN, and it is the whole point: with the device gone the two records of one drive share NO
|
||||
// runtime field — no mount, no backing device, and the Observe row's DurableID has degraded off the
|
||||
// fs-UUID. What they DO share is CONFIGURATION: the Observe row's storage path (storage.cfg) and the
|
||||
// registry row's unit `Where` (the .mount unit) are the same path, so both derive the same stable
|
||||
// guest path. That is the key both sides can still compute, which is why the dedup keys on it.
|
||||
seenGuest := make(map[string]bool, len(out))
|
||||
for _, d := range out {
|
||||
if d.MountPath != "" {
|
||||
seen[d.MountPath] = true
|
||||
}
|
||||
if d.GuestPath != "" {
|
||||
seenGuest[d.GuestPath] = true
|
||||
}
|
||||
}
|
||||
if drives, derr := s.driveTargets.Known(r.Context()); derr == nil {
|
||||
for _, d := range drives {
|
||||
if d.MountPath == "" || seen[d.MountPath] {
|
||||
continue
|
||||
}
|
||||
// Same drive as an Observe row that already carries this guest path — skip it. Suppressing
|
||||
// it rather than teaching it BackupTarget is deliberate: the registry row has a non-empty
|
||||
// MountPath (from the unit file, stale by then), and the controller reads
|
||||
// `d.BackupTarget && d.MountPath != ""` as "a real drive with its own mountpoint — HEALTHY"
|
||||
// (backup_target_offer.go:79). Putting the flag on a row with a stale MountPath would have
|
||||
// silently regressed R-114, telling the customer the backup target is fine while its drive
|
||||
// is missing. Pinned by TestAbsentTargetKeepsR114DegradedSignal.
|
||||
if gp := StablePathForRaw(d.MountPath); gp != "" && seenGuest[gp] {
|
||||
continue
|
||||
}
|
||||
di := DiskInfo{
|
||||
Name: d.Name, Type: d.Type, State: "attached",
|
||||
MountPath: d.MountPath, DurableID: d.DurableID,
|
||||
@@ -233,7 +373,13 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
// exactly like the Observe path — else the controller reads a registry drive as "Leválasztva".
|
||||
if gp := StablePathForRaw(d.MountPath); gp != "" {
|
||||
di.GuestPath = gp
|
||||
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp)
|
||||
// R-113 + R-117: same three-term conjunction as the Observe path. This path matters
|
||||
// MORE, not less — a registry drive with no PVE dir-storage is exactly the shape
|
||||
// E-2d detached, and its State is hardcoded "attached" below, so these checks are
|
||||
// the only device truth this row carries.
|
||||
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
|
||||
s.devicePresent(d.MountPath) &&
|
||||
s.bindUsable(gp, d.MountPath)
|
||||
}
|
||||
// A registry drive has no PVE `pvesm status` snapshot, so fill backing device + capacity
|
||||
// from the host directly: resolve the device by fs-UUID, and statfs the mount for size —
|
||||
@@ -241,10 +387,17 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if d.UUID != "" {
|
||||
// Resolve to the real /dev node (e.g. /dev/sdd), not the by-uuid symlink path, to match
|
||||
// how Observe-sourced rows display the backing device.
|
||||
if dev, err := storage.ResolveStorageDevice("uuid:" + d.UUID); err == nil {
|
||||
if dev, err := s.resolveStorageDevice("uuid:" + d.UUID); err == nil {
|
||||
di.BackingDevice = dev
|
||||
}
|
||||
}
|
||||
// Fix B (v0.95.0): union-path drives skip Observe's enrich, so read SMART here through the
|
||||
// same seam the dir targets use. Only set when the read actually ran (Health != "").
|
||||
if di.BackingDevice != "" && s.smart != nil {
|
||||
if sm := s.smart.SMARTForBacking(r.Context(), di.BackingDevice); sm.Health != "" {
|
||||
di.Smart = &sm
|
||||
}
|
||||
}
|
||||
if total, used, okc := statfsCapacity(d.MountPath); okc {
|
||||
di.TotalBytes, di.UsedBytes = total, used
|
||||
di.UsedFraction = float64(used) / float64(total)
|
||||
@@ -357,6 +510,11 @@ func (s *Server) handleDiskEject(w http.ResponseWriter, r *http.Request, vmid in
|
||||
writeErr(w, http.StatusForbidden, "mount is system/backup-protected — eject refused (role: "+string(role)+")")
|
||||
return
|
||||
}
|
||||
// E-2c: the role gate above passes a drive that is BOTH user-data and the vzdump target (E-1 put
|
||||
// the target on the enrolled drive's own mountpoint). Refuse specifically, naming the remedy.
|
||||
if s.refuseIfBackupTarget(r.Context(), w, "eject", vmid, req.Where) {
|
||||
return
|
||||
}
|
||||
dependents := s.dependentGuests(r.Context(), req.Where)
|
||||
// Record the EJECT intent BEFORE unmounting (the target still resolves to its durable-id) so the
|
||||
// self-heal watchdog leaves it alone — an OFFICIAL eject is the only thing that sets this (P3); an
|
||||
@@ -410,6 +568,11 @@ func (s *Server) handleDiskDecommission(w http.ResponseWriter, r *http.Request,
|
||||
writeErr(w, http.StatusForbidden, "mount is system/backup-protected — decommission refused (role: "+string(role)+")")
|
||||
return
|
||||
}
|
||||
// E-2c: same narrow gate as eject — decommission migrates data off and retires the drive, which
|
||||
// would strand the backup target just as thoroughly.
|
||||
if s.refuseIfBackupTarget(r.Context(), w, "decommission", vmid, req.Where) {
|
||||
return
|
||||
}
|
||||
dependents := s.dependentGuests(r.Context(), req.Where)
|
||||
// Resolve the durable-id BEFORE unmounting (it still resolves while mounted) for the bind prune.
|
||||
id := s.durableIDForMount(r.Context(), req.Where)
|
||||
@@ -803,9 +966,12 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
|
||||
}
|
||||
|
||||
// boundUnderParent reports whether a drive's felhom-data is bound at its stable guest path AND visible
|
||||
// inside the guest (the usable-in-guest signal the controller's gate keys on — a guest reboot leaves the
|
||||
// host bind in place but invisible to the guest until re-propagated). Injectable via s.boundCheck for
|
||||
// tests; defaults to the guest-namespace mount check.
|
||||
// inside the guest — a guest reboot leaves the host bind in place but invisible to the guest until
|
||||
// re-propagated. Injectable via s.boundCheck for tests; defaults to the guest-namespace mount check.
|
||||
//
|
||||
// This is VISIBILITY, not liveness (R-117). It compares only the mount point, so it stays true over a
|
||||
// bind whose device has gone; do not read it as "usable in the guest" — that is the whole three-term
|
||||
// conjunction at the two /disks construction sites, whose third term is bindUsable.
|
||||
func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath string) bool {
|
||||
if s.boundCheck != nil {
|
||||
return s.boundCheck(stablePath)
|
||||
@@ -816,6 +982,44 @@ func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath stri
|
||||
return s.guestAttach.GuestSeesMount(ctx, vmid, stablePath)
|
||||
}
|
||||
|
||||
// devicePresent reports whether the drive's BACKING DEVICE is still there, by asking whether its RAW
|
||||
// host mount is still a mountpoint (R-113).
|
||||
//
|
||||
// WHY THE RAW MOUNT AND NOT THE BIND. The raw mount at /mnt/<name> is a systemd mount unit bound to
|
||||
// its device: when the device goes, the unit stops and the mountpoint disappears. The agent's own bind
|
||||
// of <raw>/felhom-data under the shared parent is an ordinary bind — nothing ties it to the device, so
|
||||
// its mountinfo entry OUTLIVES the device as a stale shell. Measured live in E-2d with the device
|
||||
// pulled: `/mnt/mentes2` NOT mounted while `/mnt/felhom-drives/mentes2` still read
|
||||
// `/dev/sdb[/felhom-data]` (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Keying presence on the
|
||||
// survivor is exactly why the controller's drive-absent gate could never fire.
|
||||
//
|
||||
// An empty raw path means we have nothing to ask about — return TRUE (unknown), never false. Absent
|
||||
// stops a customer's apps, so "cannot tell" must never be reported as "gone".
|
||||
func (s *Server) devicePresent(rawMountPath string) bool {
|
||||
if rawMountPath == "" {
|
||||
return true // cannot tell → never claim absent
|
||||
}
|
||||
if s.deviceCheck != nil {
|
||||
return s.deviceCheck(rawMountPath)
|
||||
}
|
||||
return isHostMountpoint(rawMountPath)
|
||||
}
|
||||
|
||||
// bindUsable is the THIRD term of the BoundUnderParent conjunction (R-117): the bind must not only exist
|
||||
// and be guest-visible, it must actually WORK. The first two terms are path-presence tests and are both
|
||||
// satisfied by a bind that names the drive that went away — measured live, with EIO on every read and
|
||||
// write while the payload read healthy and the gate restarted the customer's apps onto it.
|
||||
//
|
||||
// UNKNOWN counts as usable, via BindLiveness.Usable — the same "cannot tell → never absent" rule
|
||||
// devicePresent applies above, and for the same reason: a false absent stops a working customer's apps.
|
||||
// Injectable via s.livenessCheck; the default reads /proc only and issues NO block I/O (CLAUDE.md).
|
||||
func (s *Server) bindUsable(stable, rawMountPath string) bool {
|
||||
if s.livenessCheck != nil {
|
||||
return s.livenessCheck(stable, rawMountPath).Usable()
|
||||
}
|
||||
return bindLiveness(stable, rawMountPath).Usable()
|
||||
}
|
||||
|
||||
// guestBoundPaths returns the set of guest mountpoint paths (the `mp=` of each entry in the guest's
|
||||
// config) — i.e. the host drives actually BOUND into the guest. F9: this is the guest-attached signal
|
||||
// (`GuestAttached`) that distinguishes a guest-usable drive from one merely present on the host. A bind
|
||||
@@ -1020,6 +1224,62 @@ func (s *Server) hostReader() storage.HostReader {
|
||||
return storage.NewProcHostReader()
|
||||
}
|
||||
|
||||
// backupTargetAt reports the configured backup TIER whose storage is mounted at `where`, or "" when
|
||||
// none is. E-2c.
|
||||
//
|
||||
// WHY THIS IS NOT A ROLE RECLASSIFICATION. The obvious fix is to make RoleForStorage return
|
||||
// RoleBackup for the target's storage, and it is wrong here: on both demo boxes the drive that now
|
||||
// holds the whole-guest archives is ALSO the enrolled user-data drive (E-1 put the vzdump target on
|
||||
// the drive's own mountpoint, beside felhom-data). Reclassifying it would refuse every legitimate
|
||||
// eject/decommission of the customer's own data drive — an over-correction that trades one silent
|
||||
// failure for a permanent obstruction. So this is a SEPARATE, narrower gate that names exactly what
|
||||
// it protects and leaves the role vocabulary alone.
|
||||
//
|
||||
// It resolves through the agent's OWN storage view (never the caller's claim) and fails OPEN — an
|
||||
// unreadable view returns "" so this gate cannot block on a transient error. That is safe because it
|
||||
// sits BEHIND the role gate, which already fails SAFE on the same error: an unresolvable mount is
|
||||
// refused there before it ever reaches this check.
|
||||
func (s *Server) backupTargetAt(ctx context.Context, where string) string {
|
||||
if where == "" || s.storage == nil {
|
||||
return ""
|
||||
}
|
||||
targets, err := s.storage.Observe(ctx)
|
||||
if err != nil {
|
||||
return "" // fail OPEN — the role gate already fails SAFE on this same error
|
||||
}
|
||||
for _, t := range s.tiers {
|
||||
if t.TargetID == "" {
|
||||
continue
|
||||
}
|
||||
for _, tgt := range targets {
|
||||
if tgt.Name == t.TargetID && tgt.MountPath == where {
|
||||
return t.TargetID
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// refuseIfBackupTarget refuses a destructive drive op when `where` backs a configured backup tier,
|
||||
// and reports whether it did. The message names the storage AND the remedy: the operation is not
|
||||
// forbidden forever, it is ordered — reassign the backup target first, then the drive is free.
|
||||
//
|
||||
// Ejecting the drive that holds the only local whole-guest backup is exactly the silent-degradation
|
||||
// class this arc has been closing: it succeeds, nothing alarms, and the box quietly loses its
|
||||
// drive-loss protection while still reporting a configured tier.
|
||||
func (s *Server) refuseIfBackupTarget(ctx context.Context, w http.ResponseWriter, op string, vmid int, where string) bool {
|
||||
target := s.backupTargetAt(ctx, where)
|
||||
if target == "" {
|
||||
return false
|
||||
}
|
||||
s.logger.Warn("local-api: protected — "+op+" refused: the mount backs a configured backup tier",
|
||||
"vmid", vmid, "where", where, "target", target)
|
||||
writeErr(w, http.StatusConflict,
|
||||
"this drive is the whole-guest backup target ("+target+") — "+op+" refused. "+
|
||||
"Reassign the backup target to another drive first, or the box loses its local drive-loss protection.")
|
||||
return true
|
||||
}
|
||||
|
||||
// roleForMountPath resolves the AUTHORITATIVE protection role of the storage mounted at `where`, from
|
||||
// the agent's OWN storage view + host topology (never the caller's claim). It mirrors deviceRole but
|
||||
// keys on the mount path (the eject input). It FAILS SAFE to system (most-protected) on any
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// R-116 — the backup-target flag must be reachable from the row the CONTROLLER keys on.
|
||||
//
|
||||
// THE DEFECT. The controller resolves the drive-absent alarm by the registered StoragePath, which for
|
||||
// an external drive is the GUEST path. When the device vanishes, Observe's exactMountDevice fails, so
|
||||
// BackingDevice becomes "" and RoleForStorage returns RoleSystem (role.go:180-181) — the guest-path
|
||||
// block is skipped and the flag-bearing row loses its guest path. It keeps its MountPath, so the union
|
||||
// loop DEDUPES the registry row away, and /disks carries NO row with that guest path at all.
|
||||
// driveTargetByPath then has no entry, isTarget[guestPath] is a MISSING KEY, and the specific
|
||||
// backup_target_absent alarm cannot fire — the generic storage_disconnected goes out instead, while
|
||||
// the RETURN (rows rejoined) fires the specific recovery. An operator gets a pair they cannot match.
|
||||
// Measured live: felhom.eu audits/SESSION-C-2026-07-29.md §5.
|
||||
//
|
||||
// These tests exercise the REAL GET /disks response and assert the emitted JSON, because the failure
|
||||
// class is "the value is on the wrong row" — a test that hand-builds rows proves nothing about which
|
||||
// row the handler actually emits.
|
||||
|
||||
// targetRowServer builds a /disks server whose primary backup tier is `primaryTarget`, over the given
|
||||
// Observe targets. boundCheck/deviceCheck are pinned so the R-113 conjunction is not the variable
|
||||
// under test here.
|
||||
func targetRowServer(t *testing.T, primaryTarget string, targets []hub.StorageTarget) *Server {
|
||||
t.Helper()
|
||||
return targetRowServerWithDrives(t, primaryTarget, targets, nil)
|
||||
}
|
||||
|
||||
// targetRowServerWithDrives additionally wires the REGISTRY union source. v0.115.0's tests left
|
||||
// DriveTargets nil, so the union loop never ran and the two-row absent shape — the actual defect — was
|
||||
// invisible to the whole suite. Any test about which row carries what MUST populate this.
|
||||
func targetRowServerWithDrives(t *testing.T, primaryTarget string, targets []hub.StorageTarget,
|
||||
drives []storage.KnownTarget) *Server {
|
||||
t.Helper()
|
||||
var known storage.KnownTargets
|
||||
if drives != nil {
|
||||
known = fakeKnownTargets{drives: drives}
|
||||
}
|
||||
srv, err := NewServer(Options{
|
||||
DriveTargets: known,
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuestsCfg{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: targets},
|
||||
// Service is REQUIRED: normalizeBackupTiers (backup_tiers.go:21-22) drops any tier with a nil
|
||||
// Service, and the legacy fallback then yields TargetID "" — which silently makes every
|
||||
// BackupTarget false and would make these tests pass for the wrong reason.
|
||||
BackupTiers: []BackupTier{{TargetID: primaryTarget, Primary: true, Service: &fakeBackups{}}},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
|
||||
DiskGate: &fakeGate{}, HostReader: sysOnSDA(),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.boundCheck = func(string) bool { return true }
|
||||
srv.deviceCheck = func(string) bool { return true }
|
||||
return srv
|
||||
}
|
||||
|
||||
// wireDisks returns the decoded /disks rows exactly as the controller receives them.
|
||||
func wireDisks(t *testing.T, srv *Server) []map[string]any {
|
||||
t.Helper()
|
||||
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
|
||||
var w struct {
|
||||
Data struct {
|
||||
Disks []map[string]any `json:"disks"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &w); err != nil {
|
||||
t.Fatalf("decode /disks: %v (%s)", err, body)
|
||||
}
|
||||
return w.Data.Disks
|
||||
}
|
||||
|
||||
// isTargetByPath reproduces the controller's driveTargetByPath EXACTLY (intermediary.go:602-616):
|
||||
// both keyings, value = backup_target. This is the map whose missing key is the whole defect, so the
|
||||
// assertion is made against a faithful copy of it rather than against a field in isolation.
|
||||
func isTargetByPath(disks []map[string]any) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, d := range disks {
|
||||
bt, _ := d["backup_target"].(bool)
|
||||
if gp, ok := d["guest_path"].(string); ok && gp != "" {
|
||||
out[gp] = bt
|
||||
}
|
||||
if mp, ok := d["mount_path"].(string); ok && mp != "" {
|
||||
out[mp] = bt
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// theAbsentTarget is the absent-target Observe row, CORRECTED in v0.116.0 to the shape the live box
|
||||
// actually produces.
|
||||
//
|
||||
// THIS FIXTURE IS WHY AN INERT FIX SHIPPED GREEN. As written for v0.115.0 it supplied
|
||||
// `MountPath: "/mnt/mentes"` — a field the real absent state does NOT have. The same exactMount failure
|
||||
// that empties BackingDevice empties MountPath (observe.go:184-190), so on the live box this row carries
|
||||
// `mount_path: ""`, and v0.115.0's `StablePathForRaw(t.MountPath)` was therefore
|
||||
// `StablePathForRaw("")` == "". The fixture handed the code a value production never supplies, the test
|
||||
// went green, and the fix was inert on real hardware — twice.
|
||||
//
|
||||
// Captured payload this now mirrors, field for field:
|
||||
// felhom.eu audits/DIAG-r116-disks-payload-2026-07-30.md §6.2.
|
||||
var theAbsentTarget = hub.StorageTarget{
|
||||
Name: "felhom-backup", Type: hub.StorageTypeLocalDir,
|
||||
MountPath: "", BackingDevice: "", ConfigPath: "/mnt/mentes",
|
||||
State: hub.StorageStateDisconnected,
|
||||
// DurableID degrades off the fs-UUID exactly as the live payload showed (`path:/mnt/cel` there).
|
||||
DurableID: "path:/mnt/mentes",
|
||||
}
|
||||
|
||||
// theAbsentRegistryRow is the OTHER half of the live absent payload — the registry/union row. Its
|
||||
// MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never consults the
|
||||
// mount table, so it survives the device intact. Its presence is what made /disks carry the drive TWICE.
|
||||
var theAbsentRegistryRow = []storage.KnownTarget{
|
||||
{Name: "9303-uuid", Type: hub.StorageTypeUSB, MountPath: "/mnt/mentes",
|
||||
DurableID: "uuid:9303", UUID: "9303"},
|
||||
}
|
||||
|
||||
// ── the observable that must move ───────────────────────────────────────────────────────────────
|
||||
|
||||
// RED-PROOF: delete the `di.GuestPath == "" && di.BackupTarget && t.BackingDevice == ""` block and
|
||||
// this fails with "the guest path the controller keys on is MISSING from /disks entirely".
|
||||
func TestAbsentBackupTargetIsResolvableByGuestPath(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
|
||||
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
|
||||
isTarget := isTargetByPath(disks)
|
||||
|
||||
const guestPath = "/mnt/felhom-drives/mentes"
|
||||
got, present := isTarget[guestPath]
|
||||
if !present {
|
||||
t.Fatalf("isTarget[%q] is a MISSING KEY — the guest path the controller keys on is missing from "+
|
||||
"/disks entirely, so notifyDriveAbsent takes the generic branch and backup_target_absent "+
|
||||
"can never fire (R-116)", guestPath)
|
||||
}
|
||||
if !got {
|
||||
t.Fatalf("isTarget[%q] = FALSE. Both rows for this drive reached the wire and the registry row — "+
|
||||
"appended last, BackupTarget defaulted false — overwrote the flag-bearing row's true. This is "+
|
||||
"the measured live defect, not a hypothetical: rows=%d", guestPath, len(disks))
|
||||
}
|
||||
}
|
||||
|
||||
// ── V2: the new guest path must NOT make the gate read the drive as PRESENT ─────────────────────
|
||||
|
||||
// This is the over-correction guard, in the exact component under test. planDriveGates computes
|
||||
// present[gp] = present[gp] || d.BoundUnderParent. If the row we now emit carried a true
|
||||
// BoundUnderParent, this fix would SILENCE the alarm it exists to raise.
|
||||
func TestAbsentTargetRowDoesNotRegisterPresence(t *testing.T) {
|
||||
srv := targetRowServer(t, "felhom-backup", []hub.StorageTarget{theAbsentTarget})
|
||||
// deviceCheck/boundCheck are pinned TRUE — the strongest possible case for a false positive.
|
||||
// The row must still report bound_under_parent=false, because that field is only ever assigned
|
||||
// inside the guest-path blocks a system-role row does not enter.
|
||||
for _, d := range wireDisks(t, srv) {
|
||||
if d["guest_path"] != "/mnt/felhom-drives/mentes" {
|
||||
continue
|
||||
}
|
||||
if bup, _ := d["bound_under_parent"].(bool); bup {
|
||||
t.Fatal("the absent backup-target row reports bound_under_parent=true — planDriveGates " +
|
||||
"would compute present=true, the Stop branch would never run, and this fix would " +
|
||||
"SUPPRESS the very alarm it exists to raise")
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Fatal("the absent target row never reached the wire")
|
||||
}
|
||||
|
||||
// ── V1: the gates, each on its own ──────────────────────────────────────────────────────────────
|
||||
|
||||
// Case B is the COMMON fresh-box shape, not an edge: the tier target is the builtin `local` on the
|
||||
// root fs. It must never acquire a guest path.
|
||||
func TestCaseBLocalTargetGetsNoGuestPath(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServer(t, "local", []hub.StorageTarget{
|
||||
{Name: "local", Type: "local", MountPath: "/var/lib/vz", BackingDevice: "", State: hub.StorageStateAttached},
|
||||
}))
|
||||
for _, d := range disks {
|
||||
if gp, _ := d["guest_path"].(string); gp != "" {
|
||||
t.Errorf("the Case B target on %v acquired guest path %q — a system-drive backup target "+
|
||||
"must not cross into the guest", d["mount_path"], gp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A storage that is RoleSystem because it is genuinely system-BACKED (non-empty BackingDevice on the
|
||||
// system disk) must be excluded — this is the case StablePathForRaw would NOT have filtered, since
|
||||
// /mnt/<name> maps to a real stable path. The BackingDevice gate is what stops it.
|
||||
func TestSystemBackedTargetUnderMntGetsNoGuestPath(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServer(t, "sysbackup", []hub.StorageTarget{
|
||||
// sysOnSDA() makes /dev/sda the system disk, so this classifies RoleSystem with a REAL device.
|
||||
{Name: "sysbackup", Type: hub.StorageTypeLocalDir, MountPath: "/mnt/sysbackup",
|
||||
BackingDevice: "/dev/sda1", State: hub.StorageStateAttached},
|
||||
}))
|
||||
for _, d := range disks {
|
||||
if gp, _ := d["guest_path"].(string); gp != "" {
|
||||
t.Errorf("a system-BACKED backup target acquired guest path %q — the BackingDevice gate "+
|
||||
"failed and the :213-214 boundary was widened", gp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── the negative ────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// A drive that is NOT the target must not acquire the flag on any row, present or absent.
|
||||
func TestNonTargetDriveNeverCarriesTheFlag(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServer(t, "felhom-backup", []hub.StorageTarget{
|
||||
{Name: "adat", Type: hub.StorageTypeLocalDir, MountPath: "/mnt/adat",
|
||||
BackingDevice: "", State: hub.StorageStateDisconnected},
|
||||
}))
|
||||
for _, d := range disks {
|
||||
if bt, _ := d["backup_target"].(bool); bt {
|
||||
t.Errorf("non-target drive %v reports backup_target=true", d["name"])
|
||||
}
|
||||
if gp, _ := d["guest_path"].(string); gp != "" {
|
||||
t.Errorf("an absent NON-target drive acquired guest path %q via the R-116 fallback — the "+
|
||||
"BackupTarget gate failed", gp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── v0.116.0 — the join, and the regression it must not cause ───────────────────────────────────
|
||||
|
||||
// THE JOIN. With the device gone the two records of one drive share no runtime field, so the dedup has
|
||||
// to key on the one thing both can still derive: the CONFIGURED path (storage.cfg's `path` on the
|
||||
// Observe side, the .mount unit's `Where` on the registry side), expressed as the stable guest path.
|
||||
// This pins that exactly one row survives — because driveTargetByPath ASSIGNS rather than ORs, so two
|
||||
// rows disagreeing on the flag is decided by append order, which is not a contract anyone should rely on.
|
||||
//
|
||||
// RED-PROOF: delete the `seenGuest[gp]` skip in the union loop and this fails with rows=2.
|
||||
func TestAbsentTargetAppearsExactlyOnce(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
|
||||
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
|
||||
|
||||
const guestPath = "/mnt/felhom-drives/mentes"
|
||||
var rows []map[string]any
|
||||
for _, d := range disks {
|
||||
if gp, _ := d["guest_path"].(string); gp == guestPath {
|
||||
rows = append(rows, d)
|
||||
}
|
||||
}
|
||||
if len(rows) != 1 {
|
||||
t.Fatalf("the absent drive is carried by %d rows, want exactly 1 — with two rows the flag the "+
|
||||
"controller reads is decided by append order, not by the fix. rows=%v", len(rows), rows)
|
||||
}
|
||||
if bt, _ := rows[0]["backup_target"].(bool); !bt {
|
||||
t.Error("the surviving row does not carry backup_target=true")
|
||||
}
|
||||
}
|
||||
|
||||
// THE REGRESSION THIS FIX MUST NOT CAUSE, and the reason neither obvious option was taken.
|
||||
//
|
||||
// The controller reads `d.BackupTarget && d.MountPath != ""` as "a real drive with its own mountpoint —
|
||||
// HEALTHY" and returns immediately (backup_target_offer.go:79). So the two candidate fixes that look
|
||||
// smallest — back-filling MountPath onto the Observe row, or teaching the registry row the flag (its
|
||||
// MountPath is non-empty, read from the stale unit file) — BOTH produce a row satisfying that predicate
|
||||
// while the drive is missing. Either would have silently regressed R-114, which shipped 2026-07-29 and
|
||||
// tells the customer „A rendszermentés meghajtója nem érhető el" in exactly this state, flipping it back
|
||||
// to a false healthy.
|
||||
//
|
||||
// R-114's correctness currently rests on the absent-state rows NOT combining the flag with a mount path.
|
||||
// That coupling was invisible until the payload was captured, and it is what this test pins.
|
||||
//
|
||||
// RED-PROOF: set `MountPath: "/mnt/mentes"` on theAbsentTarget (v0.115.0's fixture value) and this fails.
|
||||
func TestAbsentTargetKeepsR114DegradedSignal(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
|
||||
[]hub.StorageTarget{theAbsentTarget}, theAbsentRegistryRow))
|
||||
|
||||
for _, d := range disks {
|
||||
bt, _ := d["backup_target"].(bool)
|
||||
mp, _ := d["mount_path"].(string)
|
||||
if bt && mp != "" {
|
||||
t.Fatalf("row %v carries backup_target=true AND mount_path=%q while the drive is ABSENT. "+
|
||||
"resolveBackupTargetState (backup_target_offer.go:79) reads that as \"a real drive with "+
|
||||
"its own mountpoint — healthy\" and returns before its TargetAbsent branch, so the "+
|
||||
"customer is told the backup target is fine while its drive is gone. That is R-114, "+
|
||||
"regressed.", d["name"], mp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// PRESENT-STATE PARITY. The fix must change nothing when the drive is there. Present state is the
|
||||
// state every healthy box is in, so a change here reaches the whole fleet; absent state reaches only a
|
||||
// box with a problem. Both rows are supplied, exactly as on a live present box, and the pre-existing
|
||||
// MountPath dedup must still collapse them to one COMPLETE row.
|
||||
func TestPresentTargetPayloadUnchanged(t *testing.T) {
|
||||
present := hub.StorageTarget{
|
||||
Name: "felhom-backup", Type: hub.StorageTypeLocalDir,
|
||||
MountPath: "/mnt/mentes", BackingDevice: "/dev/sdb", ConfigPath: "/mnt/mentes",
|
||||
State: hub.StorageStateAttached, DurableID: "uuid:9303",
|
||||
}
|
||||
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
|
||||
[]hub.StorageTarget{present}, theAbsentRegistryRow))
|
||||
|
||||
var rows []map[string]any
|
||||
for _, d := range disks {
|
||||
if d["name"] == "felhom-backup" || d["mount_path"] == "/mnt/mentes" {
|
||||
rows = append(rows, d)
|
||||
}
|
||||
}
|
||||
if len(rows) != 1 {
|
||||
t.Fatalf("present state carries the drive on %d rows, want 1 (the MountPath dedup): %v", len(rows), rows)
|
||||
}
|
||||
r := rows[0]
|
||||
for field, want := range map[string]any{
|
||||
"mount_path": "/mnt/mentes", "guest_path": "/mnt/felhom-drives/mentes",
|
||||
"backing_device": "/dev/sdb", "role": "user-data", "state": "attached",
|
||||
"backup_target": true, "bound_under_parent": true, "durable_id": "uuid:9303",
|
||||
} {
|
||||
if got := r[field]; got != want {
|
||||
t.Errorf("present-state %s = %v, want %v — the fix altered the healthy payload", field, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The negative, with the union loop actually running: a non-target absent drive gains the flag on no row
|
||||
// and keeps its own registry row (nothing to dedup against, since no Observe row claims its guest path).
|
||||
func TestAbsentNonTargetKeepsItsRegistryRowAndNoFlag(t *testing.T) {
|
||||
disks := wireDisks(t, targetRowServerWithDrives(t, "felhom-backup",
|
||||
[]hub.StorageTarget{{Name: "adat", Type: hub.StorageTypeLocalDir, MountPath: "",
|
||||
BackingDevice: "", ConfigPath: "/mnt/adat", State: hub.StorageStateDisconnected}},
|
||||
[]storage.KnownTarget{{Name: "adat-uuid", Type: hub.StorageTypeUSB,
|
||||
MountPath: "/mnt/adat", DurableID: "uuid:1111", UUID: "1111"}}))
|
||||
|
||||
isTarget := isTargetByPath(disks)
|
||||
for k, v := range isTarget {
|
||||
if v {
|
||||
t.Errorf("isTarget[%q] = true for a NON-target drive — the BackupTarget gate failed", k)
|
||||
}
|
||||
}
|
||||
if _, ok := isTarget["/mnt/felhom-drives/adat"]; !ok {
|
||||
t.Error("the non-target drive lost its guest-path key entirely — the union row was over-suppressed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,397 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// R-117 — BoundUnderParent must mean THE BIND ACTUALLY WORKS, not "a mount by that name exists".
|
||||
//
|
||||
// THE BUG THESE PIN. GuestSeesMount (intermediary.go) and isHostMountpoint both parse a mountinfo line
|
||||
// and then test only fields[4], the mount POINT. Field 3 — major:minor — sits in the same parsed slice
|
||||
// and was discarded. So after a drive is detached and returned, the raw host mount heals onto the NEW
|
||||
// device via its fs-UUID-keyed unit while the bind still names the OLD one, and BOTH existing terms stay
|
||||
// true. Measured live: raw on 8:32 /dev/sdc, bind on 8:16 /dev/sdb with `shutdown`, BoundUnderParent
|
||||
// TRUE, EIO on every read and write, and the controller's gate taking its Return branch — restarting the
|
||||
// customer's apps onto that namespace and emailing backup_target_restored, with no alarm on any channel
|
||||
// (felhom.eu audits/SPIKE-r117-bind-liveness-2026-07-30.md §3.3, §5.2).
|
||||
//
|
||||
// AND THE HALF THAT EMITS NOTHING AT ALL (spike §9, filed R-117a): a device that fails WITHOUT
|
||||
// disappearing leaves the raw mount active, the devnos EQUAL, and the drive never Disconnected — so the
|
||||
// gate produces neither a Stop nor a Return action and nothing is emitted, indefinitely. A devno
|
||||
// comparison alone reads stale-device=false there, which is why term 3 checks the filesystem's own abort
|
||||
// flags too. TestDisks_BindLiveness_AbortedFilesystemReadsAbsent is that case; a fix that shipped only
|
||||
// the devno comparison would pass every other test in this file.
|
||||
//
|
||||
// WHY THE FIXTURES ARE REAL. Each mountinfo body below is the captured output of the spike run, not a
|
||||
// hand-written line. R-116's fix shipped green and inert because its fixture supplied a MountPath
|
||||
// production never supplies. These tests redirect procSelfMountinfo at a fixture file, so the REAL
|
||||
// parser (hostMountEntries), the REAL predicate (bindLiveness) and the REAL /disks handler all run —
|
||||
// the data is injected, the logic is not.
|
||||
//
|
||||
// RED-PROOFS (each verified to land, see REPORT.md): dropping `&& s.bindUsable(...)` from either /disks
|
||||
// construction site fails StaleBindReadsAbsent / UnionPath_StaleBindReadsAbsent and
|
||||
// AbortedFilesystemReadsAbsent; dropping the abort check so only the device comparison remains fails
|
||||
// AbortedFilesystemReadsAbsent and AbortedWins_WhenDevnosAgree ALONE — that is the P1-only fix, and it is
|
||||
// the one worth fearing; dropping `emergency_ro` from abortTokensByFS fails only the emergency_ro subtest;
|
||||
// and returning BindLive instead of BindUnknown for an unreadable table fails UnknownIsTreatedAsPresent.
|
||||
//
|
||||
// A THIRD ordering trap, caught by TestBindLiveness_Verdicts during development and worth naming because
|
||||
// it reports correctly while breaking the repair: reading the abort flag BEFORE comparing devices
|
||||
// classifies the real return state as BindAborted, since its stale bind carries `shutdown` as well as a
|
||||
// different device. BoundUnderParent still reads absent — every test in Group A still passes — but
|
||||
// AttachDrive then refuses the re-bind that actually repairs it, so the self-heal never runs. The verdict
|
||||
// must answer "would a re-bind help", which means reading the abort flag of the RAW mount (the re-bind's
|
||||
// target) in the stale case, and of the bind itself only when the devices already agree.
|
||||
|
||||
// ── fixtures, from the spike's captures ─────────────────────────────────────────────────────────
|
||||
//
|
||||
// Devices, super options, optional-field tags and root paths are verbatim. ONE substitution: the spike
|
||||
// ran against a SCRATCH shared parent (/mnt/r117-drives) so it could not disturb the live
|
||||
// /mnt/felhom-drives peer group, whereas the code under test derives the stable path itself, as
|
||||
// StableParentDir + "/" + DriveNameFromRaw(raw). So /mnt/r117-drives/sd becomes
|
||||
// /mnt/felhom-drives/r117sd. Substituting anything else would make these fixtures describe a path
|
||||
// production never produces — which is precisely how R-116's fix shipped green and inert.
|
||||
|
||||
// mountinfoHealthy is the HEALTHY state (spike §3.3): raw and bind on the SAME device, no abort token.
|
||||
const mountinfoHealthy = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512
|
||||
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512
|
||||
`
|
||||
|
||||
// mountinfoStaleBind is the R-117 RETURN state (spike §3.3 / §5.2): the drive came back as /dev/sdc
|
||||
// (8:32) and the raw mount healed onto it, while the bind still names /dev/sdb (8:16) and carries
|
||||
// `shutdown`. Both pre-R-117 terms read true here.
|
||||
const mountinfoStaleBind = `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
|
||||
814 33 8:32 / /mnt/r117sd rw,relatime shared:485 - ext4 /dev/sdc rw,stripe=512
|
||||
`
|
||||
|
||||
// mountinfoAborted is the STEADY-STATE state (spike §9): the device errored in place, so it NEVER LEFT.
|
||||
// Raw and bind are the SAME device — the device comparison cannot see this — and ext4 has done an
|
||||
// emergency remount-ro. Today this state emits nothing on any channel.
|
||||
//
|
||||
// SECOND substitution, and it is the one that nearly made this test decoration. The spike produced this
|
||||
// state on a dm device (dm is the only mechanism that can make a device error WITHOUT disappearing), so
|
||||
// the capture reads 252:11 /dev/mapper/r117cel. Transposed here onto the USB drive shape, because
|
||||
// RoleForStorage derives role="system" for a /dev/mapper backing device — and a system-role row never
|
||||
// enters the block that computes BoundUnderParent, so the field stays false by DEFAULT and the assertion
|
||||
// below passes without term 3 ever running. It did exactly that until RP1 failed to fail (see REPORT.md).
|
||||
// An in-place abort on a USB drive is the realistic customer case anyway (a link reset that recovers the
|
||||
// link after ext4 has already given up); only the super options and the matching devnos carry the claim.
|
||||
const mountinfoAborted = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
|
||||
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
|
||||
`
|
||||
|
||||
// mountinfoAbortedShutdown is the same in-place shape with the OTHER ext4 abort token, the one a device
|
||||
// removal sets. Both were measured; a check for only `shutdown` passes mountinfoAborted and a check for
|
||||
// only `emergency_ro` passes this — which is why abortTokensByFS carries both, and why RP4 exists.
|
||||
const mountinfoAbortedShutdown = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
|
||||
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw,stripe=512,shutdown
|
||||
`
|
||||
|
||||
// mountinfoUnknownFS is healthy-looking but on a filesystem whose abort vocabulary we have not measured.
|
||||
// The honest verdict is UNKNOWN — which must be treated as PRESENT, not as live and not as absent.
|
||||
const mountinfoUnknownFS = `748 33 8:16 / /mnt/r117sd rw,relatime shared:450 - btrfs /dev/sdb rw
|
||||
755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - btrfs /dev/sdb rw
|
||||
`
|
||||
|
||||
// useMountinfo points the REAL parsers at a fixture for the duration of one test.
|
||||
func useMountinfo(t *testing.T, body string) {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "mountinfo")
|
||||
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prev := procSelfMountinfo
|
||||
procSelfMountinfo = p
|
||||
t.Cleanup(func() { procSelfMountinfo = prev })
|
||||
}
|
||||
|
||||
// livenessServer builds a /disks server over one Observe target (or one registry drive) whose raw mount
|
||||
// is `raw` and stable guest path derives from it. The two PRE-R-117 terms are forced TRUE — that is the
|
||||
// whole point: they were both true in the measured defect, so term 3 is the only thing that can save us.
|
||||
func livenessServer(t *testing.T, obs []hub.StorageTarget, known []storage.KnownTarget) *Server {
|
||||
t.Helper()
|
||||
opts := Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuestsCfg{},
|
||||
Backups: &fakeBackups{},
|
||||
Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: obs},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
|
||||
DiskGate: &fakeGate{},
|
||||
HostReader: sysOnSDA(),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
}
|
||||
if known != nil {
|
||||
opts.DriveTargets = fakeKnownTargets{drives: known}
|
||||
}
|
||||
srv, err := NewServer(opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
// Terms 1 and 2 TRUE — the measured defect's own conditions. livenessCheck is left nil so the real
|
||||
// bindLiveness runs against the fixture.
|
||||
srv.boundCheck = func(string) bool { return true }
|
||||
srv.deviceCheck = func(string) bool { return true }
|
||||
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
|
||||
return srv
|
||||
}
|
||||
|
||||
var obsSD = []hub.StorageTarget{
|
||||
{Name: "sd", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb", MountPath: "/mnt/r117sd", State: hub.StorageStateAttached},
|
||||
}
|
||||
var knownSD = []storage.KnownTarget{
|
||||
{Name: "sd", Type: hub.StorageTypeUSB, MountPath: "/mnt/r117sd", DurableID: "uuid:71e1", UUID: "71e1"},
|
||||
}
|
||||
|
||||
// ── Group A — the consequence: a dead namespace reads ABSENT ────────────────────────────────────
|
||||
|
||||
// TestDisks_BindLiveness_StaleBindReadsAbsent is R-117 case (a), through the real /disks handler.
|
||||
// It asserts the CONSEQUENCE — what the controller reads off the wire — not that a comparison happened.
|
||||
func TestDisks_BindLiveness_StaleBindReadsAbsent(t *testing.T) {
|
||||
useMountinfo(t, mountinfoStaleBind)
|
||||
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("BoundUnderParent reports PRESENT over a stale bind (R-117). The bind names 8:16 /dev/sdb " +
|
||||
"while the raw mount is 8:32 /dev/sdc; every access through it returns EIO. The controller's " +
|
||||
"gate would take its Return branch (controller intermediary.go:258,:299) and restart the " +
|
||||
"customer's apps onto a dead namespace, then email backup_target_restored.")
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisks_BindLiveness_AbortedFilesystemReadsAbsent is R-117a, the steady-state half — and the test a
|
||||
// devno-only fix would fail. The device NEVER LEFT, so raw and bind agree on 252:11.
|
||||
func TestDisks_BindLiveness_AbortedFilesystemReadsAbsent(t *testing.T) {
|
||||
for _, c := range []struct{ name, body string }{
|
||||
{"emergency_ro (errors=remount-ro fired in place)", mountinfoAborted},
|
||||
{"shutdown (forced abort)", mountinfoAbortedShutdown},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
useMountinfo(t, c.body)
|
||||
// GUARD, earned: assert the row is the shape production emits BEFORE asserting the field.
|
||||
// A system-role row has no GuestPath, never runs the conjunction, and reports
|
||||
// BoundUnderParent=false by default — passing this test while proving nothing.
|
||||
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
|
||||
if di.Role != "user-data" || di.GuestPath == "" {
|
||||
t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+
|
||||
"the conjunction never runs on such a row, so any assertion below is vacuous",
|
||||
di.Role, di.GuestPath)
|
||||
}
|
||||
if di.BoundUnderParent {
|
||||
t.Error("BoundUnderParent reports PRESENT over an ABORTED filesystem (R-117a). The devnos " +
|
||||
"MATCH (the device never disappeared), so the device comparison cannot see this — only " +
|
||||
"the filesystem's own abort token can. Today this state emits NOTHING on any channel: " +
|
||||
"the drive is never Disconnected, so the gate produces neither a Stop nor a Return.")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent — the union path for the steady-state half. It
|
||||
// matters more than the Observe one here: this row's Role is hardcoded user-data and its State hardcoded
|
||||
// attached, so the conjunction is the ONLY thing on the row that can report the abort.
|
||||
func TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent(t *testing.T) {
|
||||
useMountinfo(t, mountinfoAborted)
|
||||
di := diskByMount(t, livenessServer(t, nil, knownSD), "/mnt/r117sd")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("union-path drive reports PRESENT over an ABORTED filesystem (R-117a) — and its Role and " +
|
||||
"State are both hardcoded on this row, so nothing else can contradict it")
|
||||
}
|
||||
}
|
||||
|
||||
// The union path carries no PVE dir-storage and hardcodes State:"attached", so these terms are the only
|
||||
// device truth on the row — R-113's reasoning, and it applies to term 3 identically.
|
||||
func TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent(t *testing.T) {
|
||||
useMountinfo(t, mountinfoStaleBind)
|
||||
di := diskByMount(t, livenessServer(t, nil, knownSD), "/mnt/r117sd")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("union-path drive reports PRESENT over a stale bind (R-117) — and its State is hardcoded " +
|
||||
"attached, so nothing else on the row can contradict it")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group B — no false negatives ────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestDisks_BindLiveness_HealthyReadsPresent(t *testing.T) {
|
||||
useMountinfo(t, mountinfoHealthy)
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
obs []hub.StorageTarget
|
||||
known []storage.KnownTarget
|
||||
}{
|
||||
{"observe", obsSD, nil},
|
||||
{"union", nil, knownSD},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
di := diskByMount(t, livenessServer(t, c.obs, c.known), "/mnt/r117sd")
|
||||
if !di.BoundUnderParent {
|
||||
t.Error("a healthy drive reads ABSENT — a false absent STOPS a working customer's apps, " +
|
||||
"which is strictly worse than the bug being fixed")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group C — cannot tell must never mean absent ────────────────────────────────────────────────
|
||||
|
||||
// TestDisks_BindLiveness_UnknownIsTreatedAsPresent pins the rule in every way it can be reached. The
|
||||
// workspace's false-invariant table records newestArchiveOn promising exactly this over a signature that
|
||||
// could not express it; Usable() is the one place it lives, so this is the test that keeps it honest.
|
||||
func TestDisks_BindLiveness_UnknownIsTreatedAsPresent(t *testing.T) {
|
||||
t.Run("unreadable mount table", func(t *testing.T) {
|
||||
prev := procSelfMountinfo
|
||||
procSelfMountinfo = filepath.Join(t.TempDir(), "does-not-exist")
|
||||
t.Cleanup(func() { procSelfMountinfo = prev })
|
||||
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
|
||||
t.Errorf("unreadable /proc gave %v, want BindUnknown", got)
|
||||
}
|
||||
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
|
||||
if !di.BoundUnderParent {
|
||||
t.Error("an unreadable mount table made the drive read ABSENT — cannot-tell must never stop apps")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no raw mount entry to compare against", func(t *testing.T) {
|
||||
// Only the bind is in the table. devicePresent is the term that answers device absence; this one
|
||||
// must abstain rather than double-count it.
|
||||
useMountinfo(t, `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:450 - ext4 /dev/sdb rw
|
||||
`)
|
||||
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
|
||||
t.Errorf("missing raw entry gave %v, want BindUnknown", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("empty paths", func(t *testing.T) {
|
||||
if got := bindLiveness("", "/mnt/r117sd"); got != BindUnknown {
|
||||
t.Errorf("empty stable gave %v, want BindUnknown", got)
|
||||
}
|
||||
if got := bindLiveness("/mnt/felhom-drives/r117sd", ""); got != BindUnknown {
|
||||
t.Errorf("empty raw gave %v, want BindUnknown", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("filesystem whose abort vocabulary is unmeasured", func(t *testing.T) {
|
||||
useMountinfo(t, mountinfoUnknownFS)
|
||||
if got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd"); got != BindUnknown {
|
||||
t.Errorf("btrfs bind gave %v, want BindUnknown — we cannot read its abort state, so we must "+
|
||||
"not claim LIVE either", got)
|
||||
}
|
||||
di := diskByMount(t, livenessServer(t, obsSD, nil), "/mnt/r117sd")
|
||||
if !di.BoundUnderParent {
|
||||
t.Error("an unmeasured filesystem read ABSENT — that would stop apps on every non-ext4 drive")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("Usable is the single place the rule lives", func(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
l BindLiveness
|
||||
want bool
|
||||
}{
|
||||
{BindLive, true},
|
||||
{BindUnknown, true}, // the rule
|
||||
{BindStaleDevice, false},
|
||||
{BindAborted, false},
|
||||
} {
|
||||
if got := c.l.Usable(); got != c.want {
|
||||
t.Errorf("%v.Usable() = %v, want %v", c.l, got, c.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ── Group D — the verdict itself, including the ordering that matters ───────────────────────────
|
||||
|
||||
func TestBindLiveness_Verdicts(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name, body string
|
||||
stable, raw string
|
||||
want BindLiveness
|
||||
}{
|
||||
{"healthy", mountinfoHealthy, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindLive},
|
||||
{"stale device (case a)", mountinfoStaleBind, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindStaleDevice},
|
||||
{"aborted in place, emergency_ro (case b)", mountinfoAborted, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindAborted},
|
||||
{"aborted in place, shutdown", mountinfoAbortedShutdown, "/mnt/felhom-drives/r117sd", "/mnt/r117sd", BindAborted},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
useMountinfo(t, c.body)
|
||||
if got := bindLiveness(c.stable, c.raw); got != c.want {
|
||||
t.Errorf("bindLiveness = %v, want %v", got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestBindLiveness_AbortedWins pins the ORDERING, which is load-bearing and not obvious: in the measured
|
||||
// stale-bind state the filesystem ALSO carries `shutdown`, so both P1 and P2 apply. The verdict must be
|
||||
// BindAborted-or-BindStaleDevice — never live — but more importantly the in-place state, where ONLY P2
|
||||
// applies, must not fall through to a devno comparison that reads equal. This test fails if P1 is checked
|
||||
// before P2 and returns early.
|
||||
func TestBindLiveness_AbortedWins_WhenDevnosAgree(t *testing.T) {
|
||||
useMountinfo(t, mountinfoAborted)
|
||||
got := bindLiveness("/mnt/felhom-drives/r117sd", "/mnt/r117sd")
|
||||
if got.Usable() {
|
||||
t.Fatalf("bindLiveness = %v (usable) — the devnos agree because the device never left, so a "+
|
||||
"P1-first implementation reads this as LIVE and ships R-117's silent half intact", got)
|
||||
}
|
||||
if got != BindAborted {
|
||||
t.Errorf("bindLiveness = %v, want BindAborted (the abort token is the only signal here)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group E — the parser, on a real captured table ──────────────────────────────────────────────
|
||||
|
||||
// TestHostMountEntries_ParsesDevnoAndSuperOpts pins the field extraction R-117 turned on. The optional
|
||||
// fields run (shared:NNN master:NNN) is variable-length, so the " - " separator — not a fixed index — is
|
||||
// what locates fstype and the super options.
|
||||
func TestHostMountEntries_ParsesDevnoAndSuperOpts(t *testing.T) {
|
||||
// A guest-side line with BOTH optional-field tags, the longest real shape (spike §5.2).
|
||||
useMountinfo(t, `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,shutdown
|
||||
`)
|
||||
got := hostMountEntries("/mnt/felhom-drives/r117sd")
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("got %d entries, want 1", len(got))
|
||||
}
|
||||
e := got[0]
|
||||
if e.Devno != "8:16" {
|
||||
t.Errorf("Devno = %q, want 8:16 — this is the field R-117 was lost for want of reading", e.Devno)
|
||||
}
|
||||
if e.Root != "/felhom-data" {
|
||||
t.Errorf("Root = %q, want /felhom-data", e.Root)
|
||||
}
|
||||
if e.FSType != "ext4" {
|
||||
t.Errorf("FSType = %q, want ext4 (located via the ' - ' separator, not a fixed index)", e.FSType)
|
||||
}
|
||||
if !strings.Contains(e.SuperOpts, "shutdown") {
|
||||
t.Errorf("SuperOpts = %q, want it to carry `shutdown`", e.SuperOpts)
|
||||
}
|
||||
}
|
||||
|
||||
// countHostMounts and isHostMountpoint were rewritten onto hostMountEntries; the double-bind convergence
|
||||
// AttachDrive depends on must survive that (REUSE.md: a boolean could not converge stacked binds).
|
||||
func TestHostMountEntries_CountsStackedBinds(t *testing.T) {
|
||||
useMountinfo(t, `755 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime - ext4 /dev/sdb rw
|
||||
756 118 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime - ext4 /dev/sdb rw
|
||||
`)
|
||||
if n := countHostMounts("/mnt/felhom-drives/r117sd"); n != 2 {
|
||||
t.Errorf("countHostMounts = %d, want 2 — AttachDrive's normalize leg needs the count, not a bool", n)
|
||||
}
|
||||
if !isHostMountpoint("/mnt/felhom-drives/r117sd") {
|
||||
t.Error("isHostMountpoint = false over two stacked binds")
|
||||
}
|
||||
if isHostMountpoint("/mnt/nope") {
|
||||
t.Error("isHostMountpoint = true for a path with no entry")
|
||||
}
|
||||
if n := countHostMounts("/mnt/nope"); n != 0 {
|
||||
t.Errorf("countHostMounts = %d for an absent path, want 0", n)
|
||||
}
|
||||
}
|
||||
@@ -11,9 +11,9 @@ import (
|
||||
// handleDiskCandidates splits discovery into initialize (all unclaimed) + attach (mountable-FS subset).
|
||||
func TestDiskCandidates_Split(t *testing.T) {
|
||||
d := &fakeDiskOps{candidates: []storage.CandidateDisk{
|
||||
{Device: "/dev/sdd", SizeBytes: 64 << 30, DataBearing: false}, // blank → initialize only
|
||||
{Device: "/dev/sdd", SizeBytes: 64 << 30, DataBearing: false}, // blank → initialize only
|
||||
{Device: "/dev/sde", FSType: "ext4", DataBearing: true, Mountable: true, MountSource: "/dev/sde1"}, // FS → init + attach
|
||||
{Device: "/dev/sdf", FSType: "ntfs", DataBearing: true, Mountable: false}, // ntfs → initialize only
|
||||
{Device: "/dev/sdf", FSType: "ntfs", DataBearing: true, Mountable: false}, // ntfs → initialize only
|
||||
}}
|
||||
h := newDiskServer(t, d, &fakeGate{}, nil, nil)
|
||||
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
// R-113 — BoundUnderParent must mean THE DEVICE IS THERE, not "a mount entry with this name exists".
|
||||
//
|
||||
// THE BUG THESE PIN. The drive's raw mount at /mnt/<name> is a systemd mount unit bound to its device
|
||||
// and dies with it. The agent's own bind of <raw>/felhom-data under the shared parent is an ordinary
|
||||
// bind — nothing ties it to the device — so it OUTLIVES the device as a stale shell. Before v0.114.0
|
||||
// BoundUnderParent was half 1 only, so a pulled drive kept reporting present, the controller's
|
||||
// drive-absent gate never produced a Stop action, and NOTHING fired on any channel: not
|
||||
// backup_target_absent, not the generic storage_disconnected. Measured live in E-2d with the device
|
||||
// detached — /mnt/mentes2 NOT mounted while /mnt/felhom-drives/mentes2 still read /dev/sdb[/felhom-data]
|
||||
// (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2).
|
||||
//
|
||||
// RED-PROOF. Drop `&& s.devicePresent(...)` from either construction site in disks.go and
|
||||
// TestDisks_DevicePresence_ObservePath_DeviceLossReadsAbsent / _UnionPath_... fail with
|
||||
// "reports present — the bind outlived the device (R-113)".
|
||||
//
|
||||
// These drive the REAL production path: NewServer → GET /disks through srv.Handler() → the JSON the
|
||||
// controller actually parses. The two lowest-level mount reads are injected (a unit test cannot create
|
||||
// real mounts), but nothing above them is faked, and the wire test below asserts the encoded field.
|
||||
|
||||
// presenceServer builds a /disks server over one Observe target and/or one registry drive, with the
|
||||
// bind and device checks independently controllable — the two conditions whose CONJUNCTION is the fix.
|
||||
func presenceServer(t *testing.T, obs []hub.StorageTarget, known []storage.KnownTarget, bound, device bool) *Server {
|
||||
t.Helper()
|
||||
opts := Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuestsCfg{},
|
||||
Backups: &fakeBackups{},
|
||||
Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: obs},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
Disks: &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}},
|
||||
DiskGate: &fakeGate{},
|
||||
HostReader: sysOnSDA(),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
}
|
||||
if known != nil {
|
||||
opts.DriveTargets = fakeKnownTargets{drives: known}
|
||||
}
|
||||
srv, err := NewServer(opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.boundCheck = func(string) bool { return bound }
|
||||
srv.deviceCheck = func(string) bool { return device }
|
||||
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
|
||||
return srv
|
||||
}
|
||||
|
||||
func diskByMount(t *testing.T, srv *Server, mount string) DiskInfo {
|
||||
t.Helper()
|
||||
for _, di := range decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()) {
|
||||
if di.MountPath == mount {
|
||||
return di
|
||||
}
|
||||
}
|
||||
t.Fatalf("no disk reported for mount %q", mount)
|
||||
return DiskInfo{}
|
||||
}
|
||||
|
||||
var obsUSB = []hub.StorageTarget{
|
||||
{Name: "usb", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb1", MountPath: "/mnt/felhom-usb", State: hub.StorageStateAttached},
|
||||
}
|
||||
|
||||
var knownUSB = []storage.KnownTarget{
|
||||
{Name: "mentes2", Type: hub.StorageTypeUSB, MountPath: "/mnt/mentes2", DurableID: "uuid:9303", UUID: "9303"},
|
||||
}
|
||||
|
||||
// ── Group A — device loss is seen ───────────────────────────────────────────────────────────────
|
||||
|
||||
func TestDisks_DevicePresence_ObservePath_DeviceLossReadsAbsent(t *testing.T) {
|
||||
// The exact E-2d shape: the bind survives (bound=true), the device is gone (device=false).
|
||||
di := diskByMount(t, presenceServer(t, obsUSB, nil, true, false), "/mnt/felhom-usb")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("BoundUnderParent reports present — the bind outlived the device (R-113). " +
|
||||
"The controller's gate would emit no Stop action, so no alarm can fire.")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisks_DevicePresence_UnionPath_DeviceLossReadsAbsent(t *testing.T) {
|
||||
// The union path matters MORE: a registry drive with no PVE dir-storage hardcodes State:"attached",
|
||||
// so the raw-mount check is the only device truth the row carries. This is what E-2d detached.
|
||||
di := diskByMount(t, presenceServer(t, nil, knownUSB, true, false), "/mnt/mentes2")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("union-path drive reports present — the bind outlived the device (R-113)")
|
||||
}
|
||||
if di.State != hub.StorageStateAttached {
|
||||
t.Logf("note: union-path State is %q", di.State) // hardcoded; see the OBSERVATION in the report
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group B — the healthy drive, and the return ─────────────────────────────────────────────────
|
||||
|
||||
func TestDisks_DevicePresence_HealthyReadsPresent(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
obs []hub.StorageTarget
|
||||
known []storage.KnownTarget
|
||||
mount string
|
||||
}{
|
||||
{"observe", obsUSB, nil, "/mnt/felhom-usb"},
|
||||
{"union", nil, knownUSB, "/mnt/mentes2"},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
di := diskByMount(t, presenceServer(t, c.obs, c.known, true, true), c.mount)
|
||||
if !di.BoundUnderParent {
|
||||
t.Error("a bound drive whose device is present must read PRESENT — " +
|
||||
"a false absent stops a working customer's apps (Scenario C's failure mode)")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group C — the over-correction guard: boot ordering must not regress ─────────────────────────
|
||||
|
||||
func TestDisks_DevicePresence_BootWindowStillReadsAbsent(t *testing.T) {
|
||||
// Boot ordering: the raw drive mounts EARLY (device=true), the agent binds under the parent ~18s
|
||||
// LATER (bound=false). Presence must stay FALSE in that window — unchanged from before R-113 — so
|
||||
// apps stay stopped until the bind is live and the gate's Return branch recreates them.
|
||||
di := diskByMount(t, presenceServer(t, obsUSB, nil, false, true), "/mnt/felhom-usb")
|
||||
if di.BoundUnderParent {
|
||||
t.Error("boot window reports present before the bind landed — this regresses the reboot " +
|
||||
"convergence the controller's gate comment at intermediary.go:220-224 depends on")
|
||||
}
|
||||
}
|
||||
|
||||
// ── Group D — unknown must never mean absent ────────────────────────────────────────────────────
|
||||
|
||||
func TestDisks_DevicePresence_UnknownIsNotAbsent(t *testing.T) {
|
||||
// devicePresent has nothing to ask about when there is no raw mount path. It must answer TRUE.
|
||||
// Absence of a signal is not evidence of absence of a device — and the cost of getting this
|
||||
// backwards is stopping a healthy customer's apps.
|
||||
srv := presenceServer(t, nil, nil, true, false)
|
||||
srv.deviceCheck = nil // exercise the real devicePresent, not the injected fake
|
||||
if !srv.devicePresent("") {
|
||||
t.Error("devicePresent(\"\") = false — an unanswerable question was reported as ABSENT")
|
||||
}
|
||||
}
|
||||
|
||||
// ── The wire contract — what the controller actually parses ─────────────────────────────────────
|
||||
|
||||
// TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss travels construction → HTTP handler → JSON
|
||||
// encoding and asserts the ENCODED field, because that is what crosses to the controller. A struct-level
|
||||
// assertion would not catch the field being dropped from the wire (e.g. an omitempty regression), and
|
||||
// `bound_under_parent` is the single field the controller's drive-absent gate keys on.
|
||||
func TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss(t *testing.T) {
|
||||
body := do(t, presenceServer(t, nil, knownUSB, true, false).Handler(), "GET", "/disks", "A", "").Body.Bytes()
|
||||
if !strings.Contains(string(body), `"bound_under_parent"`) {
|
||||
t.Fatalf("the wire has no bound_under_parent field at all — the controller's gate reads nothing: %s", body)
|
||||
}
|
||||
var wire struct {
|
||||
Data struct {
|
||||
Disks []map[string]any `json:"disks"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &wire); err != nil {
|
||||
t.Fatalf("decode /disks: %v", err)
|
||||
}
|
||||
var seen bool
|
||||
for _, d := range wire.Data.Disks {
|
||||
if d["mount_path"] != "/mnt/mentes2" {
|
||||
continue
|
||||
}
|
||||
seen = true
|
||||
if v, ok := d["bound_under_parent"].(bool); !ok || v {
|
||||
t.Errorf("wire bound_under_parent = %v (want false) — the device is gone", d["bound_under_parent"])
|
||||
}
|
||||
}
|
||||
if !seen {
|
||||
t.Fatalf("the drive never reached the wire: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
|
||||
)
|
||||
|
||||
func smartIP(v int) *int { return &v }
|
||||
|
||||
// TestDisks_SmartSerialized (v0.94.0) — the already-computed SMART summary is copied into the /disks
|
||||
// payload for a target that has it (Health set), including the SATA counters + temperature; a target
|
||||
// whose SMART was never read (zero-value summary, Health "") omits the field entirely.
|
||||
//
|
||||
// Red-proof: drop the `di.Smart = &sm` copy in handleDisks → the "data" disk's Smart is nil → this fails.
|
||||
func TestDisks_SmartSerialized(t *testing.T) {
|
||||
d := &fakeDiskOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}}
|
||||
sv := fakeStorage{targets: []hub.StorageTarget{
|
||||
{
|
||||
Name: "data", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb1", MountPath: "/mnt/data",
|
||||
Smart: hub.SmartSummary{
|
||||
Health: hub.SmartPassed,
|
||||
TemperatureC: smartIP(34),
|
||||
ReallocatedSectors: smartIP(3),
|
||||
PendingSectors: smartIP(0),
|
||||
},
|
||||
},
|
||||
// Zero-value SMART (never read) — Health "" → must be omitted from the payload.
|
||||
{Name: "nosmart", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdc1", MountPath: "/mnt/nosmart"},
|
||||
}}
|
||||
h := newDiskServer(t, d, &fakeGate{}, sv, nil)
|
||||
|
||||
w := do(t, h, "GET", "/disks", "A", "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("GET /disks: %d (%s)", w.Code, w.Body.String())
|
||||
}
|
||||
byName := map[string]DiskInfo{}
|
||||
for _, di := range decodeDisks(t, w.Body.Bytes()) {
|
||||
byName[di.Name] = di
|
||||
}
|
||||
|
||||
ds := byName["data"].Smart
|
||||
if ds == nil {
|
||||
t.Fatal("data disk: smart summary was not serialized")
|
||||
}
|
||||
if ds.Health != hub.SmartPassed {
|
||||
t.Errorf("data smart Health = %q, want PASSED", ds.Health)
|
||||
}
|
||||
if ds.ReallocatedSectors == nil || *ds.ReallocatedSectors != 3 {
|
||||
t.Errorf("data ReallocatedSectors = %v, want 3", ds.ReallocatedSectors)
|
||||
}
|
||||
if ds.PendingSectors == nil || *ds.PendingSectors != 0 {
|
||||
t.Errorf("data PendingSectors = %v, want 0 (a real zero, not null)", ds.PendingSectors)
|
||||
}
|
||||
if ds.TemperatureC == nil || *ds.TemperatureC != 34 {
|
||||
t.Errorf("data TemperatureC = %v, want 34", ds.TemperatureC)
|
||||
}
|
||||
|
||||
if byName["nosmart"].Smart != nil {
|
||||
t.Errorf("nosmart disk: smart must be omitted when Health is empty, got %+v", byName["nosmart"].Smart)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Fix B (v0.95.0): the /disks union path reads SMART for registry/USB drives ----
|
||||
|
||||
type fakeKnownTargets struct{ drives []storage.KnownTarget }
|
||||
|
||||
func (f fakeKnownTargets) Known(context.Context) ([]storage.KnownTarget, error) { return f.drives, nil }
|
||||
|
||||
type fakeSmartReader struct {
|
||||
byDev map[string]hub.SmartSummary
|
||||
calls []string
|
||||
}
|
||||
|
||||
func (f *fakeSmartReader) SMARTForBacking(_ context.Context, dev string) hub.SmartSummary {
|
||||
f.calls = append(f.calls, dev)
|
||||
if s, ok := f.byDev[dev]; ok {
|
||||
return s
|
||||
}
|
||||
return hub.SmartSummary{}
|
||||
}
|
||||
|
||||
func sp(s string) *string { return &s }
|
||||
|
||||
// A union-path (registry/USB) drive now gets a real SMART read + model, via the Smart seam — it used
|
||||
// to ride the enrich-free union path and show "Nincs adat".
|
||||
// Red-proof: delete the Fix-B block in handleDisks (the s.smart.SMARTForBacking call) → the union
|
||||
// drive carries no smart and this fails.
|
||||
func TestDisks_UnionPathReadsSMART(t *testing.T) {
|
||||
d := &fakeDiskOps{probe: storage.DeviceProbe{Probed: true}}
|
||||
sm := &fakeSmartReader{byDev: map[string]hub.SmartSummary{
|
||||
"/dev/sdb1": {Health: hub.SmartPassed, ModelName: sp("TOSHIBA MQ04ABF100")},
|
||||
}}
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0",
|
||||
Guests: &fakeGuests{},
|
||||
Backups: &fakeBackups{},
|
||||
Store: &fakeStore{},
|
||||
Storage: fakeStorage{}, // no Observe targets → the union drive is not deduped away
|
||||
DriveTargets: fakeKnownTargets{drives: []storage.KnownTarget{
|
||||
{Name: "data-usb", Type: hub.StorageTypeUSB, MountPath: "/mnt/hdd_1", DurableID: "uuid:47a3", UUID: "47a3"},
|
||||
}},
|
||||
Smart: sm,
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
Disks: d,
|
||||
HostReader: sysOnSDA(),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("new server: %v", err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.resolveStorageDevice = func(string) (string, error) { return "/dev/sdb1", nil }
|
||||
|
||||
disks := decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes())
|
||||
var usb *DiskInfo
|
||||
for i := range disks {
|
||||
if disks[i].Name == "data-usb" {
|
||||
usb = &disks[i]
|
||||
}
|
||||
}
|
||||
if usb == nil {
|
||||
t.Fatalf("union drive not in /disks: %+v", disks)
|
||||
}
|
||||
if usb.Smart == nil || usb.Smart.Health != hub.SmartPassed {
|
||||
t.Fatalf("union drive SMART not read: %+v", usb.Smart)
|
||||
}
|
||||
if usb.Smart.ModelName == nil || *usb.Smart.ModelName != "TOSHIBA MQ04ABF100" {
|
||||
t.Errorf("union drive model not carried: %v", usb.Smart)
|
||||
}
|
||||
if len(sm.calls) != 1 || sm.calls[0] != "/dev/sdb1" {
|
||||
t.Errorf("SMART should be read once on /dev/sdb1, got %v", sm.calls)
|
||||
}
|
||||
}
|
||||
@@ -134,9 +134,9 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
|
||||
return memoryBounds{}, fmt.Errorf("node status: %w", err)
|
||||
}
|
||||
b := memoryBounds{
|
||||
allocatedMB: cfg.Memory, // PVE config memory is already MB
|
||||
usageMB: bytesToMBUp(st.Mem), // bytes → MB, rounded up
|
||||
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
|
||||
allocatedMB: cfg.Memory, // PVE config memory is already MB
|
||||
usageMB: bytesToMBUp(st.Mem), // bytes → MB, rounded up
|
||||
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
|
||||
minMB: minGuestMemoryMB,
|
||||
running: st.Status == "running",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,270 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// F-REBOOT (Campaign 8 fault 11) — a guest that should be running and is not.
|
||||
//
|
||||
// THE OUTAGE THIS EXISTS TO KILL. A `pct reboot` issued while a vzdump was in flight completed its
|
||||
// SHUTDOWN half and never issued the start. The guest was found `stopped` with 0 containers, no
|
||||
// lock, and nothing retrying; it stayed down 9m47s until a human ran `pct start`. The backup itself
|
||||
// SUCCEEDED — so every alarm the appliance has was silent, because nothing was broken except that
|
||||
// the customer's entire appliance was off.
|
||||
//
|
||||
// WHY THE EXISTING RECOVERY MISSED IT. `RecoverStaleLockedGuests` (stalelock.go) already does
|
||||
// unlock → delete dangling snapshot → start iff onboot, and it is CORRECT. It missed this by two
|
||||
// gaps, both narrow:
|
||||
// - its predicate acts only on a guest holding a stale vzdump lock (`backup`/`snapshot-delete`);
|
||||
// fault 11's guest was stopped and UNLOCKED, so it returned early;
|
||||
// - it runs ONCE at agent startup, on the load-bearing invariant that a backup lock present then
|
||||
// is stale by definition. A guest that goes down while the agent is already up is never
|
||||
// re-examined.
|
||||
//
|
||||
// This watchdog closes exactly those two gaps and nothing more: it is periodic, and it acts on
|
||||
// "should be running, is not, and is not locked".
|
||||
//
|
||||
// ── THE TRAP, WHICH IS THE SAME SHAPE AS F-CRIT-1's ──────────────────────────────────────────
|
||||
//
|
||||
// A guest the operator deliberately stopped must NOT be auto-started. Fighting the operator makes
|
||||
// maintenance impossible and is worse than the outage — the same over-correction that F-CRIT-1's fix
|
||||
// had to avoid when it stopped whitelisting StateStopped.
|
||||
//
|
||||
// The distinction used is `onboot`, and it is deliberately NOT invented here:
|
||||
// - it is ALREADY the distinction stalelock.go uses for exactly this decision
|
||||
// (`if onboot && g.Status != "running"`), so the two paths cannot disagree;
|
||||
// - it is 1 on customer guests and 0 on scratch/golden guests (agent v0.101.0 sets scratch to 0);
|
||||
// - it is the same flag `pve-guests` itself consults at host boot, so the agent AGREES WITH THE
|
||||
// PLATFORM rather than maintaining a second, private definition of "should be running".
|
||||
//
|
||||
// The hub's desired-state `Run` (internal/desired) is a stronger signal and is wired, but it is
|
||||
// hub-dependent. `onboot` keeps working on a box that has lost hub contact — which is precisely when
|
||||
// an unattended appliance most needs to come back up.
|
||||
|
||||
const (
|
||||
// guestPowerInterval is how often the watchdog looks. Matches the guestnet watchdog's cadence so
|
||||
// the two guest-facing sweeps stay in step, and is far below the 9m47s outage the finding recorded.
|
||||
guestPowerInterval = 60 * time.Second
|
||||
|
||||
// guestPowerMaxAttempts bounds the retry. A guest that will not start must not be started in a
|
||||
// loop forever (Scenario C) — after this many failures the watchdog stops trying and raises it.
|
||||
guestPowerMaxAttempts = 3
|
||||
|
||||
// guestPowerHeartbeatEvery emits a summary line every Nth sweep. 10 x 60s = 10 minutes, matching
|
||||
// the controller's deadapp heartbeat.
|
||||
//
|
||||
// WHY THIS EXISTS, and it is a correction to this file's OWN first version (v0.107.0): the
|
||||
// watchdog logged at startup and when it ACTED, and was otherwise silent. A silent watchdog is
|
||||
// indistinguishable from a dead one — which is F-OBS, the very finding fixed in the same session
|
||||
// this file shipped in, and it is what standing rule 3 exists to prevent. An operator needs a
|
||||
// POSITIVE observable that the sweep is running; "no start lines" must not be the only evidence.
|
||||
guestPowerHeartbeatEvery = 10
|
||||
)
|
||||
|
||||
// guestPowerBackoff is the delay before each retry: 1m, 2m, 4m.
|
||||
//
|
||||
// Measured, not picked round: a healthy `pct start` of guest 9201 completed in ~25 s (observed twice
|
||||
// on 2026-07-28), so even the first 1-minute wait carries 2.4x headroom over a normal start. Three
|
||||
// attempts bound the disruption at roughly 7 minutes — inside the 9m47s outage this fixes — while
|
||||
// never becoming an unbounded loop.
|
||||
var guestPowerBackoff = []time.Duration{time.Minute, 2 * time.Minute, 4 * time.Minute}
|
||||
|
||||
// guestPowerState is one guest's recovery attempt record. In-memory on purpose, like the R-88
|
||||
// breaker: an agent restart re-attempts immediately, which is the cheap direction to fail — a
|
||||
// forgotten backoff costs one extra start attempt, whereas persisting it could carry a stale
|
||||
// "this guest won't start" verdict across the restart that fixed it.
|
||||
type guestPowerState struct {
|
||||
attempts int
|
||||
nextAt time.Time
|
||||
raised bool // the give-up fault has already been raised for this run
|
||||
}
|
||||
|
||||
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
|
||||
// the stale-lock controller is not wired (it supplies the ownership-proven guest list).
|
||||
func (s *Server) WatchGuestPower(ctx context.Context) {
|
||||
if s.staleLock == nil {
|
||||
return
|
||||
}
|
||||
s.logger.Info("guest-power: watchdog started", "interval", guestPowerInterval.String(),
|
||||
"max_attempts", guestPowerMaxAttempts)
|
||||
t := time.NewTicker(guestPowerInterval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
s.GuestPowerTick(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// GuestPowerTick performs one sweep. Exported so a test (and a live check) can drive exactly one
|
||||
// cycle instead of waiting on the ticker.
|
||||
func (s *Server) GuestPowerTick(ctx context.Context) {
|
||||
if s.staleLock == nil {
|
||||
return
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
// Unknown ownership ⇒ do nothing. Never fall back to an unfiltered list: starting a
|
||||
// co-tenant's guest would be worse than leaving ours down.
|
||||
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
|
||||
return
|
||||
}
|
||||
var stopped int
|
||||
for _, g := range guests {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if g.Status != "running" {
|
||||
stopped++
|
||||
}
|
||||
s.recoverOneStoppedGuest(ctx, g)
|
||||
}
|
||||
|
||||
s.guestPowerSweeps++
|
||||
noteGuestPowerSweep(s.logger, s.guestPowerSweeps, len(guests), stopped)
|
||||
}
|
||||
|
||||
// noteGuestPowerSweep emits the liveness observable every guestPowerHeartbeatEvery sweeps.
|
||||
//
|
||||
// It carries WHAT THE SWEEP SAW, not merely that it ran: a line saying "I am alive" cannot
|
||||
// distinguish "alive, all guests up" from "alive, one guest down and being left alone on purpose",
|
||||
// and the second is the state an operator needs to see. Pure and separately testable — the mistake
|
||||
// being corrected here was untestable precisely because it lived inline.
|
||||
func noteGuestPowerSweep(logger *slog.Logger, sweeps, evaluated, stopped int) {
|
||||
if logger == nil || sweeps <= 0 || sweeps%guestPowerHeartbeatEvery != 0 {
|
||||
return
|
||||
}
|
||||
logger.Info("guest-power: watchdog alive",
|
||||
"sweeps_since_boot", sweeps, "guests_evaluated", evaluated, "currently_stopped", stopped)
|
||||
}
|
||||
|
||||
// recoverOneStoppedGuest starts a single guest that should be running and is not.
|
||||
func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
if g.Status == "running" {
|
||||
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
||||
return
|
||||
}
|
||||
|
||||
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
||||
if err != nil {
|
||||
s.logger.Warn("guest-power: read guest config failed — skipping", "vmid", g.VMID, "err", err)
|
||||
return
|
||||
}
|
||||
|
||||
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
|
||||
// (or the golden-image provisioning) does not want this guest running.
|
||||
if !onboot {
|
||||
return
|
||||
}
|
||||
|
||||
// A locked guest belongs to another operation, mid-flight or stale. The stale-lock recovery owns
|
||||
// that case and knows how to prove a lock is stale; this watchdog must not race it or start a
|
||||
// guest whose lock means "a restore is writing my disks right now".
|
||||
if lock != "" {
|
||||
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
|
||||
"vmid", g.VMID, "lock", lock)
|
||||
return
|
||||
}
|
||||
|
||||
// Never start a guest while a vzdump is genuinely in flight for it — a stop-mode backup stops the
|
||||
// guest ON PURPOSE and starting it underneath would corrupt the backup. Fail safe on doubt.
|
||||
running, err := s.staleLock.BackupRunning(ctx, g.VMID)
|
||||
if err != nil {
|
||||
s.logger.Warn("guest-power: could not confirm no backup is running — NOT starting (fail-safe)",
|
||||
"vmid", g.VMID, "err", err)
|
||||
return
|
||||
}
|
||||
if running {
|
||||
s.logger.Info("guest-power: a vzdump is in flight — leaving the guest stopped until it finishes",
|
||||
"vmid", g.VMID)
|
||||
return
|
||||
}
|
||||
|
||||
st, due := s.guestPowerDue(g.VMID)
|
||||
if !due {
|
||||
return
|
||||
}
|
||||
if st.attempts >= guestPowerMaxAttempts {
|
||||
// SCENARIO C — bounded. Raise it ONCE and stop retrying; an infinite silent retry loop is the
|
||||
// over-correction here, and a guest that has refused three starts needs a human, not a fourth.
|
||||
if !st.raised {
|
||||
s.markGuestPowerRaised(g.VMID)
|
||||
s.logger.Error("guest-power: GIVING UP — guest should be running (onboot) but failed to start after repeated attempts; it needs operator attention",
|
||||
"vmid", g.VMID, "attempts", st.attempts, "status", g.Status)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
s.logger.Warn("guest-power: guest should be running (onboot) but is stopped and unlocked — starting it",
|
||||
"vmid", g.VMID, "status", g.Status, "attempt", st.attempts+1, "of", guestPowerMaxAttempts)
|
||||
if err := s.staleLock.Start(ctx, g.VMID); err != nil {
|
||||
s.noteGuestPowerFailure(g.VMID)
|
||||
s.logger.Error("guest-power: start failed", "vmid", g.VMID, "attempt", st.attempts+1, "err", err)
|
||||
return
|
||||
}
|
||||
s.forgetGuestPower(g.VMID)
|
||||
s.logger.Warn("guest-power: STARTED a guest that should have been running", "vmid", g.VMID)
|
||||
}
|
||||
|
||||
// ---- attempt bookkeeping (guarded by its own mutex; independent of the jobs lock) ----------
|
||||
|
||||
var guestPowerMu sync.Mutex
|
||||
|
||||
// guestPowerDue reports the guest's attempt state and whether a new attempt is due now.
|
||||
func (s *Server) guestPowerDue(vmid int) (guestPowerState, bool) {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
if s.guestPower == nil {
|
||||
s.guestPower = map[int]guestPowerState{}
|
||||
}
|
||||
st := s.guestPower[vmid]
|
||||
if st.nextAt.IsZero() || !s.now().Before(st.nextAt) {
|
||||
return st, true
|
||||
}
|
||||
return st, false
|
||||
}
|
||||
|
||||
// noteGuestPowerFailure records a failed start and arms the next backoff.
|
||||
func (s *Server) noteGuestPowerFailure(vmid int) {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
if s.guestPower == nil {
|
||||
s.guestPower = map[int]guestPowerState{}
|
||||
}
|
||||
st := s.guestPower[vmid]
|
||||
st.attempts++
|
||||
i := st.attempts - 1
|
||||
if i >= len(guestPowerBackoff) {
|
||||
i = len(guestPowerBackoff) - 1
|
||||
}
|
||||
st.nextAt = s.now().Add(guestPowerBackoff[i])
|
||||
s.guestPower[vmid] = st
|
||||
}
|
||||
|
||||
// markGuestPowerRaised records that the give-up fault has been raised, so it is logged once.
|
||||
func (s *Server) markGuestPowerRaised(vmid int) {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
st := s.guestPower[vmid]
|
||||
st.raised = true
|
||||
s.guestPower[vmid] = st
|
||||
}
|
||||
|
||||
// forgetGuestPower clears a guest's attempt history — called when it is running again, so a guest
|
||||
// that recovers does not carry its old failures into the next incident.
|
||||
func (s *Server) forgetGuestPower(vmid int) {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
if s.guestPower == nil {
|
||||
return
|
||||
}
|
||||
delete(s.guestPower, vmid)
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// A CORRECTION TO THIS PACKAGE'S OWN v0.107.0. The guest-power watchdog logged at startup and when it
|
||||
// ACTED, and was silent otherwise — so on a healthy box the only evidence it was running was the
|
||||
// absence of start lines, which is equally consistent with the sweep having died. That is F-OBS's
|
||||
// shape and what standing rule 3 forbids, shipped in the same session F-OBS was fixed.
|
||||
//
|
||||
// These tests assert the emitted LINE. Asserting that a function was called would reproduce the
|
||||
// original mistake, which was invisible precisely because nothing pinned the output.
|
||||
|
||||
// RED-PROOF: delete the noteGuestPowerSweep call at the end of GuestPowerTick (or the Info line
|
||||
// inside it) → this fails with "no liveness observable after 10 sweeps — silence is
|
||||
// indistinguishable from a dead watchdog".
|
||||
func TestGuestPowerSweep_EmitsLivenessObservable(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "running"}, {VMID: 9100, Status: "stopped"}},
|
||||
locks: map[int]string{9201: "", 9100: ""},
|
||||
onboot: map[int]bool{9201: true, 9100: false}, // 9100 is a deliberate stop
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
s.logger = slog.New(slog.NewTextHandler(&buf, nil))
|
||||
|
||||
for i := 0; i < guestPowerHeartbeatEvery; i++ {
|
||||
s.GuestPowerTick(context.Background())
|
||||
}
|
||||
|
||||
out := buf.String()
|
||||
if !strings.Contains(out, "watchdog alive") {
|
||||
t.Fatalf("no liveness observable after %d sweeps — silence is indistinguishable from a dead watchdog:\n%s",
|
||||
guestPowerHeartbeatEvery, out)
|
||||
}
|
||||
if !strings.Contains(out, "level=INFO") {
|
||||
t.Errorf("the observable is not at INFO — a box on the default level would never see it:\n%s", out)
|
||||
}
|
||||
// It must carry WHAT IT SAW. "currently_stopped=1" is the operator-relevant fact here: the sweep is
|
||||
// alive AND is deliberately leaving one guest down, which "I ran" alone cannot express.
|
||||
for _, want := range []string{"sweeps_since_boot=", "guests_evaluated=2", "currently_stopped=1"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("the observable omits %q — it proves the sweep ran but not what it found:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// It must be a summary, not a line per sweep: at 60 s that would be 1440 lines/day, which is the
|
||||
// pressure that made silence attractive in the first place.
|
||||
//
|
||||
// RED-PROOF: change the guard to `sweeps%1 != 0` → this fails with
|
||||
// "emitted 30 observables across 30 sweeps — that is the flood that made silence attractive".
|
||||
func TestGuestPowerSweep_IsASummaryNotAFlood(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
lg := slog.New(slog.NewTextHandler(&buf, nil))
|
||||
|
||||
const sweeps = 30
|
||||
for i := 1; i <= sweeps; i++ {
|
||||
noteGuestPowerSweep(lg, i, 1, 0)
|
||||
}
|
||||
|
||||
got := strings.Count(buf.String(), "watchdog alive")
|
||||
want := sweeps / guestPowerHeartbeatEvery
|
||||
if got == sweeps {
|
||||
t.Fatalf("emitted %d observables across %d sweeps — that is the flood that made silence attractive", got, sweeps)
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("emitted %d observables across %d sweeps, want %d", got, sweeps, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The heartbeat period must stay short enough that a STALLED sweep is obvious well inside the outage
|
||||
// window this watchdog exists to close (the finding's incident was 9m47s of total appliance
|
||||
// downtime). If someone widens the cadence to hours the observable stops being a liveness signal.
|
||||
func TestGuestPowerHeartbeat_StaysUsefulAsALivenessSignal(t *testing.T) {
|
||||
period := guestPowerHeartbeatEvery * int(guestPowerInterval.Seconds())
|
||||
if period > 15*60 {
|
||||
t.Errorf("heartbeat period is %ds (>15min) — too sparse to notice a stalled watchdog", period)
|
||||
}
|
||||
if guestPowerHeartbeatEvery < 2 {
|
||||
t.Errorf("heartbeat every %d sweeps is a per-sweep flood", guestPowerHeartbeatEvery)
|
||||
}
|
||||
}
|
||||
|
||||
// Off-cadence sweeps stay quiet; a nil logger must not panic (the ticker goroutine has no recovery).
|
||||
func TestGuestPowerSweep_QuietOffCadenceAndNilSafe(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
lg := slog.New(slog.NewTextHandler(&buf, nil))
|
||||
noteGuestPowerSweep(lg, guestPowerHeartbeatEvery-1, 1, 0)
|
||||
if buf.Len() != 0 {
|
||||
t.Errorf("emitted off-cadence:\n%s", buf.String())
|
||||
}
|
||||
noteGuestPowerSweep(nil, guestPowerHeartbeatEvery, 1, 0) // must not panic
|
||||
}
|
||||
|
||||
// A sweep that ABORTED on unproven ownership must NOT count as a healthy sweep — otherwise the
|
||||
// heartbeat would report liveness for a watchdog that is examining nothing, which is a worse lie than
|
||||
// silence.
|
||||
//
|
||||
// RED-PROOF: move the s.guestPowerSweeps++ above the Guests() error return → this fails with
|
||||
// "an aborted sweep was counted as healthy".
|
||||
func TestGuestPowerSweep_AbortedSweepIsNotCounted(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{guestsErr: errors.New("pool read failed")}
|
||||
s := gpServer(t, ctl, nil)
|
||||
for i := 0; i < guestPowerHeartbeatEvery*2; i++ {
|
||||
s.GuestPowerTick(context.Background())
|
||||
}
|
||||
if s.guestPowerSweeps != 0 {
|
||||
t.Errorf("an aborted sweep was counted as healthy (sweeps=%d) — the heartbeat would claim liveness for a watchdog examining nothing",
|
||||
s.guestPowerSweeps)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// F-REBOOT (Campaign 8 fault 11): a guest rebooted mid-backup never came back — stopped, unlocked,
|
||||
// nothing retrying, 9m47s of total appliance outage.
|
||||
//
|
||||
// Scenario A (a crashed guest is restarted), B (a deliberately stopped guest is left alone) and
|
||||
// C (bounded retry, then escalate). B and C are what make A safe.
|
||||
|
||||
type fakeGuestPowerCtl struct {
|
||||
guests []proxmox.Guest
|
||||
guestsErr error
|
||||
locks map[int]string // vmid -> lock ("" = unlocked)
|
||||
onboot map[int]bool
|
||||
backupRun map[int]bool
|
||||
backupErr error
|
||||
started []int
|
||||
startErr error
|
||||
}
|
||||
|
||||
func (f *fakeGuestPowerCtl) Guests(context.Context) ([]proxmox.Guest, error) {
|
||||
return f.guests, f.guestsErr
|
||||
}
|
||||
func (f *fakeGuestPowerCtl) Lock(_ context.Context, vmid int) (string, bool, error) {
|
||||
return f.locks[vmid], f.onboot[vmid], nil
|
||||
}
|
||||
func (f *fakeGuestPowerCtl) BackupRunning(_ context.Context, vmid int) (bool, error) {
|
||||
return f.backupRun[vmid], f.backupErr
|
||||
}
|
||||
func (f *fakeGuestPowerCtl) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
|
||||
func (f *fakeGuestPowerCtl) Unlock(context.Context, int) error { return nil }
|
||||
func (f *fakeGuestPowerCtl) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
|
||||
func (f *fakeGuestPowerCtl) Start(_ context.Context, vmid int) error {
|
||||
f.started = append(f.started, vmid)
|
||||
return f.startErr
|
||||
}
|
||||
|
||||
func gpServer(t *testing.T, ctl StaleLockController, now func() time.Time) *Server {
|
||||
t.Helper()
|
||||
s := &Server{staleLock: ctl, logger: slog.New(slog.NewTextHandler(discardW{}, nil))}
|
||||
if now != nil {
|
||||
s.now = now
|
||||
} else {
|
||||
s.now = func() time.Time { return time.Now().UTC() }
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
type discardW struct{}
|
||||
|
||||
func (discardW) Write(p []byte) (int, error) { return len(p), nil }
|
||||
|
||||
// Scenario A — a guest that should be running (onboot) and is stopped-and-unlocked IS started.
|
||||
//
|
||||
// RED-PROOF: delete the `s.staleLock.Start(...)` call in recoverOneStoppedGuest (or make the whole
|
||||
// function return before it) → started is empty and this fails with
|
||||
// "guest 9201 was NOT started — this is F-REBOOT".
|
||||
func TestGuestPower_StoppedOnbootGuestIsStarted(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
|
||||
s.GuestPowerTick(context.Background())
|
||||
|
||||
if len(ctl.started) != 1 || ctl.started[0] != 9201 {
|
||||
t.Fatalf("guest 9201 was NOT started — this is F-REBOOT (started=%v)", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — a DELIBERATELY stopped guest (onboot:0) is never started. This is the trap: fighting
|
||||
// the operator makes maintenance impossible and is worse than the outage being fixed.
|
||||
//
|
||||
// RED-PROOF: remove the `if !onboot { return }` guard → the golden/scratch guest is started and this
|
||||
// fails with "a deliberately stopped guest (onboot:0) was started".
|
||||
func TestGuestPower_DeliberatelyStoppedGuestIsLeftAlone(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9100, Status: "stopped"}, {VMID: 990000, Status: "stopped"}},
|
||||
locks: map[int]string{9100: "", 990000: ""},
|
||||
onboot: map[int]bool{9100: false, 990000: false}, // golden + scratch
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
|
||||
s.GuestPowerTick(context.Background())
|
||||
|
||||
if len(ctl.started) != 0 {
|
||||
t.Errorf("a deliberately stopped guest (onboot:0) was started: %v", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// A LOCKED stopped guest belongs to the stale-lock path, which knows how to prove a lock is stale.
|
||||
// This watchdog must not race it.
|
||||
func TestGuestPower_LockedGuestIsLeftToTheStaleLockPath(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: "snapshot-delete"},
|
||||
onboot: map[int]bool{9201: true},
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 0 {
|
||||
t.Errorf("started a LOCKED guest: %v — that races the stale-lock recovery", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// A guest whose vzdump is genuinely in flight must be left stopped — a stop-mode backup stops the
|
||||
// guest ON PURPOSE, and starting it underneath would corrupt the backup.
|
||||
func TestGuestPower_InFlightBackupIsNotDisturbed(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
backupRun: map[int]bool{9201: true},
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 0 {
|
||||
t.Errorf("started a guest with a vzdump in flight: %v", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// Fail-safe: if we cannot confirm no backup is running, do NOT start.
|
||||
func TestGuestPower_UnconfirmableBackupFailsSafe(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
backupErr: errors.New("task list unavailable"),
|
||||
}
|
||||
s := gpServer(t, ctl, nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 0 {
|
||||
t.Errorf("started despite being unable to confirm no backup is running: %v", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// Unknown ownership ⇒ act on nothing. Starting a co-tenant's guest is worse than leaving ours down.
|
||||
func TestGuestPower_UnprovenOwnershipActsOnNothing(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{guestsErr: errors.New("pool read failed")}
|
||||
s := gpServer(t, ctl, nil)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 0 {
|
||||
t.Errorf("acted with unproven ownership: %v", ctl.started)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario C — bounded retry, then escalate. A guest that will not start must NOT be retried forever.
|
||||
//
|
||||
// RED-PROOF: remove the `if st.attempts >= guestPowerMaxAttempts` branch → the sweep keeps starting
|
||||
// on every tick and this fails with "start attempted N times, want at most 3 — infinite retry loop".
|
||||
func TestGuestPower_RetryIsBoundedThenEscalates(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("cannot start: storage offline"),
|
||||
}
|
||||
base := time.Now().UTC()
|
||||
now := base
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
|
||||
// Drive many ticks, advancing well past every backoff each time.
|
||||
for i := 0; i < 12; i++ {
|
||||
s.GuestPowerTick(context.Background())
|
||||
now = now.Add(10 * time.Minute)
|
||||
}
|
||||
|
||||
if len(ctl.started) > guestPowerMaxAttempts {
|
||||
t.Errorf("start attempted %d times, want at most %d — this is an infinite retry loop",
|
||||
len(ctl.started), guestPowerMaxAttempts)
|
||||
}
|
||||
if len(ctl.started) != guestPowerMaxAttempts {
|
||||
t.Errorf("start attempted %d times, want exactly %d before giving up", len(ctl.started), guestPowerMaxAttempts)
|
||||
}
|
||||
}
|
||||
|
||||
// The backoff must actually hold a retry back — otherwise "bounded" is only bounded by luck.
|
||||
func TestGuestPower_BackoffDefersTheNextAttempt(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("boom"),
|
||||
}
|
||||
base := time.Now().UTC()
|
||||
now := base
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
|
||||
s.GuestPowerTick(context.Background()) // attempt 1, arms a 1m backoff
|
||||
if len(ctl.started) != 1 {
|
||||
t.Fatalf("precondition: want 1 attempt, got %d", len(ctl.started))
|
||||
}
|
||||
now = base.Add(30 * time.Second) // still inside the 1m backoff
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 1 {
|
||||
t.Errorf("retried inside the backoff window (%d attempts) — the bound is not being honoured", len(ctl.started))
|
||||
}
|
||||
now = base.Add(90 * time.Second) // past it
|
||||
s.GuestPowerTick(context.Background())
|
||||
if len(ctl.started) != 2 {
|
||||
t.Errorf("did not retry after the backoff lapsed (%d attempts)", len(ctl.started))
|
||||
}
|
||||
}
|
||||
|
||||
// A guest that comes back healthy must lose its attempt history, so it does not carry old failures
|
||||
// into the next incident.
|
||||
func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("boom"),
|
||||
}
|
||||
base := time.Now().UTC()
|
||||
now := base
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
s.GuestPowerTick(context.Background())
|
||||
|
||||
if _, due := s.guestPowerDue(9201); due {
|
||||
t.Error("precondition: a backoff should be armed after a failed start")
|
||||
}
|
||||
// it comes back up
|
||||
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
|
||||
s.GuestPowerTick(context.Background())
|
||||
if _, due := s.guestPowerDue(9201); !due {
|
||||
t.Error("attempt history survived the guest coming back healthy")
|
||||
}
|
||||
}
|
||||
@@ -233,7 +233,33 @@ func (b *GuestBinder) AttachDrive(ctx context.Context, vmid int, where string) (
|
||||
// makes this converge a double-bind to one (the old umount-one+mount-one never did).
|
||||
n := countHostMounts(stable)
|
||||
if n == 1 && b.GuestSeesMount(ctx, vmid, stable) {
|
||||
return stable, nil // exactly one bind + guest-visible → fully live, no-op
|
||||
// R-117: "one bind + the guest sees it" is NOT liveness. Both of those are path-presence tests, so
|
||||
// this early return declared a namespace that EIO'd on every call "fully live" and defeated the
|
||||
// three call sites that already invoke this repair — the 20 s reconcile ticker, agent startup, and
|
||||
// the controller's Return branch BEFORE it restarts the apps (spike §8.2). The verdict decides:
|
||||
switch lv := bindLiveness(stable, where); lv {
|
||||
case BindStaleDevice:
|
||||
// Case (a). The raw mount has healed onto the returning device; re-binding this stale shell
|
||||
// onto it REPAIRS the namespace live, with no guest restart (proven, spike §8.1). Fall through
|
||||
// to the normalize+rebind below. WARN not INFO-per-tick: this fires once, then it is fixed.
|
||||
b.logger.Warn("guest-attach: bind is STALE — it names a different device than the raw mount; re-binding",
|
||||
"vmid", vmid, "where", where, "stable", stable, "verdict", lv.String())
|
||||
case BindAborted:
|
||||
// Case (b), the Q7 steady-state case. The raw mount is the SAME aborted superblock, so a
|
||||
// re-bind produces a fresh bind to a still-dead filesystem — and because this runs every 20 s
|
||||
// it would be an infinite silent retry: exactly the silence Q7 found, with more CPU. Leave the
|
||||
// mount alone and let the truth travel in BoundUnderParent, which now reads false, so the
|
||||
// drive gate stops the apps and raises the alarm. Clearing an aborted filesystem needs a
|
||||
// remount or a fsck — an operator decision, never an automatic one (R-117a).
|
||||
//
|
||||
// DEBUG, not WARN: this repeats every tick, and the operator-facing signal is the /disks
|
||||
// payload plus the customer alarm. Per logging-conventions, INFO is for state changes.
|
||||
b.logger.Debug("guest-attach: filesystem under the bind has ABORTED — not re-binding (a re-bind cannot clear it); reported not-live instead",
|
||||
"vmid", vmid, "where", where, "stable", stable, "verdict", lv.String())
|
||||
return stable, nil
|
||||
default: // BindLive, BindUnknown — genuinely live, or we cannot tell. Unchanged behaviour.
|
||||
return stable, nil
|
||||
}
|
||||
}
|
||||
for i := 0; i < 16 && countHostMounts(stable) > 0; i++ {
|
||||
if err := b.run(ctx, "umount", stable); err != nil {
|
||||
@@ -250,23 +276,7 @@ func (b *GuestBinder) AttachDrive(ctx context.Context, vmid int, where string) (
|
||||
|
||||
// countHostMounts returns how many times `path` appears as a mount target in /proc/self/mountinfo (i.e.
|
||||
// how many stacked binds are at it). 0 = not mounted; >1 = stacked duplicates. Used to normalize to one.
|
||||
func countHostMounts(path string) int {
|
||||
f, err := os.Open("/proc/self/mountinfo")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer f.Close()
|
||||
n := 0
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
fields := strings.Fields(sc.Text())
|
||||
if len(fields) >= 5 && fields[4] == path {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
func countHostMounts(path string) int { return len(hostMountEntries(path)) }
|
||||
|
||||
// GuestSeesMount reports whether vmid's guest currently has `path` as a mount target in ITS mount
|
||||
// namespace (read from /proc/<guest-init-pid>/mountinfo). This is the GUEST-side truth the host-side
|
||||
@@ -278,7 +288,7 @@ func (b *GuestBinder) GuestSeesMount(ctx context.Context, vmid int, path string)
|
||||
if pid == "" {
|
||||
return false
|
||||
}
|
||||
data, err := os.ReadFile("/proc/" + pid + "/mountinfo")
|
||||
data, err := os.ReadFile(procGuestMountinfo(pid))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
@@ -391,20 +401,212 @@ func (b *GuestBinder) DetachDrive(ctx context.Context, where string) error {
|
||||
// isHostMountpoint reports whether path is currently a mount target in the host's mount table
|
||||
// (/proc/self/mountinfo). Pure read — used for idempotency (skip re-binding) and the BoundUnderParent
|
||||
// report. A read error → false (treat as not-mounted; AttachDrive then (re)binds, which is safe).
|
||||
func isHostMountpoint(path string) bool {
|
||||
f, err := os.Open("/proc/self/mountinfo")
|
||||
func isHostMountpoint(path string) bool { return len(hostMountEntries(path)) > 0 }
|
||||
|
||||
// procSelfMountinfo is the host mount table every predicate in this file reads. It is a package var
|
||||
// ONLY so a test can point the REAL parsers at a captured fixture — production never reassigns it, and a
|
||||
// test that does must restore it (t.Cleanup). Injecting the DATA rather than the verdict is what keeps
|
||||
// the R-117 tests non-hollow: the parser, the predicate and the /disks handler all run for real.
|
||||
var procSelfMountinfo = "/proc/self/mountinfo"
|
||||
|
||||
// procGuestMountinfo resolves a guest's init PID to its mount-table path. A package var for the same
|
||||
// single reason as procSelfMountinfo: so a test can point the REAL GuestSeesMount at a captured guest
|
||||
// mount table. Production never reassigns it.
|
||||
var procGuestMountinfo = func(pid string) string { return "/proc/" + pid + "/mountinfo" }
|
||||
|
||||
// mountEntry is the parsed subset of a mountinfo line the liveness predicate needs. Field numbers are
|
||||
// the kernel's 1-based numbering (proc(5) "/proc/<pid>/mountinfo"): 3 = major:minor, 4 = root within the
|
||||
// filesystem, 5 = mount point; after the " - " separator come fstype, source and the per-superblock
|
||||
// options. Mount points containing spaces are octal-escaped by the kernel, so strings.Fields is safe.
|
||||
type mountEntry struct {
|
||||
// Devno is field 3, the backing device as major:minor. THIS is the field R-117 was lost for want of
|
||||
// reading: it sat in the same parsed slice as the mount point and was discarded.
|
||||
Devno string
|
||||
// Root is field 4 — which subtree of the filesystem is mounted (e.g. /felhom-data for our binds).
|
||||
Root string
|
||||
// FSType is the filesystem driver, needed to know whether SuperOpts' vocabulary is one we can read.
|
||||
FSType string
|
||||
// SuperOpts is the per-superblock option list — where ext4 records that it has stopped serving I/O.
|
||||
SuperOpts string
|
||||
}
|
||||
|
||||
// hostMountEntries returns every entry in the host mount table whose mount point is `path`. There is
|
||||
// more than one when binds are stacked (the double-bind case AttachDrive normalizes). A read error
|
||||
// yields nil — callers treat that as "not mounted"/"cannot tell", never as a positive.
|
||||
//
|
||||
// Pure /proc read, NO BLOCK I/O, per CLAUDE.md's health-check rule: a probe that touches a wedged
|
||||
// device enters uninterruptible sleep and survives SIGKILL (measured, R-117 spike §6.3).
|
||||
func hostMountEntries(path string) []mountEntry {
|
||||
f, err := os.Open(procSelfMountinfo)
|
||||
if err != nil {
|
||||
return false
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
var out []mountEntry
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
// mountinfo field 5 (0-indexed 4) is the mount point.
|
||||
fields := strings.Fields(sc.Text())
|
||||
if len(fields) >= 5 && fields[4] == path {
|
||||
return true
|
||||
if len(fields) < 5 || fields[4] != path {
|
||||
continue
|
||||
}
|
||||
e := mountEntry{Devno: fields[2], Root: fields[3]}
|
||||
// The optional-fields run is variable-length; the " - " separator terminates it.
|
||||
for i := 5; i < len(fields); i++ {
|
||||
if fields[i] != "-" {
|
||||
continue
|
||||
}
|
||||
if len(fields) > i+1 {
|
||||
e.FSType = fields[i+1]
|
||||
}
|
||||
if len(fields) > i+3 {
|
||||
e.SuperOpts = fields[i+3]
|
||||
}
|
||||
break
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// BindLiveness is the THREE-state answer to "is the bind at the stable path actually usable?".
|
||||
//
|
||||
// Three states and not a bool, deliberately. The R-117 fix must be able to say "cannot tell", and the
|
||||
// cost of getting that wrong is asymmetric: reporting a live drive absent STOPS a customer's apps. The
|
||||
// workspace's false-invariant table records `newestArchiveOn` promising "errors degrade to unknown,
|
||||
// never to no-backup" over a (value, bool) signature that made it unrepresentable — the comment was a
|
||||
// wish. Read every verdict through Usable() and no caller can repeat that.
|
||||
type BindLiveness int
|
||||
|
||||
const (
|
||||
// BindUnknown — liveness could not be established (unreadable /proc, no raw mount to compare
|
||||
// against, or a filesystem whose abort vocabulary we have not measured). TREATED AS PRESENT by
|
||||
// Usable(), the same rule devicePresent applies to an empty path (disks.go).
|
||||
BindUnknown BindLiveness = iota
|
||||
// BindLive — the bind names the same device as the raw mount and its filesystem has not aborted.
|
||||
BindLive
|
||||
// BindStaleDevice — R-117 case (a), the detach/return case. The bind still references the superblock
|
||||
// of the drive that went away, while the raw mount has healed onto the returning device via its
|
||||
// fs-UUID-keyed unit. Every access through the bind fails. RE-BINDING REPAIRS THIS.
|
||||
BindStaleDevice
|
||||
// BindAborted — R-117 case (b), the Q7 steady-state case. The filesystem under the bind has given up:
|
||||
// ext4 sets `shutdown` when its device vanished, `emergency_ro` when errors=remount-ro fired in place.
|
||||
// The raw mount is the SAME aborted superblock, so RE-BINDING CANNOT REPAIR THIS — it must surface as
|
||||
// not-live so the drive gate stops the apps and alarms. See AttachDrive's switch.
|
||||
BindAborted
|
||||
)
|
||||
|
||||
// Usable is the ONLY sanctioned way to turn a verdict into a yes/no, so the unknown-is-present rule
|
||||
// lives in exactly one place. Pinned by TestBindLiveness_UnknownIsTreatedAsPresent.
|
||||
func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown }
|
||||
|
||||
func (l BindLiveness) String() string {
|
||||
switch l {
|
||||
case BindLive:
|
||||
return "live"
|
||||
case BindStaleDevice:
|
||||
return "stale-device"
|
||||
case BindAborted:
|
||||
return "filesystem-aborted"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
// abortTokensByFS maps a filesystem driver to the per-superblock option tokens it sets when it has
|
||||
// stopped serving I/O. BOTH ext4 tokens are load-bearing and BOTH were measured (R-117 spike §4):
|
||||
// `shutdown` when the device was removed, `emergency_ro` when errors=remount-ro fired with the device
|
||||
// still present. A check for only `shutdown` passes the ENTIRE Q7 state, which is the silent half.
|
||||
//
|
||||
// ext2/ext3 are served by the ext4 driver on this kernel, so they emit the same tokens. Anything else is
|
||||
// a customer-supplied filesystem whose vocabulary we have not measured — it yields UNKNOWN, never LIVE
|
||||
// (the agent itself only ever formats ext4).
|
||||
var abortTokensByFS = map[string][]string{
|
||||
"ext4": {"shutdown", "emergency_ro"},
|
||||
"ext3": {"shutdown", "emergency_ro"},
|
||||
"ext2": {"shutdown", "emergency_ro"},
|
||||
}
|
||||
|
||||
// fsAborted reports whether the entry's filesystem has aborted, and whether we could tell at all.
|
||||
// `known` false means the fstype is not in abortTokensByFS — the caller must degrade to BindUnknown
|
||||
// rather than infer health from the absence of a token it does not know how to look for.
|
||||
func fsAborted(e mountEntry) (aborted, known bool) {
|
||||
toks, ok := abortTokensByFS[e.FSType]
|
||||
if !ok {
|
||||
return false, false
|
||||
}
|
||||
for _, opt := range strings.Split(e.SuperOpts, ",") {
|
||||
for _, t := range toks {
|
||||
if opt == t {
|
||||
return true, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
return false, true
|
||||
}
|
||||
|
||||
// bindLiveness is the R-117 predicate: does the bind at `stable` actually work? `raw` is the drive's RAW
|
||||
// host mount (/mnt/<name>). Reads /proc only — NO block I/O, per CLAUDE.md's health-check rule.
|
||||
//
|
||||
// WHY THIS EXISTS. GuestSeesMount and isHostMountpoint both compare only field 5 (the mount point) of a
|
||||
// mountinfo line, so both answer "does a mount by that name exist" and neither can see that the bind and
|
||||
// the raw mount name DIFFERENT devices. Measured live: raw on 8:32 /dev/sdc while the bind read
|
||||
// 8:16 /dev/sdb with `shutdown`, BoundUnderParent true, EIO on every read and write, and the gate
|
||||
// restarting the customer's apps onto it (R-117 spike §5.2).
|
||||
//
|
||||
// HOST-SIDE ONLY, deliberately: the host bind and the guest's view of it are the same mount in one
|
||||
// propagation peer group and carry identical devno and super options (measured, spike §5.2), so this
|
||||
// needs no lxc-info fork. Guest VISIBILITY is a different question and stays with GuestSeesMount.
|
||||
func bindLiveness(stable, raw string) BindLiveness {
|
||||
if stable == "" || raw == "" {
|
||||
return BindUnknown // nothing to compare — never claim absent
|
||||
}
|
||||
binds := hostMountEntries(stable)
|
||||
if len(binds) == 0 {
|
||||
return BindUnknown // nothing bound here; that is isHostMountpoint's question, not this one
|
||||
}
|
||||
rawEntries := hostMountEntries(raw)
|
||||
if len(rawEntries) == 0 {
|
||||
return BindUnknown // the raw mount is gone — devicePresent already reports that as absent
|
||||
}
|
||||
// The two cases are distinguished by WHETHER THE DEVICES AGREE, and the abort flag is read off a
|
||||
// DIFFERENT entry in each. Getting this backwards is a live trap, caught here by
|
||||
// TestBindLiveness_Verdicts: in the real return state the stale bind carries `shutdown` AND names a
|
||||
// different device, so an abort-first rule classifies it BindAborted — which reports correctly but
|
||||
// refuses the re-bind that actually repairs it. The question a verdict must answer for AttachDrive is
|
||||
// not "is something aborted" but "would a re-bind help".
|
||||
rawEntry := rawEntries[0]
|
||||
for _, b := range binds {
|
||||
if b.Devno != rawEntry.Devno {
|
||||
// P1 — case (a). The bind references a superblock that is NOT the one the raw mount now has:
|
||||
// the drive went away and came back, and the raw mount healed onto it via its fs-UUID-keyed
|
||||
// unit. Sound rather than heuristic — a stale bind pins the dead superblock, which keeps the
|
||||
// old device index allocated, which FORCES the returning device onto a different number
|
||||
// (measured both ways, including the control test where releasing the bind let the letter be
|
||||
// reused, spike §3.4).
|
||||
//
|
||||
// Whether a re-bind repairs it depends on the RAW mount, which is what a re-bind would point
|
||||
// at — not on the stale bind's own abort flag.
|
||||
if aborted, known := fsAborted(rawEntry); known && aborted {
|
||||
return BindAborted // re-binding would land on another dead filesystem
|
||||
}
|
||||
return BindStaleDevice // re-binding lands on the healthy returning device: repairable
|
||||
}
|
||||
}
|
||||
// Same superblock on both sides, so a re-bind is a no-op by construction. Only the filesystem's own
|
||||
// abort state can tell us anything — and this is R-117's steady-state half (spike §9), where the
|
||||
// device NEVER LEFT so the devnos above agree and P1 alone reads healthy.
|
||||
for _, b := range binds {
|
||||
if aborted, known := fsAborted(b); known && aborted {
|
||||
return BindAborted
|
||||
}
|
||||
}
|
||||
// Devices agree and nothing aborted. If we cannot read this filesystem's abort vocabulary we must not
|
||||
// call it live — say unknown, which Usable() treats as present.
|
||||
for _, b := range binds {
|
||||
if _, known := fsAborted(b); !known {
|
||||
return BindUnknown
|
||||
}
|
||||
}
|
||||
return BindLive
|
||||
}
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-117 §2.2 — WHAT AttachDrive DOES with each verdict, which is a RULING and not a detail.
|
||||
//
|
||||
// The two dead states have DIFFERENT repairs, and the existing self-heal only fits one:
|
||||
//
|
||||
// - STALE DEVICE (the detach/return case). The raw mount has healed onto the returning device via its
|
||||
// fs-UUID-keyed unit, so umount + re-bind lands the namespace on a HEALTHY superblock. Repair is
|
||||
// correct, and it happens live with no guest restart (proven on hardware, spike §8.1). It MUST run:
|
||||
// three call sites already invoke it — the 20 s reconcile ticker, agent startup, and the controller's
|
||||
// Return branch BEFORE it restarts the apps (spike §8.2) — and before v0.117.0 all three were
|
||||
// short-circuited by `if n == 1 && GuestSeesMount(...)` declaring the dead namespace "fully live".
|
||||
//
|
||||
// - ABORTED FILESYSTEM (the steady-state case, R-117a). The raw mount is the SAME aborted superblock,
|
||||
// so a re-bind produces a fresh bind to a still-dead filesystem. It MUST NOT run: AttachDrive is
|
||||
// called every 20 s, so re-binding here is an infinite silent retry — the exact silence R-117a
|
||||
// found, with more CPU — and it would mask the state instead of surfacing it. The truth travels in
|
||||
// BoundUnderParent (now false), so the drive gate stops the apps and alarms. Clearing an aborted
|
||||
// filesystem needs a remount or a fsck; that is an operator decision, never an automatic one.
|
||||
//
|
||||
// These tests assert the CONSEQUENCE — which privileged commands were issued — not that a verdict was
|
||||
// computed. RED-PROOF: make the BindAborted arm fall through to the re-bind instead of returning, and
|
||||
// TestAttachDrive_AbortedFilesystem_DoesNotRebind fails on the recorded umount/mount calls.
|
||||
|
||||
// attachRecorder is a proxmox.Runner that records every privileged call and answers `lxc-info -p` with a
|
||||
// fixed PID, so the REAL GuestSeesMount runs against a captured guest mount table.
|
||||
type attachRecorder struct {
|
||||
calls [][]string
|
||||
pid string
|
||||
}
|
||||
|
||||
func (r *attachRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
if name == "lxc-info" {
|
||||
return []byte(r.pid + "\n"), nil, nil
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r *attachRecorder) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
// mountOps returns just the mount-table-mutating calls — the ones that constitute "a repair ran".
|
||||
func (r *attachRecorder) mountOps() []string {
|
||||
var out []string
|
||||
for _, c := range r.calls {
|
||||
switch c[0] {
|
||||
case "umount", "mount":
|
||||
out = append(out, strings.Join(c, " "))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// attachFixture points BOTH mount tables at fixtures: the host one (procSelfMountinfo, which
|
||||
// countHostMounts and bindLiveness read) and the guest one (procGuestMountinfo, which the REAL
|
||||
// GuestSeesMount reads). Only the data is injected — every predicate runs for real.
|
||||
func attachFixture(t *testing.T, hostBody, guestBody string) *attachRecorder {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
hp := filepath.Join(dir, "host-mountinfo")
|
||||
gp := filepath.Join(dir, "guest-mountinfo")
|
||||
if err := os.WriteFile(hp, []byte(hostBody), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(gp, []byte(guestBody), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prevHost, prevGuest := procSelfMountinfo, procGuestMountinfo
|
||||
procSelfMountinfo = hp
|
||||
procGuestMountinfo = func(string) string { return gp }
|
||||
t.Cleanup(func() { procSelfMountinfo, procGuestMountinfo = prevHost, prevGuest })
|
||||
return &attachRecorder{pid: "9301"}
|
||||
}
|
||||
|
||||
// guestSeesStale / guestSeesHealthy / guestSeesAborted are the GUEST-side captures — note `master:450`,
|
||||
// the slave-of-the-shared-parent tag that proves propagation was wired (spike §3.1). The guest carries the
|
||||
// same devno and super options as the host bind, because it IS the same mount.
|
||||
const guestSeesStale = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,shutdown
|
||||
`
|
||||
const guestSeesHealthy = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512
|
||||
`
|
||||
const guestSeesAborted = `759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - ext4 /dev/sdb rw,stripe=512,emergency_ro
|
||||
`
|
||||
|
||||
func attachBinder(rec *attachRecorder) *GuestBinder {
|
||||
return NewGuestBinder(rec, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
}
|
||||
|
||||
// TestAttachDrive_StaleBind_Rebinds is Q6's consequence: the repair that was short-circuited for three
|
||||
// releases now runs. Exactly the state measured on hardware — bind on 8:16, raw healed onto 8:32.
|
||||
func TestAttachDrive_StaleBind_Rebinds(t *testing.T) {
|
||||
rec := attachFixture(t, mountinfoStaleBind, guestSeesStale)
|
||||
b := attachBinder(rec)
|
||||
|
||||
got, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd")
|
||||
if err != nil {
|
||||
t.Fatalf("AttachDrive: %v", err)
|
||||
}
|
||||
if got != "/mnt/felhom-drives/r117sd" {
|
||||
t.Errorf("stable path = %q", got)
|
||||
}
|
||||
ops := rec.mountOps()
|
||||
if len(ops) == 0 {
|
||||
t.Fatal("NO repair ran over a stale bind — this is the R-117 short-circuit: `n == 1 && " +
|
||||
"GuestSeesMount` declared an EIO namespace \"fully live\", so the 20 s ticker, agent startup " +
|
||||
"and the controller's pre-restart re-attach all did nothing")
|
||||
}
|
||||
var sawUmount, sawBind bool
|
||||
for _, o := range ops {
|
||||
if strings.HasPrefix(o, "umount /mnt/felhom-drives/r117sd") {
|
||||
sawUmount = true
|
||||
}
|
||||
if o == "mount --bind /mnt/r117sd/felhom-data /mnt/felhom-drives/r117sd" {
|
||||
sawBind = true
|
||||
}
|
||||
}
|
||||
if !sawUmount || !sawBind {
|
||||
t.Errorf("repair did not umount-then-rebind; ops=%v", ops)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAttachDrive_AbortedFilesystem_DoesNotRebind is the §2.2 ruling. A re-bind here cannot repair
|
||||
// anything (the raw mount is the same aborted superblock) and AttachDrive runs every 20 s, so re-binding
|
||||
// would be an infinite silent retry that also masks the state.
|
||||
func TestAttachDrive_AbortedFilesystem_DoesNotRebind(t *testing.T) {
|
||||
rec := attachFixture(t, mountinfoAborted, guestSeesAborted)
|
||||
b := attachBinder(rec)
|
||||
|
||||
got, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd")
|
||||
if err != nil {
|
||||
t.Fatalf("AttachDrive returned an error for an aborted filesystem: %v — it must be a quiet no-op; "+
|
||||
"an error here would log `reconcile: AttachDrive failed` every 20 s", err)
|
||||
}
|
||||
if got != "/mnt/felhom-drives/r117sd" {
|
||||
t.Errorf("stable path = %q", got)
|
||||
}
|
||||
if ops := rec.mountOps(); len(ops) != 0 {
|
||||
t.Errorf("AttachDrive re-bound an ABORTED filesystem: %v\n"+
|
||||
"A re-bind lands on the SAME dead superblock, and this runs every 20 s — an infinite silent "+
|
||||
"retry, which is R-117a's silence with more CPU. The state must SURFACE via "+
|
||||
"BoundUnderParent=false so the gate stops the apps and alarms.", ops)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAttachDrive_Healthy_IsStillANoOp — the idempotency the reconcile ticker depends on. If this
|
||||
// regressed, every tick would umount and re-bind a working drive, re-firing propagation into the guest
|
||||
// 4320 times a day.
|
||||
func TestAttachDrive_Healthy_IsStillANoOp(t *testing.T) {
|
||||
rec := attachFixture(t, mountinfoHealthy, guestSeesHealthy)
|
||||
b := attachBinder(rec)
|
||||
|
||||
if _, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd"); err != nil {
|
||||
t.Fatalf("AttachDrive: %v", err)
|
||||
}
|
||||
if ops := rec.mountOps(); len(ops) != 0 {
|
||||
t.Errorf("a healthy bind was disturbed: %v — the 20 s reconcile must stay a no-op", ops)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAttachDrive_UnknownLiveness_IsANoOp — cannot-tell must not trigger churn either. An unreadable
|
||||
// mount table making the agent umount and re-bind every 20 s would be a self-inflicted outage.
|
||||
func TestAttachDrive_UnknownLiveness_IsANoOp(t *testing.T) {
|
||||
// Host table healthy (so n == 1) but on a filesystem whose abort vocabulary we cannot read.
|
||||
rec := attachFixture(t, mountinfoUnknownFS,
|
||||
`759 1176 8:16 /felhom-data /mnt/felhom-drives/r117sd rw,relatime shared:459 master:450 - btrfs /dev/sdb rw
|
||||
`)
|
||||
b := attachBinder(rec)
|
||||
|
||||
if _, err := b.AttachDrive(context.Background(), 9301, "/mnt/r117sd"); err != nil {
|
||||
t.Fatalf("AttachDrive: %v", err)
|
||||
}
|
||||
if ops := rec.mountOps(); len(ops) != 0 {
|
||||
t.Errorf("an UNKNOWN verdict caused a re-bind: %v — cannot-tell must never churn a live mount", ops)
|
||||
}
|
||||
}
|
||||
@@ -73,6 +73,11 @@ func TestDisks_GuestPathAndBoundUnderParent(t *testing.T) {
|
||||
srv.baseCtx = context.Background()
|
||||
// felhom-usb is bound under the parent; felhom-flash is not.
|
||||
srv.boundCheck = func(p string) bool { return p == "/mnt/felhom-drives/felhom-usb" }
|
||||
// R-113 (v0.114.0): BoundUnderParent is now `bound && device present`. This test's subject is the
|
||||
// BIND half, so hold the device half constant at present — otherwise the fixture would be asserting
|
||||
// a drive that is bound with no raw mount underneath it, which is the absent state, not this test's
|
||||
// case. Device presence has its own tests (TestDisks_DevicePresence*).
|
||||
srv.deviceCheck = func(string) bool { return true }
|
||||
|
||||
disks := decodeDisks(t, do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes())
|
||||
byMount := map[string]DiskInfo{}
|
||||
|
||||
@@ -41,12 +41,12 @@ type netStorageAddRequest struct {
|
||||
Name string `json:"name"`
|
||||
Protocol string `json:"protocol"` // "nfs" | "smb"
|
||||
Server string `json:"server"`
|
||||
Export string `json:"export"` // NFS export path | SMB share name
|
||||
MappedUID int `json:"mapped_uid"` // container uid (e.g. 1000)
|
||||
MappedGID int `json:"mapped_gid"` // container gid
|
||||
IdleTimeoutSec int `json:"idle_timeout_sec"` // automount idle-unmount window; 0 → default
|
||||
Username string `json:"username,omitempty"` // SMB only (secret — written to creds file)
|
||||
Password string `json:"password,omitempty"` // SMB only (secret — written to creds file)
|
||||
Export string `json:"export"` // NFS export path | SMB share name
|
||||
MappedUID int `json:"mapped_uid"` // container uid (e.g. 1000)
|
||||
MappedGID int `json:"mapped_gid"` // container gid
|
||||
IdleTimeoutSec int `json:"idle_timeout_sec"` // automount idle-unmount window; 0 → default
|
||||
Username string `json:"username,omitempty"` // SMB only (secret — written to creds file)
|
||||
Password string `json:"password,omitempty"` // SMB only (secret — written to creds file)
|
||||
}
|
||||
|
||||
// handleNetStorageAdd installs a NAS share host-side — verify-before-commit (SPIKE-nas-verify).
|
||||
|
||||
@@ -148,8 +148,8 @@ func TestNetVerify_TruthTable(t *testing.T) {
|
||||
t.Run("ReadDir ok but NOT mounted = FAILED (empty-dir false positive guard)", func(t *testing.T) {
|
||||
n := &fakeNetOps{}
|
||||
srv := newVerifyServer(t, n, t.TempDir(), verifySeams{
|
||||
trigger: func(string) error { return nil }, // the read "worked" (empty dir)
|
||||
mounted: func(string) bool { return false }, // …but no nfs4/cifs in /proc/mounts
|
||||
trigger: func(string) error { return nil }, // the read "worked" (empty dir)
|
||||
mounted: func(string) bool { return false }, // …but no nfs4/cifs in /proc/mounts
|
||||
journal: func(context.Context, string) (string, error) { return "", nil },
|
||||
})
|
||||
h := srv.Handler()
|
||||
|
||||
+542
-98
@@ -16,6 +16,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
||||
@@ -33,6 +34,12 @@ type GuestAPI interface {
|
||||
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
|
||||
}
|
||||
|
||||
// PrivilegedRunner runs a fenced root wrapper. The seam exists so the backup-target move is testable
|
||||
// without sudo: the wrapper IS the security boundary, so tests substitute it, never bypass it.
|
||||
type PrivilegedRunner interface {
|
||||
Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
|
||||
}
|
||||
|
||||
// BackupService enqueues a vzdump/PBS backup of a guest. Satisfied by *backup.BackupRunner.
|
||||
// BackupWithSnapshotHook (8B.2) invokes onSnapshot once mid-backup when the storage snapshot is
|
||||
// taken (snapshot mode only) so the controller can resume its app early; in stop mode it is never
|
||||
@@ -41,6 +48,36 @@ type BackupService interface {
|
||||
BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error)
|
||||
}
|
||||
|
||||
// BackupArchiveLister is an OPTIONAL extension to BackupService: "when did a backup last LAND on
|
||||
// this tier's storage?", answered by the storage rather than by memory. *backup.BackupRunner
|
||||
// satisfies it.
|
||||
//
|
||||
// R-84: the agent's backup Store is in-memory, so after every restart the due-check saw nothing and
|
||||
// the controller took a redundant backup — a wasted multi-hour WAN upload on the offsite tier after
|
||||
// every agent deploy. Consulting the storage makes the cold path truthful without persisting
|
||||
// anything, and it self-corrects: a pruned archive correctly stops counting.
|
||||
type BackupArchiveLister interface {
|
||||
NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error)
|
||||
}
|
||||
|
||||
// BackupTier (R-82) binds ONE backup tier's runner to its own policy. The agent builds one per
|
||||
// resolved config tier; the local API serves each independently so "local daily + PBS weekly" is
|
||||
// expressible over the wire, not just in config.
|
||||
//
|
||||
// COMPATIBILITY CONTRACT (load-bearing — the agent and controller deploy independently):
|
||||
// the tier whose Primary is true is what EVERY untargeted endpoint acts on. An old controller
|
||||
// never sends `?target=`, so it sees exactly the pre-R-82 behaviour and response bytes.
|
||||
type BackupTier struct {
|
||||
TargetID string
|
||||
Cadence time.Duration
|
||||
// WaitTimeout bounds the fire-and-forget backup context. It MUST be >= the runner's own wait
|
||||
// bound, or the outer context cancels first and the tier reports a false failure while the
|
||||
// vzdump keeps running (observed live 2026-07-26 with a fixed 2h outer bound).
|
||||
WaitTimeout time.Duration
|
||||
Primary bool
|
||||
Service BackupService
|
||||
}
|
||||
|
||||
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
|
||||
type BackupStore interface {
|
||||
RecordBackup(hub.Backup)
|
||||
@@ -54,6 +91,13 @@ type StorageView interface {
|
||||
Observe(ctx context.Context) ([]hub.StorageTarget, error)
|
||||
}
|
||||
|
||||
// SmartReader (v0.95.0, Fix B) reads per-disk SMART for the /disks union path so registry/USB drives
|
||||
// that ride the union (not Observe's enrich) still get a health verdict. A zero-value summary
|
||||
// (Health "") means "could not read". Satisfied by *storage.SmartReader.
|
||||
type SmartReader interface {
|
||||
SMARTForBacking(ctx context.Context, backingDevice string) hub.SmartSummary
|
||||
}
|
||||
|
||||
// TokenAuthority resolves a presented bearer token to its guest VMID. Satisfied by *TokenStore.
|
||||
type TokenAuthority interface {
|
||||
Lookup(token string) (int, bool)
|
||||
@@ -77,11 +121,22 @@ type Options struct {
|
||||
// DriveTargets (Impl-2a, optional) yields registry+units-sourced drives for the /disks view, so a
|
||||
// drive with NO PVE storage still appears. Unioned with Storage.Observe (deduped by mount path).
|
||||
DriveTargets storage.KnownTargets
|
||||
Tokens TokenAuthority
|
||||
// Smart (v0.95.0, Fix B) reads per-disk SMART for the /disks UNION path — registry/USB drives ride
|
||||
// the union (not Observe's enrich), so without this they carry no health verdict. OPTIONAL; nil →
|
||||
// union rows have no SMART (pre-v0.95.0 behavior). Satisfied by *storage.SmartReader.
|
||||
Smart SmartReader
|
||||
Tokens TokenAuthority
|
||||
// BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest
|
||||
// is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a
|
||||
// safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence.
|
||||
// When BackupTiers is supplied this is IGNORED (the primary tier carries its own cadence).
|
||||
BackupCadence time.Duration
|
||||
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
|
||||
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
|
||||
BackupTiers []BackupTier
|
||||
// InFlight (R-85) is the host-wide one-heavy-operation gate shared with the restore-test
|
||||
// scheduler. OPTIONAL: nil → no cross-gating (pre-R-85 behaviour). See backup.InFlight.
|
||||
InFlight *backup.InFlight
|
||||
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
|
||||
// are served; otherwise they report "not configured". DiskGate authorizes the destructive
|
||||
// (data-bearing) format path; Guests lists guests for the eject dependent-warning.
|
||||
@@ -97,6 +152,14 @@ type Options struct {
|
||||
// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
|
||||
// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
|
||||
NetStorage NetworkStorageOps
|
||||
// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
|
||||
// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
|
||||
Privileged PrivilegedRunner
|
||||
// ConfigPath is agent.json, so the backup-target move can repoint the primary tier. "" (env-only
|
||||
// config) → the move reports it cannot persist rather than pretending it did.
|
||||
ConfigPath string
|
||||
// StateDir is where a pre-write recovery copy of agent.json is parked. "" → no copy is parked.
|
||||
StateDir string
|
||||
// SmbCredsDir is where the agent writes the 0600 SMB credentials files (out-of-band). "" →
|
||||
// /var/lib/felhom-agent/smb-creds.
|
||||
SmbCredsDir string
|
||||
@@ -172,36 +235,56 @@ type backupJob struct {
|
||||
Error string
|
||||
}
|
||||
|
||||
// backupJobKey identifies one guest's job on ONE tier (R-82).
|
||||
type backupJobKey struct {
|
||||
vmid int
|
||||
target string
|
||||
}
|
||||
|
||||
// Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf
|
||||
// and authorizes every request against the token's guest only.
|
||||
type Server struct {
|
||||
addr string
|
||||
cert tls.Certificate
|
||||
guests GuestAPI
|
||||
backups BackupService
|
||||
store BackupStore
|
||||
addr string
|
||||
cert tls.Certificate
|
||||
guests GuestAPI
|
||||
backups BackupService
|
||||
store BackupStore
|
||||
storage StorageView
|
||||
driveTargets storage.KnownTargets // Impl-2a: registry+units drives for /disks (optional)
|
||||
tokens TokenAuthority
|
||||
cadence time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
smart SmartReader // v0.95.0 Fix B: SMART for the /disks union path (optional)
|
||||
tokens TokenAuthority
|
||||
cadence time.Duration
|
||||
// tiers (R-82) is the resolved backup-tier list, PRIMARY FIRST. Always non-empty: when the
|
||||
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
|
||||
// is the pre-R-82 shape.
|
||||
tiers []BackupTier
|
||||
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
|
||||
inFlight *backup.InFlight
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
guestList GuestLister // slice 8C (optional)
|
||||
guestAttach GuestAttacher // slice 10 P2 (optional)
|
||||
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
|
||||
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
|
||||
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
guestList GuestLister // slice 8C (optional)
|
||||
guestAttach GuestAttacher // slice 10 P2 (optional)
|
||||
mem MemoryOps // v0.90.0 R-24 guest RAM resize (optional)
|
||||
memMu sync.Mutex // single-flight around a resize apply (one customer per host)
|
||||
netStorage NetworkStorageOps // Part A1: NAS network mounts (optional)
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||
guestPower map[int]guestPowerState
|
||||
|
||||
// guestPowerSweeps counts completed guest-power sweeps, for the liveness observable. Touched only
|
||||
// from GuestPowerTick, which the ticker calls serially.
|
||||
guestPowerSweeps int
|
||||
host storage.HostReader // role classification source (optional; defaults to ProcHostReader)
|
||||
|
||||
hostMetrics HostMetricsProvider // slice 9 (optional)
|
||||
hostID string // slice 10B: for the data-bearing-format pending-op hint
|
||||
@@ -212,6 +295,10 @@ type Server struct {
|
||||
// inline customer-confirmed wipe (durable id → current device, re-derive+match,
|
||||
// re-inspect). Defaults to s.reresolveDurableForWipe (real storage funcs); tests
|
||||
// override it to avoid touching real /dev.
|
||||
// resolveStorageDevice maps a durable id (uuid:<fs-uuid>) to its /dev node for the /disks union
|
||||
// path. Defaults to storage.ResolveStorageDevice (hits /dev/disk/by-*); tests override it.
|
||||
resolveStorageDevice func(durableID string) (string, error)
|
||||
|
||||
reresolveWipe func(ctx context.Context, durableID string) (string, error)
|
||||
|
||||
// reresolveBlank is the BLANK-format sibling (audit D3): same anti-retarget
|
||||
@@ -230,8 +317,27 @@ type Server struct {
|
||||
// Optional — nil defaults to the real host mount-table read (isHostMountpoint); tests inject a fake.
|
||||
boundCheck func(string) bool
|
||||
|
||||
// deviceCheck reports whether a drive's RAW host mount is still mounted — the agent's device-presence
|
||||
// signal (R-113). Deliberately separate from boundCheck: the raw mount is device-bound (a systemd
|
||||
// mount unit that dies with its device) while the agent's own bind under the shared parent is NOT,
|
||||
// so only the raw mount distinguishes "device present" from "the bind outlived the device".
|
||||
// Optional — nil defaults to isHostMountpoint; tests inject a fake.
|
||||
deviceCheck func(string) bool
|
||||
|
||||
// livenessCheck answers whether the bind at a stable guest path is USABLE, not merely present — the
|
||||
// third term of the BoundUnderParent conjunction (R-117). Deliberately separate from boundCheck and
|
||||
// deviceCheck because it is the only one of the three that compares them: boundCheck asks "does the
|
||||
// guest see a mount by that name", deviceCheck asks "is the raw mount still there", and BOTH are
|
||||
// satisfied by a bind that names the drive that went away while the raw mount healed onto the
|
||||
// returning one. Optional — nil defaults to bindLiveness. Prefer redirecting procSelfMountinfo at a
|
||||
// captured fixture over injecting here: that exercises the real parser and predicate.
|
||||
livenessCheck func(stable, raw string) BindLiveness
|
||||
|
||||
jobsMu sync.Mutex
|
||||
jobs map[int]*backupJob // per-guest backup job state (slice 8B)
|
||||
// jobs is per-guest-PER-TARGET backup job state (slice 8B; keyed by target too since R-82).
|
||||
// Keying by vmid alone would let a PBS backup started inside the same quiesce window collide
|
||||
// with the local one's single-flight and hand the caller the WRONG job id.
|
||||
jobs map[backupJobKey]*backupJob
|
||||
|
||||
// agentic controller update (Phase 1): the swapper + per-guest single-flight gate.
|
||||
swap *ControllerSwapper
|
||||
@@ -255,15 +361,18 @@ type Server struct {
|
||||
// holder. R lives ONLY in escrowR (never in the job struct — snapshots must be structurally
|
||||
// incapable of carrying it) and is zeroed on claim, supersede, or TTL expiry. See
|
||||
// escrow_ceremony.go for the custody rules.
|
||||
escrowCeremony *EscrowCeremonyConfig
|
||||
escrowMu sync.Mutex
|
||||
escrowJob *escrowCeremonyJob
|
||||
escrowR []byte
|
||||
escrowRClaimed bool
|
||||
escrowRExpiry time.Time
|
||||
escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it)
|
||||
escrowCeremony *EscrowCeremonyConfig
|
||||
escrowMu sync.Mutex
|
||||
escrowJob *escrowCeremonyJob
|
||||
escrowR []byte
|
||||
escrowRClaimed bool
|
||||
escrowRExpiry time.Time
|
||||
escrowDone <-chan struct{} // closes when the detached job finishes (tests wait on it)
|
||||
// ceremonyRun executes the fixed-argv sudo self-invocation (tests inject canned JSON).
|
||||
ceremonyRun ceremonyRunner
|
||||
privileged PrivilegedRunner
|
||||
configPath string
|
||||
stateDir string
|
||||
// escrowSudoCheck is the preflight's list-mode grant probe (`sudo -n -l -- <argv>`).
|
||||
escrowSudoCheck func(ctx context.Context) error
|
||||
// escrowLookPath resolves a binary on PATH for preflight (tests inject).
|
||||
@@ -290,37 +399,50 @@ func NewServer(o Options) (*Server, error) {
|
||||
cadence = defaultBackupCadence
|
||||
}
|
||||
s := &Server{
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
addr: o.ListenAddr,
|
||||
cert: o.Cert,
|
||||
guests: o.Guests,
|
||||
backups: o.Backups,
|
||||
store: o.Store,
|
||||
storage: o.Storage,
|
||||
driveTargets: o.DriveTargets,
|
||||
smart: o.Smart,
|
||||
tokens: o.Tokens,
|
||||
cadence: cadence,
|
||||
logger: o.Logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
disks: o.Disks,
|
||||
diskGate: o.DiskGate,
|
||||
guestList: o.Guests2,
|
||||
guestAttach: o.GuestAttach,
|
||||
mem: o.Memory,
|
||||
netStorage: o.NetStorage,
|
||||
privileged: o.Privileged,
|
||||
configPath: o.ConfigPath,
|
||||
stateDir: o.StateDir,
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[int]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
staleLock: o.StaleLock,
|
||||
host: o.HostReader,
|
||||
hostMetrics: o.HostMetrics,
|
||||
hostID: o.HostID,
|
||||
agentVersion: o.AgentVersion,
|
||||
logRing: o.LogRing,
|
||||
jobs: map[backupJobKey]*backupJob{},
|
||||
swapInFlight: map[int]bool{},
|
||||
}
|
||||
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
|
||||
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
|
||||
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
|
||||
// the primary is always first, because that is what the untargeted endpoints act on.
|
||||
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
|
||||
s.inFlight = o.InFlight
|
||||
if s.backups == nil && len(s.tiers) > 0 {
|
||||
s.backups = s.tiers[0].Service
|
||||
}
|
||||
if s.escrowStagePath == "" {
|
||||
s.escrowStagePath = escrow.StagedResticPasswordPath()
|
||||
@@ -328,6 +450,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
s.reresolveWipe = s.reresolveDurableForWipe
|
||||
s.reresolveBlank = s.reresolveDurableForBlankFormat
|
||||
s.deviceDurableID = storage.DeviceDurableID
|
||||
s.resolveStorageDevice = storage.ResolveStorageDevice
|
||||
s.netTrigger = triggerNetMount
|
||||
s.netMounted = storage.NetworkMountedAt
|
||||
s.netJournal = readUnitJournal
|
||||
@@ -354,6 +477,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback))
|
||||
mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup))
|
||||
mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue))
|
||||
mux.HandleFunc("GET /backup/tiers", s.withGuest(s.handleBackupTiers))
|
||||
mux.HandleFunc("POST /backup/target", s.withGuest(s.handleSetBackupTarget))
|
||||
mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus))
|
||||
mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus))
|
||||
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
|
||||
@@ -622,6 +747,8 @@ type BackupResponse struct {
|
||||
VMID int `json:"vmid"`
|
||||
JobID string `json:"job_id"`
|
||||
Phase string `json:"phase"`
|
||||
// Target (R-82) echoes the tier; empty + omitted for an untargeted request (pre-R-82 bytes).
|
||||
Target string `json:"target,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
@@ -635,17 +762,63 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
return
|
||||
}
|
||||
}
|
||||
// Single-flight per guest: if a backup is already running for this guest, return that job
|
||||
// (don't start a second concurrent vzdump). The controller polls /backup/status on it.
|
||||
s.jobsMu.Lock()
|
||||
if cur := s.jobs[vmid]; cur != nil && cur.Phase == PhaseRunning {
|
||||
job := *cur
|
||||
s.jobsMu.Unlock()
|
||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase}, "")
|
||||
tier, echo, ok := s.tierFromRequest(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||
|
||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||
// could not succeed anyway — but without this guard it would be ATTEMPTED, fail on the lock, and
|
||||
// record a spurious failure that leaves the tier permanently due.
|
||||
//
|
||||
// Two distinct cases, deliberately answered differently:
|
||||
// - SAME tier already in flight → return THAT job (202). Idempotent: the caller re-polls it.
|
||||
// - DIFFERENT tier in flight → 409. Not a new job, and NOT the other tier's job either —
|
||||
// handing back a foreign job id is how a caller comes to believe its own backup ran.
|
||||
//
|
||||
// "In flight" includes `snapshotted`, not just `running`: after the storage snapshot the vzdump
|
||||
// is still uploading and still holding the lock. Checking only `running` (the pre-R-82 code)
|
||||
// left a window where a second POST would start a real second vzdump.
|
||||
s.jobsMu.Lock()
|
||||
if cur := s.jobs[key]; cur != nil && backupInFlight(cur.Phase) {
|
||||
job := *cur
|
||||
s.jobsMu.Unlock()
|
||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
|
||||
return
|
||||
}
|
||||
if busyTarget, busyJob, busy := s.otherTierInFlight(vmid, tier.TargetID); busy {
|
||||
s.jobsMu.Unlock()
|
||||
s.logger.Info("local-api: backup refused — another tier is still in flight",
|
||||
"vmid", vmid, "requested_target", tier.TargetID, "busy_target", busyTarget, "busy_job", busyJob)
|
||||
writeStatus(w, http.StatusConflict, false, nil,
|
||||
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
|
||||
return
|
||||
}
|
||||
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
|
||||
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
|
||||
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
|
||||
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
|
||||
// only the additive tiers carry the target segment.
|
||||
// R-85 Scenario F: a backup and a restore-test must never run together — both move multi-GB over
|
||||
// the same tunnel. Acquired here (still holding jobsMu is fine: TryAcquire never blocks) and
|
||||
// released when the fire-and-forget goroutine finishes.
|
||||
release, busy, free := s.inFlight.TryAcquire("backup:" + tier.TargetID)
|
||||
if !free {
|
||||
s.jobsMu.Unlock()
|
||||
s.logger.Info("local-api: backup refused — a heavy operation is already in flight",
|
||||
"vmid", vmid, "requested_target", tier.TargetID, "busy", busy)
|
||||
writeStatus(w, http.StatusConflict, false, nil,
|
||||
"a heavy operation is already in flight ("+busy+") — only one runs at a time on this host")
|
||||
return
|
||||
}
|
||||
|
||||
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
|
||||
s.jobs[vmid] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
|
||||
if !tier.Primary && tier.TargetID != "" {
|
||||
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
|
||||
}
|
||||
s.jobs[key] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
|
||||
s.jobsMu.Unlock()
|
||||
|
||||
// Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the
|
||||
@@ -658,46 +831,78 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
base = context.Background()
|
||||
}
|
||||
go func() {
|
||||
bctx, cancel := context.WithTimeout(base, 2*time.Hour)
|
||||
defer release() // R-85: free the host-wide gate when this backup finishes, however it ends
|
||||
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
|
||||
// A fixed 2h here would silently cap a 6h offsite tier.
|
||||
outer := tier.WaitTimeout
|
||||
if outer <= 0 {
|
||||
outer = 2 * time.Hour
|
||||
}
|
||||
bctx, cancel := context.WithTimeout(base, outer+15*time.Minute)
|
||||
defer cancel()
|
||||
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
|
||||
// controller resumes its app early (snapshot mode only; in stop mode this never fires).
|
||||
b, err := s.backups.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(vmid, jobID) })
|
||||
b, err := tier.Service.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(key, jobID) })
|
||||
if err != nil {
|
||||
b.VMID = vmid
|
||||
b.Success = false
|
||||
if b.Error == "" {
|
||||
b.Error = err.Error()
|
||||
}
|
||||
s.logger.Error("local-api: backup job failed", "vmid", vmid, "job", jobID, "err", err)
|
||||
// TargetID is what the hub attributes the record to; a failed run must still say which
|
||||
// tier failed, and the runner may not have set it on the error path.
|
||||
if b.TargetID == "" {
|
||||
b.TargetID = tier.TargetID
|
||||
}
|
||||
s.logger.Error("local-api: backup job failed", "vmid", vmid, "target", tier.TargetID, "job", jobID, "err", err)
|
||||
} else {
|
||||
s.logger.Info("local-api: backup job complete", "vmid", vmid, "job", jobID, "archive", b.Archive)
|
||||
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
|
||||
}
|
||||
s.store.RecordBackup(b)
|
||||
s.finishJob(vmid, jobID, b)
|
||||
s.finishJob(key, jobID, b)
|
||||
}()
|
||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
|
||||
}
|
||||
|
||||
// backupInFlight reports whether a phase means "this backup still holds the guest".
|
||||
// `snapshotted` counts: the storage snapshot is taken but the vzdump is still uploading.
|
||||
func backupInFlight(phase string) bool {
|
||||
return phase == PhaseRunning || phase == PhaseSnapshotted
|
||||
}
|
||||
|
||||
// otherTierInFlight reports whether a DIFFERENT tier has an in-flight backup for this guest.
|
||||
// Caller must hold s.jobsMu.
|
||||
func (s *Server) otherTierInFlight(vmid int, target string) (busyTarget, busyJob string, busy bool) {
|
||||
for k, j := range s.jobs {
|
||||
if k.vmid != vmid || k.target == target || j == nil {
|
||||
continue
|
||||
}
|
||||
if backupInFlight(j.Phase) {
|
||||
return k.target, j.JobID, true
|
||||
}
|
||||
}
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
|
||||
// still the current job and still running (don't regress done/failed, and don't touch a newer job).
|
||||
func (s *Server) markSnapshotted(vmid int, jobID string) {
|
||||
func (s *Server) markSnapshotted(key backupJobKey, jobID string) {
|
||||
s.jobsMu.Lock()
|
||||
defer s.jobsMu.Unlock()
|
||||
cur := s.jobs[vmid]
|
||||
cur := s.jobs[key]
|
||||
if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning {
|
||||
return
|
||||
}
|
||||
cur.Phase = PhaseSnapshotted
|
||||
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", vmid, "job", jobID)
|
||||
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", key.vmid, "target", key.target, "job", jobID)
|
||||
}
|
||||
|
||||
// finishJob transitions the guest's job to done/failed (only if it is still the current job — a
|
||||
// later job started after a single-flight gap must not be overwritten by an older one's result).
|
||||
func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
|
||||
func (s *Server) finishJob(key backupJobKey, jobID string, b hub.Backup) {
|
||||
s.jobsMu.Lock()
|
||||
defer s.jobsMu.Unlock()
|
||||
cur := s.jobs[vmid]
|
||||
cur := s.jobs[key]
|
||||
if cur == nil || cur.JobID != jobID {
|
||||
return
|
||||
}
|
||||
@@ -712,10 +917,10 @@ func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
|
||||
}
|
||||
|
||||
// jobSnapshot returns a copy of the guest's current job (ok=false if none).
|
||||
func (s *Server) jobSnapshot(vmid int) (backupJob, bool) {
|
||||
func (s *Server) jobSnapshot(key backupJobKey) (backupJob, bool) {
|
||||
s.jobsMu.Lock()
|
||||
defer s.jobsMu.Unlock()
|
||||
if j := s.jobs[vmid]; j != nil {
|
||||
if j := s.jobs[key]; j != nil {
|
||||
return *j, true
|
||||
}
|
||||
return backupJob{}, false
|
||||
@@ -725,31 +930,185 @@ func (s *Server) jobSnapshot(vmid int) (backupJob, bool) {
|
||||
// recorded OR the newest successful one is older than the agent-local cadence. A successful
|
||||
// POST /backup flips this to false for the window, so the controller won't re-quiesce in a loop.
|
||||
// The hub-served policy is slice 10.
|
||||
// BackupAgeState (R-88 Part 2) says WHY AgeSecs is what it is — the distinction the type system
|
||||
// could not previously express.
|
||||
//
|
||||
// Before this, a storage read ERROR and a genuine never-backed-up both produced a nil AgeSecs with
|
||||
// the same `Reason`, byte-identical on the wire. The controller therefore fired its window-gate
|
||||
// safety valve ("no backup yet — never withhold the first one") on an unreadable storage, quiescing
|
||||
// customer app stacks OUTSIDE the backup window. Absence of a signal, read as a specific value —
|
||||
// the fourth instance of that class in this codebase.
|
||||
//
|
||||
// A STRING enum, not a bool: the zero value must mean "legacy agent, no information", and "" says
|
||||
// that unambiguously where `false` would silently masquerade as a real answer.
|
||||
type BackupAgeState string
|
||||
|
||||
const (
|
||||
// AgeStateKnown — AgeSecs is set and meaningful.
|
||||
AgeStateKnown BackupAgeState = "known"
|
||||
// AgeStateAbsent — a POSITIVE determination that no backup has ever landed for this tier. This is
|
||||
// the only state that may fire the controller's safety valve.
|
||||
AgeStateAbsent BackupAgeState = "absent"
|
||||
// AgeStateUnknown — the agent could not determine the age (storage unreadable, timestamp
|
||||
// unparseable). Still DUE (an unreadable storage must never suppress a backup), but the window
|
||||
// gate must NOT be bypassed on it.
|
||||
AgeStateUnknown BackupAgeState = "unknown"
|
||||
)
|
||||
|
||||
type BackupDueResponse struct {
|
||||
VMID int `json:"vmid"`
|
||||
Due bool `json:"due"`
|
||||
Reason string `json:"reason"`
|
||||
AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none
|
||||
// Target (R-82) echoes the tier this verdict is about. EMPTY (and omitted) for an untargeted
|
||||
// request, which is what keeps the pre-R-82 response bytes identical for old controllers.
|
||||
Target string `json:"target,omitempty"`
|
||||
// AgeState (R-88 Part 2) disambiguates a nil AgeSecs. Additive: an OLD controller ignores it and
|
||||
// behaves exactly as before. An EMPTY value on the wire means the agent is pre-v0.105.0 — the
|
||||
// controller must treat that as "legacy, no information", never as AgeStateUnknown.
|
||||
AgeState BackupAgeState `json:"age_state,omitempty"`
|
||||
}
|
||||
|
||||
// archiveLookup is the three-state result of asking a tier's storage when a backup last landed.
|
||||
// It exists because the old (time.Time, bool) signature could not distinguish "nothing there" from
|
||||
// "I could not look" — the doc comment on newestArchiveOn promised that distinction for months while
|
||||
// the type made it impossible.
|
||||
type archiveLookup int
|
||||
|
||||
const (
|
||||
archiveFound archiveLookup = iota // a backup exists; the time is valid
|
||||
archiveAbsent // read succeeded, no backup for this guest on this tier
|
||||
archiveUnknown // could not read (error, or the service has no lister)
|
||||
)
|
||||
|
||||
func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
latest := s.latestSuccessfulBackupFor(r.Context(), vmid)
|
||||
if latest == nil {
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet"})
|
||||
return
|
||||
}
|
||||
age, ok := backupAge(latest.StartedAt, s.now())
|
||||
tier, echo, ok := s.tierFromRequest(w, r)
|
||||
if !ok {
|
||||
// Unparseable timestamp: fail safe toward "due" so a backup still happens.
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due"})
|
||||
return
|
||||
}
|
||||
// A tier whose TARGET STORAGE does not exist yet is DEFERRED, not due (R-82 Slice D).
|
||||
//
|
||||
// A fresh box carries the offsite tier in its installer defaults, but `felhom-pbs` only appears
|
||||
// when the hub provisions the DR tier (`felhom-pbs-apply`). Reporting "due" in that window would
|
||||
// have the controller quiesce the apps and fire a vzdump at a storage that does not exist —
|
||||
// every cadence, until provisioning happens. Deferring keeps the tier silent until it is real,
|
||||
// and it goes live with NO restart the moment the storage appears.
|
||||
//
|
||||
// Fail-safe: a storage-view ERROR does not defer. Unknown must never suppress a backup.
|
||||
if tier.TargetID != "" && !s.targetStoragePresent(r.Context(), tier.TargetID) {
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: false,
|
||||
Reason: "target storage not present yet — tier deferred until it is provisioned", Target: echo})
|
||||
return
|
||||
}
|
||||
// Newest backup for THIS tier: the in-memory record if this process took one, otherwise the
|
||||
// storage itself (R-84 — see BackupArchiveLister). Whichever is newer wins.
|
||||
var newest time.Time
|
||||
var haveNewest bool
|
||||
var unparseable bool
|
||||
if latest := s.latestSuccessfulBackupForTarget(r.Context(), vmid, tier.TargetID); latest != nil {
|
||||
if t, ok2 := backupAge2(latest.StartedAt); ok2 {
|
||||
newest, haveNewest = t, true
|
||||
} else {
|
||||
unparseable = true
|
||||
}
|
||||
}
|
||||
t, lookup := s.newestArchiveOn(r.Context(), tier, vmid)
|
||||
if lookup == archiveFound && (!haveNewest || t.After(newest)) {
|
||||
newest, haveNewest = t, true
|
||||
unparseable = false // ground truth supersedes an unreadable in-memory timestamp
|
||||
}
|
||||
if !haveNewest {
|
||||
// R-88 Part 2: THREE distinct reasons for a nil age, each with its own state. Only ABSENT is a
|
||||
// positive claim of "never backed up"; only that one may license the controller to bypass its
|
||||
// backup window. All three stay DUE — an agent that cannot tell must never suppress a backup.
|
||||
switch {
|
||||
case unparseable:
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateUnknown,
|
||||
Reason: "last backup time unparseable — treating as due", Target: echo})
|
||||
case lookup == archiveUnknown:
|
||||
// The storage could not be read AND this process holds no record. Previously this emitted
|
||||
// "no successful backup recorded yet" — a positive claim built out of two absences, which
|
||||
// is what fired the window-gate valve during the 2026-07-27 PBS outage.
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateUnknown,
|
||||
Reason: "backup storage unreadable and no in-memory record — age UNKNOWN, treating as due", Target: echo})
|
||||
default:
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeState: AgeStateAbsent,
|
||||
Reason: "no successful backup recorded yet", Target: echo})
|
||||
}
|
||||
return
|
||||
}
|
||||
age := s.now().Sub(newest)
|
||||
ageSecs := int64(age.Seconds())
|
||||
if age >= s.cadence {
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs})
|
||||
if age >= tier.Cadence {
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
|
||||
Reason: "older than cadence", Target: echo})
|
||||
return
|
||||
}
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs})
|
||||
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, AgeSecs: &ageSecs, AgeState: AgeStateKnown,
|
||||
Reason: "within cadence window", Target: echo})
|
||||
}
|
||||
|
||||
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
|
||||
// A controller that gets 404 here is talking to a PRE-R-82 agent and must fall back to the single
|
||||
// untargeted tier — that 404 is the designed capability probe.
|
||||
type BackupTiersResponse struct {
|
||||
VMID int `json:"vmid"`
|
||||
Tiers []BackupTierInfo `json:"tiers"`
|
||||
}
|
||||
|
||||
// BackupTierInfo is one tier as advertised to the controller.
|
||||
type BackupTierInfo struct {
|
||||
Target string `json:"target"`
|
||||
CadenceSeconds int64 `json:"cadence_seconds"`
|
||||
Primary bool `json:"primary"`
|
||||
}
|
||||
|
||||
func (s *Server) handleBackupTiers(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
resp := BackupTiersResponse{VMID: vmid, Tiers: make([]BackupTierInfo, 0, len(s.tiers))}
|
||||
for _, t := range s.tiers {
|
||||
resp.Tiers = append(resp.Tiers, BackupTierInfo{
|
||||
Target: t.TargetID,
|
||||
CadenceSeconds: int64(t.Cadence.Seconds()),
|
||||
Primary: t.Primary,
|
||||
})
|
||||
}
|
||||
writeOK(w, resp)
|
||||
}
|
||||
|
||||
// tierFromRequest resolves the `?target=` query parameter to a tier.
|
||||
//
|
||||
// THE COMPATIBILITY RULE (§4): NO target parameter → the PRIMARY tier, and the echoed target is
|
||||
// EMPTY so the response marshals byte-identically to pre-R-82 (BackupDueResponse.Target is
|
||||
// omitempty). An old controller cannot tell this agent from the old one.
|
||||
//
|
||||
// An UNKNOWN target is a 400, never a silent fallback to the primary: a controller asking about a
|
||||
// tier this agent does not serve must find out, not be told about a different tier's freshness.
|
||||
func (s *Server) tierFromRequest(w http.ResponseWriter, r *http.Request) (BackupTier, string, bool) {
|
||||
want := strings.TrimSpace(r.URL.Query().Get("target"))
|
||||
if want == "" {
|
||||
return s.primaryTier(), "", true
|
||||
}
|
||||
for _, t := range s.tiers {
|
||||
if t.TargetID == want {
|
||||
return t, t.TargetID, true
|
||||
}
|
||||
}
|
||||
writeStatus(w, http.StatusBadRequest, false, nil, "unknown backup target: "+want)
|
||||
return BackupTier{}, "", false
|
||||
}
|
||||
|
||||
// primaryTier returns the tier every untargeted endpoint acts on. tiers is never empty (New
|
||||
// synthesizes one), but this stays defensive: a zero tier would silently disable backups.
|
||||
func (s *Server) primaryTier() BackupTier {
|
||||
for _, t := range s.tiers {
|
||||
if t.Primary {
|
||||
return t
|
||||
}
|
||||
}
|
||||
if len(s.tiers) > 0 {
|
||||
return s.tiers[0]
|
||||
}
|
||||
return BackupTier{TargetID: "", Cadence: defaultBackupCadence, Service: s.backups}
|
||||
}
|
||||
|
||||
// BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest
|
||||
@@ -760,11 +1119,20 @@ type BackupStatusResponse struct {
|
||||
JobID string `json:"job_id,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest
|
||||
// Target (R-82) echoes the tier; empty + omitted when untargeted (pre-R-82 bytes).
|
||||
Target string `json:"target,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Backup: s.latestBackupFor(r.Context(), vmid)}
|
||||
if job, ok := s.jobSnapshot(vmid); ok {
|
||||
tier, echo, ok0 := s.tierFromRequest(w, r)
|
||||
if !ok0 {
|
||||
return
|
||||
}
|
||||
// Untargeted keeps the pre-R-82 meaning EXACTLY: the primary tier's job, and the newest backup
|
||||
// across ANY target (echo == "" → pickLatestBackup's match-any path).
|
||||
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Target: echo,
|
||||
Backup: s.pickLatestBackup(r.Context(), vmid, false, echo)}
|
||||
if job, ok := s.jobSnapshot(backupJobKey{vmid: vmid, target: tier.TargetID}); ok {
|
||||
resp.Phase = job.Phase
|
||||
resp.JobID = job.JobID
|
||||
resp.Error = job.Error
|
||||
@@ -787,21 +1155,34 @@ func (s *Server) handleRestoreTestStatus(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
// latestBackupFor returns this guest's most recent backup from the store (nil if none).
|
||||
func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup {
|
||||
return s.pickLatestBackup(ctx, vmid, false)
|
||||
return s.pickLatestBackup(ctx, vmid, false, "")
|
||||
}
|
||||
|
||||
// latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) —
|
||||
// the basis for /backup/due (a failed backup must not satisfy the cadence).
|
||||
func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup {
|
||||
return s.pickLatestBackup(ctx, vmid, true)
|
||||
return s.pickLatestBackup(ctx, vmid, true, "")
|
||||
}
|
||||
|
||||
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool) *hub.Backup {
|
||||
// latestSuccessfulBackupForTarget is the R-82 per-tier twin: a tier's due-ness must be judged
|
||||
// against ITS OWN newest successful backup. The store is already keyed by target, so this is a
|
||||
// filter, not a data-model change — but WITHOUT it a fresh local backup would satisfy the PBS
|
||||
// tier's cadence and the DR tier would never run.
|
||||
func (s *Server) latestSuccessfulBackupForTarget(ctx context.Context, vmid int, target string) *hub.Backup {
|
||||
return s.pickLatestBackup(ctx, vmid, true, target)
|
||||
}
|
||||
|
||||
// pickLatestBackup returns the newest matching record. target "" matches ANY target (the pre-R-82
|
||||
// behaviour, kept for the untargeted status endpoint).
|
||||
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool, target string) *hub.Backup {
|
||||
var latest *hub.Backup
|
||||
for _, b := range s.store.Backups(ctx) {
|
||||
if b.VMID != vmid || (successOnly && !b.Success) {
|
||||
continue
|
||||
}
|
||||
if target != "" && b.TargetID != target {
|
||||
continue
|
||||
}
|
||||
bb := b
|
||||
if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically
|
||||
latest = &bb
|
||||
@@ -810,6 +1191,47 @@ func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly boo
|
||||
return latest
|
||||
}
|
||||
|
||||
// newestArchiveOn asks THIS TIER's storage when a backup last landed (R-84). Errors and
|
||||
// unsupported services degrade to "unknown", never to "no backup" — an unreadable storage must not
|
||||
// make the tier look freshly backed up, and it must not suppress a backup either: the caller falls
|
||||
// back to the in-memory record, whose absence means DUE.
|
||||
func (s *Server) newestArchiveOn(ctx context.Context, tier BackupTier, vmid int) (time.Time, archiveLookup) {
|
||||
lister, ok := tier.Service.(BackupArchiveLister)
|
||||
if !ok {
|
||||
// NO LISTER = the pre-R-84 world, and it must stay ABSENT — not unknown.
|
||||
//
|
||||
// "Unknown" is the tempting answer (we cannot consult storage, so we do not know) and it is
|
||||
// WRONG here, because it would regress Scenario D: the controller fires its first-backup
|
||||
// safety valve only on ABSENT, so a genuinely new box on a no-lister build would never take
|
||||
// its first backup outside the window, and nobody would notice for weeks. A loud bug traded
|
||||
// for a silent one.
|
||||
//
|
||||
// The honest reading: on this path the in-memory record is the ONLY registry that exists, so
|
||||
// its absence means "no backup recorded" in the only terms available — exactly the claim this
|
||||
// path has always made. UNKNOWN is reserved for a lister that was asked and could not answer.
|
||||
return time.Time{}, archiveAbsent
|
||||
}
|
||||
t, found, err := lister.NewestArchiveTime(ctx, vmid)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: could not read the backup storage for the due-check — falling back to the in-memory record",
|
||||
"vmid", vmid, "target", tier.TargetID, "err", err)
|
||||
return time.Time{}, archiveUnknown
|
||||
}
|
||||
if !found {
|
||||
return time.Time{}, archiveAbsent
|
||||
}
|
||||
return t, archiveFound
|
||||
}
|
||||
|
||||
// backupAge2 parses an RFC3339 backup start time, returning it as a time.
|
||||
func backupAge2(startedAt string) (time.Time, bool) {
|
||||
t, err := time.Parse(time.RFC3339, startedAt)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return t.UTC(), true
|
||||
}
|
||||
|
||||
// backupAge parses an RFC3339 backup start time and returns its age relative to now.
|
||||
func backupAge(startedAt string, now time.Time) (time.Duration, bool) {
|
||||
t, err := time.Parse(time.RFC3339, startedAt)
|
||||
@@ -819,6 +1241,28 @@ func backupAge(startedAt string, now time.Time) (time.Duration, bool) {
|
||||
return now.Sub(t), true
|
||||
}
|
||||
|
||||
// targetStoragePresent reports whether a backup target exists on this host RIGHT NOW.
|
||||
//
|
||||
// Returns TRUE on a storage-view error: "I could not check" must never be read as "not there", or a
|
||||
// transient probe failure would silently suppress backups — the absence-is-not-failure rule this
|
||||
// project keeps relearning (R-80, R-81).
|
||||
func (s *Server) targetStoragePresent(ctx context.Context, target string) bool {
|
||||
if s.storage == nil {
|
||||
return true
|
||||
}
|
||||
targets, err := s.storage.Observe(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: storage view unavailable for the backup-target presence check — assuming present", "target", target, "err", err)
|
||||
return true
|
||||
}
|
||||
for _, t := range targets {
|
||||
if t.Name == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// classByStorage builds a storage-id → class(fast|slow|"") map from the host storage view. A
|
||||
// view error is logged and yields an empty map (class falls back to "" — a hint, not load-bearing).
|
||||
func (s *Server) classByStorage(ctx context.Context) map[string]string {
|
||||
|
||||
@@ -55,7 +55,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
|
||||
|
||||
// The whole point: a recipe built from the live read carries the pbs coord.
|
||||
h := hub.BuildDRRecipeHostHalf(nil,
|
||||
[]hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}}, got)
|
||||
[]hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}}, got,
|
||||
hub.ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
|
||||
if h.PBS == nil {
|
||||
t.Fatal("pbs coord absent despite a reachable PBS — the gap this fixes")
|
||||
}
|
||||
@@ -67,7 +68,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
|
||||
bare := NewSnapshotStore()
|
||||
h2 := hub.BuildDRRecipeHostHalf(nil,
|
||||
[]hub.StorageTarget{{Name: "felhom-pbs", Type: hub.StorageTypePBS, Content: "backup"}},
|
||||
bare.PBSSnapshots(context.Background()))
|
||||
bare.PBSSnapshots(context.Background()),
|
||||
hub.ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
|
||||
if h2.PBS != nil {
|
||||
t.Fatal("companion sanity: the bare store should yield NO pbs coord (proves the live read is load-bearing)")
|
||||
}
|
||||
|
||||
@@ -47,6 +47,16 @@ type RestoreLXCOptions struct {
|
||||
// 'mpN' to bind mount is only possible for root"); replacing it with a throwaway volume needs
|
||||
// no root and the boot-verify doesn't need the drive's data.
|
||||
MountOverrides map[string]string
|
||||
// ConfigOverrides sets arbitrary guest-config params AT RESTORE TIME (they take precedence over
|
||||
// the archive's own values), for settings that must hold from the instant the guest exists —
|
||||
// before any post-restore SetConfig could run.
|
||||
//
|
||||
// The restore-test uses it for `onboot=0`. A restore that fails BEFORE the post-restore config
|
||||
// step leaves a scratch guest carrying the SOURCE guest's config verbatim, including
|
||||
// `onboot: 1` — so a leaked scratch would auto-start on the next host reboot, with the source's
|
||||
// MAC, static island IP and hostname. Observed live 2026-07-26. The normal path link-downs every
|
||||
// NIC before boot, so this is defence in depth for the ABNORMAL path, where the leak happens.
|
||||
ConfigOverrides map[string]string
|
||||
}
|
||||
|
||||
// RestoreLXC restores an LXC from a vzdump/PBS archive via POST /nodes/{node}/lxc
|
||||
@@ -71,6 +81,9 @@ func (c *Client) RestoreLXC(ctx context.Context, opts RestoreLXCOptions) (string
|
||||
for k, val := range opts.MountOverrides {
|
||||
v.Set(k, val) // e.g. mp0 -> "local-lvm:1,mp=/data,backup=0" (overrides the archive's mp0)
|
||||
}
|
||||
for k, val := range opts.ConfigOverrides {
|
||||
v.Set(k, val) // e.g. onboot -> "0" (a leaked scratch must never auto-start)
|
||||
}
|
||||
return c.dataString(ctx, http.MethodPost, "/nodes/"+c.node+"/lxc", v)
|
||||
}
|
||||
|
||||
|
||||
@@ -136,6 +136,39 @@ func (p *Privileged) CreateGoldenLXC(ctx context.Context, spec GoldenLXCSpec) er
|
||||
return p.run(ctx, "pct", args...)
|
||||
}
|
||||
|
||||
// DestroyScratchLXC destroys a restore-test scratch guest through the fenced root path, refusing any
|
||||
// vmid outside the caller-supplied scratch band.
|
||||
//
|
||||
// WHY THIS CANNOT BE THE API — and this is the fourth fenced exception, so the reasoning is recorded
|
||||
// in full. A restore-test whose restore FAILS leaves a scratch guest the API token cannot destroy:
|
||||
// `FelhomAgentGuest` is granted at /pool/felhom, and a guest joins that pool only when its restore
|
||||
// COMPLETES. A failed restore therefore leaves a guest that exists, is in no pool, and is out of
|
||||
// reach (403 VM.Allocate) while holding its disks.
|
||||
//
|
||||
// TWO API-SIDE FIXES WERE BUILT AND BOTH REFUTED BY LIVE TEST on 2026-07-28:
|
||||
// - Adopt the stranded guest into the pool, then retry. `PUT /pools/{pool}` ALSO requires
|
||||
// VM.Allocate on the VM being added, so pool membership cannot bootstrap its own authority.
|
||||
// - Grant the role per-path at /vms/990000..990009. Durable for exactly ONE use per slot: PVE's own
|
||||
// destroy calls AccessControl::remove_vm_access (LXC.pm:906), deleting every ACL at /vms/<vmid>
|
||||
// (AccessControl.pm:1898). The grant is consumed by the operation it authorises.
|
||||
//
|
||||
// The band ACLs are still provisioned (host-install v1.21.0) and the API path is still tried FIRST —
|
||||
// this is the fallback that makes teardown deterministic rather than once-per-slot.
|
||||
//
|
||||
// THE FENCE. The band is enforced in THREE places, deliberately: sudoers matches the vmid literally
|
||||
// (`pct destroy 99000[0-9] --purge` — even a compromised agent asking for 9201 is refused by sudo
|
||||
// itself), this method re-checks it before exec, and the caller checks its own journal provenance.
|
||||
// Unlike an ACL, none of these is consumed by use.
|
||||
func (p *Privileged) DestroyScratchLXC(ctx context.Context, vmid, bandMin, bandMax int) error {
|
||||
if bandMin <= 0 || bandMax < bandMin {
|
||||
return fmt.Errorf("proxmox: DestroyScratchLXC needs a configured scratch band, got [%d,%d]", bandMin, bandMax)
|
||||
}
|
||||
if vmid < bandMin || vmid > bandMax {
|
||||
return fmt.Errorf("proxmox: refusing to destroy vmid %d — outside the scratch band [%d,%d]", vmid, bandMin, bandMax)
|
||||
}
|
||||
return p.run(ctx, "pct", "destroy", strconv.Itoa(vmid), "--purge")
|
||||
}
|
||||
|
||||
// MountUSBByUUID mounts a filesystem by UUID at target (creating the mountpoint).
|
||||
//
|
||||
// WHY THIS CANNOT BE THE API: a physical host mount is not a Proxmox API op; it is
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// F-LEAK (Campaign 8): the fourth root-fenced exception. Its whole justification is that the band is
|
||||
// enforced rather than assumed, so these tests are about the REFUSALS, not the happy path.
|
||||
//
|
||||
// The band is checked in three independent places on purpose: sudoers matches the vmid literally
|
||||
// (`pct destroy 99000[0-9] --purge`), this method re-checks before exec, and the caller checks journal
|
||||
// provenance. These tests pin the middle one; the sudoers glob is proven live.
|
||||
|
||||
type recordingRunner struct {
|
||||
calls [][]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *recordingRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return nil, nil, r.err
|
||||
}
|
||||
|
||||
func (r *recordingRunner) RunStdin(_ context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return nil, nil, r.err
|
||||
}
|
||||
|
||||
// A vmid inside the band is destroyed, with --purge so no config/ACL/firewall residue survives.
|
||||
//
|
||||
// RED-PROOF: drop "--purge" from the args → this fails with "destroy is not --purge", and the live
|
||||
// sudoers rule (which matches the FULL vector including --purge) would refuse the call outright.
|
||||
func TestDestroyScratchLXC_InBandDestroysWithPurge(t *testing.T) {
|
||||
r := &recordingRunner{}
|
||||
if err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), 990003, 990000, 990009); err != nil {
|
||||
t.Fatalf("in-band destroy failed: %v", err)
|
||||
}
|
||||
if len(r.calls) != 1 {
|
||||
t.Fatalf("want exactly 1 exec, got %d: %v", len(r.calls), r.calls)
|
||||
}
|
||||
got := strings.Join(r.calls[0], " ")
|
||||
if got != "pct destroy 990003 --purge" {
|
||||
t.Errorf("exec vector = %q, want %q (it must match the sudoers rule byte for byte)",
|
||||
got, "pct destroy 990003 --purge")
|
||||
}
|
||||
}
|
||||
|
||||
// THE ONE THAT MATTERS. A vmid outside the band must be refused WITHOUT EXECUTING ANYTHING — a real
|
||||
// customer guest, the golden image, a co-tenant's VM.
|
||||
//
|
||||
// RED-PROOF: remove the `vmid < bandMin || vmid > bandMax` check → this fails with
|
||||
// "REFUSAL FAILED: executed [pct destroy 9201 --purge] for out-of-band vmid 9201".
|
||||
func TestDestroyScratchLXC_RefusesOutOfBandWithoutExecuting(t *testing.T) {
|
||||
for _, vmid := range []int{
|
||||
1, // arbitrary
|
||||
9201, // the LIVE customer guest on both demo boxes
|
||||
9100, // golden image
|
||||
9999, // reserved
|
||||
989999, // one below the band
|
||||
990010, // one ABOVE the band — the off-by-one
|
||||
} {
|
||||
r := &recordingRunner{}
|
||||
err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), vmid, 990000, 990009)
|
||||
if err == nil {
|
||||
t.Errorf("vmid %d was NOT refused — the fence is open", vmid)
|
||||
}
|
||||
if len(r.calls) != 0 {
|
||||
t.Errorf("REFUSAL FAILED: executed %v for out-of-band vmid %d", r.calls, vmid)
|
||||
}
|
||||
if err != nil && !strings.Contains(err.Error(), "outside the scratch band") {
|
||||
t.Errorf("vmid %d refused with an unhelpful error: %v", vmid, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An unconfigured or inverted band must refuse everything rather than defaulting to something. A zero
|
||||
// band is what a mis-wired caller looks like, and "destroy vmid 0" must never become reachable.
|
||||
//
|
||||
// RED-PROOF: drop the `bandMin <= 0 || bandMax < bandMin` check → the [0,0] case admits vmid 0 and
|
||||
// this fails with "an unconfigured band admitted vmid 0".
|
||||
func TestDestroyScratchLXC_RefusesUnconfiguredBand(t *testing.T) {
|
||||
cases := []struct{ vmid, min, max int }{
|
||||
{0, 0, 0}, // nothing configured at all
|
||||
{990000, 0, 0}, // band absent, real scratch vmid
|
||||
{990000, 0, 990009}, // min unset
|
||||
{990005, 990009, 990000}, // inverted
|
||||
{990000, -1, 990009}, // negative
|
||||
}
|
||||
for _, c := range cases {
|
||||
r := &recordingRunner{}
|
||||
if err := NewPrivileged(r, "").DestroyScratchLXC(context.Background(), c.vmid, c.min, c.max); err == nil {
|
||||
t.Errorf("band [%d,%d] admitted vmid %d — an unconfigured band must refuse", c.min, c.max, c.vmid)
|
||||
}
|
||||
if len(r.calls) != 0 {
|
||||
t.Errorf("an unconfigured band admitted vmid %d and EXECUTED %v", c.vmid, r.calls)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -51,10 +51,15 @@ const DefaultDataVolMount = "mp0"
|
||||
// Single source of truth for both restore sites (provision bring-up + restore-test).
|
||||
const DefaultPool = "felhom"
|
||||
|
||||
// DefaultSysDataMount is the mpN slot the golden bakes the SSD user-data volume (/mnt/sys_drive) at.
|
||||
// This is the controller's system_data_path; provision grows it (SysDataGrowGB) like the Docker-data
|
||||
// volume. mp1 is the natural next bring-up slot (mp8/mp9 are added by the provision back-half).
|
||||
const DefaultSysDataMount = "mp1"
|
||||
// DefaultSysDataMount is RETIRED (agent v0.120.0, R-165 / decision D-a). The golden no longer bakes a
|
||||
// second volume: since build-golden.sh v3.0.0 there is ONE data volume at /var/lib/felhom (mp0) and
|
||||
// both /var/lib/docker and /mnt/sys_drive are binds of subdirectories of it, so there is no mp1 to
|
||||
// resize. The constant is kept, and deliberately points at nothing, so that a stale caller fails
|
||||
// loudly at review rather than silently resizing a slot that does not exist.
|
||||
//
|
||||
// SysDataGrowGB itself is NOT removed — see its field comment: the host installer still passes
|
||||
// `-sysdata-grow`, and its GiB are FOLDED INTO the single volume's grow rather than dropped.
|
||||
const DefaultSysDataMount = ""
|
||||
|
||||
// Structural host-bind mountpoints every provisioned guest carries (GL-5; verdict of
|
||||
// SPIKE-dr-bindmount-source-2026-07-07): the permanent drives parent bind (mp8,
|
||||
@@ -183,16 +188,28 @@ type BringUpSpec struct {
|
||||
DataVolGrowGB int
|
||||
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
|
||||
DataVolMount string
|
||||
// SysDataGrowGB grows the golden-carried SSD user-data volume (SysDataMount, default mp1, mounted at
|
||||
// /mnt/sys_drive = the controller's system_data_path) to the per-customer target. Same online,
|
||||
// grow-only mechanism as DataVolGrowGB. 0 = skip (keep the golden's small size — the volume is still
|
||||
// a separate mount, so the controller's "not a separate drive" warning clears regardless of grow).
|
||||
// SysDataGrowGB is a COMPATIBILITY INPUT since agent v0.120.0 (R-165). There is no longer a second
|
||||
// volume to grow — but `felhom.eu/scripts/felhom-host-install.sh` computes and passes
|
||||
// `-sysdata-grow` (its step_grows derives both numbers from the thin pool's free space), and an
|
||||
// installer and an agent do not upgrade in the same instant.
|
||||
//
|
||||
// SO ITS GiB ARE FOLDED INTO THE SINGLE VOLUME'S GROW RATHER THAN DROPPED. Dropping them would
|
||||
// silently shrink every appliance by the user-data share — on the ≥300 GiB branch that is 42 of
|
||||
// 250 GiB — which is exactly the "a knob that silently does nothing" outcome R-165 was told to
|
||||
// avoid. Folding keeps total capacity identical whichever installer version runs.
|
||||
SysDataGrowGB int
|
||||
// SysDataMount is the mpN slot of the golden's user-data volume to grow; "" → DefaultSysDataMount ("mp1").
|
||||
// SysDataMount is RETIRED and ignored (see DefaultSysDataMount). Kept so an older caller still
|
||||
// compiles; it selects nothing.
|
||||
SysDataMount string
|
||||
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
||||
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
|
||||
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
|
||||
// IslandBridge + IslandGuestAddr (R-50): when BOTH are set, the guest gets a static net1 on the
|
||||
// host-internal island bridge, so the controller reaches the agent over a fixed private address
|
||||
// that survives any LAN/DHCP/site move (the F1 fix). Empty (default) = no net1, byte-for-byte the
|
||||
// pre-R-50 config. Set from cfg.LocalAPI (island_bridge/island_guest_addr) at both call sites.
|
||||
IslandBridge string // e.g. "vmbr9"
|
||||
IslandGuestAddr string // guest net1 CIDR, e.g. "169.254.253.2/30"
|
||||
}
|
||||
|
||||
// BringUpResult is the outcome. It reuses the restore-test's WARNINGS surface
|
||||
@@ -401,12 +418,20 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
||||
// online (storage-split B4); its OWN call like the rootfs resize. The volume + baked images
|
||||
// came in with the restore, so we grow it rather than attach a fresh one that would shadow
|
||||
// the baked images.
|
||||
if spec.DataVolGrowGB > 0 {
|
||||
//
|
||||
// R-165: ONE volume, therefore ONE grow. `SysDataGrowGB` is FOLDED IN here rather than driving
|
||||
// a second resize — see its field comment. This is the only arithmetic the merge added.
|
||||
growGB := spec.DataVolGrowGB + spec.SysDataGrowGB
|
||||
if growGB > 0 {
|
||||
mount := spec.DataVolMount
|
||||
if mount == "" {
|
||||
mount = DefaultDataVolMount
|
||||
}
|
||||
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.DataVolGrowGB))
|
||||
if spec.SysDataGrowGB > 0 {
|
||||
e.logger.Info("bring-up: folding the retired sys-data grow into the single data volume (R-165)",
|
||||
"data_grow_gb", spec.DataVolGrowGB, "sysdata_grow_gb", spec.SysDataGrowGB, "total_gb", growGB, "mount", mount)
|
||||
}
|
||||
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", growGB))
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize (%s): %w", mount, err)
|
||||
return
|
||||
@@ -417,25 +442,9 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
||||
}
|
||||
}
|
||||
|
||||
// 4c. Grow the golden-carried SSD user-data volume (mp1, /mnt/sys_drive = the controller's
|
||||
// system_data_path) to the per-customer target. Same shape as the Docker-data grow: grow-only,
|
||||
// online, its OWN call. The volume came in with the restore (separate mount, backup=1), so we
|
||||
// grow it rather than attach a fresh one.
|
||||
if spec.SysDataGrowGB > 0 {
|
||||
mount := spec.SysDataMount
|
||||
if mount == "" {
|
||||
mount = DefaultSysDataMount
|
||||
}
|
||||
supid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.SysDataGrowGB))
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
if _, err := e.waitTask(ctx, supid, proxmox.WaitOptions{}); err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize task (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
// 4c. RETIRED (R-165). There is no second volume: the golden ships ONE, and the sys-data grow is
|
||||
// folded into 4b above. Deliberately left as a comment rather than silently vanishing, so a
|
||||
// reader of a v0.119.0 archive's provision log can see where the second resize went.
|
||||
|
||||
// 4d. DR structural-bind swap (GL-5): replace the two restore-time throwaway volumes (see the
|
||||
// restore call) with the REAL host binds, then delete the displaced volumes so a KEPT DR
|
||||
@@ -606,6 +615,15 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
|
||||
params["net0"] = withoutHwaddr(net0) // omit hwaddr → PVE generates a fresh MAC (F1)
|
||||
}
|
||||
}
|
||||
// R-50 island NIC: attach a static net1 on the host-internal bridge so the control plane
|
||||
// (controller→agent local API) rides a fixed private address, immune to any LAN/DHCP/site move.
|
||||
// Both modes: a provisioned guest AND a DR-restored guest need to reach the island-bound agent on
|
||||
// the target host. No hwaddr → PVE mints a fresh per-guest MAC (the /30 is one guest per host, so
|
||||
// a MAC would not collide either way, but a fresh one keeps net1 symmetric with net0). Additive:
|
||||
// omitted entirely when the island is not configured, keeping non-island hosts unchanged.
|
||||
if strings.TrimSpace(spec.IslandBridge) != "" && strings.TrimSpace(spec.IslandGuestAddr) != "" {
|
||||
params["net1"] = fmt.Sprintf("name=eth1,bridge=%s,ip=%s", spec.IslandBridge, spec.IslandGuestAddr)
|
||||
}
|
||||
if spec.Mode == ModeProvision && spec.Hostname != "" {
|
||||
params["hostname"] = spec.Hostname
|
||||
}
|
||||
|
||||
@@ -165,6 +165,36 @@ func TestBuildBringUpConfig_ResourceCaps(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// R-50: with the island configured, bring-up attaches a static net1 on the island bridge; with it
|
||||
// unset (or half-set), NO net1 is emitted — byte-for-byte the pre-R-50 config on non-island hosts.
|
||||
// Pure-function check on buildBringUpConfig (the derivation that makes fresh installs F1-immune).
|
||||
func TestBuildBringUpConfig_IslandNIC(t *testing.T) {
|
||||
// island set → net1 present, exact shape, no hwaddr (PVE mints a fresh per-guest MAC)
|
||||
island := buildBringUpConfig(BringUpSpec{
|
||||
Mode: ModeProvision, IslandBridge: "vmbr9", IslandGuestAddr: "169.254.253.2/30",
|
||||
}, scratchCfg())
|
||||
if got, want := island["net1"], "name=eth1,bridge=vmbr9,ip=169.254.253.2/30"; got != want {
|
||||
t.Errorf("island net1 mismatch:\n got %q\nwant %q", got, want)
|
||||
}
|
||||
// DR mode too — a restored customer guest must also reach the island-bound agent on the host.
|
||||
dr := buildBringUpConfig(BringUpSpec{
|
||||
Mode: ModeDRGuestLoss, KeepMAC: true, IslandBridge: "vmbr9", IslandGuestAddr: "169.254.253.2/30",
|
||||
}, scratchCfg())
|
||||
if _, ok := dr["net1"]; !ok {
|
||||
t.Errorf("DR bring-up must also attach the island net1, got none")
|
||||
}
|
||||
// island unset → NO net1 key (non-island hosts unchanged; the pre-R-50 default)
|
||||
none := buildBringUpConfig(BringUpSpec{Mode: ModeProvision}, scratchCfg())
|
||||
if v, ok := none["net1"]; ok {
|
||||
t.Errorf("net1 must be ABSENT when the island is not configured, got %q", v)
|
||||
}
|
||||
// half-configured (bridge only) → still no net1 (all-or-nothing; config.Validate rejects the config too)
|
||||
half := buildBringUpConfig(BringUpSpec{Mode: ModeProvision, IslandBridge: "vmbr9"}, scratchCfg())
|
||||
if v, ok := half["net1"]; ok {
|
||||
t.Errorf("net1 must be ABSENT when only the bridge is set, got %q", v)
|
||||
}
|
||||
}
|
||||
|
||||
// Both restore sites allocate the guest INTO the felhom pool (SPIKE 3b): the provision bring-up
|
||||
// threads spec.Pool, and the restore-test hardcodes DefaultPool — else a pool-scoped token 403s on
|
||||
// the created guest's config/start/destroy. Asserts via the fakeAPI's captured RestoreLXCOptions.
|
||||
@@ -235,10 +265,16 @@ func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The golden-carried SSD user-data volume (/mnt/sys_drive) is grown via a SEPARATE resize on its
|
||||
// mpN slot (mp1), independent of the rootfs and Docker-data grows. With SysDataGrowGB=0 NO mp1
|
||||
// resize is issued (the volume stays at the golden size, still a separate mount).
|
||||
func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
|
||||
// R-165 RETARGETED THIS TEST, and the retarget IS the contract change. There is no longer a second
|
||||
// volume, so `SysDataGrowGB` no longer drives its own resize on mp1 — its GiB are FOLDED INTO the
|
||||
// single volume's grow.
|
||||
//
|
||||
// FOLDED, NOT DROPPED, and that is the whole point. `felhom-host-install.sh` computes and passes
|
||||
// `-sysdata-grow` from the thin pool's free space, and an installer and an agent do not upgrade in
|
||||
// the same instant. Dropping the value would silently shrink every appliance built by an older
|
||||
// installer by the user-data share — 42 of 250 GiB on the standard branch — which is precisely the
|
||||
// "a knob that silently does nothing" outcome this work was told to avoid.
|
||||
func TestRunBringUp_StorageSplit_SysDataGrowIsFoldedIn(t *testing.T) {
|
||||
const vmid = 8051
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
|
||||
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||
@@ -247,26 +283,29 @@ func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
|
||||
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
|
||||
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051",
|
||||
DataVolGrowGB: 240, SysDataGrowGB: 42, // grows mp0 AND mp1 (DefaultSysDataMount)
|
||||
DataVolGrowGB: 240, SysDataGrowGB: 42, // ONE volume: 240 + 42 = 282
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
t.Fatalf("provision must pass, got %+v", res)
|
||||
}
|
||||
// TWO resizes here: Docker-data mp0 +240G and the user-data volume mp1 +42G (no rootfs grow).
|
||||
if len(api.resizes) != 2 {
|
||||
t.Fatalf("expected data-volume + sys-data resizes, got %+v", api.resizes)
|
||||
// EXACTLY ONE resize. A second one would mean an mp1 the golden no longer ships.
|
||||
if len(api.resizes) != 1 {
|
||||
t.Fatalf("expected exactly ONE data-volume resize (there is no mp1 since R-165), got %+v", api.resizes)
|
||||
}
|
||||
var sawData, sawSys bool
|
||||
for _, r := range api.resizes {
|
||||
if r.disk == "mp0" && r.size == "+240G" {
|
||||
sawData = true
|
||||
}
|
||||
if r.disk == "mp1" && r.size == "+42G" {
|
||||
sawSys = true
|
||||
}
|
||||
r := api.resizes[0]
|
||||
if r.disk != "mp0" {
|
||||
t.Fatalf("resized %q, want mp0 — the single data volume", r.disk)
|
||||
}
|
||||
if !sawData || !sawSys {
|
||||
t.Errorf("want mp0 +240G AND mp1 +42G, got %+v", api.resizes)
|
||||
if r.size != "+282G" {
|
||||
t.Fatalf("resized %s, want +282G (240 data + 42 folded sys-data). Anything less means the "+
|
||||
"retired knob's GiB were DROPPED, silently shrinking every appliance an older "+
|
||||
"felhom-host-install.sh provisions", r.size)
|
||||
}
|
||||
for _, rr := range api.resizes {
|
||||
if rr.disk == "mp1" {
|
||||
t.Fatalf("an mp1 resize was issued (%+v) — the golden ships no second volume, so this "+
|
||||
"would fail on a real box", rr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -198,7 +198,7 @@ func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestS
|
||||
launched := false
|
||||
defer func() {
|
||||
if launched {
|
||||
e.teardownScratch(ctx, base)
|
||||
e.teardownScratch(ctx, base, spec.ScratchMin, spec.ScratchMax)
|
||||
return
|
||||
}
|
||||
e.append(withState(base, OpFailed))
|
||||
@@ -231,6 +231,12 @@ func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestS
|
||||
// Pool=DefaultPool so the scratch guest is created INTO the felhom pool — else a pool-scoped
|
||||
// token 403s on the scratch guest's config/start/destroy (SPIKE residual #2).
|
||||
VMID: vmid, Archive: spec.Archive, Storage: spec.RestoreStorage, MountOverrides: mountOverrides, Pool: DefaultPool,
|
||||
// onboot=0 from the instant the guest exists. Step 2 below link-downs every NIC before the
|
||||
// guest is ever started, so the NORMAL path cannot conflict with the live source. This
|
||||
// covers the ABNORMAL path: a restore that fails before step 2 (e.g. the wait expiring) can
|
||||
// leave a scratch carrying the source's `onboot: 1` plus its MAC/static island IP/hostname —
|
||||
// which a host reboot would then start alongside the original. Observed live 2026-07-26.
|
||||
ConfigOverrides: map[string]string{"onboot": "0"},
|
||||
})
|
||||
if err != nil {
|
||||
if pveAlreadyExists(err) {
|
||||
@@ -440,7 +446,12 @@ func sizeToGB(s string) int {
|
||||
|
||||
// teardownScratch destroys the scratch guest (benign, gated) and records the entry terminal.
|
||||
// On any teardown failure it leaves the entry in-flight so Recover reaps the guest later.
|
||||
func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) {
|
||||
//
|
||||
// F-LEAK (Campaign 8): the API destroy is tried FIRST and is the normal path. It fails on a scratch
|
||||
// left by a FAILED restore, because such a guest never joined /pool/felhom and the token's
|
||||
// VM.Allocate lives there — so a band-scoped fallback through the fenced root path follows. See
|
||||
// proxmox.DestroyScratchLXC for the two API-side fixes that were built and refuted live.
|
||||
func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry, scratchMin, scratchMax int) {
|
||||
// Cancel-immune + bounded, so a shutdown mid-test still tears down.
|
||||
tctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
|
||||
defer cancel()
|
||||
@@ -453,6 +464,14 @@ func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) {
|
||||
}
|
||||
upid, err := e.api.DestroyLXC(tctx, base.VMID)
|
||||
if err != nil {
|
||||
// A 403 "missing privilege VM.Allocate" here means this scratch is not a felhom-pool member: a
|
||||
// FAILED restore never completes the `--pool` association, and the token's VM.Allocate is
|
||||
// granted at /pool/felhom. Fall back to the band-scoped fenced destroy — WITHOUT it the guest
|
||||
// leaks and holds its disks until a human removes it.
|
||||
if e.destroyScratchPrivileged(tctx, base.VMID, scratchMin, scratchMax, err) {
|
||||
e.append(withState(base, OpSucceeded))
|
||||
return
|
||||
}
|
||||
e.logger.Error("restore-test: scratch teardown failed; left for Recover", "vmid", base.VMID, "err", err)
|
||||
return
|
||||
}
|
||||
@@ -555,3 +574,37 @@ func withUPID(base JournalEntry, upid string, state OpState) JournalEntry {
|
||||
base.At = time.Now().UTC()
|
||||
return base
|
||||
}
|
||||
|
||||
// destroyScratchPrivileged is the F-LEAK fallback: destroy a stranded scratch through the fenced root
|
||||
// path when the API token cannot. Reports whether the guest is gone.
|
||||
//
|
||||
// It refuses unless the guest is BOTH agent-created scratch provenance (this journal entry) and inside
|
||||
// the configured band. That is the innermost of three checks — sudoers matches the vmid literally and
|
||||
// proxmox.DestroyScratchLXC re-checks the band — because this op DESTROYS and the band must not rest
|
||||
// on a single guard.
|
||||
func (e *Engine) destroyScratchPrivileged(ctx context.Context, vmid, bandMin, bandMax int, apiErr error) bool {
|
||||
if e.hostRun == nil {
|
||||
e.logger.Warn("restore-test: no host-root runner wired — cannot reclaim the stranded scratch",
|
||||
"vmid", vmid, "api_err", apiErr)
|
||||
return false
|
||||
}
|
||||
if bandMin <= 0 || bandMax < bandMin {
|
||||
e.logger.Error("restore-test: scratch band is not configured — refusing the privileged teardown",
|
||||
"vmid", vmid, "min", bandMin, "max", bandMax)
|
||||
return false
|
||||
}
|
||||
if vmid < bandMin || vmid > bandMax {
|
||||
e.logger.Error("restore-test: refusing the privileged teardown — vmid is outside the scratch band",
|
||||
"vmid", vmid, "min", bandMin, "max", bandMax)
|
||||
return false
|
||||
}
|
||||
e.logger.Warn("restore-test: API teardown failed (stranded scratch is in no pool) — reclaiming via the fenced root path",
|
||||
"vmid", vmid, "api_err", apiErr)
|
||||
if err := proxmox.NewPrivileged(e.hostRun, "").DestroyScratchLXC(ctx, vmid, bandMin, bandMax); err != nil {
|
||||
e.logger.Error("restore-test: privileged scratch teardown ALSO failed; left for Recover",
|
||||
"vmid", vmid, "err", err)
|
||||
return false
|
||||
}
|
||||
e.logger.Warn("restore-test: stranded scratch guest reclaimed via the fenced root path", "vmid", vmid)
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -667,3 +667,27 @@ func TestRunRestoreTest_RefusalsPropagate(t *testing.T) {
|
||||
t.Fatalf("never restore a partial guest to verify it: %+v", apiBind.restores)
|
||||
}
|
||||
}
|
||||
|
||||
// A leaked scratch guest must never AUTO-START. The normal path link-downs every NIC before boot
|
||||
// (TestRestoreTest… above), so the source can never be conflicted with on the happy path. This
|
||||
// covers the abnormal one: a restore that fails BEFORE the link-down step leaves a scratch carrying
|
||||
// the SOURCE guest's config verbatim — including `onboot: 1`, its MAC, its static island IP and its
|
||||
// hostname. Observed live 2026-07-26, when a wait-timeout left exactly such a guest on demo-felhom.
|
||||
// onboot=0 is therefore set AT RESTORE TIME, not after: after is too late for the path that leaks.
|
||||
func TestRestoreTest_RestoreSetsOnbootZero(t *testing.T) {
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
|
||||
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||
defer q.Close()
|
||||
|
||||
_ = e.RunRestoreTest(context.Background(), RestoreTestSpec{
|
||||
Archive: "local:backup/x.tar.zst", RestoreStorage: "local-lvm",
|
||||
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
|
||||
})
|
||||
if len(api.restores) != 1 {
|
||||
t.Fatalf("want one restore, got %+v", api.restores)
|
||||
}
|
||||
if got := api.restores[0].ConfigOverrides["onboot"]; got != "0" {
|
||||
t.Fatalf("the restore MUST set onboot=0 so a leaked scratch cannot auto-start; got %q (%#v)",
|
||||
got, api.restores[0].ConfigOverrides)
|
||||
}
|
||||
}
|
||||
|
||||
+67
-13
@@ -58,6 +58,9 @@ type observed struct {
|
||||
known KnownTarget
|
||||
src proxmox.Storage
|
||||
cat storageCategory
|
||||
// smartHint (v0.95.0) is a SMART-ONLY whole-disk device for a dir-storage whose own backing is
|
||||
// empty (the builtin `local` on the shared LVM root). Never assigned to BackingDevice/durable_id.
|
||||
smartHint string
|
||||
}
|
||||
|
||||
// Observe builds the reported []hub.StorageTarget. A non-nil error means the Proxmox read
|
||||
@@ -84,13 +87,22 @@ func (o *Observer) enrich(ctx context.Context, ob observed) hub.StorageTarget {
|
||||
if o.ops == nil {
|
||||
return t
|
||||
}
|
||||
// SMART: only for dir-backed targets with a resolvable whole-disk device.
|
||||
if ob.cat == catDir && t.BackingDevice != "" {
|
||||
if dev, ok := smartDeviceFor(t.BackingDevice); ok {
|
||||
if sm, err := o.ops.SMART(ctx, dev); err != nil {
|
||||
o.logger.Warn("storage: SMART read failed; health UNKNOWN", "device", dev, "err", err)
|
||||
} else {
|
||||
t.Smart = sm
|
||||
// SMART: for dir-backed targets. The device is the target's own backing (a USB/local-dir exact
|
||||
// mount) OR, for a dir on a shared filesystem whose backing is deliberately empty (the builtin
|
||||
// `local` on the LVM root — removable-safety guard in build), the SMART-only hint build() resolved
|
||||
// from the containing filesystem. smartDeviceFor then resolves dm/LVM/partition to the whole disk.
|
||||
if ob.cat == catDir {
|
||||
smartDev := t.BackingDevice
|
||||
if smartDev == "" {
|
||||
smartDev = ob.smartHint
|
||||
}
|
||||
if smartDev != "" {
|
||||
if dev, ok := smartDeviceFor(smartDev); ok {
|
||||
if sm, err := o.ops.SMART(ctx, dev); err != nil {
|
||||
o.logger.Warn("storage: SMART read failed; health UNKNOWN", "device", dev, "err", err)
|
||||
} else {
|
||||
t.Smart = sm
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -223,9 +235,21 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
Content: s.Content,
|
||||
MountPath: mountPath,
|
||||
BackingDevice: backingDevice,
|
||||
ClassHint: classHint,
|
||||
Role: "", // hub-owned; not derivable from a Proxmox def (slice 10)
|
||||
Smart: hub.SmartSummary{Health: hub.SmartUnknown},
|
||||
// R-116: the CONFIGURED path, carried verbatim and never resolved. This is emphatically NOT the
|
||||
// fallthrough the comment above forbids — that prohibition is about resolving a device or a UUID
|
||||
// from the CONTAINING filesystem when the target is not its own mount, which would hand back
|
||||
// root's identity and mis-target a DR re-attach. `s.Path` is the storage's own declaration of
|
||||
// where it lives; it identifies nothing but itself, and it is not used for device or UUID
|
||||
// resolution anywhere. MountPath stays empty when the mount is gone, which is the truth.
|
||||
ConfigPath: s.Path,
|
||||
// R-106: the CONFIGURED PBS namespace, carried verbatim from storage.cfg. Empty for every
|
||||
// non-pbs storage (Proxmox only emits it on pbs), and empty for a pbs storage in the root
|
||||
// namespace — the DR recipe distinguishes those two cases by the storage's TYPE, never by
|
||||
// guessing from this string.
|
||||
PBSNamespace: s.Namespace,
|
||||
ClassHint: classHint,
|
||||
Role: "", // hub-owned; not derivable from a Proxmox def (slice 10)
|
||||
Smart: hub.SmartSummary{Health: hub.SmartUnknown},
|
||||
}
|
||||
|
||||
// Thin-pool DATA fill: surfaced prominently for lvmthin (metadata fill is Phase B/lvs).
|
||||
@@ -238,10 +262,23 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
}
|
||||
}
|
||||
|
||||
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
|
||||
// builtin `local` on the LVM root) has empty backing by design, yet we can still read the PHYSICAL
|
||||
// disk's SMART by resolving its containing filesystem. Gated to catDir + no own backing + reachable,
|
||||
// so an UNPLUGGED removable (disconnected) never reads root's SMART, and a mounted removable uses
|
||||
// its own backing instead.
|
||||
smartHint := ""
|
||||
if category == catDir && backingDevice == "" && reachable {
|
||||
if dev, ok := containingMountDevice(mounts, s.Path); ok {
|
||||
smartHint = dev
|
||||
}
|
||||
}
|
||||
|
||||
return observed{
|
||||
target: tgt,
|
||||
src: s,
|
||||
cat: category,
|
||||
target: tgt,
|
||||
src: s,
|
||||
cat: category,
|
||||
smartHint: smartHint,
|
||||
known: KnownTarget{
|
||||
Name: s.Storage,
|
||||
Type: typ,
|
||||
@@ -260,6 +297,12 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
// smartctl (which targets the disk, not the partition). Returns ok=false when the result
|
||||
// isn't a recognized raw disk (e.g. device-mapper / LVM), so SMART is simply skipped.
|
||||
func smartDeviceFor(device string) (string, bool) {
|
||||
// Device-mapper / LVM (Fix A, v0.95.0): resolve to the single backing whole disk via sysfs
|
||||
// slaves. This is what finally covers the system SSD under `pve-root`. The sysfs resolution IS
|
||||
// the existence check, so we do NOT re-run ValidateSMARTDevice on its result.
|
||||
if strings.HasPrefix(device, "/dev/dm-") || strings.HasPrefix(device, "/dev/mapper/") {
|
||||
return dmWholeDisk(device)
|
||||
}
|
||||
dev := device
|
||||
if m := reNVMePart.FindStringSubmatch(device); m != nil {
|
||||
dev = m[1] // /dev/nvme0n1p2 -> /dev/nvme0n1
|
||||
@@ -445,5 +488,16 @@ func mergeConfig(node, cluster proxmox.Storage) proxmox.Storage {
|
||||
if node.Content == "" {
|
||||
node.Content = cluster.Content
|
||||
}
|
||||
// R-106: the pbs namespace. `NodeStorage` does not return it AT ALL — it is cluster-config only —
|
||||
// so without this line `Namespace` is always empty on the merged entry and every consumer sees a
|
||||
// root-namespace box. Found by LIVE VALIDATION, not by the unit tests: the DR-recipe tests supply
|
||||
// StorageTarget values directly, so they never crossed this merge.
|
||||
//
|
||||
// This function is a copy-only-what-is-needed allow-list, which is exactly how the gap arose. If you
|
||||
// add a consumer of any other type-specific field (`Username` is the remaining unmerged one), add it
|
||||
// here too and pin it in TestMergeConfig_CarriesPBSNamespace's table.
|
||||
if node.Namespace == "" {
|
||||
node.Namespace = cluster.Namespace
|
||||
}
|
||||
return node
|
||||
}
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// Fix A (A2): the builtin `local` dir lives on the LVM root, so its backing is empty by design — but
|
||||
// enrich now resolves the containing filesystem (/ → /dev/mapper/pve-root) and, via the dm sysfs
|
||||
// slaves, the physical disk (/dev/sda). The system disk stops reading "Nincs adat".
|
||||
// Red-proof: drop the `smartDev = ob.smartHint` fallback in enrich → local stays UNKNOWN and SMART
|
||||
// is never called on /dev/sda.
|
||||
func TestObserve_SystemDirSMARTViaContainingFS(t *testing.T) {
|
||||
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
|
||||
|
||||
api := &fakeStorageAPI{
|
||||
node: "demo-felhom",
|
||||
cluster: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz"}},
|
||||
nodeSt: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1}},
|
||||
}
|
||||
host := &fakeHostReader{
|
||||
mounts: []Mount{{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"}},
|
||||
}
|
||||
ops := &fakeHostOps{smartByDevice: map[string]hub.SmartSummary{"/dev/sda": {Health: hub.SmartPassed, ModelName: strptr("AirDisk 512GB SSD")}}}
|
||||
|
||||
got, err := NewObserver(api, host, ops, quietLogger()).Observe(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
local := byName(got)["local"]
|
||||
if local.Smart.Health != hub.SmartPassed {
|
||||
t.Errorf("system disk SMART not enriched via the containing fs: health=%q", local.Smart.Health)
|
||||
}
|
||||
if local.Smart.ModelName == nil || *local.Smart.ModelName != "AirDisk 512GB SSD" {
|
||||
t.Errorf("model not carried: %v", local.Smart.ModelName)
|
||||
}
|
||||
// SMART must have run on the resolved PHYSICAL disk, never the dm/mapper node.
|
||||
if len(ops.smartDevices) != 1 || ops.smartDevices[0] != "/dev/sda" {
|
||||
t.Errorf("SMART should target /dev/sda, got %v", ops.smartDevices)
|
||||
}
|
||||
// The backing device / durable_id must stay untouched by the SMART-only resolution.
|
||||
if local.BackingDevice != "" {
|
||||
t.Errorf("system-dir SMART resolution leaked into BackingDevice: %q", local.BackingDevice)
|
||||
}
|
||||
}
|
||||
|
||||
// The watchdog Known() path MUST remain enrich-free (its slow root-shelling reads are the reason it
|
||||
// exists as a separate fast path). Known must never invoke SMART.
|
||||
// Red-proof: route Known through enrich → smartDevices is non-empty and this fails.
|
||||
func TestKnown_NeverInvokesSMART(t *testing.T) {
|
||||
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
|
||||
api := &fakeStorageAPI{
|
||||
node: "demo-felhom",
|
||||
cluster: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz"}},
|
||||
nodeSt: []proxmox.Storage{{Storage: "local", Type: "dir", Path: "/var/lib/vz", Active: 1}},
|
||||
}
|
||||
host := &fakeHostReader{mounts: []Mount{{Device: "/dev/mapper/pve-root", MountPoint: "/", FSType: "ext4"}}}
|
||||
ops := &fakeHostOps{smartByDevice: map[string]hub.SmartSummary{"/dev/sda": {Health: hub.SmartPassed}}}
|
||||
|
||||
if _, err := NewObserver(api, host, ops, quietLogger()).Known(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(ops.smartDevices) != 0 {
|
||||
t.Errorf("Known() invoked SMART %d time(s) — it must stay enrich-free: %v", len(ops.smartDevices), ops.smartDevices)
|
||||
}
|
||||
}
|
||||
|
||||
func strptr(s string) *string { return &s }
|
||||
@@ -292,3 +292,66 @@ func TestObserve_MountReadFailureDegradesNotFatal(t *testing.T) {
|
||||
t.Errorf("lvmthin still derivable without mounts: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// R-106 — the pbs namespace must survive the cluster/node merge.
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
// TestObserve_CarriesPBSNamespaceThroughMerge pins the gap that shipped in agent v0.118.0 and was caught
|
||||
// by LIVE VALIDATION rather than by tests: `mergeConfig` copies a hand-listed set of type-specific fields
|
||||
// from the CLUSTER config onto the NODE entry, and `Namespace` was not on that list. `NodeStorage` does
|
||||
// not return the namespace at all — it is cluster-config only — so `StorageTarget.PBSNamespace` was
|
||||
// always empty and the DR recipe reported the root namespace on every per-customer box, exactly the
|
||||
// R-106 symptom the fix was supposed to remove.
|
||||
//
|
||||
// The DR-recipe tests could not catch it: they construct StorageTarget values directly, so nothing
|
||||
// crossed this merge. This test drives the REAL Observe path with the split PVE returns reproduced —
|
||||
// namespace present in the cluster list, absent from the node list, which is what PVE actually does.
|
||||
func TestObserve_CarriesPBSNamespaceThroughMerge(t *testing.T) {
|
||||
api := &fakeStorageAPI{
|
||||
node: "demo-felhom",
|
||||
// Cluster config: carries the type-specific fields, as /storage does.
|
||||
cluster: []proxmox.Storage{{
|
||||
Storage: "felhom-pbs", Type: "pbs", Content: "backup",
|
||||
Server: "10.77.0.1", Datastore: "felhom-offsite", Namespace: "demo-felhom",
|
||||
}},
|
||||
// Node entry: live usage + active flag, and NO namespace — the shape that made the bug invisible.
|
||||
nodeSt: []proxmox.Storage{{
|
||||
Storage: "felhom-pbs", Type: "pbs", Content: "backup",
|
||||
Total: 100, Used: 10, Avail: 90, Active: 1, Enabled: 1,
|
||||
}},
|
||||
}
|
||||
o := NewObserver(api, &fakeHostReader{}, nil, quietLogger())
|
||||
|
||||
targets, err := o.Observe(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Observe: %v", err)
|
||||
}
|
||||
if len(targets) != 1 {
|
||||
t.Fatalf("want 1 target, got %d", len(targets))
|
||||
}
|
||||
if got := targets[0].PBSNamespace; got != "demo-felhom" {
|
||||
t.Errorf("PBSNamespace=%q, want %q — the namespace was lost in mergeConfig, so the DR recipe "+
|
||||
"reports the root namespace on a per-customer box (R-106)", got, "demo-felhom")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMergeConfig_CarriesPBSNamespace is the direct table over the merge itself: a node entry that omits
|
||||
// a field takes the cluster's value, and a node entry that HAS one keeps its own (never clobbered).
|
||||
func TestMergeConfig_CarriesPBSNamespace(t *testing.T) {
|
||||
cluster := proxmox.Storage{Storage: "felhom-pbs", Type: "pbs", Namespace: "demo-hp", Datastore: "felhom-offsite"}
|
||||
|
||||
// Node omits the namespace (the real PVE shape) → it must be filled from the cluster config.
|
||||
if got := mergeConfig(proxmox.Storage{Storage: "felhom-pbs"}, cluster).Namespace; got != "demo-hp" {
|
||||
t.Errorf("namespace absent on the node entry: got %q, want it merged from the cluster config", got)
|
||||
}
|
||||
// Node already has one → keep it (the merge is fill-if-empty, never overwrite).
|
||||
nodeOwn := proxmox.Storage{Storage: "felhom-pbs", Namespace: "node-wins"}
|
||||
if got := mergeConfig(nodeOwn, cluster).Namespace; got != "node-wins" {
|
||||
t.Errorf("merge clobbered the node's own namespace: got %q", got)
|
||||
}
|
||||
// No cluster row at all → the node entry passes through untouched.
|
||||
if got := mergeConfig(proxmox.Storage{Storage: "x", Namespace: "keep"}, proxmox.Storage{}).Namespace; got != "keep" {
|
||||
t.Errorf("empty cluster row altered the node entry: got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
// presence so an absent section (e.g. NVMe fields on a SATA disk, or no SMART at all on a
|
||||
// USB bridge) decodes cleanly to nil and we degrade to UNKNOWN.
|
||||
type smartctlJSON struct {
|
||||
ModelName *string `json:"model_name"`
|
||||
SmartStatus *struct {
|
||||
Passed bool `json:"passed"`
|
||||
} `json:"smart_status"`
|
||||
@@ -64,6 +65,9 @@ func parseSMART(raw []byte) hub.SmartSummary {
|
||||
s.Health = hub.SmartFailing
|
||||
}
|
||||
}
|
||||
if j.ModelName != nil && *j.ModelName != "" {
|
||||
s.ModelName = j.ModelName
|
||||
}
|
||||
if j.Temperature != nil && j.Temperature.Current != nil {
|
||||
s.TemperatureC = j.Temperature.Current
|
||||
}
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// sysBlockRoot is the sysfs block directory. A package var so tests can point it at a fixture tree.
|
||||
var sysBlockRoot = "/sys/block"
|
||||
|
||||
// dmWholeDisk resolves a device-mapper / LVM device to its SINGLE backing whole disk, recursing
|
||||
// through stacked dm layers via /sys/block/<dm>/slaves (Fix A, SPIKE-smart-coverage-2026-07-25).
|
||||
// Returns ok=false when the device is not dm, sysfs is missing, there are no slaves, or the slaves
|
||||
// span MORE THAN ONE physical disk — in that last case we deliberately skip rather than guess which
|
||||
// of two disks to SMART (e.g. a mirrored LV).
|
||||
func dmWholeDisk(device string) (string, bool) {
|
||||
name := dmName(device)
|
||||
if name == "" {
|
||||
return "", false
|
||||
}
|
||||
disks := map[string]bool{}
|
||||
if !collectSlaveDisks(name, disks, 0) {
|
||||
return "", false
|
||||
}
|
||||
if len(disks) != 1 {
|
||||
return "", false // no disk, or an ambiguous multi-disk dm — never guess
|
||||
}
|
||||
for d := range disks {
|
||||
return "/dev/" + d, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// dmName maps a dm device path to its sysfs name (dm-N). Handles /dev/dm-N (and a bare dm-N)
|
||||
// directly, and /dev/mapper/<name> by matching /sys/block/dm-*/dm/name.
|
||||
func dmName(device string) string {
|
||||
base := filepath.Base(device)
|
||||
if strings.HasPrefix(base, "dm-") {
|
||||
return base
|
||||
}
|
||||
if strings.HasPrefix(device, "/dev/mapper/") {
|
||||
entries, err := os.ReadDir(sysBlockRoot)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, e := range entries {
|
||||
if !strings.HasPrefix(e.Name(), "dm-") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(sysBlockRoot, e.Name(), "dm", "name"))
|
||||
if err == nil && strings.TrimSpace(string(b)) == base {
|
||||
return e.Name()
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// collectSlaveDisks fills `disks` with the whole-disk names backing dm `name`, recursing through
|
||||
// nested dm. Returns false on missing sysfs, no slaves, or excessive nesting (loop guard).
|
||||
func collectSlaveDisks(name string, disks map[string]bool, depth int) bool {
|
||||
if depth > 8 {
|
||||
return false
|
||||
}
|
||||
entries, err := os.ReadDir(filepath.Join(sysBlockRoot, name, "slaves"))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
return false
|
||||
}
|
||||
for _, e := range entries {
|
||||
s := e.Name()
|
||||
if strings.HasPrefix(s, "dm-") {
|
||||
if !collectSlaveDisks(s, disks, depth+1) {
|
||||
return false
|
||||
}
|
||||
continue
|
||||
}
|
||||
disks[wholeDiskName(s)] = true
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// wholeDiskName strips a partition suffix to the whole-disk name (sda3→sda, nvme0n1p3→nvme0n1).
|
||||
func wholeDiskName(part string) string {
|
||||
if m := reNVMePartName.FindStringSubmatch(part); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
if m := reSDPartName.FindStringSubmatch(part); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
return part
|
||||
}
|
||||
|
||||
var (
|
||||
reNVMePartName = regexp.MustCompile(`^(nvme[0-9]+n[0-9]+)p[0-9]+$`)
|
||||
reSDPartName = regexp.MustCompile(`^((?:sd|hd|vd)[a-z]+)[0-9]+$`)
|
||||
)
|
||||
|
||||
// containingMountDevice returns the device of the mount whose mountpoint is the LONGEST prefix of
|
||||
// path — the filesystem that actually holds `path`. Used ONLY to pick a whole-disk device for a
|
||||
// SMART read of a dir-storage that lives inside a shared filesystem (the builtin `local` on the LVM
|
||||
// root); it never feeds durable_id / backing_device (which stay empty for such targets by design —
|
||||
// the removable-safety guard in build()).
|
||||
func containingMountDevice(mounts []Mount, path string) (string, bool) {
|
||||
clean := cleanMountPath(path)
|
||||
best, bestLen := "", -1
|
||||
for _, m := range mounts {
|
||||
if m.Device == "" {
|
||||
continue
|
||||
}
|
||||
mp := cleanMountPath(m.MountPoint)
|
||||
if mp == clean || mp == "/" || strings.HasPrefix(clean, strings.TrimRight(mp, "/")+"/") {
|
||||
if len(mp) > bestLen {
|
||||
best, bestLen = m.Device, len(mp)
|
||||
}
|
||||
}
|
||||
}
|
||||
return best, best != ""
|
||||
}
|
||||
|
||||
// SmartReader reads per-disk SMART for a backing device, resolving partition / dm / LVM down to the
|
||||
// whole disk. It exists so the localapi /disks UNION path (registry/USB drives that skip Observe's
|
||||
// enrich) gets the SAME SMART read the dir targets get, without duplicating smartDeviceFor (Fix B).
|
||||
// A zero-value summary (Health "") means "could not read" (nil ops, unresolvable device, or a read
|
||||
// error) — distinct from a read that returned UNKNOWN — so the caller can omit it exactly like the
|
||||
// dir path does.
|
||||
type SmartReader struct{ ops HostOps }
|
||||
|
||||
// NewSmartReader wraps a HostOps for the localapi union path.
|
||||
func NewSmartReader(ops HostOps) *SmartReader { return &SmartReader{ops: ops} }
|
||||
|
||||
// SMARTForBacking reads SMART for backingDevice (partition/dm/whole-disk). Never returns an error;
|
||||
// on any failure the summary's Health is "".
|
||||
func (r *SmartReader) SMARTForBacking(ctx context.Context, backingDevice string) hub.SmartSummary {
|
||||
if r == nil || r.ops == nil {
|
||||
return hub.SmartSummary{}
|
||||
}
|
||||
dev, ok := smartDeviceFor(backingDevice)
|
||||
if !ok {
|
||||
return hub.SmartSummary{}
|
||||
}
|
||||
sm, err := r.ops.SMART(ctx, dev)
|
||||
if err != nil {
|
||||
return hub.SmartSummary{}
|
||||
}
|
||||
return sm
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// fixtureSysfs builds a /sys/block-shaped tree and points sysBlockRoot at it. `slaves` maps a dm
|
||||
// name to its slave entries; `dmNames` maps a dm name to its /dm/name content (for /dev/mapper/*).
|
||||
func fixtureSysfs(t *testing.T, slaves map[string][]string, dmNames map[string]string) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
for dm, sl := range slaves {
|
||||
for _, s := range sl {
|
||||
if err := os.MkdirAll(filepath.Join(root, dm, "slaves", s), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
for dm, name := range dmNames {
|
||||
dir := filepath.Join(root, dm, "dm")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "name"), []byte(name+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
old := sysBlockRoot
|
||||
sysBlockRoot = root
|
||||
t.Cleanup(func() { sysBlockRoot = old })
|
||||
}
|
||||
|
||||
// Fix A dm/LVM resolution. Red-proof: remove the `len(disks) != 1` all-same-disk guard in
|
||||
// dmWholeDisk → the "mirror over two disks" case resolves to one of them instead of skipping.
|
||||
func TestDMWholeDisk(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
slaves map[string][]string
|
||||
dmNames map[string]string
|
||||
in string
|
||||
want string
|
||||
ok bool
|
||||
}{
|
||||
{"single SATA slave", map[string][]string{"dm-1": {"sda3"}}, nil, "/dev/dm-1", "/dev/sda", true},
|
||||
{"single NVMe slave", map[string][]string{"dm-0": {"nvme0n1p3"}}, nil, "/dev/dm-0", "/dev/nvme0n1", true},
|
||||
{"stacked dm → one disk", map[string][]string{"dm-2": {"dm-1"}, "dm-1": {"sda3"}}, nil, "/dev/dm-2", "/dev/sda", true},
|
||||
{"mapper name → dm-1", map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"}, "/dev/mapper/pve-root", "/dev/sda", true},
|
||||
{"mirror over two disks → skip", map[string][]string{"dm-1": {"sda3", "sdb3"}}, nil, "/dev/dm-1", "", false},
|
||||
{"no slaves → skip", map[string][]string{"dm-1": {}}, nil, "/dev/dm-1", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
fixtureSysfs(t, c.slaves, c.dmNames)
|
||||
got, ok := dmWholeDisk(c.in)
|
||||
if ok != c.ok || got != c.want {
|
||||
t.Errorf("%s: dmWholeDisk(%q) = (%q,%v), want (%q,%v)", c.name, c.in, got, ok, c.want, c.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// smartDeviceFor routes dm/mapper devices through the resolver, and whole-disk/partition through the
|
||||
// regex path unchanged.
|
||||
func TestSmartDeviceFor_DMBranch(t *testing.T) {
|
||||
fixtureSysfs(t, map[string][]string{"dm-1": {"sda3"}}, map[string]string{"dm-1": "pve-root"})
|
||||
if dev, ok := smartDeviceFor("/dev/mapper/pve-root"); !ok || dev != "/dev/sda" {
|
||||
t.Errorf("smartDeviceFor(/dev/mapper/pve-root) = (%q,%v), want (/dev/sda,true)", dev, ok)
|
||||
}
|
||||
// missing sysfs → skip, not a guess
|
||||
fixtureSysfs(t, map[string][]string{}, nil)
|
||||
if _, ok := smartDeviceFor("/dev/dm-9"); ok {
|
||||
t.Error("smartDeviceFor should skip an unresolvable dm device")
|
||||
}
|
||||
}
|
||||
|
||||
func TestContainingMountDevice(t *testing.T) {
|
||||
mounts := []Mount{
|
||||
{Device: "/dev/mapper/pve-root", MountPoint: "/"},
|
||||
{Device: "/dev/sda2", MountPoint: "/boot/efi"},
|
||||
{Device: "/dev/sdb1", MountPoint: "/mnt/usb"},
|
||||
}
|
||||
// A dir inside root resolves to root's device (longest prefix wins over "/").
|
||||
if dev, ok := containingMountDevice(mounts, "/var/lib/vz"); !ok || dev != "/dev/mapper/pve-root" {
|
||||
t.Errorf("containing(/var/lib/vz) = (%q,%v), want /dev/mapper/pve-root", dev, ok)
|
||||
}
|
||||
// A path under a more-specific mount picks that mount, not root.
|
||||
if dev, ok := containingMountDevice(mounts, "/mnt/usb/data"); !ok || dev != "/dev/sdb1" {
|
||||
t.Errorf("containing(/mnt/usb/data) = (%q,%v), want /dev/sdb1", dev, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// Model capture (v0.95.0) — smartctl's model_name flows into SmartSummary; absent → nil.
|
||||
func TestParseSMART_ModelName(t *testing.T) {
|
||||
withModel := parseSMART([]byte(`{"model_name":"TOSHIBA MQ04ABF100","smart_status":{"passed":true}}`))
|
||||
if withModel.ModelName == nil || *withModel.ModelName != "TOSHIBA MQ04ABF100" {
|
||||
t.Errorf("ModelName = %v, want TOSHIBA MQ04ABF100", withModel.ModelName)
|
||||
}
|
||||
if withModel.Health != hub.SmartPassed {
|
||||
t.Errorf("health = %q, want PASSED", withModel.Health)
|
||||
}
|
||||
noModel := parseSMART([]byte(`{"smart_status":{"passed":true}}`))
|
||||
if noModel.ModelName != nil {
|
||||
t.Errorf("absent model_name should be nil, got %v", noModel.ModelName)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""agent_gates.py — THE entry point for this repo's gates. Run from the repo root:
|
||||
|
||||
python3 scripts/agent_gates.py # every gate
|
||||
python3 scripts/agent_gates.py --fast # only gates that touch no network and no container
|
||||
# runtime (what .githooks/pre-push runs)
|
||||
|
||||
Gates (all must pass; **non-zero exit on any failure**):
|
||||
|
||||
1. reuse-refs every path cited by this repo's REUSE.md still resolves
|
||||
2. published every `v<semver>` tag has a downloadable package AND a tag tree that serves
|
||||
the agent's configs (R-115). NEEDS NETWORK, so it is **not** in `--fast` and
|
||||
the pre-push hook does not run it — a push must not fail because Gitea blinked
|
||||
or because someone is offline on a train. CI runs the FULL set for exactly this
|
||||
reason: it is the machine that can afford a network check, and it is the half
|
||||
that emails when something is wrong.
|
||||
|
||||
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
|
||||
|
||||
A census of all thirteen gate scripts across the four felhom repos found one clean correlation:
|
||||
**every check a CLAUDE.md tells a person to run was passing, and two of the four nobody is told
|
||||
to run were failing** — one since 14 July. This repo was the extreme case: nothing at all ran
|
||||
against it, and its REUSE.md — 90 cited paths — was checked by no one. This file exists so the
|
||||
agent is not the one repo with nowhere to put a gate, and so the pre-push hook has the same entry
|
||||
point in all four repos. It grows when the agent grows a second check.
|
||||
|
||||
THE SHARED CHECKER. `reuse_refs_check.py` lives in ONE place — `felhom.eu/scripts/` — and is
|
||||
invoked here across the workspace at `<repo-root>/../felhom.eu/scripts/`. It is deliberately NOT
|
||||
copied into this repo: duplicating it would recreate exactly the drift it exists to detect. If the
|
||||
sibling clone is absent the gate FAILS and prints the path it tried — fail-closed, because a
|
||||
runner that quietly skips a gate is the inert-seam failure this project has shipped four times.
|
||||
|
||||
EXIT CODES. Each gate returns 0 clean / 1 convicted / 2 inconclusive. This runner exits non-zero
|
||||
if any gate is non-zero, and reports 2 distinctly as INCONCLUSIVE — an undetermined result is
|
||||
never a pass, but it is not a conviction either.
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reuse_refs_check.py")
|
||||
|
||||
# (label, absolute script path, args, fast)
|
||||
GATES = [
|
||||
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
||||
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
|
||||
def hooks_armed_note(root):
|
||||
"""Print a WARNING (never a failure) when this clone's pre-push hook is not switched on.
|
||||
|
||||
core.hooksPath is local config and a clone does not carry it, so an unarmed clone is silent
|
||||
by construction — this is the only place it becomes visible.
|
||||
"""
|
||||
try:
|
||||
val = subprocess.check_output(["git", "config", "--get", "core.hooksPath"],
|
||||
cwd=root, stderr=subprocess.DEVNULL).decode().strip()
|
||||
except Exception:
|
||||
val = ""
|
||||
norm = val.replace("\\", "/").rstrip("/")
|
||||
if norm == ".githooks" or norm.endswith("/.githooks"):
|
||||
return
|
||||
print("WARNING: this clone is UNARMED — core.hooksPath is %s, so the pre-push hook will not\n"
|
||||
" run here. Switch it on once with: git config core.hooksPath .githooks"
|
||||
% (("'" + val + "'") if val else "unset"))
|
||||
|
||||
|
||||
def run_gate(label, path, args):
|
||||
if not os.path.exists(path):
|
||||
print("\nFAIL: gate '%s' is MISSING — tried %s" % (label, path))
|
||||
print(" A missing gate is a failure, never a skip (fail-closed). The reuse-refs")
|
||||
print(" checker is shared and lives in the felhom.eu sibling clone; it is never copied.")
|
||||
return 1
|
||||
print("\n" + "=" * 78)
|
||||
print("== gate: %s (%s%s)" % (label, os.path.basename(path),
|
||||
(" " + " ".join(args)) if args else ""))
|
||||
print("=" * 78, flush=True)
|
||||
# stream the gate's own output rather than capturing it — its diagnostics are the point.
|
||||
return subprocess.call([sys.executable, path] + args, cwd=ROOT)
|
||||
|
||||
|
||||
def main(argv):
|
||||
fast = "--fast" in argv
|
||||
unknown = [a for a in argv if a != "--fast"]
|
||||
if unknown:
|
||||
print("unknown argument(s): %s" % " ".join(unknown))
|
||||
print("usage: python3 scripts/agent_gates.py [--fast]")
|
||||
return 2
|
||||
|
||||
selected = [g for g in GATES if g[3] or not fast]
|
||||
skipped = [g[0] for g in GATES if not (g[3] or not fast)]
|
||||
print("agent_gates — %d gate(s)%s" % (len(selected), " [--fast]" if fast else ""))
|
||||
if skipped:
|
||||
print(" --fast SKIPPED (deliberate periodic runs, never in a hook): %s" % ", ".join(skipped))
|
||||
hooks_armed_note(ROOT)
|
||||
|
||||
results = [(label, run_gate(label, path, args)) for label, path, args, _f in selected]
|
||||
|
||||
print("\n" + "=" * 78)
|
||||
print("== summary")
|
||||
print("=" * 78)
|
||||
worst = 0
|
||||
for label, rc in results:
|
||||
print(" %-18s %-13s (exit %d)" % (label, VERDICT.get(rc, "ERROR"), rc))
|
||||
if rc != 0:
|
||||
worst = 1 if rc == 1 or worst == 1 else 2
|
||||
if worst == 0:
|
||||
print("\nall agent gates OK")
|
||||
return 0
|
||||
convicted = [l for l, rc in results if rc == 1]
|
||||
undecided = [l for l, rc in results if rc not in (0, 1)]
|
||||
if convicted:
|
||||
print("\nCONVICTED: %s" % ", ".join(convicted))
|
||||
if undecided:
|
||||
print("UNDETERMINED (never a pass): %s" % ", ".join(undecided))
|
||||
return worst
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""check-published-versions.py — a released agent version must be INSTALLABLE, not merely built.
|
||||
|
||||
R-115. A box installs the agent from a Gitea generic package the hub vouches, never from git, and
|
||||
since R-110/R-183 it also fetches the agent's sixteen config files from `raw/tag/v<version>/`.
|
||||
Nothing in the build, deploy or session-end path checked that either existed, so "deployed" and
|
||||
"installable" were independent states that drifted silently. **Three instances in five days:**
|
||||
|
||||
* R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published — a new customer
|
||||
would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
* 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
* 0.120.0 (2026-08-03) built, committed, deployed to BOTH demo hosts, never published. A
|
||||
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
agent — and would have *succeeded* while doing it.
|
||||
|
||||
THE INVARIANTS — there are TWO now, and the second is R-188's price.
|
||||
|
||||
(1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
|
||||
and the tag must serve the agent's configs.
|
||||
|
||||
(2) No PUBLISHED version may be missing its tag.
|
||||
|
||||
Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
|
||||
AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
|
||||
package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
|
||||
needs a token this gate does not have. That reasoning was sound and the ordering was still wrong —
|
||||
the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
|
||||
seconds before its own package existed and mailing the operator a failure for a release that worked
|
||||
(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
|
||||
|
||||
Moving the push does not get to trade invariant (2) away, so it is asserted here instead — WITHOUT a
|
||||
token, and therefore as a BOUNDED PROBE rather than an enumeration:
|
||||
|
||||
* the FRONTIER — the versions immediately above the highest tag. This is the realistic failure the
|
||||
new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
|
||||
version beyond the newest tag.
|
||||
* the GAPS — patch versions that fall between two existing tags and have no tag of their own.
|
||||
|
||||
Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
|
||||
token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
|
||||
coverage is invisible reads as a guarantee it is not making.
|
||||
|
||||
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
|
||||
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
|
||||
and that was measured rather than assumed:** the hub's artifact manifest
|
||||
(`GET /api/v1/artifacts/<customer>`) answers **401** without a per-customer retrieval passphrase,
|
||||
and the Gitea package LISTING api answers **401** without a token, while the package DOWNLOAD url
|
||||
and the git TAGS api are both anonymous. So a credential-free gate can ask *"is this version
|
||||
installable"* but not *"which version is vouched"*. Putting an operator credential into CI to close
|
||||
that gap is the operator's call, not a gate author's — it is recorded as a limitation below and as a
|
||||
backlog row rather than quietly assumed away.
|
||||
|
||||
**What this invariant does catch: all three instances above.** `release-agent.sh` creates the tag and
|
||||
publishes in one act, so a release whose publish was skipped, failed, or was forgotten leaves a tag
|
||||
with no package — which is exactly what this refuses. It needs no version floor: tags begin at
|
||||
v0.120.0, which is published.
|
||||
|
||||
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
|
||||
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
|
||||
Nor does the converse probe prove that NO untagged package exists — only that none exists at the
|
||||
probed versions, which are printed. Closing that properly needs a read token in CI (→ R-184).
|
||||
|
||||
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
|
||||
INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
|
||||
standing rule this project earned twice, and a gate that green-lights on its own blindness is worse
|
||||
than no gate, because it looks like coverage.
|
||||
|
||||
Pure python3 + urllib, NO curl and no third-party module: the CI runner is a host-mode container
|
||||
carrying python3 and git and nothing else, and an earlier workflow step died on
|
||||
`curl: command not found`.
|
||||
|
||||
python3 scripts/check-published-versions.py
|
||||
|
||||
Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
|
||||
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
|
||||
OWNER = "admin"
|
||||
REPO = "felhom-agent"
|
||||
PKG = "felhom-agent"
|
||||
TIMEOUT = 25
|
||||
|
||||
# One config the installer fetches. Its presence proves the TAG's tree carries the configs the
|
||||
# sixteen `fetch_raw` calls will ask for — a tag that exists but predates them would 404 a box
|
||||
# mid-install, on a virgin machine, as root.
|
||||
PROBE_CONFIG = "configs/felhom-agent.service"
|
||||
|
||||
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
def _get(url, want_body=False):
|
||||
"""GET a URL. Returns (status, body_or_None). Network failure raises."""
|
||||
tried.append(url)
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
body = r.read() if want_body else None
|
||||
return r.status, body
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, None
|
||||
|
||||
|
||||
def inconclusive(msg):
|
||||
print("INCONCLUSIVE:", msg)
|
||||
print(" URLs tried (a 'no access' claim must name its attempts):")
|
||||
for u in tried:
|
||||
print(" ", u)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def _pkg_exists(version):
|
||||
"""True iff the generic package for `version` is downloadable anonymously."""
|
||||
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
|
||||
status, _ = _get(url)
|
||||
return status == 200, url
|
||||
|
||||
|
||||
def untagged_probe_set(versions):
|
||||
"""The versions to probe for invariant (2), as (version, why) pairs.
|
||||
|
||||
Bounded on purpose and printed by the caller: the package listing api needs a token (401,
|
||||
re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
|
||||
places an orphan would actually land.
|
||||
|
||||
FRONTIER — a publish that succeeded followed by a tag push that failed leaves the orphan
|
||||
exactly one version past the newest tag. This is the failure mode the R-188
|
||||
reordering makes possible, so it is the one that must not be guesswork.
|
||||
GAPS — a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
|
||||
tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
|
||||
"""
|
||||
parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
|
||||
have = set(parsed)
|
||||
out = []
|
||||
if not parsed:
|
||||
return out
|
||||
|
||||
hi = parsed[-1]
|
||||
for cand, why in (
|
||||
((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
|
||||
((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
|
||||
((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
|
||||
((hi[0] + 1, 0, 0), "next major after the newest tag"),
|
||||
):
|
||||
if cand not in have:
|
||||
out.append(("%d.%d.%d" % cand, why))
|
||||
|
||||
MAX_GAP_PROBES = 12
|
||||
for a, b in zip(parsed, parsed[1:]):
|
||||
if a[0] != b[0] or a[1] != b[1]:
|
||||
continue # a minor/major step is not a patch gap
|
||||
for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
|
||||
cand = (a[0], a[1], patch)
|
||||
if cand not in have:
|
||||
out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
print("check-published-versions — every released agent version must be INSTALLABLE")
|
||||
print(" gitea:", GITEA_BASE)
|
||||
|
||||
tags_url = "%s/api/v1/repos/%s/%s/tags?limit=200" % (GITEA_BASE, OWNER, REPO)
|
||||
try:
|
||||
status, body = _get(tags_url, want_body=True)
|
||||
except Exception as e:
|
||||
inconclusive("cannot reach Gitea to list tags: %s" % e)
|
||||
if status != 200 or not body:
|
||||
inconclusive("tags api returned HTTP %s — cannot enumerate releases" % status)
|
||||
try:
|
||||
tags = [t["name"] for t in json.loads(body.decode("utf-8"))]
|
||||
except Exception as e:
|
||||
inconclusive("tags api response is not the expected JSON: %s" % e)
|
||||
|
||||
versions = sorted(m.group(1) for m in (TAG_RE.match(t) for t in tags) if m)
|
||||
if not versions:
|
||||
# Not a failure: a repo legitimately has no version tags before its first release. Say so
|
||||
# loudly rather than reporting a vacuous pass — an empty check that prints OK is how an
|
||||
# unexamined thing becomes a documented-clean one.
|
||||
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
||||
print("\ncheck-published-versions: NOTHING TO CHECK")
|
||||
return 0
|
||||
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
|
||||
|
||||
bad = []
|
||||
for v in versions:
|
||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||
try:
|
||||
pkg_status, _ = _get(pkg_url)
|
||||
raw_status, _ = _get(raw_url)
|
||||
except Exception as e:
|
||||
inconclusive("network failure while checking v%s: %s" % (v, e))
|
||||
|
||||
problems = []
|
||||
if pkg_status != 200:
|
||||
problems.append("binary NOT downloadable (HTTP %s at %s)" % (pkg_status, pkg_url))
|
||||
if raw_status != 200:
|
||||
problems.append("tag does not serve %s (HTTP %s) — a box would 404 mid-install"
|
||||
% (PROBE_CONFIG, raw_status))
|
||||
if problems:
|
||||
bad.append((v, problems))
|
||||
print(" FAIL v%s:" % v)
|
||||
for p in problems:
|
||||
print(" -", p)
|
||||
else:
|
||||
print(" ok v%s: binary downloadable + tag serves its configs" % v)
|
||||
|
||||
# ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
|
||||
probes = untagged_probe_set(versions)
|
||||
orphans = []
|
||||
print()
|
||||
print(" converse probe — a published version with no tag (bounded; the package listing api")
|
||||
print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
|
||||
for v, why in probes:
|
||||
try:
|
||||
exists, url = _pkg_exists(v)
|
||||
except Exception as e:
|
||||
inconclusive("network failure while probing v%s: %s" % (v, e))
|
||||
mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
|
||||
print(" %-10s %-42s %s" % (v, why, mark))
|
||||
if exists:
|
||||
orphans.append((v, url))
|
||||
|
||||
print()
|
||||
if bad or orphans:
|
||||
if orphans:
|
||||
print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
|
||||
for v, url in orphans:
|
||||
print(" v%s is downloadable at %s but has no git tag." % (v, url))
|
||||
print(" A release publishes and then pushes its tag; a package with no tag means the")
|
||||
print(" push failed or was skipped. The local tag is probably still in the release")
|
||||
print(" clone — finish it with:")
|
||||
for v, _ in orphans:
|
||||
print(" git push origin v%s" % v)
|
||||
print(" (and if the tag is gone, re-create it on the released commit before pushing.)")
|
||||
if bad:
|
||||
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
|
||||
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
|
||||
print(" the R-115 defect, three times in five days. Publish it with:")
|
||||
print(" scripts/release-agent.sh <version>")
|
||||
if bad or orphans:
|
||||
return 1
|
||||
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Regular → Executable
+5
-1
@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
|
||||
BIN="$(mktemp -t felhom-agent.XXXXXX)"
|
||||
CLEANUP_BIN="$BIN"
|
||||
log "building felhom-agent $VERSION from $REPO_ROOT …"
|
||||
( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
||||
# These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
|
||||
# They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
|
||||
# the one the release path built for the same version. One version name must mean one binary
|
||||
# whichever entry point produced it.
|
||||
( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
||||
fi
|
||||
[[ -f "$BIN" ]] || die "binary not found: $BIN"
|
||||
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
|
||||
|
||||
Executable
+212
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env bash
|
||||
# release-agent.sh — THE way to release a felhom-agent version. One act: build → tag → publish →
|
||||
# verify by independent download.
|
||||
#
|
||||
# WHY THIS EXISTS (R-115). Publishing used to be a step someone had to remember, and it was
|
||||
# forgotten THREE TIMES IN FIVE DAYS:
|
||||
#
|
||||
# * R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published, so a new customer
|
||||
# would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
# * 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
# * 0.120.0 (2026-08-03) built, committed and deployed to BOTH demo hosts, never published. A
|
||||
# documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
# agent — and would have *succeeded* while doing it, because the current `step_grows`
|
||||
# sets SYSDATA_GROW=0 so the older agent's fatal mp1 resize never fires.
|
||||
#
|
||||
# R-111's own closing line said publishing should join the release train rather than stay a
|
||||
# remembered step. It closed SHIPPED without that leg, and the leg recurred the same afternoon —
|
||||
# which is the evidence that a note is not a mechanism. This file is the mechanism. The
|
||||
# documentation now points here instead of at a raw `go build` line, so there is ONE documented way
|
||||
# to release and it cannot complete without publishing.
|
||||
#
|
||||
# WHY IT TAGS (R-183). Since felhom-host-install.sh pins its sixteen agent-config fetches to
|
||||
# `raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root, on a
|
||||
# virgin machine. The tag and the package are two halves of one release and are created together.
|
||||
#
|
||||
# WHY IT DOES NOT VOUCH. Vouching is what points machines at a version, and it stays the operator's
|
||||
# deliberate act — the same prove-then-vouch principle that governed the golden two sessions ago.
|
||||
# This script prints the version and sha to vouch; a human decides when.
|
||||
#
|
||||
# Usage:
|
||||
# GITEA_USER=admin GITEA_TOKEN=<token> ./scripts/release-agent.sh <version>
|
||||
#
|
||||
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
||||
# GITEA_BASE / GITEA_OWNER override the defaults.
|
||||
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
||||
set -euo pipefail
|
||||
|
||||
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
||||
GITEA_OWNER="${GITEA_OWNER:-admin}"
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
die() { echo "[release-agent] ERROR: $*" >&2; exit 1; }
|
||||
log() { echo "[release-agent] $*" >&2; }
|
||||
|
||||
VERSION="${1:-}"
|
||||
[[ -n "$VERSION" ]] || die "version required (usage: GITEA_USER=.. GITEA_TOKEN=.. $0 <version>)"
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "version must be bare semver X.Y.Z (got '$VERSION')"
|
||||
TAG="v$VERSION"
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# ── 1. Clean-tree gate ──────────────────────────────────────────────────────────────────────────
|
||||
# An unpushed change does not exist. Releasing a dirty tree publishes a binary whose source nobody
|
||||
# else can obtain, and tags a commit that does not contain what was built.
|
||||
if [[ "${RELEASE_ALLOW_DIRTY:-0}" != "1" ]]; then
|
||||
[[ -z "$(git status --porcelain)" ]] || die "working tree is dirty — commit and push first"
|
||||
local_head="$(git rev-parse HEAD)"
|
||||
git fetch -q origin main
|
||||
[[ "$local_head" == "$(git rev-parse origin/main)" ]] \
|
||||
|| die "HEAD != origin/main — push first (an unpushed change does not exist)"
|
||||
fi
|
||||
|
||||
# ── 2. Refuse to re-release a version that already exists ───────────────────────────────────────
|
||||
# Silently overwriting a published artifact is how "the same version" comes to mean two different
|
||||
# binaries on two different boxes.
|
||||
if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
|
||||
die "tag $TAG already exists — releasing over it would make one version name two binaries"
|
||||
fi
|
||||
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
|
||||
|
||||
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
|
||||
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
||||
trap 'rm -f "$BIN"' EXIT
|
||||
log "building $VERSION …"
|
||||
# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
|
||||
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
|
||||
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
|
||||
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
|
||||
#
|
||||
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
||||
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
|
||||
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
||||
#
|
||||
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
|
||||
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
|
||||
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
|
||||
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
|
||||
#
|
||||
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
|
||||
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
|
||||
# version — one version name, two binaries, by whichever entry point was used. Both now build the
|
||||
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
|
||||
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
||||
|| die "go build failed"
|
||||
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
||||
[[ "$built_ver" == "$VERSION" ]] \
|
||||
|| die "the built binary reports '$built_ver', not '$VERSION' — the ldflag did not take"
|
||||
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
||||
log "built ok: sha256 $BUILT_SHA"
|
||||
|
||||
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
|
||||
#
|
||||
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
|
||||
#
|
||||
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
|
||||
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
|
||||
# because the Gitea package LISTING api needs a token the gate does not have.
|
||||
#
|
||||
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
|
||||
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
|
||||
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
|
||||
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
|
||||
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
|
||||
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
|
||||
#
|
||||
# So the tag is still created HERE, before anything is published — the build and the tag still
|
||||
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
|
||||
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
|
||||
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
|
||||
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
|
||||
# now checked rather than one being arranged for.
|
||||
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
||||
git tag -a "$TAG" -m "agent $TAG
|
||||
|
||||
Released by scripts/release-agent.sh.
|
||||
sha256 of the published binary: $BUILT_SHA
|
||||
|
||||
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
||||
so this tag is part of the released artifact, not a bookmark (R-183)."
|
||||
|
||||
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
||||
log "publishing …"
|
||||
# Invoked through `bash` DELIBERATELY, not as an executable. On 2026-08-03 the first real release
|
||||
# through this script died here — `publish-agent.sh` has been mode 0644 since it was created on
|
||||
# 2026-06-28, because every earlier caller ran it as `bash scripts/publish-agent.sh`. So the one leg
|
||||
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
|
||||
# the same commit; this line makes the release independent of it, because a file mode is exactly the
|
||||
# kind of thing that is lost again by a checkout, an archive, or a copy.
|
||||
if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
|
||||
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
|
||||
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
|
||||
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
|
||||
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
|
||||
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
if [[ "$now_published" == "200" ]]; then
|
||||
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
|
||||
else
|
||||
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
|
||||
fi
|
||||
die "publish failed"
|
||||
fi
|
||||
|
||||
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
|
||||
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
|
||||
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
|
||||
#
|
||||
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
|
||||
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
|
||||
# The recovery is one line and it is printed rather than described.
|
||||
log "pushing $TAG …"
|
||||
if ! git push origin "$TAG"; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
|
||||
|
||||
version : $VERSION
|
||||
sha256 : $BUILT_SHA
|
||||
|
||||
The tag exists in this clone only. Nothing installs from an untagged version (the installer
|
||||
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
|
||||
FAIL on it as a published version with no tag. Finish the release with:
|
||||
|
||||
git push origin $TAG
|
||||
|
||||
EOF
|
||||
die "tag push failed after a successful publish — see above"
|
||||
fi
|
||||
|
||||
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
||||
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
||||
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
||||
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
||||
# returns the right sha are different claims.
|
||||
log "verifying by independent download …"
|
||||
DL="$(mktemp -t felhom-agent-dl-XXXXXX)"
|
||||
trap 'rm -f "$BIN" "$DL"' EXIT
|
||||
curl -fsS -o "$DL" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" \
|
||||
|| die "round-trip GET failed — the version is NOT installable"
|
||||
DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
|
||||
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|
||||
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
|
||||
|
||||
# The tag must also serve the configs the installer will fetch from it.
|
||||
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
|
||||
[[ "$cfg_code" == "200" ]] \
|
||||
|| die "tag $TAG does not serve configs/felhom-agent.service (HTTP $cfg_code) — a box would 404 mid-install"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
|
||||
version : $VERSION
|
||||
tag : $TAG
|
||||
sha256 : $BUILT_SHA
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
EOF
|
||||
@@ -0,0 +1,67 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Seam test for scripts/agent_gates.py.
|
||||
|
||||
Run from the repo root: python3 scripts/test_agent_gates.py
|
||||
|
||||
WHY THIS EXISTS. An entry point is a seam by definition: a runner that LISTS a gate but never
|
||||
executes it is inert and fully green, and this project has shipped an inert seam four times. So
|
||||
the assertion is on the member gate's OWN distinctive stdout — never on the runner's summary
|
||||
line, which the runner can print without ever calling anything — plus the exit code, which is a
|
||||
runner's actual effect.
|
||||
|
||||
The second test is the one that matters here: this repo's only gate lives in a SIBLING clone, so
|
||||
"the sibling is missing" must be a FAILURE and not a quiet skip.
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ENTRY = os.path.join(ROOT, "scripts", "agent_gates.py")
|
||||
|
||||
|
||||
class AgentGatesTest(unittest.TestCase):
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
p = subprocess.run([sys.executable, ENTRY, "--fast"], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
cls.rc = p.returncode
|
||||
cls.out = p.stdout.decode("utf-8", "replace")
|
||||
|
||||
def test_exit_code_is_zero(self):
|
||||
self.assertEqual(self.rc, 0, self.out)
|
||||
|
||||
def test_member_gate_actually_ran(self):
|
||||
self.assertIn("cited paths — exact", self.out,
|
||||
"the reuse-refs gate is listed but its own output never appeared — an inert "
|
||||
"runner prints the summary without calling anything:\n%s" % self.out)
|
||||
|
||||
def test_missing_shared_checker_is_a_failure_not_a_skip(self):
|
||||
"""Fail-closed. Copy the entry point into a lone directory with no felhom.eu sibling and
|
||||
confirm it CONVICTS rather than reporting green with nothing run."""
|
||||
tmp = tempfile.mkdtemp(prefix="agent-gates-")
|
||||
try:
|
||||
lone = os.path.join(tmp, "felhom-agent", "scripts")
|
||||
os.makedirs(lone)
|
||||
shutil.copy(ENTRY, os.path.join(lone, "agent_gates.py"))
|
||||
p = subprocess.run([sys.executable, os.path.join(lone, "agent_gates.py"), "--fast"],
|
||||
cwd=os.path.dirname(lone),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
out = p.stdout.decode("utf-8", "replace")
|
||||
self.assertNotEqual(p.returncode, 0, out)
|
||||
self.assertIn("is MISSING — tried", out)
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
def test_unknown_argument_is_rejected(self):
|
||||
p = subprocess.run([sys.executable, ENTRY, "--nope"], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
self.assertEqual(p.returncode, 2, p.stdout.decode("utf-8", "replace"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
Reference in New Issue
Block a user