Compare commits
26 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 703db166e7 | |||
| aa74294a7d | |||
| 5b2666e3a2 | |||
| 062a7027ab | |||
| a2e914f683 | |||
| 0404f60e6a | |||
| 3f5f61b716 | |||
| 6d7904786c | |||
| 856a127cd6 | |||
| 257c4d85c0 | |||
| 72161f6cf0 | |||
| 03b58cec0a | |||
| fe14bc62c0 | |||
| 0b28eae7bb | |||
| 7581f8140a | |||
| 3d0a1d615d | |||
| 77e2cc4583 | |||
| cd1b087db7 | |||
| 53d0c6bfc4 | |||
| 4d82591052 | |||
| 4618169036 | |||
| 1b14cfd0b4 | |||
| 0db77666c6 | |||
| dd2d1feb6e | |||
| 9dfd89cb94 | |||
| 4bb84fc3ca |
@@ -0,0 +1,46 @@
|
||||
---
|
||||
paths: ["internal/backup/**", "internal/pbs/**", "internal/pbsdr/**", "internal/dr/**"]
|
||||
---
|
||||
|
||||
# Backup, PBS and DR
|
||||
|
||||
`internal/backup/` is the vzdump runner, restore-test scheduler and report store. `internal/pbs/` is
|
||||
the fingerprint-pinned PBS-API client plus the verify maintenance loop. `internal/pbsdr/` and
|
||||
`internal/dr/` carry the DR tier and recipe halves.
|
||||
|
||||
## The three PBS laws
|
||||
|
||||
1. **Set-only.** `pvesm remove` **DELETES the encryption key**. Re-apply configuration; never remove
|
||||
and re-add a PBS storage to change it.
|
||||
2. **Secret on stdin.** A token secret is passed on stdin, never as an argv the process table shows.
|
||||
3. **Verify the pin BEFORE consuming the secret.** A fingerprint check after the secret has been sent
|
||||
protects nothing.
|
||||
|
||||
## Verify is server-side, and its default skips the work
|
||||
|
||||
The agent drives verification **remotely** via the PBS API; `proxmox-backup-client` has **no** verify
|
||||
subcommand. `POST .../verify` defaults to **`ignore-verified=true`, which SKIPS already-verified
|
||||
snapshots** — send `ignore-verified=false` to actually re-read and detect corruption. A verify that
|
||||
skipped everything reports success.
|
||||
|
||||
## Presence is not success
|
||||
|
||||
A timestamp recording an **attempt** is not evidence of a **result**. Where a status field travels
|
||||
beside a timestamp, the verdict must consult **both** — or the timestamp must record only successes.
|
||||
Ask of any timestamp: *what exactly must have happened for this to be set?* If the answer is "we
|
||||
tried", it cannot answer "did it work".
|
||||
|
||||
**Corollary:** when a verdict changes which field it counts from, the alarm text changes with it.
|
||||
Leaving a message reading `last run 8h ago` while alarming on a six-day-old **success** turns a true
|
||||
alarm into one the operator dismisses.
|
||||
|
||||
## Prune is server-side now
|
||||
|
||||
`DatastoreBackup` carries **no** `Datastore.Prune`. Boxes set `keep_last: 0` and the off-site endpoint
|
||||
runs the prune jobs. **Box tokens stay write-only — never widen that grant** (R-89).
|
||||
|
||||
<!--
|
||||
The ignore-verified default is the sharpest instance of the "absent log line" class in this repo: a
|
||||
verify that silently skipped every snapshot completes fast, exits clean, and reports the same shape
|
||||
as one that read every byte.
|
||||
-->
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
paths: ["internal/capability/**", "internal/storage/**", "internal/localapi/**", "internal/hub/**", "internal/guesthook/**"]
|
||||
---
|
||||
|
||||
# A health check issues no block I/O
|
||||
|
||||
No `statfs`, no `getdents`, no read, write or `fsync` — **not even behind a timeout**.
|
||||
|
||||
A probe that touches a wedged device enters uninterruptible sleep, survives `SIGKILL`, and cannot be
|
||||
recovered until the device returns or the host reboots — so `systemctl restart` hangs too. A timeout
|
||||
protects the caller's control flow and nothing else: the blocked thread remains.
|
||||
|
||||
**Liveness is decided from `/proc` and the kernel's own state**, never by reading or writing the
|
||||
filesystem.
|
||||
|
||||
<!--
|
||||
Measured, R-117 spike §6.3 (felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md):
|
||||
a probe stayed in D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT
|
||||
needs journal access); and statfs/getdents returned HEALTHY on a namespace that EIOs every byte —
|
||||
fast, and wrong.
|
||||
|
||||
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
|
||||
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
|
||||
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
|
||||
file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
|
||||
-->
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
paths: ["internal/localapi/**", "internal/authz/**", "internal/guesthook/**"]
|
||||
---
|
||||
|
||||
# Local API, authz and guest hooks — the per-guest blast radius
|
||||
|
||||
`internal/localapi/` is the narrow per-guest local API: token store, disks/format, guest binds,
|
||||
controller swap, stale-lock recovery, pinned self-signed leaf. `internal/authz/` is the operator
|
||||
signed-op verifier (SSHSIG) plus the durable nonce store. `internal/guesthook/` installs the
|
||||
pre-start self-heal hookscript.
|
||||
|
||||
> **Overlap note:** `health-checks.md` also matches `internal/localapi/**` and
|
||||
> `internal/guesthook/**`. That is deliberate — both rules apply there and both load. Neither
|
||||
> supersedes the other.
|
||||
|
||||
## Scoping is the whole security property
|
||||
|
||||
This API is reachable **from inside a customer guest**. Every route must be scoped to the guest that
|
||||
called it — a route that can name another guest's id has escaped its blast radius. Fail **safe to
|
||||
protected**: an unrecognised or unresolvable caller gets less access, never more.
|
||||
|
||||
## Replay protection must survive a restart
|
||||
|
||||
**`authz.MemoryNonceStore` on a real host is a defect** — replay protection dies on restart. Use
|
||||
`authz.FileNonceStore`. The memory store exists for tests.
|
||||
|
||||
## The token is a hash on disk, plaintext only at mint
|
||||
|
||||
The store keeps **hashes**. The plaintext token exists in exactly one place, `bootstrap.json` on the
|
||||
PVE host — so a "read the token" step means reading that file, and a lost token is re-minted, never
|
||||
recovered.
|
||||
|
||||
## Binds can brick guest boot
|
||||
|
||||
| Do not | Because | Use |
|
||||
|---|---|---|
|
||||
| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can **brick guest boot** (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
|
||||
| `isHostMountpoint` to reconcile bind state | a boolean cannot converge stacked double-binds (the `/mnt` doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
|
||||
|
||||
<!--
|
||||
Why fail-safe-to-protected rather than fail-closed: this API also carries the recovery paths. A hard
|
||||
refusal on an unresolvable caller would make a half-broken guest unrecoverable through the very
|
||||
interface built to recover it. Less access, never none.
|
||||
-->
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
paths: ["internal/proxmox/**", "internal/reconcile/**", "internal/signedjobs/**"]
|
||||
---
|
||||
|
||||
# Proxmox — the API contract, and how destructive work is gated
|
||||
|
||||
`internal/proxmox/` is the API-first `Client` plus the fenced root-CLI `Privileged`.
|
||||
`internal/reconcile/` is the reconcile engine, reversibility gate, op journal and crash recovery.
|
||||
`internal/signedjobs/` holds the operator-signed destructive executors (wipe, decommission).
|
||||
|
||||
## A 200 on the POST is not success
|
||||
|
||||
**Every mutating op is async**: it returns a **UPID**, and `WaitTask` must assert
|
||||
`exitstatus == "OK"`. Authorization can fail at *task execution* long after the HTTP call returned
|
||||
200. Treating the POST's status as the result is how a failed destroy reads as a successful one.
|
||||
|
||||
## The privsep token gotcha
|
||||
|
||||
A `--privsep 1` token's rights are the **intersection** of the backing user's permissions **and** the
|
||||
token's own ACLs. The role must be granted on **both** or every call 403s. The same intersection rule
|
||||
bites on PBS (`token ∩ user`).
|
||||
|
||||
## TLS
|
||||
|
||||
**SHA-256 leaf-cert pinning** against the self-signed host cert. **No insecure default**, ever. The
|
||||
pin is the raw leaf-DER sha — the SAN is never checked, so a cert rotation changes the pin and the
|
||||
agent must be re-pinned.
|
||||
|
||||
## The destructive path — never the direct call
|
||||
|
||||
| Do not | Because | Use |
|
||||
|---|---|---|
|
||||
| `Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
|
||||
| add a method to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers `Cmnd_Alias` + `validate.go`-style checks |
|
||||
| treat `ListLXC` output as "guests we own" | audit A1 — pre-v0.62.0 the stale-lock reaper did exactly this, contained only by the pool-scoped token | intersect with `Client.Pool` membership (`staleLockController.Guests()`); **fail safe on read failure** |
|
||||
|
||||
Full trap table: `REUSE.md` §3. Every guest joins the `felhom` pool — `VM.Audit` comes from the
|
||||
`/pool` grant, not from a per-guest ACL.
|
||||
|
||||
<!--
|
||||
The fence is not stylistic. It is what makes this component auditable: two types, one of which can
|
||||
only speak HTTP and one of which can only shell out, with a test asserting neither crosses. A single
|
||||
convenience method on Privileged that also makes an HTTP call would end that property silently.
|
||||
-->
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
paths: ["internal/storage/**", "internal/escrow/**"]
|
||||
---
|
||||
|
||||
# Storage and escrow — format safety and zero-knowledge recovery
|
||||
|
||||
`internal/storage/` is the storage observer, durable IDs, role/claim classifiers, `SudoHostOps` and
|
||||
the watchdog. `internal/escrow/` is the PBS-key escrow with its zero-knowledge recovery code.
|
||||
|
||||
> **Overlap note:** `health-checks.md` also matches `internal/storage/**`. Deliberate — both rules
|
||||
> apply there and both load.
|
||||
|
||||
## Never format the device you inspected
|
||||
|
||||
**AGENT-001 is a TOCTOU:** acting on the caller's `req.Device` (or any remembered `/dev` path) after
|
||||
inspection lets `/dev` re-enumeration retarget the node to a **different physical disk**. Format the
|
||||
**re-resolved** device — `Server.reresolveWipe` / `reresolveBlank`.
|
||||
|
||||
**Never exec raw `mkfs.*`** (including `Binaries.MkfsExt4`/`MkfsXfs`): sudoers no longer allowlists
|
||||
raw mkfs, and going direct bypasses the claim filter and the wrapper's re-checks. Use
|
||||
`SudoHostOps.Format`, which routes through `felhom-mkfs-guarded`.
|
||||
|
||||
## The two durable-ID schemes refuse each other
|
||||
|
||||
They are not interchangeable, and each returns a `binding_mismatch` for the other's scheme:
|
||||
|
||||
| Purpose | Scheme | Resolver |
|
||||
|---|---|---|
|
||||
| wipe confirmation | `byid:` / `byuuid:` | `ResolveDurableDevice`, `DiskInfo.WipeDurableID` |
|
||||
| enrolled-storage remount | `uuid:` | `ResolveStorageDevice` |
|
||||
|
||||
Using `DiskInfo.DurableID` (a `uuid:`) as a wipe-confirmation id is F20-BUG2.
|
||||
|
||||
## Drive data is never taken by force
|
||||
|
||||
Plain `umount` only — **never `-l`, never `-f`**, and never any format operation under
|
||||
`/mnt/felhom-drives`.
|
||||
|
||||
## Escrow is zero-knowledge, and a fetch failure is not a wrong code
|
||||
|
||||
The server holds no client key; a no-key restore fails with `missing key`. **A fetch failure must
|
||||
never be reported as a wrong recovery code** — that told a customer their correct code was bad, in
|
||||
hundredths of a second, when checking a code actually takes about one. Distinguish "we could not
|
||||
reach the store" from "the code did not match", always.
|
||||
|
||||
<!--
|
||||
The escrow recovery-code "flake" was a REAL defect, not a flake. "Known flake, re-run" needs evidence
|
||||
before it is said out loud — that phrase cost this project a real finding once.
|
||||
-->
|
||||
@@ -43,7 +43,19 @@ jobs:
|
||||
- name: Run the gate entry point
|
||||
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
|
||||
# exit code IS the result: no `|| true`, no pipe that could swallow it.
|
||||
run: cd ws/felhom-agent && python3 scripts/agent_gates.py --fast
|
||||
#
|
||||
# THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
|
||||
# container runtime" and exists for `.githooks/pre-push`, where a push must not fail
|
||||
# because Gitea blinked or because someone is on a train. CI is the opposite machine: it
|
||||
# has the network, it is not in anyone's way, and it is the half that emails. The
|
||||
# published-versions gate — the R-115 mechanism, which asks Gitea whether a released
|
||||
# version can actually be downloaded — is network-bound and therefore runs ONLY here.
|
||||
# Leaving `--fast` in place would have registered that gate and never run it, which is the
|
||||
# built-but-never-wired failure this project has shipped four times.
|
||||
env:
|
||||
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
|
||||
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
|
||||
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
|
||||
|
||||
- name: Alarm on failure
|
||||
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
|
||||
|
||||
@@ -29,6 +29,41 @@ root=$(git rev-parse --show-toplevel 2>/dev/null) || {
|
||||
}
|
||||
cd "$root" || exit 1
|
||||
|
||||
# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
|
||||
# Refuse a push from a clone outside the felhom workspace.
|
||||
#
|
||||
# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
|
||||
# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
|
||||
# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
|
||||
# fixed by writing it down again — it has to be asserted where it can bite.
|
||||
#
|
||||
# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
|
||||
# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
|
||||
#
|
||||
# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
|
||||
# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
|
||||
# failed — this hook must not brick a legitimate clone on a different host.
|
||||
#
|
||||
# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
|
||||
FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
|
||||
if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
|
||||
ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
|
||||
root_real=$(pwd -P) || root_real=""
|
||||
if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
|
||||
case "$root_real/" in
|
||||
"$ws_real"/*) : ;; # inside the workspace — proceed
|
||||
*)
|
||||
echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
|
||||
echo " clone: $root_real" >&2
|
||||
echo " expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
|
||||
echo " Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
|
||||
echo " and state that you did in the session report." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
|
||||
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
|
||||
|
||||
+525
@@ -1,3 +1,528 @@
|
||||
## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220)
|
||||
|
||||
**After a rebuild the customer's own drives could not be re-attached, and the refusal named an action
|
||||
they could not perform.** `GET /api/disks/candidates` returned `initialize: [], attach: []` while both
|
||||
drives sat there, and the deploy refused with *"choose an attached drive from the list"* — a list that
|
||||
was empty. Measured live **three times**: CAMPAIGN-11 Phase 1, and twice on the R-201 re-walk.
|
||||
|
||||
**The mechanism.** Enrolment mounts a drive **twice** — at the managed `/mnt/felhom-drives/<name>` and
|
||||
at the raw `/mnt/<name>` it creates on the host. **The host survives a guest rebuild; the controller's
|
||||
registry does not.** So `classifyClaim` saw a mount outside the managed prefix and correctly concluded
|
||||
"claimed by something else" — about Felhom's own mount.
|
||||
|
||||
**The fix is CORROBORATED, not a widened prefix.** A mountpoint outside `/mnt/felhom-drives` is
|
||||
forgiven **only when the same device is ALSO mounted under the managed path** — a pairing that only
|
||||
Felhom's own enrolment produces. A disk another system is using, at `/srv/data` or `/media/x` or even
|
||||
`/mnt/someone-elses-disk`, has no such counterpart and **is still refused**. That fence has its own
|
||||
test, and its red-proof shows an over-wide fix offering `/mnt/someone-elses-disk` for formatting.
|
||||
|
||||
**Read from `/proc/mounts`, deliberately.** The lsblk invocation is pinned **verbatim** in
|
||||
`configs/felhom-agent.sudoers` (`lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/*`), so switching it to
|
||||
the plural `MOUNTPOINTS` would have meant shipping a sudoers change with the binary — a far larger
|
||||
blast radius than this finding warrants. `/proc/mounts` is world-readable: **no sudo, no new allowlisted
|
||||
command, no config change.**
|
||||
|
||||
**Fail-safe:** an unreadable mount table corroborates **nothing**, so the device classifies exactly as
|
||||
it did before this change — refused. "We could not corroborate" must never read as "it is ours".
|
||||
|
||||
Tests: `claim_r220_test.go` — the own-drive case, the foreign-mount fence over four paths, and the
|
||||
corroboration itself (both mounts required; a lone raw mount vouches for nothing; another device's
|
||||
managed mount does not vouch for this one; an unreadable table corroborates nothing).
|
||||
**Red-proofs:** removing the exemption refuses the customer's own drive again
|
||||
(*"device is mounted at /mnt/adatok (sdb)"*); over-widening it to any `/mnt/*` path breaks the fence.
|
||||
|
||||
## docs — CLAUDE.md becomes a core plus path-scoped rules (2026-08-06, R-229 leg (b)) — no version bump
|
||||
|
||||
**Documentation only. No Go changed, nothing built, nothing deployed.** `go build`/`vet`/`test` green
|
||||
and unchanged.
|
||||
|
||||
**175 -> 99 effective lines** (207 -> 103 raw, 14,093 -> 6,267 bytes). The release/publish-train
|
||||
section was the largest block and the `felhom-build-deploy` skill already carries the procedure, so
|
||||
the core points at it instead of restating a table that drifts from the script. The package layout
|
||||
went the same way as the controller's: `REUSE.md` and the tree are its home, and the per-package
|
||||
annotations that were doing real work moved into the rule file for the area they describe rather than
|
||||
being deleted.
|
||||
|
||||
**New:** `.claude/rules/{proxmox,localapi,backup,storage}.md`, all `paths:`-scoped, all <=46 effective
|
||||
lines, joining the existing `health-checks.md`.
|
||||
|
||||
**Kept in the core deliberately** — it is the only part re-injected after `/compact`: the root-CLI
|
||||
fence and its three named exceptions (breaching it is how this component stops being auditable), the
|
||||
destructive-op gate, the prove-ownership rule from audit A1, the gate entry point, the F9
|
||||
live-validation fence, trunk-based with its revert-and-report escape hatch, and the end-of-session
|
||||
checklist.
|
||||
|
||||
**Glob overlap, stated rather than silently resolved:** `health-checks.md` matches
|
||||
`internal/{localapi,guesthook}/**` and `internal/storage/**`, which `localapi.md` and `storage.md`
|
||||
also match. Both rules load in those directories and neither supersedes the other; each new file says
|
||||
so in its own text so a reader who sees two rules fire is not left guessing which wins.
|
||||
|
||||
## docs — the "CI is still owed" claim was stale; corrected (2026-08-06, R-229 part 2) — no version bump
|
||||
|
||||
**One sentence, no code.** This file asserted that continuous integration was still owed
|
||||
(`felhom.eu` `OPEN-ITEMS.md` R-168). **R-168 was CLOSED on 2026-08-02** — a Gitea Actions runner
|
||||
re-runs each repo's gate entry point on every push and emails the operator on failure. Found while
|
||||
confirming this session's own push by run ID, which is the check that caught it.
|
||||
|
||||
The same stale sentence was in four instruction files across all four repos and is corrected in all
|
||||
four. In `felhom-agent/CLAUDE.md` it **contradicted the same file's release section**, which already
|
||||
said R-168 mails the failure — a contradiction inside one instruction file, which is the exact class
|
||||
the R-229 work exists to find.
|
||||
|
||||
## docs — expired and contradictory blocks removed from CLAUDE.md (2026-08-06, R-229) — no version bump
|
||||
|
||||
**Documentation and gate registration only. No Go changed, nothing built, nothing deployed.**
|
||||
|
||||
Surgical corrections; the file was deliberately **not** restructured (that is deferred, R-229).
|
||||
|
||||
- **Deleted the expired TEMPORARY block.** It read *"felhom-pve is at a remote site (until
|
||||
~2026-08-02) … Delete this block on return"* and was still being read as current fact on
|
||||
**2026-08-06**, four days past its own deadline — while `felhom-controller/CLAUDE.md` asserted the
|
||||
opposite. The location-independence fact worth keeping (`localapi` binds `169.254.253.1:8443` on
|
||||
`vmbr9` since the R-50 island migration) moved to an HTML comment.
|
||||
- **Every component version literal is gone** from effective text, including
|
||||
`felhom-agent --version → 0.115.0` and the `go.mod` Go directive. Versions change several times a
|
||||
day; ask the hub's `/hosts` + `/configs` or the box.
|
||||
- The drill-VM claim and the host addresses now point at `documentation/operations/nodes.md`, which
|
||||
already stated both correctly. **This file's drill-VM claim was the correct one** — confirmed by
|
||||
`qm list` on demo-hp.
|
||||
- The R-115/R-188/R-186 release **narratives** moved to an HTML comment and to the
|
||||
`felhom-build-deploy` skill; the **directives** stayed (never hand-roll the build; the
|
||||
build → tag → publish → push order; reproducible `-trimpath -buildvcs=false`).
|
||||
- The health-check block-I/O rule became `.claude/rules/health-checks.md`, scoped to the five
|
||||
packages where health checks are written. It had been duplicated from `felhom.eu/CLAUDE.md` *with a
|
||||
note explaining that that file does not load in an agent-only session* — correct reasoning, made
|
||||
obsolete by path-scoped rules.
|
||||
|
||||
`agent_gates.py` now registers **`instructions`** (shared, `felhom.eu/scripts/`, never copied).
|
||||
|
||||
Full accounting: `felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md`.
|
||||
|
||||
## v0.126.0 — a fetch failure is not a wrong recovery code (2026-08-06, R-224)
|
||||
|
||||
**A hub the agent could not reach was reported to the customer as a bad recovery code.** Measured live
|
||||
on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and a **correct,
|
||||
current** recovery code, the customer was told their code did not open their package — **in 0.0556 s**,
|
||||
against ~1.0 s for a genuine unseal. No unseal was attempted. F4 produced the same message in 0.0299 s
|
||||
with this agent stopped.
|
||||
|
||||
**The discriminator existed here the whole time and this boundary threw it away.** `recover.go` fails
|
||||
at four distinguishable points; the local-api handler had cases for two of them and a `default` that
|
||||
answered *"the recovery code did not open the sealed bundle, or the bundle could not be fetched"* —
|
||||
one sentence for two situations, only one of which is the customer's doing.
|
||||
|
||||
**The fix is a value, not a log line.** `escrow.ErrBundleFetch` joins the fetch leg's error, and the
|
||||
handler routes it to **502** with its own words: *"the sealed recovery bundle could not be fetched from
|
||||
the hub — the recovery code was NOT used and nothing was written."* 502 rather than 4xx because the
|
||||
request was not bad; an upstream dependency failed. The `default` now carries **only** the fail-closed
|
||||
wrong-code case and says so without the "or".
|
||||
|
||||
Four situations, four statuses — **502** fetch failed · **400** the bundle was fetched and refused the
|
||||
code · **404** the hub holds no bundle · **409** the bundle predates the repository-password field.
|
||||
The controller classifies on the STATUS and must never parse these sentences.
|
||||
|
||||
⚠ **A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.**
|
||||
`TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct` has said since v0.125.0 that *"the operator must
|
||||
not be sent to re-read their recovery code because the hub was unreachable"* — and it passed
|
||||
throughout, because it asserted this package's error **string** one layer below where the merge
|
||||
happened, and a string is something no caller can branch on. It now asserts the sentinel, and its
|
||||
consequence-level twin asserts the STATUS at the boundary the customer's message is derived from.
|
||||
**Prefer the test that asserts the consequence over the one that asserts the mechanism.**
|
||||
|
||||
Tests: `recover_test.go` (fetch classifies as `ErrBundleFetch`; a wrong code does **not**; an absent
|
||||
blob keeps its own identity) and `localapi/escrow_recover_class_test.go` (each situation's status, and
|
||||
a standalone assertion that fetch-failure and wrong-code never share one). **Red-proofs:** removing the
|
||||
`%w` join fails the sentinel test; deleting the handler case makes both answer `400` with the
|
||||
wrong-code sentence — the exact pre-fix code, and the exact defect CAMPAIGN-11 measured.
|
||||
|
||||
## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 7–8)
|
||||
|
||||
**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed
|
||||
since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the
|
||||
recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest
|
||||
writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token` —
|
||||
an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||
offsite repository password into the same bundle. Anyone reading that output would conclude the
|
||||
repository password was not there. It now names what THIS bundle actually carried and what it did not.
|
||||
|
||||
**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the
|
||||
customer's recovery code, the agent fetches this host's own sealed blob from the hub
|
||||
(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and
|
||||
returns **only the offsite restic repository password** plus its sha256.
|
||||
|
||||
**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG
|
||||
private key. The controller is a trust tier down and needs none of them; returning them would widen
|
||||
the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not
|
||||
recoverable later.
|
||||
|
||||
**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately
|
||||
absent from the controller image; the blob is a host-scoped object whose only writer is this agent
|
||||
under the per-host key, so the read is that write's mirror.
|
||||
|
||||
**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned
|
||||
channel, is held in memory for one call, is cleared on the success path AND every failure path, is
|
||||
never written to disk, never an argument in a process list, never logged at any level including
|
||||
inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered
|
||||
plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty
|
||||
afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later
|
||||
call overwriting the leaked file, which is exactly how the first version of that test passed its own
|
||||
red-proof while R sat on disk.
|
||||
|
||||
Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a
|
||||
bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and
|
||||
a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to
|
||||
re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids.
|
||||
|
||||
**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main` → `runDaemon` →
|
||||
`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as
|
||||
`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping
|
||||
that makes cross-host retrieval impossible is a property of which key is used). This project's
|
||||
built-but-never-wired count is six and links 6–7 were two of them; the fix must not become the seventh.
|
||||
|
||||
## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190)
|
||||
|
||||
**v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The
|
||||
capability reported degraded for "one cycle" — meaning the probe call that performed the repair. But
|
||||
`probeAll` is invoked **independently** by the periodic self-check log and by the collector building a
|
||||
host-report. On the demo box the repairing call was the log's (`09:39:34`, journal shows
|
||||
`GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED` and `degraded=1`), and the host-report built three
|
||||
seconds later found the grant present and sent **`ok`**. The agent's journal had the record; the hub
|
||||
had nothing; the operator would have learned nothing.
|
||||
|
||||
That is the exact silence R-190 is about, re-created inside its own mitigation — and every unit test
|
||||
passed while it was true.
|
||||
|
||||
**The fix is a latch on TIME rather than on call count.** A confirmed repair is reported for
|
||||
`storeGrantRepairReportWindow` (20 minutes), which comfortably exceeds the 900 s host-report interval,
|
||||
so at least one report must carry the transition. It clears on its own — a permanently degraded
|
||||
capability would be its own false alarm — and it is per tier.
|
||||
|
||||
**Two hollow tests were caught and fixed on the way**, both the same shape this repo keeps finding: a
|
||||
test asserting a value it constructed itself, and a test asserting the latch HELPER rather than the
|
||||
path that consumes it — whose red-proof duly passed. The decisions now live in
|
||||
`storeGrantHealthyVerdict` and `storeGrantRepairedVerdict`, and the tests call those.
|
||||
|
||||
## v0.124.0 — a lost storage grant repairs itself, and says that it was lost (2026-08-04, R-190)
|
||||
|
||||
**R-190 is a grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24** — with a
|
||||
host reinstall, logged `pveum` activity and cluster-log entries all ruled out by measurement. The
|
||||
cause is still open. The resilience does not have to wait for it.
|
||||
|
||||
**Everything needed already existed and had only ever been called once.** The root wrapper
|
||||
(`felhom-backup-target-apply grant <id>`), its sudoers vector (`grant *`, any storage id, confirmed
|
||||
not assumed), and the exact command were all in place — and the `grant` verb had only ever run at
|
||||
storage CREATION. That is the *built but never wired* shape, in a verb rather than a seam, and it is
|
||||
this project's seventh instance.
|
||||
|
||||
**What v0.124.0 does:** when the store-grant probe finds the grant absent on a tier the box depends
|
||||
on, it runs that wrapper and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including
|
||||
its restraint: one attempt, one confirmation, and anything still wrong stays loudly wrong.
|
||||
|
||||
**THE RECORD IS THE POINT, AND IT IS THE HALF R-190 IS ACTUALLY ABOUT.** A repair that leaves only
|
||||
`ok` behind destroys the only evidence a permission vanished, so a recurring loss becomes undetectable
|
||||
forever — strictly worse than the fault it fixes. So a confirmed repair reports **DEGRADED for exactly
|
||||
one cycle**, with the explanation in `Feature`:
|
||||
|
||||
```
|
||||
backup tier felhom-backup: the agent's storage grant was MISSING and has been AUTOMATICALLY
|
||||
RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)
|
||||
```
|
||||
|
||||
**Nothing new was built to carry it.** The hub's existing ok→degraded→ok edge is the channel — it
|
||||
alerts and e-mails on the first edge and logs the recovery on the next cycle, so one loss produces
|
||||
exactly one alert pair. No wire change, no hub change, no new event type. `Feature` carries the text
|
||||
because that is the field the hub interpolates into the operator's e-mail; `Reason` does not travel.
|
||||
|
||||
**Bounded (Scenario F):** one attempt per tier per hour, in memory. A storage can be unreadable for
|
||||
reasons an ACL cannot fix, and a re-grant on every report cycle is a repair loop wearing a fix's
|
||||
clothes. An agent restart re-arms it, which is correct — a restart is exactly when a box should
|
||||
re-check what it depends on.
|
||||
|
||||
**A failed repair never masks the fault:** the capability stays degraded with the failure in its
|
||||
reason, and a repair that "succeeded" but did not survive the re-read is reported as needing a human.
|
||||
|
||||
## v0.123.0 — a tier the box cannot READ now says so (2026-08-03, R-185)
|
||||
|
||||
**The missing permission is one command. The silence was the defect.** On demo-felhom the agent's PVE
|
||||
token held `FelhomAgentStore` on `local`, `local-lvm` and `felhom-pbs` — and **not** on
|
||||
`felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked for
|
||||
that storage's content the API answers `{"data":[]}` while root sees three archives (6.1–6.3 GB,
|
||||
08-01/02/03).
|
||||
|
||||
**An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return.** `pickForThisRun` skips
|
||||
an empty tier — correctly, because a fresh offsite tier legitimately has nothing yet — and reports
|
||||
*"no settled archive yet"*. So that tier was never restore-testable on that box and nothing ever
|
||||
mentioned it. This project's own rule, in a new place: an empty answer is not evidence that there is
|
||||
nothing there.
|
||||
|
||||
**The permission question, unlike the listing, has a definite answer — so it is asked directly.**
|
||||
`Client.Permissions` reads `GET /access/permissions?path=/storage/<target>` **as the agent's own
|
||||
token** (asking as root answers a different question and always says yes), and one
|
||||
`capability.Status` per configured tier reports the result. It composes *around* the sudo prober, the
|
||||
way the pool-read check already does — an API read does not belong inside a sudo-policy probe.
|
||||
|
||||
**MEASURED FIRST, and the obvious reading is wrong.** The ungranted path does not answer empty and
|
||||
does not 403:
|
||||
|
||||
```
|
||||
/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
|
||||
/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
|
||||
```
|
||||
|
||||
It answers with the privileges **inherited** from the box-wide `/` grant. A probe asking *"is the
|
||||
path present?"* or *"does it have `Datastore.Audit`?"* would report the blinded storage healthy — so
|
||||
the probe tests for `Datastore.AllocateSpace` specifically, and a red-proof pins that.
|
||||
|
||||
**Decisions, each weighed once:**
|
||||
|
||||
- **The probed set comes from the box's own config** (`BackupTiers()`), not a fixed list. A hardcoded
|
||||
probe list is exactly the defect being fixed, reproduced inside the fix.
|
||||
- **CRITICAL**, because the hub alerts only on critical and a non-critical entry would ride the
|
||||
report and alert nobody — the same silence with extra steps. **Except** the `local` fallback
|
||||
target, which host-install's own comment calls the DEGRADED configuration: it is still probed and
|
||||
still reported, but it does not page, because turning an ordinary documented setup into an alert
|
||||
is how a signal becomes something an operator archives unread.
|
||||
- **It never looks at content**, so it cannot alarm on a newborn tier by construction.
|
||||
- **It never reports ok when it could not ask.** An unreachable PVE is degraded: a self-check that
|
||||
fails open converts *"I do not know"* into *"fine"*.
|
||||
|
||||
The wire shape (`capability.Status`) is unchanged, so the hub's existing critical-degraded alert
|
||||
applies with no hub change and no hub bump.
|
||||
|
||||
## v0.122.0 — three ways the signals lied about themselves (2026-08-03, R-189 · R-188 · R-186)
|
||||
|
||||
All three are the reporting and release path misreporting its own work. **No customer machine, no
|
||||
backup, no restore, no disk layout, no data.** The restore-test itself and when it runs are unchanged
|
||||
from v0.121.1.
|
||||
|
||||
### R-189 — a passing restore-test no longer vanishes on a restart
|
||||
|
||||
`restore_tests[]` came only from the in-memory `backup.Store`, whose own comment read *"lost on
|
||||
restart; the cadence re-populates"*. That was true under a timer. It stopped being true when R-86 made
|
||||
the agent refuse to re-test an archive it has already proven: a proof lost to a restart is not
|
||||
repeated for a whole archive generation — **a week on the offsite tier** — and the hub calls the tier
|
||||
unproven for all of it.
|
||||
|
||||
**Observed, not predicted (2026-08-03):** a real 14.5 GB offsite restore-test PASSED at 15:25:14, the
|
||||
agent was restarted 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two
|
||||
host-reports.
|
||||
|
||||
The durable proof already existed — `RestoreTestState`, on disk, per tier, with the archive since
|
||||
R-86 — and `Snapshot()` had carried the doc comment *"for the host-report gauge"* since the day it was
|
||||
written **with no caller at all**: a seam built, documented, and never connected. It now carries the
|
||||
`tier` and what was `verified` as well (stored at proof time, when they are known for certain, rather
|
||||
than derived later by a storage lookup that can fail), and `ProvenRestoreTests` renders them as report
|
||||
entries which the collector merges.
|
||||
|
||||
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** A fresh failure beats a stored
|
||||
success — the failure is the news and lives nowhere else; a stored success beats a stale in-memory
|
||||
entry after a restart; a tier never appears twice, which the hub would read as two tests. An
|
||||
unparseable timestamp counts as older, so a malformed entry cannot displace a good one.
|
||||
- **It refuses to lie.** A record missing the archive or the tier produces NO entry, and run mechanics
|
||||
(scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one
|
||||
would be. An unproven tier reading as proven would be worse than the defect being fixed.
|
||||
- **Only successes are persisted, and that asymmetry is now written down where it will be read:** a
|
||||
success suppresses future work, so losing it leaves the system quietly less tested than it believes;
|
||||
a failure causes future work and heals itself at the next evaluation.
|
||||
- The `Store` comment that stopped being true is corrected in place rather than left to mislead.
|
||||
|
||||
### R-188 — a correct release no longer emails a failure
|
||||
|
||||
`on: [push]` fires the gates workflow on the **tag** push, and the release pushed its tag *before*
|
||||
publishing, so CI ran the published-versions gate in the seconds before the package existed and
|
||||
correctly reported it missing. Measured across two releases in one session: runs 12/13 and 17/18, same
|
||||
sha each time, opposite results — a race, not a rule. R-168 made that mail the thing that cannot be
|
||||
missed; one that is wrong half the time is one you stop reading.
|
||||
|
||||
**Only the tag PUSH moved** (build → tag locally → publish → push tag). The tag is still created before
|
||||
anything is published, so the build and the tag still describe the same commit; it simply becomes
|
||||
*visible* — to CI, and to any `raw/tag/…` fetch — once the package is downloadable.
|
||||
|
||||
The invariant the old order protected is **not traded away**: `check-published-versions.py` now asserts
|
||||
the converse directly — **no published version may be missing its tag** — as a bounded probe of the
|
||||
frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set every
|
||||
run because a check whose coverage is invisible reads as a guarantee it is not making. The package
|
||||
listing api still answers **401** without a token (re-measured), so absence cannot be enumerated, and
|
||||
the script says so.
|
||||
|
||||
A publish that succeeds and a tag push that then fails now **dies loudly**, printing the one-line
|
||||
recovery; and a publish that *fails* removes the local-only tag so the release can simply be retried
|
||||
instead of colliding with itself.
|
||||
|
||||
### R-186 — a released binary can now be verified by rebuilding it
|
||||
|
||||
`go build` stamps a module version derived from VCS state, so a build made before the tag existed and
|
||||
a rebuild made after it were different binaries. Measured at one commit, same source, same toolchain:
|
||||
|
||||
```
|
||||
default flags, no tag yet ... 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
||||
default flags, tagged ....... 4a38f394… 14 085 440 B (mod v0.121.99)
|
||||
-trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
||||
```
|
||||
|
||||
The stamp is removed rather than sequenced around — nothing in this repo reads it (no `ReadBuildInfo`
|
||||
caller) and the version comes from the explicit `-X main.version` ldflag. `-trimpath` additionally
|
||||
makes a rebuild from a different checkout directory match.
|
||||
|
||||
**A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and
|
||||
therefore produced a binary **74 KB smaller** than the release path built for the same version — one
|
||||
version name, two binaries, decided by which entry point was used. Both now build identically.
|
||||
|
||||
`CLAUDE.md` records the exact command an operator can run to verify a published binary independently.
|
||||
|
||||
## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3)
|
||||
|
||||
**Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that
|
||||
had just shipped.** Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
|
||||
construction. After it, *"nothing is due"* is the NORMAL outcome — and it was logged at **DEBUG**,
|
||||
which journald drops. An empty journal would then have been equally consistent with a healthy loop
|
||||
and with a dead goroutine: the exact shape the R-88 watcher was retired for, re-created in a new
|
||||
place by making the quiet path the common one.
|
||||
|
||||
A not-due evaluation now logs one **INFO** line naming every tier's verdict:
|
||||
|
||||
```
|
||||
backup: restore-test evaluated — nothing due
|
||||
verdicts="felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven;
|
||||
felhom-backup: no settled archive yet — nothing to prove (newborn or still settling)"
|
||||
```
|
||||
|
||||
Four lines a day at the 6 h default, and the answer to *"why did nothing run last night?"* is in the
|
||||
log instead of being re-derived. A tier whose storage cannot be listed reads `UNKNOWN` with its error
|
||||
in the same line, so a lookup failure can never present as "nothing due".
|
||||
|
||||
Red-proved by reverting to the bare `Debug` line: the test asserts what the SCHEDULER emits on a real
|
||||
`tick`, not what the helper returns — a helper-level test would have passed against a tick that never
|
||||
called it.
|
||||
|
||||
## v0.121.0 — a restore-test proves each BACKUP, not the clock (2026-08-03, R-86)
|
||||
|
||||
**The trigger changed; the restore-test did not.** `Scheduler.Run` still has a ticker, but it is now
|
||||
the **evaluation interval** — how often "is anything due?" is asked — and no longer the thing that
|
||||
decides a test happens. What decides is a per-archive due-check
|
||||
(`internal/backup/restoretest_due.go`):
|
||||
|
||||
> Let **A** = the newest archive on this tier that has settled for at least `settle` (default 24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
A daily tier is therefore proved once a day, on yesterday's archive; a weekly tier once a week, on
|
||||
its own; a newborn tier is UNKNOWN and never a fault. Per-archive due-ness IS the pacing — one test
|
||||
per archive generation and no more — so there is deliberately no second rate limiter on top of it.
|
||||
|
||||
**The trap this avoided, recorded because it is the version a reasonable person writes.** R-86's own
|
||||
wording ("~24 h after its own newest archive") implemented literally is *"due when the newest archive
|
||||
is ≥ 24 h old"* — and on a **daily** tier that is never true, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches 24 h. The literal rule silently switches
|
||||
restore-testing OFF for the tier that matters most. It has its own red-proof, which was observed
|
||||
failing with **0 runs over 5 simulated days**.
|
||||
|
||||
**What the fix rests on**
|
||||
|
||||
- **The state records WHICH archive was proven** (`restoretest_state.go`), not merely when a tier last
|
||||
passed — a time cannot answer "have we proven *this* archive". A pre-R-86 state file keeps its time
|
||||
(rotation ordering survives the upgrade) and yields **no** proven archive, so each tier is due
|
||||
exactly once after the upgrade: one extra test per tier, once, which is the safe direction.
|
||||
- **Two knobs replace one, and the old one is not silently repurposed.**
|
||||
`restore_test_eval_interval_seconds` (how often due-ness is asked; default **6 h**) and
|
||||
`restore_test_settle_seconds` (how long an archive must sit; default **24 h**). The deprecated
|
||||
`restore_test_cadence_seconds` keeps its DISABLE meaning (negative) verbatim, and a positive value
|
||||
now seeds the **settle lag** — with a start-up WARN naming both replacements.
|
||||
- **6 h is bounded from both sides, not picked.** MEASURED cost of one evaluation on demo-felhom
|
||||
(Part 1.4): local dir storage **18 ms**, the PBS tier over the WAN to ep0 **392 ms**, both together
|
||||
**430 ms** — cheap enough for minutes, so cost is not the constraint. The **ceiling** is: a tier
|
||||
whose restore-test keeps failing stays due, so the evaluation interval is also its RETRY interval,
|
||||
and a retry is a multi-GB restore.
|
||||
- **The due-check runs BEFORE the heavy-operation gate is taken.** Evaluations are frequent now, and
|
||||
holding that gate for a read that answers "nothing to do" would open a window at every evaluation
|
||||
in which a starting backup cannot acquire — and a backup that cannot acquire records a failure and
|
||||
pages the operator (F-A1). Nothing heavy starts before the gate.
|
||||
- **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete
|
||||
1-byte phantom (F-CRIT-2's artefact, which server-side prune does not collect) would be picked
|
||||
forever, fail forever, never earn proof, and leave the tier due at EVERY evaluation — turning the
|
||||
evaluation interval into the retry rate for a multi-GB restore. `PickRestoreCandidateOn` now
|
||||
delegates to the settle-aware picker, so both callers agree.
|
||||
|
||||
**Unchanged, deliberately:** the restore-test itself (restore → boot → verify → destroy the scratch),
|
||||
its journal, crash recovery, the scratch VMID band, the one-heavy-operation gate, success-only proof
|
||||
credit, and oldest-proven ordering — which survives as the tie-break **between due tiers**.
|
||||
|
||||
**New:** `--selftest=restore-test-due` — read-only; prints the per-tier due verdict the scheduler
|
||||
would act on, with the measured cost of the lookup.
|
||||
|
||||
**Live finding, pre-existing and NOT caused by this change (filed as R-185):** on demo-felhom the
|
||||
agent's PVE token has no ACL on `/storage/felhom-backup`, so the API returns an EMPTY content listing
|
||||
for that storage (root sees three archives). The host tier has therefore never been restore-testable
|
||||
on that box, and both R-85's rotation and R-86's due-check report it indistinguishably from "newborn"
|
||||
("no settled archive yet"). Verified live against `local` (grant present → 3 archives) and
|
||||
`felhom-backup` (no grant → `{"data":[]}`).
|
||||
|
||||
## Releasing publishes, and an unreleasable version cannot pass CI (2026-08-03, R-115 + R-183) — **NO VERSION BUMP**
|
||||
|
||||
**No Go code changed, so nothing is bumped and nothing was built.** This is the release path and a
|
||||
gate; the agent stays **v0.120.0**.
|
||||
|
||||
**`scripts/release-agent.sh` — THE way to release.** Build → **tag** → publish → **verify by an
|
||||
independent download**. Publishing used to be a separate remembered step and was **forgotten three
|
||||
times in five days** (R-111's seventeen stranded releases, 0.114.0, and 0.120.0 — which sat deployed
|
||||
on both demo hosts and undownloadable, so a documented-path reinstall would have silently downgraded
|
||||
them to the pre-merge agent *while reporting success*). R-111's own closing line named this leg and
|
||||
closed SHIPPED without it; it recurred the same afternoon, which is the evidence that a note is not a
|
||||
mechanism.
|
||||
|
||||
- It **tags** because `felhom-host-install.sh` now fetches this version's sixteen config files from
|
||||
`raw/tag/v<version>/` (R-183) — a released version without a tag 404s a box mid-install, as root.
|
||||
- It **verifies by downloading what it just published** and comparing the sha to what it built. The
|
||||
publish step's own success is a report on its own write; a fetch returning the right bytes is a
|
||||
different claim, and it is the one that matters.
|
||||
- It **refuses** a dirty or unpushed tree, and refuses to re-release an existing version — one
|
||||
version name must never mean two binaries.
|
||||
- It **does NOT vouch.** Vouching points machines at a version and stays the operator's act.
|
||||
|
||||
**`scripts/check-published-versions.py` — the gate (R-115 mechanism (b)).** Every `v<semver>` tag
|
||||
must have a downloadable package AND a tag tree that serves the agent's configs. Registered in
|
||||
`agent_gates.py` as **not `--fast`** (it needs network, and a push must not fail because Gitea
|
||||
blinked), and **the CI workflow now runs the FULL gate set** rather than `--fast` — otherwise the
|
||||
gate would have been registered and never run, which is the built-but-never-wired failure this
|
||||
project has shipped four times.
|
||||
|
||||
**The invariant is NOT the one the task specified, and the reason was measured.** The task asked for
|
||||
*"the version the hub tells machines to install must be downloadable"*. That is the better invariant
|
||||
and CI cannot see it: the hub's artifact manifest answers **401** without a per-customer passphrase,
|
||||
and Gitea's package LISTING api answers **401** without a token, while the package DOWNLOAD url and
|
||||
the git TAGS api are both anonymous. Putting an operator credential into CI to close that gap is the
|
||||
operator's call, not a gate author's. The tag-based invariant needs no credential and **catches all
|
||||
three recorded instances**, because the release script creates the tag and publishes in one act.
|
||||
**What it does not catch — the hub vouching a version that was never released at all — is recorded
|
||||
as R-184 rather than assumed away.**
|
||||
|
||||
## docs — v0.120.0 PUBLISHED + vouched, and proven on two reinstalled boxes (2026-08-03, R-178) — **no version bump, nothing built**
|
||||
|
||||
**Nothing shipped in this entry.** It records an operational fact the version history could not
|
||||
otherwise carry: **v0.120.0 had been built, committed at `cd6e267` and deployed to both demo hosts,
|
||||
but never published.** `GET https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.120.0/felhom-agent`
|
||||
returned **HTTP 404** (0.119.0 → 200), and the hub's Day-0 manifest accordingly vouched **0.119.0**.
|
||||
|
||||
**Why that mattered more than it looks.** `felhom-host-install.sh` step 5 skips the binary install
|
||||
only when `installed == vouched` *exactly*, so a documented-path reinstall would have replaced the
|
||||
merge-aware 0.120.0 with the pre-merge 0.119.0 — **and would have succeeded**, because the current
|
||||
`step_grows` passes `-sysdata-grow 0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error)
|
||||
therefore never fires. Both demo boxes would have come back on an agent that predates the merge while
|
||||
every log line read green.
|
||||
|
||||
Published this session on an operator ruling, from a clean tree (`git status --porcelain` empty,
|
||||
`HEAD == origin/main == 4bb84fc3`): `scripts/publish-agent.sh 0.120.0` → upload **HTTP 201**,
|
||||
round-trip GET verified, `AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`;
|
||||
then vouched in the hub manifest, which resolved the sha authoritatively from Gitea rather than
|
||||
trusting the submitted value.
|
||||
|
||||
Both reinstalls then fetched and sha-verified it over the real customer path —
|
||||
`verified sha256 a7763d31b55b5ce7… matches the hub manifest` — and 4b's single-volume grow was
|
||||
exercised live on both boxes (`data +46G (->70G, ONE volume)` on demo-hp,
|
||||
`+226G (->250G)` on demo-felhom), each producing `mp0` at `/var/lib/felhom` with **no `mp1`**.
|
||||
|
||||
**Filed as the third instance of R-115** (*"publishing is a remembered step"*), which has been
|
||||
WAITING-ON-OPERATOR since 2026-07-29. Full evidence: `felhom.eu/REPORT.md`.
|
||||
|
||||
## v0.120.0 — one data volume (2026-08-03, R-165 · decision D-a · variant V-c) — `build-golden.sh` 2.1.0 → **3.0.0**
|
||||
|
||||
**The dedicated backup partition stops existing.** A golden built by `build-golden.sh` v3.0.0 ships a
|
||||
|
||||
@@ -1,179 +1,103 @@
|
||||
# CLAUDE.md — `felhom-agent`
|
||||
|
||||
> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
|
||||
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
|
||||
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
|
||||
> Stable orientation only — **current state lives in `CONTEXT.md` and the top of `CHANGELOG.md`**,
|
||||
> never here. Cross-repo conventions (artifact taxonomy, access, clean-tree gate, secrets,
|
||||
> CHANGELOG/REPORT): workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`. Path-scoped detail:
|
||||
> `.claude/rules/`.
|
||||
|
||||
## What this repo is
|
||||
|
||||
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
|
||||
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
|
||||
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
|
||||
The operator-tier **host agent**, one per Proxmox host, owning **all** Proxmox interaction:
|
||||
provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a
|
||||
narrow per-guest local API. It is the **most privilege-sensitive component in the system**.
|
||||
|
||||
- Renamed former `proxmox-controller` repo.
|
||||
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
|
||||
creds). Do not confuse them.
|
||||
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
|
||||
- Renamed from `proxmox-controller`.
|
||||
- **Distinct from `felhom-controller`** — that is the *in-guest* controller, Docker-only, holding no
|
||||
Proxmox credentials. Do not confuse them.
|
||||
- **Control plane, not data plane:** if the agent dies, apps keep serving; only management degrades.
|
||||
- Pure Go stdlib + `golang.org/x/crypto`. No web frameworks.
|
||||
|
||||
## Read before writing code
|
||||
## Doing X → read Y
|
||||
|
||||
- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
|
||||
in the same commit that changes a shared helper or pattern.
|
||||
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
|
||||
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
|
||||
`felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
|
||||
| Doing | Read |
|
||||
|---|---|
|
||||
| writing any new code | `REUSE.md` — helpers, format-safety guards, traps, seams |
|
||||
| needing current state / open threads | `CONTEXT.md` + the top `CHANGELOG.md` entry |
|
||||
| Proxmox, reconcile or signed jobs | loads itself: `.claude/rules/proxmox.md` |
|
||||
| local API, authz or guest hooks | loads itself: `.claude/rules/localapi.md` |
|
||||
| backup, PBS or DR | loads itself: `.claude/rules/backup.md` |
|
||||
| storage or escrow | loads itself: `.claude/rules/storage.md` |
|
||||
| writing a health check | loads itself: `.claude/rules/health-checks.md` |
|
||||
| **release, build, publish, deploy, verify a version** | the **`felhom-build-deploy`** skill — **never hand-roll it** |
|
||||
| writing or reviewing a test, fixing a bug | the **`felhom-testing`** skill |
|
||||
| host addresses, break-glass, node facts | `felhom.eu/documentation/operations/nodes.md` — never restate them |
|
||||
| which box may I break | `felhom.eu/documentation/runbooks/target-selection.md` |
|
||||
| what version is live anywhere | ask the hub (`/hosts`, `/configs`) or the box — **never a doc** |
|
||||
| the authoritative design | `felhom.eu/documentation/architecture/03-host-agent.md` (locked) |
|
||||
|
||||
## Layout (verified against the tree)
|
||||
## The root-CLI fence — API-first, exactly three exceptions
|
||||
|
||||
```
|
||||
cmd/felhom-agent/ main + flags + --selftest modes + the daemon entry
|
||||
cmd/felhom-opsign/ offline operator signing CLI (SSHSIG)
|
||||
internal/authz/ operator signed-op verifier (SSHSIG) + durable FileNonceStore
|
||||
internal/backup/ vzdump backup runner + restore-test scheduler + report store
|
||||
internal/capability/ live sudo-policy capability probe (degradation visibility)
|
||||
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
|
||||
internal/desired/ hub desired-state syncer (envelope observer)
|
||||
internal/escrow/ PBS-key escrow (zero-knowledge recovery code)
|
||||
internal/guesthook/ pre-start self-heal hookscript install
|
||||
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
|
||||
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
|
||||
internal/localapi/ per-guest local API: token store, disks/format, guest binds, controller swap,
|
||||
stale-lock recovery, pinned self-signed leaf
|
||||
internal/log/ slog setup
|
||||
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop
|
||||
internal/provision/ guest bootstrap back-half (token mint → bootstrap.json → pct bind)
|
||||
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
|
||||
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
|
||||
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
|
||||
internal/storage/ storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
|
||||
```
|
||||
|
||||
## Build / run
|
||||
|
||||
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
|
||||
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
|
||||
DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
|
||||
host — build and run live tests locally.
|
||||
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
|
||||
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
|
||||
|
||||
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
|
||||
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
|
||||
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
|
||||
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
|
||||
> elsewhere).
|
||||
|
||||
| Step | Where | One-liner |
|
||||
|---|---|---|
|
||||
| Build | DooPlex (local) | `cd /mnt/5_hdd/felhom.eu/git/felhom-agent && git pull && go build -ldflags '-X main.version=<v>' -o /tmp/felhom-agent-<v> ./cmd/felhom-agent` |
|
||||
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
||||
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
||||
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
||||
| Publish | DooPlex (local) | `scripts/publish-agent.sh <ver> <bin>` (REGISTRY_* creds); hub Day-0 manifest vouch = operator follow-up |
|
||||
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
||||
|
||||
## Proxmox model (the load-bearing rules)
|
||||
This is in the core because breaching it is how this component stops being auditable.
|
||||
|
||||
- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
|
||||
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
|
||||
`Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
|
||||
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
|
||||
the POST is **not** success; authorization can fail at task execution.
|
||||
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
|
||||
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
|
||||
AND the token's ACLs — the role must be granted on **both**, or every call 403s.
|
||||
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
|
||||
`Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
|
||||
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors.
|
||||
- **`Client` never shells out; `Privileged` never makes HTTP calls** — asserted by `routing_test.go`.
|
||||
Adding a method to `proxmox.Privileged` breaks the fence; use `proxmox.Runner` plus a new sudoers
|
||||
`Cmnd_Alias` and `validate.go`-style checks (`REUSE.md` §3).
|
||||
- **Destructive ops go through the reconcile gate / signed-jobs path.** Never call
|
||||
`Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc — that skips classification, signature,
|
||||
per-guest serialization and crash recovery.
|
||||
- **Ownership must be PROVEN, never assumed.** A raw `ListLXC` list is not "guests the agent owns";
|
||||
intersect with `Client.Pool` membership and fail safe on a read failure (audit A1).
|
||||
|
||||
## Demo host (for live tests)
|
||||
## Gates — ONE entry point
|
||||
|
||||
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
|
||||
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
|
||||
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
|
||||
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
|
||||
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
|
||||
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
|
||||
**Which box is safe to break, and what may be done to each:
|
||||
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
|
||||
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
|
||||
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
|
||||
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
|
||||
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
|
||||
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
|
||||
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
|
||||
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
|
||||
that is all of them. **A missing gate is a FAILURE, never a skip.**
|
||||
|
||||
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
|
||||
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
|
||||
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
|
||||
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
|
||||
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
|
||||
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
|
||||
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
|
||||
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
|
||||
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
||||
>
|
||||
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
|
||||
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
|
||||
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
|
||||
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
|
||||
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
|
||||
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
|
||||
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
|
||||
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
|
||||
when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the session
|
||||
report when you use it** — CI re-runs the same entry point on every push and **emails the operator on
|
||||
failure**, so a bypass is noticed even though it is not blocked (R-168, CLOSED 2026-08-02).
|
||||
|
||||
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
|
||||
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
|
||||
> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
|
||||
> (`cygpath -w` for the local scp path; CRLF hazard on config files).
|
||||
<!--
|
||||
WHY ONE ENTRY POINT (2026-08-02, R-29): a census of all gates across the four repos found every check
|
||||
a CLAUDE.md names was passing, and two of the four nobody is told to run were failing. This repo was
|
||||
the extreme case — nothing ran against it at all, and 90 cited paths were checked by no one.
|
||||
-->
|
||||
|
||||
## Live validation — the fence
|
||||
|
||||
Exercise the **SERVER-SIDE PIPELINE** a real user triggers, end-to-end. **The forbidden shortcut is
|
||||
BYPASSING it** — the F9 episode was a raw guest-attach with hand-set state, and it proved nothing.
|
||||
|
||||
`claude-in-chrome` is NOT available on DooPlex. Invoking the exact endpoint the UI invokes is an
|
||||
acceptable proxy — **say which method was used**. Low-level mechanism tests where the direct call IS
|
||||
the mechanism are exempt.
|
||||
|
||||
## Conventions
|
||||
|
||||
### Trunk-based — no branches
|
||||
|
||||
All shippable work commits **directly to `main`**; `main` equals what is deployed.
|
||||
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
|
||||
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
|
||||
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
|
||||
|
||||
> **In every repository where you make a change, update both files in that repo:**
|
||||
> - **`CHANGELOG.md`** — cumulative log, newest on top.
|
||||
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
|
||||
>
|
||||
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
|
||||
|
||||
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
|
||||
guess** when you'd otherwise invent input/output.
|
||||
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
|
||||
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
|
||||
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
|
||||
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
|
||||
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
|
||||
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
|
||||
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
|
||||
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
|
||||
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
|
||||
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
|
||||
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
|
||||
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
|
||||
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
|
||||
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
|
||||
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
|
||||
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
|
||||
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
|
||||
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
|
||||
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
|
||||
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
|
||||
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
|
||||
- **Trunk-based — no branches.** All shippable work commits directly to `main`; `main` equals what is
|
||||
deployed. Report-only artifacts (audits, findings, fixspecs) go to `felhom.eu/documentation/`.
|
||||
- **Unattended escape hatch:** if a fix cannot be cleanly verified and shipped, **revert and report**
|
||||
— never park it on a branch.
|
||||
- **Logging**: the slog logger fans out to journald (configured level) plus the always-DEBUG
|
||||
`applog.Ring` (remote pulls). English, keys-never-values, durations on outcomes. Full rules:
|
||||
`felhom.eu/documentation/runbooks/logging-conventions.md`.
|
||||
- Update `REUSE.md` in the same commit that adds, changes or deprecates a shared helper or pattern.
|
||||
|
||||
### Live validation
|
||||
## End-of-session checklist
|
||||
|
||||
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
|
||||
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
|
||||
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
|
||||
mechanism tests where the direct call IS the mechanism are exempt.
|
||||
|
||||
## Workflow & artifacts
|
||||
|
||||
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
|
||||
CHANGELOG + REPORT.md.
|
||||
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
|
||||
for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
|
||||
felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
|
||||
decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
|
||||
real customer data — demo scratch guests are fair game.
|
||||
- **`CHANGELOG.md`** (cumulative, newest on top) and **`REPORT.md`** (overwritten with this run only)
|
||||
— in every repo touched.
|
||||
- **`CONTEXT.md`** — decisions, state, what is next.
|
||||
- **`REUSE.md`** — if a shared helper or pattern moved.
|
||||
- **A finding goes in `felhom.eu/documentation/backlog/OPEN-ITEMS.md` first**, never only in a report
|
||||
or an audit.
|
||||
- **Confirm your own last push's CI run went green, by run ID** — CI mails on failure, which is a PUSH
|
||||
signal; this is the PULL check that catches a lost or unread mail. An unchecked green is an
|
||||
assumption, not an observation.
|
||||
|
||||
+121
@@ -3,8 +3,129 @@
|
||||
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
|
||||
> entry = current); the end-of-task detail lives in `REPORT.md`.
|
||||
|
||||
## R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked
|
||||
|
||||
`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
|
||||
customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
|
||||
(`hub.Client.FetchIdentityEscrow` → `GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
|
||||
per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
|
||||
repository password plus its sha256.
|
||||
|
||||
**Rules that must not erode:**
|
||||
- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
|
||||
trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
|
||||
- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
|
||||
no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
|
||||
- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
|
||||
never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
|
||||
log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
|
||||
- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
|
||||
the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
|
||||
at age's KDF, nothing written).
|
||||
- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
|
||||
`main` → `runDaemon` → `buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
|
||||
`Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
|
||||
self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
|
||||
Links 6 and 7 were two of this project's six built-but-never-wired instances.
|
||||
|
||||
**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
|
||||
A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
|
||||
recovered password, reopens a repository, or restores a file.
|
||||
|
||||
**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
|
||||
"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
|
||||
into the same bundle — anyone reading it would conclude the password was not there. It now names what
|
||||
THIS bundle carried and what it did not.
|
||||
|
||||
## Current
|
||||
|
||||
- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
|
||||
`FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
|
||||
storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
|
||||
through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
|
||||
archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
|
||||
restore-testable and nothing said so.
|
||||
- **The permission question is asked directly**, because unlike the listing it has a definite
|
||||
answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
|
||||
own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
|
||||
composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
|
||||
belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
|
||||
degraded alert applies with **no hub change**.
|
||||
- **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
|
||||
403 — it carries the privileges INHERITED from the box-wide `/` grant
|
||||
(`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
|
||||
blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
|
||||
changing that constant (`storeGrantRequiredPriv`, red-proved).
|
||||
- **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
|
||||
defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
|
||||
does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
|
||||
ok when it could not ask.
|
||||
- **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
|
||||
grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
|
||||
and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
|
||||
correctly picking the 08-02 archive (08-03 had not settled 24 h).
|
||||
- **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
|
||||
grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.
|
||||
|
||||
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
|
||||
None touches data; all three cost attention, which every other signal depends on.
|
||||
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
|
||||
from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
|
||||
timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
|
||||
a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
|
||||
twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
|
||||
still read, and a record missing archive-or-tier is NOT reported), exposes
|
||||
`ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
|
||||
newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
|
||||
twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
|
||||
"host-report gauge" in its doc comment and had **no caller** for weeks.
|
||||
- **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
|
||||
future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
|
||||
tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
|
||||
- **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
|
||||
locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
|
||||
made the gate correctly fail a correct release ~half the time. The old order's invariant is now
|
||||
asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
|
||||
a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
|
||||
- **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
|
||||
bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
|
||||
`CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
|
||||
identical. The verification command is in `CLAUDE.md`.
|
||||
|
||||
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
|
||||
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
|
||||
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
|
||||
weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
|
||||
literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
|
||||
tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
|
||||
where it matters most. Red-proved at 0 runs over 5 simulated days.
|
||||
- **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
|
||||
keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
|
||||
upgrade, deliberately.
|
||||
- **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
|
||||
(verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
|
||||
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
|
||||
(default 24 h). It is NOT carried into the evaluation interval.
|
||||
- **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
|
||||
both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
|
||||
evaluation interval is also its retry interval for a multi-GB restore.
|
||||
- The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
|
||||
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
|
||||
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
|
||||
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
|
||||
- **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
|
||||
DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
|
||||
a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
|
||||
evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
|
||||
with its error in that same line.
|
||||
- **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
|
||||
encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
|
||||
named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
|
||||
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
|
||||
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
|
||||
host tier has never been restore-testable there, and the due-check cannot distinguish that from
|
||||
a newborn tier.
|
||||
|
||||
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
|
||||
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
|
||||
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
|
||||
|
||||
@@ -1,41 +1,41 @@
|
||||
# REPORT — CI runs the gate entry point on every push (R-168, 2026-08-02)
|
||||
# REPORT — felhom-agent v0.127.0: a mount Felhom made is not foreign (R-220)
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (session 1's gate entry point, same day) have their durable record in `CHANGELOG.md`.
|
||||
|
||||
**No version bump, no build, no deploy.** This adds `.gitea/workflows/gates.yml` and a CHANGELOG
|
||||
entry. Nothing compiled changed.
|
||||
**Scope: the host half of R-220.** The customer-facing refusal message is the controller's half and
|
||||
ships as felhom-controller v0.203.0.
|
||||
|
||||
## What changed
|
||||
|
||||
`.gitea/workflows/gates.yml` — on every push, a Gitea Actions runner obtains this repo at the
|
||||
**exact pushed SHA** (shallow `git fetch`, no `uses:` step anywhere) and runs
|
||||
`python3 scripts/agent_gates.py --fast` and nothing else. The exit code is the job's result: no `|| true`, no
|
||||
pipe that could swallow it.
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `internal/storage/claim.go` | `claimFacts.felhomOwnedMounts`; `classifyClaim` forgives a non-managed mountpoint **only when corroborated**; `felhomOwnedMounts()` + `procMounts()` |
|
||||
| `internal/storage/hostops.go` | `mountTable` seam (nil ⇒ real `/proc/mounts`) |
|
||||
| `internal/storage/claim_r220_test.go` | new — the own-drive case, the fence, and the corroboration's four edges |
|
||||
|
||||
**It REPORTS, it cannot REFUSE**, and the workflow header says so in the pre-push hook's voice: this
|
||||
repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand
|
||||
at. The refusing half is `.githooks/pre-push` (per-clone, `--no-verify`-able); this half notices when
|
||||
that was skipped. Making CI blocking needs branch protection plus a PR workflow — an operator
|
||||
decision, tracked as `felhom.eu` `OPEN-ITEMS.md` **R-169**.
|
||||
## The shape chosen, and why (§7.3)
|
||||
|
||||
**A failed run emails the operator.** Probe P5 measured that Gitea itself sends **nothing** on a
|
||||
failed run — no mail, no notification row, no log line — so the workflow sends its own via Resend and
|
||||
prints the provider's accepted id, making "a message left the machine" an observable. Demonstrated on
|
||||
a real red run in `felhom.eu`: `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
|
||||
**Candidate (b): the claimed check distinguishes a mount Felhom made from a foreign one** — the task
|
||||
called it "nearer the truth" and it is, because the host and its knowledge survive the rebuild while
|
||||
the guest's registry does not. Candidate (a) — having the rebuild path clear the raw mounts — would
|
||||
have made correctness depend on a cleanup step running, and a cleanup that does not run leaves exactly
|
||||
today's defect.
|
||||
|
||||
**CI reproduces the workspace's SIBLING LAYOUT on purpose.** This repo's entry point invokes the
|
||||
shared `reuse_refs_check.py` that lives in the `felhom.eu` clone next door and is deliberately never
|
||||
copied here, and this repo's `REUSE.md` cites `hub/internal/store/dr_recipe.go`, which lives in the hub. The workflow clones
|
||||
`felhom.eu` as a sibling; without it the gate fails **closed** with `gate is MISSING` — correctly,
|
||||
but for the wrong reason.
|
||||
**The discriminator is corroboration, not a path prefix**: the same device must ALSO be mounted under
|
||||
`/mnt/felhom-drives`. Only enrolment produces that pairing.
|
||||
|
||||
## Verification
|
||||
**`/proc/mounts` rather than `lsblk MOUNTPOINTS`**, because the lsblk invocation is pinned verbatim in
|
||||
the sudoers file; changing it would have coupled this fix to a config rollout. `/proc/mounts` is
|
||||
world-readable and needs neither.
|
||||
|
||||
First run: run #1, id=9, conclusion **success**, sha `eb991445`.
|
||||
## Green gate
|
||||
|
||||
**CI and the local pre-push hook AGREE**, which is the check that mattered: `90 cited paths — exact 88, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: felhom.eu)` — identical to the local run.
|
||||
`go build` · `go vet` clean · `go test ./...` → **29 packages ok** · `agent_gates.py --fast` → all OK.
|
||||
|
||||
The runner is unprivileged host-mode, shared with the other three repos on a single owner-scoped
|
||||
registration (measured: all four repos' tasks claimed by `felhom-gates-runner`). Full probe detail,
|
||||
the security posture and the teardown: `felhom.eu/documentation/audits/SPIKE-ci-runner-2026-08-02.md`
|
||||
and `felhom.eu/REPORT.md`.
|
||||
| Red-proof | Result |
|
||||
|---|---|
|
||||
| remove the `felhomOwnedMounts` exemption | **FAILS** — "device is mounted at /mnt/adatok (sdb)", the pre-fix refusal |
|
||||
| over-widen the exemption to any `/mnt/*` | **FAILS** — "/mnt/someone-elses-disk was offered for formatting" |
|
||||
|
||||
## Not changed
|
||||
|
||||
No sudoers, no allowlisted command, no PVE surface, no format path. Every other claim signal
|
||||
(system disk, read-only, LVM PV, ZFS member, member FSTYPEs, empty-topology backstop) is untouched.
|
||||
|
||||
@@ -148,8 +148,10 @@
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,t)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test rotation state, persisted (atomic tmp+rename) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. |
|
||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target) (string,error)` | The per-run restore-test spec + per-tier candidate lookup (R-85) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. |
|
||||
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
|
||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
|
||||
// grepping for a string.
|
||||
//
|
||||
// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
|
||||
// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
|
||||
// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
|
||||
// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
|
||||
// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
|
||||
// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
|
||||
// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
|
||||
|
||||
func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
|
||||
t.Helper()
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
|
||||
if err != nil {
|
||||
t.Fatalf("parsing main.go: %v", err)
|
||||
}
|
||||
return fset, f
|
||||
}
|
||||
|
||||
// callsWithin returns the set of function names called (directly, by identifier or selector) inside
|
||||
// the named top-level function.
|
||||
func callsWithin(f *ast.File, fnName string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn := ce.Fun.(type) {
|
||||
case *ast.Ident:
|
||||
out[fn.Name] = true
|
||||
case *ast.SelectorExpr:
|
||||
if x, ok := fn.X.(*ast.Ident); ok {
|
||||
out[x.Name+"."+fn.Sel.Name] = true
|
||||
}
|
||||
out[fn.Sel.Name] = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
|
||||
func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
|
||||
_, f := parseMain(t)
|
||||
|
||||
// 1. main() reaches runDaemon.
|
||||
if !callsWithin(f, "main")["runDaemon"] {
|
||||
t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
|
||||
}
|
||||
// 2. runDaemon reaches buildLocalAPIServer.
|
||||
if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||
t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
|
||||
}
|
||||
|
||||
// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
|
||||
// an escrow.OffsiteKeyRecoverer is constructed there.
|
||||
var optionsHasField, recovererConstructed bool
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
cl, ok := n.(*ast.CompositeLit)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := cl.Type.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
pkg, _ := sel.X.(*ast.Ident)
|
||||
if pkg == nil {
|
||||
return true
|
||||
}
|
||||
switch pkg.Name + "." + sel.Sel.Name {
|
||||
case "localapi.Options":
|
||||
for _, el := range cl.Elts {
|
||||
kv, ok := el.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
|
||||
optionsHasField = true
|
||||
}
|
||||
}
|
||||
case "escrow.OffsiteKeyRecoverer":
|
||||
recovererConstructed = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !recovererConstructed {
|
||||
t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
|
||||
"production assembly point (the built-but-never-wired shape, seventh instance)")
|
||||
}
|
||||
if !optionsHasField {
|
||||
t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
|
||||
"exists and the route would answer 503 forever")
|
||||
}
|
||||
}
|
||||
|
||||
// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
|
||||
// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
|
||||
// key is used.
|
||||
func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
|
||||
fset, f := parseMain(t)
|
||||
var fetchUsesHubClient bool
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
|
||||
return true
|
||||
}
|
||||
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
|
||||
fetchUsesHubClient = true
|
||||
} else {
|
||||
t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
|
||||
fset.Position(ce.Pos()))
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !fetchUsesHubClient {
|
||||
t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
|
||||
"not wired, or it uses a client whose credentials are not this host's")
|
||||
}
|
||||
}
|
||||
|
||||
// The route itself must be registered on the local API. A handler with no route is the same defect
|
||||
// one layer down, and it has shipped here before.
|
||||
func TestRecoverRouteIsRegistered(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parsing localapi/server.go: %v", err)
|
||||
}
|
||||
var registered bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
ce, ok := n.(*ast.CallExpr)
|
||||
if !ok || len(ce.Args) < 2 {
|
||||
return true
|
||||
}
|
||||
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "HandleFunc" {
|
||||
return true
|
||||
}
|
||||
lit, ok := ce.Args[0].(*ast.BasicLit)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
|
||||
registered = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !registered {
|
||||
t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
|
||||
"exists and nothing can reach it")
|
||||
}
|
||||
}
|
||||
+461
-10
@@ -24,6 +24,7 @@ import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -165,7 +166,7 @@ func main() {
|
||||
showVersion bool
|
||||
)
|
||||
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
||||
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
|
||||
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
|
||||
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
|
||||
@@ -238,6 +239,8 @@ func main() {
|
||||
os.Exit(runSelftestBackup(context.Background(), cfg, logger, vmid))
|
||||
case "restore-test":
|
||||
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
|
||||
case "restore-test-due":
|
||||
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
|
||||
case "pbs-verify":
|
||||
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
|
||||
case "lanresolver":
|
||||
@@ -409,6 +412,318 @@ func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
|
||||
return s
|
||||
}
|
||||
|
||||
// storeGrantStatuses probes whether the agent's OWN TOKEN may read the storages this box depends
|
||||
// on — one capability.Status per configured backup tier (R-185).
|
||||
//
|
||||
// ── WHY THIS EXISTS, AND WHY IT IS NOT A CONTENT LISTING ─────────────────────────────────────
|
||||
//
|
||||
// On demo-felhom the token had FelhomAgentStore on local, local-lvm and felhom-pbs — and NOT on
|
||||
// `felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked
|
||||
// for that storage's content the API answers `{"data":[]}` while root sees three archives.
|
||||
//
|
||||
// **An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return**, and no care at that
|
||||
// call site can separate them: `pickForThisRun` skips an empty tier (correctly — a fresh offsite
|
||||
// tier legitimately has nothing) and says "no settled archive yet". So the host tier on that box was
|
||||
// never restore-testable and nothing ever mentioned it. That is this project's own rule failing in a
|
||||
// new place: an empty answer is not evidence that there is nothing there.
|
||||
//
|
||||
// The permission question, unlike the listing, has a DEFINITE answer — so it is asked directly.
|
||||
//
|
||||
// ── WHAT IS PROBED, AND WHY NOT A FIXED LIST ─────────────────────────────────────────────────
|
||||
//
|
||||
// The tiers come from this box's own config (`BackupTiers()`), because a hardcoded probe list is
|
||||
// precisely the defect being fixed — the installer's hardcoded ACL set is what drifted from the
|
||||
// target it went on to configure. Probing what the box says it depends on cannot drift from it.
|
||||
//
|
||||
// CRITICAL, deliberately: a tier the agent cannot read is a tier whose backups are invisible to it
|
||||
// and which is never restore-tested. The hub alerts only on Critical, and a non-critical entry here
|
||||
// would ride the report and alert nobody — the same silence with extra steps.
|
||||
//
|
||||
// One exception, so an ordinary configuration is not turned into an alarm: a box with no dedicated
|
||||
// target (`local_backup_target: "local"`, which host-install's own comment calls the DEGRADED
|
||||
// fallback) is not treated as critical for that tier — see storeGrantCritical.
|
||||
func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Config, repair *storeGrantRepairer) []capability.Status {
|
||||
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
|
||||
out := make([]capability.Status, 0, len(tiers))
|
||||
for _, t := range tiers {
|
||||
out = append(out, storeGrantStatus(ctx, px, t.TargetID, storeGrantCritical(t.TargetID), repair))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// storeGrantRepairReportWindow is how long after a repair the capability keeps reporting the
|
||||
// transition. It MUST exceed the hub report interval, or the record never reaches the operator.
|
||||
//
|
||||
// FOUND BY THE LIVE RUN, NOT BY THE TESTS (2026-08-04). The first implementation reported degraded
|
||||
// for exactly "one cycle" — the probe call that did the repair. But `probeAll` is invoked
|
||||
// INDEPENDENTLY by the startup/periodic self-check log and by the collector building a host report,
|
||||
// so the repairing call was the LOG's, and the report built three seconds later found the grant
|
||||
// present and reported `ok`. The agent's journal had the record; the hub had nothing; the operator
|
||||
// would have learned nothing. That is precisely the silence R-190 is about, re-created inside its own
|
||||
// mitigation.
|
||||
//
|
||||
// A latch on TIME rather than on call count fixes it: 20 minutes comfortably exceeds the 900 s report
|
||||
// interval, so at least one host-report must carry the transition, and it still clears on its own.
|
||||
const storeGrantRepairReportWindow = 20 * time.Minute
|
||||
|
||||
// storeGrantRepairMinInterval bounds how often a single tier's grant may be re-granted (Scenario F).
|
||||
//
|
||||
// A storage can be unreadable for reasons an ACL cannot fix — the storage is gone, PVE is wedged,
|
||||
// the wrapper is missing. Without a bound the probe would re-grant on every report cycle forever: a
|
||||
// repair loop is a new defect wearing a fix's clothes. One attempt per tier per hour is frequent
|
||||
// enough that a real loss is repaired within one backup window, and rare enough that a permanent
|
||||
// fault produces attempts you can count on one hand per day.
|
||||
const storeGrantRepairMinInterval = time.Hour
|
||||
|
||||
// storeGrantRepairer bounds and records the self-repair. It is deliberately in-memory: an agent
|
||||
// restart re-arms the repair, which is correct — a restart is exactly when a box should re-check
|
||||
// everything it depends on.
|
||||
type storeGrantRepairer struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
|
||||
log *slog.Logger
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||
}
|
||||
|
||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||
func (r *storeGrantRepairer) noteRepaired(target string, now time.Time) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.repaired == nil {
|
||||
r.repaired = map[string]time.Time{}
|
||||
}
|
||||
r.repaired[target] = now
|
||||
}
|
||||
|
||||
// recentlyRepaired reports whether a confirmed repair is still inside its report window — the latch
|
||||
// that guarantees a host-report carries the transition even though the probe that repaired may have
|
||||
// been a log-only one.
|
||||
func (r *storeGrantRepairer) recentlyRepaired(target string, now time.Time) bool {
|
||||
if r == nil {
|
||||
return false
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
t, ok := r.repaired[target]
|
||||
return ok && now.Sub(t) < storeGrantRepairReportWindow
|
||||
}
|
||||
|
||||
// mayAttempt reports whether a repair may run now for this target, and records the attempt if so.
|
||||
func (r *storeGrantRepairer) mayAttempt(target string, now time.Time) bool {
|
||||
if r == nil || r.run == nil {
|
||||
return false
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.last == nil {
|
||||
r.last = map[string]time.Time{}
|
||||
}
|
||||
if t, ok := r.last[target]; ok && now.Sub(t) < storeGrantRepairMinInterval {
|
||||
return false
|
||||
}
|
||||
r.last[target] = now
|
||||
return true
|
||||
}
|
||||
|
||||
// repair runs the EXISTING root wrapper's `grant` verb for this storage. It adds no privileged
|
||||
// surface: `felhom-backup-target-apply grant *` is already in the sudoers allowlist for any storage
|
||||
// id (configs/felhom-agent.sudoers), and the verb already grants BOTH the user and the token — a
|
||||
// privsep token's rights are the intersection, so granting one of the two grants nothing usable.
|
||||
//
|
||||
// This is the pbsdr shape (internal/pbsdr/manager.go, the R-22 self-grant): on a refusal, run the
|
||||
// root wrapper and RE-READ ONCE rather than dead-locking. Its restraint is copied too — one attempt,
|
||||
// one confirmation, and anything still wrong stays loudly wrong.
|
||||
func (r *storeGrantRepairer) repair(ctx context.Context, target string) error {
|
||||
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
_, errOut, err := r.run(rctx, localapi.BackupTargetWrapperPath, "grant", target)
|
||||
if err != nil {
|
||||
r.log.Error("store-grant: SELF-REPAIR FAILED — the tier stays unreadable",
|
||||
"target", target, "err", err, "stderr", strings.TrimSpace(string(errOut)))
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// storeGrantRequiredPriv is the privilege whose ABSENCE was measured to blind the content listing.
|
||||
//
|
||||
// Measured on demo-felhom 2026-08-03: the two storages that list through the token hold
|
||||
// Datastore.Allocate + Datastore.AllocateSpace (the FelhomAgentStore role); the one that answers
|
||||
// empty holds only what the box-wide grant propagates (Sys.Audit, SDN.Use, Datastore.Audit). It is
|
||||
// NOT Datastore.Audit that is missing — checking for that would report the blinded storage healthy.
|
||||
const storeGrantRequiredPriv = "Datastore.AllocateSpace"
|
||||
|
||||
// storeGrantCritical decides whether a missing grant on this target is Critical (operator-paged).
|
||||
//
|
||||
// "local" is host-install's DEGRADED fallback target — a box with no dedicated backup storage is a
|
||||
// known, ordinary configuration, and turning it into a critical alert is how a signal becomes
|
||||
// something an operator archives unread. It is still probed and still reported; only the paging
|
||||
// differs.
|
||||
func storeGrantCritical(targetID string) bool { return targetID != "local" }
|
||||
|
||||
// storeGrantStatus is one tier's grant probe. It NEVER reports ok when it could not ask: a
|
||||
// self-check that fails open is worse than none, because it converts "I do not know" into "fine".
|
||||
func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string, critical bool, repair *storeGrantRepairer) capability.Status {
|
||||
s := capability.Status{
|
||||
Name: "pve:store-grant:" + targetID,
|
||||
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
|
||||
Critical: critical,
|
||||
Status: capability.StatusOK,
|
||||
}
|
||||
if px == nil {
|
||||
s.Status, s.Reason = capability.StatusDegraded, "not configured"
|
||||
return s
|
||||
}
|
||||
if targetID == "" {
|
||||
s.Status, s.Reason = capability.StatusDegraded, "tier has no target id"
|
||||
return s
|
||||
}
|
||||
pctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
privs, err := px.Permissions(pctx, "/storage/"+targetID)
|
||||
s = storeGrantVerdict(targetID, critical, privs, err)
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
if s.Status != capability.StatusDegraded {
|
||||
// Healthy — but if this tier was repaired moments ago, keep REPORTING the transition until a
|
||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||
// comment).
|
||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||
}
|
||||
|
||||
// ── R-190 mitigation: the grant is missing — repair it, and SAY that it was missing ──────────
|
||||
//
|
||||
// R-190 is a grant that demonstrably worked at 04:44 and was gone by 09:24, with a reinstall,
|
||||
// logged pveum activity and cluster-log entries all ruled out. The cause is still open; the
|
||||
// resilience does not have to wait for it. Everything needed already exists — the root wrapper,
|
||||
// its sudoers vector for any storage id, and the exact command — and until now the `grant` verb
|
||||
// had only ever been called at CREATION. That is the "built but never wired" shape, in a verb
|
||||
// rather than a seam.
|
||||
if !repair.mayAttempt(targetID, time.Now()) {
|
||||
// Bounded (Scenario F): an earlier attempt did not hold and it is too soon to try again. Stay
|
||||
// degraded and say why — a quiet "we already tried" is how a permanent fault becomes silence.
|
||||
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||
" and a self-repair was attempted within the last " + storeGrantRepairMinInterval.String() +
|
||||
" without holding — NOT retrying yet; this needs a human"
|
||||
return s
|
||||
}
|
||||
if rerr := repair.repair(ctx, targetID); rerr != nil {
|
||||
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||
" and the self-repair FAILED (" + rerr.Error() + ") — this tier's archives are INVISIBLE to the agent"
|
||||
return s // Scenario E: a failed repair must never mask the degraded state.
|
||||
}
|
||||
// Re-read ONCE to confirm, exactly as pbsdr does — the wrapper reporting success is a claim about
|
||||
// its own write; the grant being readable is a different claim, and it is the one that matters.
|
||||
cctx, ccancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer ccancel()
|
||||
privs2, err2 := px.Permissions(cctx, "/storage/"+targetID)
|
||||
if err2 != nil || privs2[storeGrantRequiredPriv] != 1 {
|
||||
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||
" and the self-repair did not take (re-read says it is still missing) — this needs a human"
|
||||
return s
|
||||
}
|
||||
|
||||
// REPAIRED — and reported as DEGRADED for exactly this one cycle, deliberately.
|
||||
//
|
||||
// The tier works again, so "ok" would be true of this instant and would throw away the only
|
||||
// evidence that anything happened. R-190's own words: the probe sees the STATE, nothing sees the
|
||||
// TRANSITION. A silent self-repair makes a recurring loss undetectable forever, which is strictly
|
||||
// worse than the fault it fixes.
|
||||
//
|
||||
// §8.5 asked whether the hub's existing degraded↔ok edge suffices before building anything new.
|
||||
// It does — as a CHANNEL — but only if the agent deliberately reports one degraded cycle: the hub
|
||||
// alerts and e-mails on the ok→degraded edge and logs the degraded→ok recovery, so one loss
|
||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||
repair.noteRepaired(targetID, time.Now())
|
||||
s = storeGrantRepairedVerdict(targetID, critical)
|
||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
return s
|
||||
}
|
||||
|
||||
// storeGrantHealthyVerdict decides what a HEALTHY probe reports — which is not always "ok".
|
||||
//
|
||||
// Split out so the tests exercise this decision rather than a copy of it. An earlier version of this
|
||||
// guard lived inline and its red-proof PASSED, because the test asserted the latch helper instead of
|
||||
// the path that consumes it — the same hollow shape this file has now caught twice.
|
||||
//
|
||||
// If the tier was repaired inside the report window, the transition is reported even though the grant
|
||||
// is present: the probe that repaired may have been a log-only one, and without this the host-report
|
||||
// carries `ok` and the operator never learns the permission vanished (measured live 2026-08-04).
|
||||
func storeGrantHealthyVerdict(targetID string, critical bool, healthy capability.Status, repairedRecently bool) capability.Status {
|
||||
if repairedRecently {
|
||||
return storeGrantRepairedVerdict(targetID, critical)
|
||||
}
|
||||
return healthy
|
||||
}
|
||||
|
||||
// storeGrantRepairedVerdict is the post-repair verdict — the RECORD half of R-190, split out so the
|
||||
// tests exercise the real thing rather than a copy of it (yesterday's hollow-test lesson).
|
||||
//
|
||||
// It reports DEGRADED although the tier now works, and that is the whole point: "ok" would be true of
|
||||
// this instant and would throw away the only evidence that a permission vanished. The hub raises its
|
||||
// ok→degraded edge (an operator e-mail) and logs the degraded→ok recovery on the next cycle, so one
|
||||
// loss produces exactly one alert pair. Nothing new was built for this — no wire change, no hub
|
||||
// change, no new event type.
|
||||
//
|
||||
// The explanation lives in FEATURE because that is the field the hub interpolates into the operator's
|
||||
// e-mail (`monitor/host_capability.go` emitTransition builds its message from the capability names
|
||||
// and features; Reason does not travel). Putting it in Reason alone would be a record nobody reads.
|
||||
func storeGrantRepairedVerdict(targetID string, critical bool) capability.Status {
|
||||
return capability.Status{
|
||||
Name: "pve:store-grant:" + targetID,
|
||||
Critical: critical,
|
||||
Status: capability.StatusDegraded,
|
||||
Feature: "backup tier " + targetID + ": the agent's storage grant was MISSING and has been " +
|
||||
"AUTOMATICALLY RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)",
|
||||
Reason: "grant absent at probe time; `felhom-backup-target-apply grant " + targetID +
|
||||
"` re-applied it and a re-read confirms " + storeGrantRequiredPriv + " is present again",
|
||||
}
|
||||
}
|
||||
|
||||
// repairLogger returns the repairer's logger, or the default — the record must survive a nil.
|
||||
func repairLogger(r *storeGrantRepairer) *slog.Logger {
|
||||
if r != nil && r.log != nil {
|
||||
return r.log
|
||||
}
|
||||
return slog.Default()
|
||||
}
|
||||
|
||||
// storeGrantVerdict is the DECISION, split out from the API call so the tests exercise the real
|
||||
// thing rather than a copy of it. A test that re-implements this branch would pass while production
|
||||
// diverged — which is the hollow shape this project keeps finding in its own tests.
|
||||
func storeGrantVerdict(targetID string, critical bool, privs map[string]int, err error) capability.Status {
|
||||
s := capability.Status{
|
||||
Name: "pve:store-grant:" + targetID,
|
||||
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
|
||||
Critical: critical,
|
||||
Status: capability.StatusOK,
|
||||
}
|
||||
if err != nil {
|
||||
// Unreachable PVE is UNKNOWN, and unknown is reported as degraded rather than ok: a
|
||||
// self-check that fails open converts "I do not know" into "fine".
|
||||
s.Status, s.Reason = capability.StatusDegraded, "could not read own permissions: "+err.Error()
|
||||
return s
|
||||
}
|
||||
if privs[storeGrantRequiredPriv] != 1 {
|
||||
// Name the storage AND the missing role: "a storage grant is missing" without saying which
|
||||
// one costs a diagnosis at 07:00.
|
||||
s.Status, s.Reason = capability.StatusDegraded,
|
||||
"the agent token lacks "+storeGrantRequiredPriv+" on /storage/"+targetID+
|
||||
" (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// logCapabilities logs the privileged-capability self-check at startup: one INFO summary, plus an
|
||||
// ERROR per degraded capability naming the gated feature (so a missing grant is loud at cutover,
|
||||
// not days later). Inactive (config-gated off, plumbing healthy — v0.86.0) is counted in the
|
||||
@@ -490,8 +805,19 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// A1 (v0.62.0): compose the PVE pool-read check AROUND the sudo prober (an API read does not
|
||||
// belong inside the sudo-policy probe). Non-critical: a degraded pool read means the stale-lock
|
||||
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
|
||||
// R-185: the store-grant probes compose around the sudo prober the same way the pool read does
|
||||
// (an API read does not belong inside the sudo-policy probe — the v0.62.0 A1 precedent).
|
||||
// R-190: the store-grant probe also REPAIRS a missing grant, through the root wrapper that
|
||||
// already exists and is already sudoers-permitted for any storage id — and reports the loss.
|
||||
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||
// invoked through the privileged path, which prepends sudo itself.
|
||||
grantRepairer := &storeGrantRepairer{
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
}
|
||||
probeAll := func(ctx context.Context) []capability.Status {
|
||||
return append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
||||
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
||||
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
|
||||
}
|
||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||
// already carries the gated view — v0.86.0.)
|
||||
@@ -660,6 +986,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
heavyOps := &backup.InFlight{}
|
||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
||||
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
||||
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
||||
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
||||
// for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB
|
||||
// offsite restore-test reached no host-report at all.
|
||||
collector.SetProvenRestoreTests(rtState)
|
||||
|
||||
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
|
||||
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
|
||||
@@ -767,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
return false
|
||||
},
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localTokens != nil {
|
||||
defer localTokens.Close()
|
||||
}
|
||||
@@ -1273,13 +1605,21 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
|
||||
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
|
||||
// machinery still works on-demand via --selftest=restore-test.
|
||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
|
||||
cadence := cfg.Backup.RestoreTestCadence()
|
||||
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
|
||||
// per-archive due-check in internal/backup/restoretest_due.go.
|
||||
cadence := cfg.Backup.RestoreTestEvalInterval()
|
||||
if cadence > 0 {
|
||||
if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
|
||||
logger.Warn("daemon: restore-test cadence disabled (config invalid)", "err", err)
|
||||
logger.Warn("daemon: restore-test disabled (config invalid)", "err", err)
|
||||
cadence = 0
|
||||
}
|
||||
}
|
||||
if cadence > 0 && cfg.Backup.RestoreTestLegacyCadenceInUse() {
|
||||
// Said ONCE, at start-up, naming both replacements: a key whose meaning changed under a box
|
||||
// without a word is the silent repurposing R-86 §8.3 forbids.
|
||||
logger.Warn("daemon: backup.restore_test_cadence_seconds is DEPRECATED — R-86 replaced the interval trigger with a per-archive due-check; this value now seeds the SETTLE lag only. Set backup.restore_test_settle_seconds and backup.restore_test_eval_interval_seconds explicitly",
|
||||
"settle", cfg.Backup.RestoreTestSettle(), "eval_interval", cadence)
|
||||
}
|
||||
min, max := cfg.Backup.ScratchBand()
|
||||
target := cfg.Backup.BackupTarget()
|
||||
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
|
||||
@@ -1315,13 +1655,18 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
}
|
||||
},
|
||||
Cadence: cadence,
|
||||
Logger: logger,
|
||||
// R-86: the settle lag — how long an archive must have sat before it is a candidate. With
|
||||
// the per-archive due-check, this plus the archive rhythm is the whole schedule.
|
||||
Settle: cfg.Backup.RestoreTestSettle(),
|
||||
Logger: logger,
|
||||
|
||||
// R-85: rotate across EVERY configured tier, oldest-proven first (operator ruling, Option 1).
|
||||
// Before this the scheduler only ever saw cfg.Backup.BackupTarget(), so the offsite tier's
|
||||
// archives were never candidates and the DR tier went unproven for its whole existence.
|
||||
// R-86 demoted that ordering to the tie-break BETWEEN DUE TIERS and widened this picker to
|
||||
// the settle-aware one, which is what makes due-ness per archive generation.
|
||||
Tiers: tierIDs,
|
||||
TierPick: runner.PickRestoreCandidateOn,
|
||||
TierPick: runner.PickSettledRestoreCandidateOn,
|
||||
State: rtState,
|
||||
InFlight: inFlight,
|
||||
})
|
||||
@@ -1332,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
||||
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
|
||||
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
|
||||
// fixed. The opened token store is returned via outTokens so the caller can Close it.
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||
if !cfg.LocalAPI.Enabled() {
|
||||
return nil
|
||||
}
|
||||
@@ -1404,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
gaMode = proxmox.RunnerSudo
|
||||
}
|
||||
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
|
||||
// R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's
|
||||
// own hub client (per-host key, self-scoped server-side), so the recoverer can never read another
|
||||
// host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config
|
||||
// cannot reach this line (the daemon exits above), so the seam is always live in production —
|
||||
// which is the point: links 6 and 7 spent months existing without a caller.
|
||||
escrowRecoverer := escrow.OffsiteKeyRecoverer{
|
||||
Fetch: func(ctx context.Context) ([]byte, bool, error) {
|
||||
resp, ferr := hubClient.FetchIdentityEscrow(ctx)
|
||||
if ferr != nil {
|
||||
return nil, false, ferr
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return nil, false, nil
|
||||
}
|
||||
blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if derr != nil {
|
||||
return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)")
|
||||
}
|
||||
return blob, true, nil
|
||||
},
|
||||
}
|
||||
srv, err := localapi.NewServer(localapi.Options{
|
||||
EscrowRecovery: escrowRecoverer,
|
||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||
Cert: cert,
|
||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||
@@ -1751,6 +2118,67 @@ func runSelftestBackup(ctx context.Context, cfg config.Config, logger *slog.Logg
|
||||
// running → teardown) of -archive (or the newest backup on the local target) into a scratch
|
||||
// guest. Standalone (no hub). Runs engine.Recover first so a leaked scratch from a prior
|
||||
// crashed test is reaped before this run.
|
||||
// runSelftestRestoreTestDue prints the per-tier DUE verdict the scheduler would act on, and what
|
||||
// each evaluation COST — read-only, so it is safe on any box at any time.
|
||||
//
|
||||
// It exists for two reasons R-86 needed and could not get from a log line. First, the due-check's
|
||||
// verdict is the whole schedule now: "why did nothing run last night?" is answerable only by asking
|
||||
// the same question the scheduler asks, against the same storages, in the same order. Second, the
|
||||
// evaluation interval had to be chosen from a MEASURED cost rather than a guess — an offsite tier's
|
||||
// candidate lookup crosses the WAN, and a monitoring loop that costs more than it is worth is how a
|
||||
// check becomes the load. It reuses the daemon's own construction path (buildRestoreTestScheduler),
|
||||
// so what it prints is what the daemon would decide, not a re-derivation of it.
|
||||
func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *slog.Logger) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
return 1
|
||||
}
|
||||
px, err := newProxmoxClient(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err)
|
||||
return 1
|
||||
}
|
||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
|
||||
|
||||
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
|
||||
start := time.Now()
|
||||
verdicts := sched.EvaluateDue(ctx)
|
||||
total := time.Since(start)
|
||||
if len(verdicts) == 0 {
|
||||
fmt.Println("no tiers configured for restore-testing (or rotation not wired)")
|
||||
return 0
|
||||
}
|
||||
rc := 0
|
||||
for _, v := range verdicts {
|
||||
proven, _ := rtState.ProvenArchive(v.Target)
|
||||
fmt.Printf("tier=%-16s due=%-5v archive=%q landed=%s proven=%q\n reason: %s\n",
|
||||
v.Target, v.Due, v.Archive, formatOrDash(v.Landed), proven, v.Reason)
|
||||
if v.Err != nil {
|
||||
// A tier we could not list is UNKNOWN, and it is a non-zero exit: an unreadable tier is
|
||||
// a real condition, not a quiet "nothing to do".
|
||||
fmt.Printf(" ERROR: %v\n", v.Err)
|
||||
rc = 3
|
||||
}
|
||||
}
|
||||
// Per-tier timing, measured one tier at a time so the WAN leg is attributable (R-86 Part 1.4).
|
||||
for _, v := range verdicts {
|
||||
t0 := time.Now()
|
||||
_ = sched.EvaluateDueTier(ctx, v.Target)
|
||||
fmt.Printf("cost tier=%-16s one_lookup=%s\n", v.Target, time.Since(t0).Round(time.Millisecond))
|
||||
}
|
||||
fmt.Printf("cost all_tiers=%s\n", total.Round(time.Millisecond))
|
||||
return rc
|
||||
}
|
||||
|
||||
// formatOrDash renders a time, or "-" when it is zero (no archive).
|
||||
func formatOrDash(t time.Time) string {
|
||||
if t.IsZero() {
|
||||
return "-"
|
||||
}
|
||||
return t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
@@ -2465,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger *
|
||||
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
|
||||
return 1
|
||||
}
|
||||
fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest)
|
||||
// R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was
|
||||
// accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||
// offsite repository password into the same bundle. Anyone reading the old output would conclude the
|
||||
// repository password was not there, and that is part of how the chain's extraction link came to be
|
||||
// described as missing for a month. Name what was recovered from THIS bundle, and name what is
|
||||
// absent, rather than reciting a fixed list.
|
||||
recovered := []string{"tunnel_token", "pbs_token"}
|
||||
var absent []string
|
||||
if bundle.WGPrivateKey != "" {
|
||||
recovered = append(recovered, "wg_private_key")
|
||||
} else {
|
||||
absent = append(absent, "wg_private_key")
|
||||
}
|
||||
if bundle.ResticRepoPassword != "" {
|
||||
recovered = append(recovered, "restic_repo_password")
|
||||
} else {
|
||||
absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)")
|
||||
}
|
||||
fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest)
|
||||
if len(absent) > 0 {
|
||||
fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; "))
|
||||
}
|
||||
|
||||
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
|
||||
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
|
||||
@@ -2940,6 +3389,8 @@ func (f *selftestFlag) Set(v string) error {
|
||||
f.mode = "backup"
|
||||
case "restore-test":
|
||||
f.mode = "restore-test"
|
||||
case "restore-test-due":
|
||||
f.mode = "restore-test-due"
|
||||
case "pbs-verify":
|
||||
f.mode = "pbs-verify"
|
||||
case "lanresolver":
|
||||
@@ -2957,7 +3408,7 @@ func (f *selftestFlag) Set(v string) error {
|
||||
case "controller-swap":
|
||||
f.mode = "controller-swap"
|
||||
default:
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-86 Scenario I — the seam-discipline test for the due-check.
|
||||
//
|
||||
// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
|
||||
// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
|
||||
// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
|
||||
// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
|
||||
// tested, never called — and this repo has shipped it four times.
|
||||
//
|
||||
// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
|
||||
// match, and a comment is not a caller.
|
||||
func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var settlePicker, oldPicker, settleWired, evalInterval bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.SelectorExpr:
|
||||
// runner.PickSettledRestoreCandidateOn passed as a value (not called).
|
||||
switch node.Sel.Name {
|
||||
case "PickSettledRestoreCandidateOn":
|
||||
settlePicker = true
|
||||
case "PickRestoreCandidateOn":
|
||||
oldPicker = true
|
||||
}
|
||||
case *ast.KeyValueExpr:
|
||||
key, ok := node.Key.(*ast.Ident)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if key.Name == "Settle" {
|
||||
settleWired = true
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
|
||||
evalInterval = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !settlePicker {
|
||||
t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
|
||||
"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
|
||||
}
|
||||
if oldPicker {
|
||||
t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
|
||||
"two pickers means the one under test is not the one running")
|
||||
}
|
||||
if !settleWired {
|
||||
t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
|
||||
"and every freshly-landed archive would be an immediate candidate")
|
||||
}
|
||||
if !evalInterval {
|
||||
t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
|
||||
"the retired cadence knob")
|
||||
}
|
||||
}
|
||||
|
||||
// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
|
||||
// RUN, and the shared heavy-operation gate is still handed to the scheduler.
|
||||
func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var inFlightWired, specIsAFunc bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
kv, ok := n.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
key, ok := kv.Key.(*ast.Ident)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch key.Name {
|
||||
case "InFlight":
|
||||
inFlightWired = true
|
||||
case "Spec":
|
||||
// A FuncLit means it is evaluated per run; anything else is a frozen value.
|
||||
if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
|
||||
specIsAFunc = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !inFlightWired {
|
||||
t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
|
||||
"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
|
||||
}
|
||||
if !specIsAFunc {
|
||||
t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
|
||||
"(the tier and its timeout evaluated once at daemon start, forever)")
|
||||
}
|
||||
}
|
||||
|
||||
func parseMainForWiring(t *testing.T) *ast.File {
|
||||
t.Helper()
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
|
||||
//
|
||||
// This test exists because the method it feeds is the project's own cautionary tale:
|
||||
// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
|
||||
// was written and **had no caller at all** — a seam built, documented and never connected, found
|
||||
// only when a live restore-test's PASS reached no host-report. The fix must not become the next
|
||||
// instance, so the wiring is asserted rather than trusted.
|
||||
//
|
||||
// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
|
||||
// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
|
||||
func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var wired, feedsState bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
|
||||
return true
|
||||
}
|
||||
wired = true
|
||||
// ...and it must be fed the PERSISTED state, not the in-memory store.
|
||||
if len(call.Args) == 1 {
|
||||
if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
|
||||
feedsState = true
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
|
||||
if !wired {
|
||||
t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
|
||||
"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
|
||||
}
|
||||
if wired && !feedsState {
|
||||
t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
|
||||
"that does NOT survive a restart, so wiring it here would fix nothing")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,417 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"go/ast"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
|
||||
)
|
||||
|
||||
// R-185 — a tier the box cannot READ must say so.
|
||||
//
|
||||
// THE OBSERVATION (demo-felhom, 2026-08-03, reproduced at the start of this session): root lists
|
||||
// three archives on `felhom-backup`; the agent's own token gets `{"data":[]}` from the same
|
||||
// endpoint; and `local`, which has the grant, lists through that same token. The token is the
|
||||
// variable, not the storage.
|
||||
//
|
||||
// The defect is NOT the missing grant — that is one command. It is that an empty content listing is
|
||||
// what a FORBIDDEN tier and a NEWBORN tier both return, so the box could not tell them apart and
|
||||
// said nothing. These tests pin the distinction.
|
||||
|
||||
// permAnswer is the shape /access/permissions really returns, taken from the live measurement:
|
||||
// an UNGRANTED path answers with the privileges inherited from the box-wide grant — NOT empty, and
|
||||
// NOT a 403.
|
||||
var (
|
||||
permGranted = map[string]int{"Datastore.Allocate": 1, "Datastore.AllocateSpace": 1}
|
||||
permUngranted = map[string]int{"Sys.Audit": 1, "SDN.Use": 1, "Datastore.Audit": 1}
|
||||
)
|
||||
|
||||
// probeWith calls the PRODUCTION decision with a permissions answer. **Naming the seam:** everything
|
||||
// below is true up to `storeGrantVerdict`; that the live call feeds it the real API answer is what
|
||||
// Part 0's measurement established and what the live run on the box demonstrates. An earlier draft
|
||||
// of this file re-implemented the branch here — it passed, and would have kept passing while
|
||||
// production diverged, which is the hollow shape this project keeps catching in its own tests.
|
||||
func probeWith(privs map[string]int, targetID string, critical bool) capability.Status {
|
||||
return storeGrantVerdict(targetID, critical, privs, nil)
|
||||
}
|
||||
|
||||
// ── SCENARIO A — a forbidden storage is REPORTED, not passed over ────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): delete the store-grant probes from `probeAll` in
|
||||
// main.go — i.e. restore `append(capProber.Probe(ctx), poolReadStatus(ctx, px))` — and
|
||||
// TestMainWiresTheStoreGrantProbe fails with "main.go never calls storeGrantStatuses". That is
|
||||
// today's behaviour on the live box: complete silence about a tier it cannot read.
|
||||
func TestStoreGrant_ForbiddenStorageIsDegradedAndNamed(t *testing.T) {
|
||||
s := probeWith(permUngranted, "felhom-backup", true)
|
||||
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a storage the agent may not read must be DEGRADED, not %q — silence is the defect", s.Status)
|
||||
}
|
||||
if !s.Critical {
|
||||
t.Fatal("it must be CRITICAL: the hub alerts only on critical, so a non-critical entry is the same silence with extra steps")
|
||||
}
|
||||
if !strings.Contains(s.Reason, "felhom-backup") {
|
||||
t.Fatalf("the reason must NAME the storage — 'a grant is missing' costs a diagnosis at 07:00; got %q", s.Reason)
|
||||
}
|
||||
if !strings.Contains(s.Reason, "FelhomAgentStore") {
|
||||
t.Fatalf("the reason must name the ROLE to grant, so the fix is in the alert; got %q", s.Reason)
|
||||
}
|
||||
}
|
||||
|
||||
// THE TRAP THE LIVE MEASUREMENT CAUGHT, pinned so it cannot be re-introduced: the ungranted answer
|
||||
// is not empty and not a 403 — it carries the INHERITED box-wide privileges. A probe that asked
|
||||
// "did the path come back?" or "does it have Datastore.Audit?" would report the blinded storage
|
||||
// healthy.
|
||||
func TestStoreGrant_InheritedPrivilegesAreNotAGrant(t *testing.T) {
|
||||
if len(permUngranted) == 0 {
|
||||
t.Fatal("fixture wrong: the ungranted answer is NOT empty — that is the whole trap")
|
||||
}
|
||||
if permUngranted["Datastore.Audit"] != 1 {
|
||||
t.Fatal("fixture wrong: the ungranted path DOES carry Datastore.Audit, inherited box-wide")
|
||||
}
|
||||
if s := probeWith(permUngranted, "felhom-backup", true); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("checking for the wrong privilege reports a blinded storage healthy; got %q", s.Status)
|
||||
}
|
||||
// ...and the privilege actually checked is the one whose absence was measured to blind listing.
|
||||
if storeGrantRequiredPriv != "Datastore.AllocateSpace" {
|
||||
t.Fatalf("the probed privilege changed to %q — re-measure before trusting it", storeGrantRequiredPriv)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B — a newborn tier is still silent ──────────────────────────────────────────────
|
||||
//
|
||||
// A storage the agent IS allowed to read but which simply holds no archives yet is HEALTHY. The
|
||||
// probe must not look at content at all, or every freshly provisioned box alarms and the signal dies.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make the probe degrade on an empty content listing instead of on
|
||||
// the permission — a granted-but-empty storage then reports degraded, i.e. every newborn box alarms.
|
||||
func TestStoreGrant_GrantedButEmptyIsHealthy(t *testing.T) {
|
||||
s := probeWith(permGranted, "felhom-pbs", true)
|
||||
if s.Status != capability.StatusOK {
|
||||
t.Fatalf("a readable tier is healthy whether or not it holds archives yet; got %q (%s)", s.Status, s.Reason)
|
||||
}
|
||||
if s.Reason != "" {
|
||||
t.Fatalf("a healthy probe carries no reason; got %q", s.Reason)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO C — the two states are distinguishable at a glance ──────────────────────────────
|
||||
func TestStoreGrant_ForbiddenAndNewbornAreDistinguishable(t *testing.T) {
|
||||
forbidden := probeWith(permUngranted, "felhom-backup", true)
|
||||
newborn := probeWith(permGranted, "felhom-pbs", true)
|
||||
|
||||
if forbidden.Status == newborn.Status {
|
||||
t.Fatalf("the two states must differ — today both read as 'no settled archive yet'; got %q for both", forbidden.Status)
|
||||
}
|
||||
if forbidden.Name == newborn.Name {
|
||||
t.Fatalf("each tier needs its own capability id, or one tier's fault hides another's; got %q twice", forbidden.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// §8.3, weighed once and pinned: a box with NO dedicated target ("local" — host-install's own
|
||||
// DEGRADED fallback) must not turn an ordinary configuration into an operator page. It is still
|
||||
// probed and still reported; only the paging differs.
|
||||
func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
|
||||
if storeGrantCritical("local") {
|
||||
t.Fatal("a box whose backup target is the 'local' fallback must not page the operator about " +
|
||||
"an ordinary, documented configuration")
|
||||
}
|
||||
for _, dedicated := range []string{"felhom-backup", "felhom-pbs", "some-nvme"} {
|
||||
if !storeGrantCritical(dedicated) {
|
||||
t.Fatalf("a DEDICATED target that cannot be read is user-facing and must be critical; %q was not", dedicated)
|
||||
}
|
||||
}
|
||||
// The fallback is still reported — silence for it would be the original defect, scoped smaller.
|
||||
if s := probeWith(permUngranted, "local", storeGrantCritical("local")); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("the fallback target must still report degraded when unreadable; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
|
||||
// probe, because it looks like coverage.
|
||||
func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
|
||||
s := storeGrantStatus(context.Background(), nil, "felhom-backup", true, nil)
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
|
||||
}
|
||||
if s.Reason == "" {
|
||||
t.Fatal("it must say why it could not ask")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// This project's "built but never wired" count reached six last week. The fix for a SILENCE must not
|
||||
// itself be silent. AST, not grep: a commented-out call still contains the string.
|
||||
func TestMainWiresTheStoreGrantProbe(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var wired bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" {
|
||||
wired = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !wired {
|
||||
t.Error("main.go never calls storeGrantStatuses — the probe would exist and report to nobody, " +
|
||||
"which is precisely the silence R-185 is about")
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-190 — the grant repairs itself, and the repair is VISIBLE ──────────────────────────────
|
||||
//
|
||||
// R-190 is a storage grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24,
|
||||
// with a host reinstall, logged `pveum` activity and cluster-log entries all ruled out. The cause is
|
||||
// open; the resilience is not conditional on it.
|
||||
//
|
||||
// The half that matters is the RECORD. R-190's own words: the probe sees the state, nothing sees the
|
||||
// transition. A self-repair that leaves only "ok" behind destroys the only evidence a loss happened,
|
||||
// so a recurring loss becomes undetectable forever — strictly worse than the fault it fixes.
|
||||
|
||||
// fakeRepairRunner records wrapper invocations and can be made to fail.
|
||||
type fakeRepairRunner struct {
|
||||
calls [][]string
|
||||
fail bool
|
||||
}
|
||||
|
||||
func (f *fakeRepairRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
f.calls = append(f.calls, append([]string{name}, args...))
|
||||
if f.fail {
|
||||
return nil, []byte("pveum: refused"), errors.New("exit status 2")
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func newRepairer(f *fakeRepairRunner) *storeGrantRepairer {
|
||||
return &storeGrantRepairer{run: f.Run, log: slog.New(slog.NewTextHandler(io.Discard, nil))}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — the repair is BOUNDED ───────────────────────────────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-04): make mayAttempt always return true (drop the
|
||||
// storeGrantRepairMinInterval check) →
|
||||
//
|
||||
// --- FAIL: TestGrantRepair_IsBounded
|
||||
// storegrant_test.go: a repair must not run on every cycle; 5 cycles produced 5 attempt(s)
|
||||
//
|
||||
// which is a re-grant every report cycle, forever, against a fault an ACL cannot fix. Restored.
|
||||
func TestGrantRepair_IsBounded(t *testing.T) {
|
||||
f := &fakeRepairRunner{}
|
||||
r := newRepairer(f)
|
||||
// Jittered, so the series never lands exactly on the interval boundary — a perfectly regular
|
||||
// series is how a threshold test passes its own mutation, which has happened here before.
|
||||
base := time.Date(2026, 8, 4, 9, 17, 43, 0, time.UTC)
|
||||
offsets := []time.Duration{0, 13*time.Minute + 7*time.Second, 27*time.Minute + 51*time.Second,
|
||||
41*time.Minute + 19*time.Second, 55*time.Minute + 3*time.Second}
|
||||
attempts := 0
|
||||
for _, off := range offsets {
|
||||
if r.mayAttempt("felhom-backup", base.Add(off)) {
|
||||
attempts++
|
||||
}
|
||||
}
|
||||
if attempts != 1 {
|
||||
t.Fatalf("a repair must not run on every cycle; %d cycles produced %d attempt(s) within %s",
|
||||
len(offsets), attempts, storeGrantRepairMinInterval)
|
||||
}
|
||||
// ...and once the interval has genuinely passed, it may try again — a bound is not a ban.
|
||||
if !r.mayAttempt("felhom-backup", base.Add(storeGrantRepairMinInterval+2*time.Minute+11*time.Second)) {
|
||||
t.Fatal("after the interval a repair must be allowed again — otherwise one failure disables the repair forever")
|
||||
}
|
||||
// A DIFFERENT tier is not throttled by this one's attempt.
|
||||
if !r.mayAttempt("felhom-pbs", base.Add(time.Minute)) {
|
||||
t.Fatal("the bound must be per tier — one tier's attempt must not suppress another's")
|
||||
}
|
||||
}
|
||||
|
||||
// A nil repairer (or one with no runner) never attempts, and never panics.
|
||||
func TestGrantRepair_NilIsSafe(t *testing.T) {
|
||||
var r *storeGrantRepairer
|
||||
if r.mayAttempt("felhom-backup", time.Now()) {
|
||||
t.Fatal("a nil repairer must never claim an attempt")
|
||||
}
|
||||
if (&storeGrantRepairer{}).mayAttempt("felhom-backup", time.Now()) {
|
||||
t.Fatal("a repairer with no runner must never claim an attempt")
|
||||
}
|
||||
}
|
||||
|
||||
// The repair calls the EXISTING wrapper verb, with the storage id — no new privileged surface.
|
||||
func TestGrantRepair_CallsTheExistingWrapperVerb(t *testing.T) {
|
||||
f := &fakeRepairRunner{}
|
||||
r := newRepairer(f)
|
||||
if err := r.repair(context.Background(), "felhom-backup"); err != nil {
|
||||
t.Fatalf("repair should succeed with a healthy runner: %v", err)
|
||||
}
|
||||
if len(f.calls) != 1 {
|
||||
t.Fatalf("exactly one wrapper invocation expected; got %d", len(f.calls))
|
||||
}
|
||||
got := f.calls[0]
|
||||
want := []string{"/usr/local/sbin/felhom-backup-target-apply", "grant", "felhom-backup"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("wrapper argv = %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("wrapper argv = %v, want %v — the sudoers vector is `grant *`; anything else is a policy change", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A repair that FAILS must surface the failure, not swallow it (Scenario E's precondition).
|
||||
func TestGrantRepair_FailureIsReturned(t *testing.T) {
|
||||
f := &fakeRepairRunner{fail: true}
|
||||
if err := newRepairer(f).repair(context.Background(), "felhom-backup"); err == nil {
|
||||
t.Fatal("a failed wrapper run must return its error — a repair that cannot run must never read as done")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D (the half that matters) — the REPAIR MUST BE VISIBLE ──────────────────────────
|
||||
//
|
||||
// A repair that leaves only "ok" behind is worse than the fault: the tier works, and the fact that a
|
||||
// permission vanished is gone with it. R-190 exists because nothing saw the transition.
|
||||
//
|
||||
// The channel is the hub's EXISTING ok→degraded→ok edge (§8.5) — nothing new was built. That only
|
||||
// works if the agent deliberately reports ONE degraded cycle after repairing, and if the explanation
|
||||
// rides the field the hub actually puts in the operator's e-mail. The hub's message is built from the
|
||||
// capability NAME and FEATURE (`internal/monitor/host_capability.go` emitTransition) — **not** from
|
||||
// Reason — so the Feature must carry it.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-04): after a successful repair, report ok instead —
|
||||
//
|
||||
// s.Status = capability.StatusOK; s.Feature unchanged
|
||||
//
|
||||
// → --- FAIL: TestGrantRepair_ARepairedGrantIsReportedAsATransition
|
||||
//
|
||||
// storegrant_test.go: a self-repair must still report DEGRADED for one cycle so the hub raises
|
||||
// its edge; got "ok" — the loss would be invisible
|
||||
//
|
||||
// i.e. exactly the silence R-190 is about. Restored.
|
||||
func TestGrantRepair_ARepairedGrantIsReportedAsATransition(t *testing.T) {
|
||||
// THE PRODUCTION verdict, not a copy of it. An earlier draft of this test built the Status
|
||||
// itself and asserted its own construction — it would have passed while production reported ok,
|
||||
// which is precisely the silence being guarded against.
|
||||
if pre := probeWith(permUngranted, "felhom-backup", true); pre.Status != capability.StatusDegraded {
|
||||
t.Fatalf("precondition: a missing grant is degraded; got %q", pre.Status)
|
||||
}
|
||||
s := storeGrantRepairedVerdict("felhom-backup", true)
|
||||
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a self-repair must still report DEGRADED for one cycle so the hub raises its edge; "+
|
||||
"got %q — the loss would be invisible", s.Status)
|
||||
}
|
||||
// The hub e-mails the FEATURE text. If the explanation is not there, the operator is told a
|
||||
// capability was degraded and never learns it repaired itself or that anything vanished.
|
||||
for _, want := range []string{"MISSING", "RESTORED", "felhom-backup", "R-190"} {
|
||||
if !strings.Contains(s.Feature, want) {
|
||||
t.Fatalf("the Feature text is what the hub puts in the operator's e-mail; it must contain %q. Got: %s", want, s.Feature)
|
||||
}
|
||||
}
|
||||
if !s.Critical {
|
||||
t.Fatal("the transition must be CRITICAL or the hub does not alert on it at all")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The wrapper's `grant` verb is itself a "built but never wired" example: it exists, is
|
||||
// sudoers-permitted for any id, and had only ever been called at storage CREATION. The repair must
|
||||
// not become the seventh instance. AST, not grep — a commented-out call still contains the string.
|
||||
func TestMainWiresTheGrantRepair(t *testing.T) {
|
||||
f := parseMainForWiring(t)
|
||||
|
||||
var built, passed bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.CompositeLit:
|
||||
if id, ok := node.Type.(*ast.Ident); ok && id.Name == "storeGrantRepairer" {
|
||||
built = true
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if id, ok := node.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" && len(node.Args) == 4 {
|
||||
if a, ok := node.Args[3].(*ast.Ident); ok && a.Name == "grantRepairer" {
|
||||
passed = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !built {
|
||||
t.Error("main.go never constructs a storeGrantRepairer — nothing would ever repair a lost grant")
|
||||
}
|
||||
if !passed {
|
||||
t.Error("storeGrantStatuses is not passed the repairer — the probe would detect the loss and " +
|
||||
"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
|
||||
}
|
||||
}
|
||||
|
||||
// The transition must survive a probe that is NOT the one feeding the hub.
|
||||
//
|
||||
// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
|
||||
// production while every unit test passed: `probeAll` is called independently by the self-check LOG
|
||||
// and by the collector building a host-report. The repairing call was the log's; the report three
|
||||
// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
|
||||
// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
|
||||
//
|
||||
// --- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
|
||||
// storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
|
||||
// the host-report would carry ok and the operator would never learn the grant vanished
|
||||
//
|
||||
// Restored.
|
||||
func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||
r := newRepairer(&fakeRepairRunner{})
|
||||
// Jittered, never landing on the window boundary.
|
||||
repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
|
||||
r.noteRepaired("felhom-backup", repairedAt)
|
||||
|
||||
// The DECISION a later probe makes — the production function, not the helper it calls. An
|
||||
// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
|
||||
// the latch's USE left the helper untouched.
|
||||
healthy := probeWith(permGranted, "felhom-backup", true)
|
||||
if healthy.Status != capability.StatusOK {
|
||||
t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
|
||||
}
|
||||
got := storeGrantHealthyVerdict("felhom-backup", true,
|
||||
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
|
||||
if got.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
|
||||
"would carry ok and the operator would never learn the grant vanished", got.Status)
|
||||
}
|
||||
if !strings.Contains(got.Feature, "RESTORED") {
|
||||
t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
|
||||
}
|
||||
// Outside the window it reports plain ok again.
|
||||
late := storeGrantHealthyVerdict("felhom-backup", true,
|
||||
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
|
||||
if late.Status != capability.StatusOK {
|
||||
t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
|
||||
"would be its own false alarm", late.Status)
|
||||
}
|
||||
if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
|
||||
t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
|
||||
}
|
||||
// ...and it clears on its own rather than latching a box degraded forever.
|
||||
if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
|
||||
t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
|
||||
}
|
||||
// It is per tier.
|
||||
if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
|
||||
t.Fatal("one tier's repair must not latch another tier's status")
|
||||
}
|
||||
// The window MUST exceed the report interval — the property, asserted rather than assumed.
|
||||
if storeGrantRepairReportWindow <= 15*time.Minute {
|
||||
t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
|
||||
"be missed entirely", storeGrantRepairReportWindow)
|
||||
}
|
||||
}
|
||||
@@ -135,10 +135,11 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
const big = 4 << 30 // a plausible whole-guest archive
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "a", Content: "backup", CTime: 10},
|
||||
{VolID: "b", Content: "backup", CTime: 99},
|
||||
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
|
||||
{VolID: "a", Content: "backup", CTime: 10, Size: big},
|
||||
{VolID: "b", Content: "backup", CTime: 99, Size: big},
|
||||
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
@@ -152,6 +153,26 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
|
||||
// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
|
||||
// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
|
||||
// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
|
||||
// PickSettledRestoreCandidateOn and this fails with
|
||||
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
|
||||
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
|
||||
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
if err != nil || vol != "real" {
|
||||
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- scheduler ---
|
||||
|
||||
type fakeRTRunner struct {
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-86 — a restore-test follows the BACKUP, not the clock.
|
||||
//
|
||||
// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
|
||||
// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
|
||||
// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
|
||||
// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
|
||||
// sometimes twice on the same archive.
|
||||
//
|
||||
// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
|
||||
//
|
||||
// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
|
||||
// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
|
||||
// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
|
||||
// The naive rule silently switches restore-testing off for the tier that matters most, and it is
|
||||
// the version a reasonable person would write. It has a red-proof of its own
|
||||
// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
|
||||
//
|
||||
// The rule implemented here:
|
||||
//
|
||||
// Let A = the newest archive on this tier that is at least `settle` old.
|
||||
// The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
|
||||
//
|
||||
// daily tier → A is yesterday's archive; a new one settles each day → proved once per day
|
||||
// weekly tier → A is last week's until the next settles → proved once per week
|
||||
// newborn tier → A does not exist → UNKNOWN, never a fault
|
||||
//
|
||||
// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
|
||||
// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
|
||||
// produce a cadence nobody can predict from either.
|
||||
//
|
||||
// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
|
||||
// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
|
||||
// the trigger changed.
|
||||
|
||||
// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
|
||||
// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
|
||||
type DueVerdict struct {
|
||||
Target string // the tier's storage target id
|
||||
|
||||
// Due is true only when Archive is set and has not been proven.
|
||||
Due bool
|
||||
// Archive is the settled candidate A ("" when the tier holds none).
|
||||
Archive string
|
||||
// Landed is when A landed on the tier (zero when Archive is "").
|
||||
Landed time.Time
|
||||
// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
|
||||
ProvenArchive string
|
||||
// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
|
||||
// NEVER reported as "not due", which would silently retire a tier the moment its storage
|
||||
// stopped answering. Due stays false (we have no archive to test) and the error travels.
|
||||
Err error
|
||||
// Reason is the one-line human account of this verdict.
|
||||
Reason string
|
||||
}
|
||||
|
||||
// String renders a verdict for the operator log / selftest output.
|
||||
func (v DueVerdict) String() string {
|
||||
return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
|
||||
}
|
||||
|
||||
// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
|
||||
//
|
||||
// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
|
||||
// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
|
||||
// still never be starved — a tier that has waited longest is served first — and keeping it as the
|
||||
// order rather than as the trigger is the whole of this change.
|
||||
func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
|
||||
if !s.rotating() {
|
||||
return nil
|
||||
}
|
||||
order := s.tiers
|
||||
if s.rtState != nil {
|
||||
order = s.rtState.OldestFirst(s.tiers)
|
||||
}
|
||||
cutoff := s.settleCutoff()
|
||||
out := make([]DueVerdict, 0, len(order))
|
||||
for _, target := range order {
|
||||
out = append(out, s.evaluateTier(ctx, target, cutoff))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// settleCutoff is the newest landing time an archive may have and still count as settled.
|
||||
func (s *Scheduler) settleCutoff() time.Time {
|
||||
if s.settle <= 0 {
|
||||
return time.Time{} // no settle requirement configured → any archive is a candidate
|
||||
}
|
||||
return s.now().Add(-s.settle)
|
||||
}
|
||||
|
||||
// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
|
||||
// unit-tested directly rather than inferred from whether a fake runner happened to be called.
|
||||
func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
|
||||
v := DueVerdict{Target: target}
|
||||
archive, landed, err := s.tierPick(ctx, target, cutoff)
|
||||
if err != nil {
|
||||
// UNKNOWN, never "not due", and never silent.
|
||||
v.Err = err
|
||||
v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
|
||||
return v
|
||||
}
|
||||
v.Archive, v.Landed = archive, landed
|
||||
if archive == "" {
|
||||
v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
|
||||
return v
|
||||
}
|
||||
proven, ok := "", false
|
||||
if s.rtState != nil {
|
||||
proven, ok = s.rtState.ProvenArchive(target)
|
||||
}
|
||||
v.ProvenArchive = proven
|
||||
if ok && proven == archive {
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
|
||||
return v
|
||||
}
|
||||
v.Due = true
|
||||
switch {
|
||||
case !ok && proven == "":
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
|
||||
default:
|
||||
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
|
||||
// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
|
||||
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
|
||||
return s.evaluateTier(ctx, target, s.settleCutoff())
|
||||
}
|
||||
|
||||
// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
|
||||
//
|
||||
// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
|
||||
// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
|
||||
// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
|
||||
// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
|
||||
// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
|
||||
func (s *Scheduler) verdictSummary(ctx context.Context) string {
|
||||
out := ""
|
||||
for _, v := range s.EvaluateDue(ctx) {
|
||||
if out != "" {
|
||||
out += "; "
|
||||
}
|
||||
switch {
|
||||
case v.Err != nil:
|
||||
out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
|
||||
default:
|
||||
out += v.Target + ": " + v.Reason
|
||||
}
|
||||
}
|
||||
if out == "" {
|
||||
return "no tiers configured"
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,593 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-86 — the restore-test follows the BACKUP, not the clock.
|
||||
//
|
||||
// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
|
||||
// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
|
||||
// asserted "no error" would pass against a scheduler that never ran anything at all — which is
|
||||
// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
|
||||
// archive, and did a second evaluation correctly run NOTHING.
|
||||
|
||||
// ── the fake tier storage ────────────────────────────────────────────────────────────────────
|
||||
|
||||
// archiveStub is one archive on a tier: its volid and when it landed.
|
||||
type archiveStub struct {
|
||||
volid string
|
||||
landed time.Time
|
||||
}
|
||||
|
||||
// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
|
||||
// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
|
||||
// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
|
||||
// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
|
||||
// picker; that the real picker obeys the same rule is asserted there, not here.
|
||||
type tierStorage struct {
|
||||
archives map[string][]archiveStub
|
||||
err map[string]error // target → lookup failure
|
||||
}
|
||||
|
||||
func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
|
||||
if e, ok := ts.err[target]; ok && e != nil {
|
||||
return "", time.Time{}, e
|
||||
}
|
||||
var best archiveStub
|
||||
for _, a := range ts.archives[target] {
|
||||
if !notAfter.IsZero() && a.landed.After(notAfter) {
|
||||
continue // not settled yet
|
||||
}
|
||||
if best.volid == "" || a.landed.After(best.landed) {
|
||||
best = a
|
||||
}
|
||||
}
|
||||
return best.volid, best.landed, nil
|
||||
}
|
||||
|
||||
// dueHarness is a scheduler with a driven clock over a fake tier storage.
|
||||
type dueHarness struct {
|
||||
s *Scheduler
|
||||
rr *rotRunner
|
||||
st *RestoreTestState
|
||||
ts *tierStorage
|
||||
clock time.Time
|
||||
path string
|
||||
}
|
||||
|
||||
func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
|
||||
t.Helper()
|
||||
return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
|
||||
}
|
||||
|
||||
func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
|
||||
t.Helper()
|
||||
h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
|
||||
h.st = NewRestoreTestState(statePath)
|
||||
h.s = NewScheduler(SchedulerOptions{
|
||||
Runner: h.rr,
|
||||
Store: NewStore(),
|
||||
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
|
||||
},
|
||||
Cadence: time.Hour,
|
||||
Settle: settle,
|
||||
Logger: quiet(),
|
||||
Tiers: tiers,
|
||||
TierPick: ts.pick,
|
||||
State: h.st,
|
||||
InFlight: &InFlight{},
|
||||
})
|
||||
h.s.now = func() time.Time { return h.clock }
|
||||
return h
|
||||
}
|
||||
|
||||
// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
|
||||
func (h *dueHarness) advance(step, total time.Duration) {
|
||||
for elapsed := time.Duration(0); elapsed < total; elapsed += step {
|
||||
h.clock = h.clock.Add(step)
|
||||
h.s.tick(context.Background())
|
||||
}
|
||||
}
|
||||
|
||||
var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
|
||||
|
||||
// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
|
||||
func dailyArchives(tier string, n int) []archiveStub {
|
||||
out := make([]archiveStub, 0, n)
|
||||
for d := 0; d < n; d++ {
|
||||
out = append(out, archiveStub{
|
||||
volid: fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
|
||||
landed: day0.AddDate(0, 0, d),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
|
||||
//
|
||||
// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
|
||||
// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
|
||||
// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
|
||||
// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
|
||||
// the tier that matters most.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
|
||||
// was replaced by the naive age rule:
|
||||
//
|
||||
// - if ok && proven == archive { … not due … }
|
||||
// - if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
|
||||
//
|
||||
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
|
||||
// old enough". Result:
|
||||
//
|
||||
// --- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
|
||||
// restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
|
||||
//
|
||||
// Zero runs — restore-testing off. Restored immediately afterwards.
|
||||
func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
|
||||
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
// Five days, evaluated hourly.
|
||||
h.advance(time.Hour, 5*24*time.Hour)
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 5 {
|
||||
t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
|
||||
}
|
||||
// And each run must be on the archive that settled that day — day0's on day 1, and so on.
|
||||
for i, a := range got {
|
||||
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
|
||||
if a != want {
|
||||
t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
|
||||
}
|
||||
}
|
||||
// The newest archive is NEVER the one tested: it has not settled.
|
||||
if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
|
||||
t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
|
||||
func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
|
||||
{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
|
||||
{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
|
||||
{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
|
||||
}}}
|
||||
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
|
||||
// Three weeks, evaluated every 6 hours — 84 evaluations.
|
||||
h.advance(6*time.Hour, 21*24*time.Hour)
|
||||
|
||||
got := h.rr.seen()
|
||||
want := []string{
|
||||
"felhom-pbs:backup/ct/9201/w0",
|
||||
"felhom-pbs:backup/ct/9201/w1",
|
||||
"felhom-pbs:backup/ct/9201/w2",
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
|
||||
//
|
||||
// This is the defect a person actually notices: today every deploy restarts the ticker, so a
|
||||
// restore-test runs one interval after each deploy regardless of what has already been proven.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
|
||||
// ProvenArchive ignore the stored archive —
|
||||
//
|
||||
// - if !ok || p.Archive == "" { return "", false }
|
||||
// - return "", false // per-tier time only, the pre-R-86 state
|
||||
//
|
||||
// → --- FAIL: TestDue_RestartRunsNothing
|
||||
//
|
||||
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
|
||||
// produced 4 run(s)
|
||||
//
|
||||
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
|
||||
// today's behaviour with the ticker's phase reset by the deploy. Restored.
|
||||
func TestDue_RestartRunsNothing(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "rt.json")
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
|
||||
start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
|
||||
|
||||
h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
|
||||
}
|
||||
|
||||
// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
|
||||
total := 0
|
||||
for i := 0; i < 2; i++ {
|
||||
h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
h2.s.tick(context.Background())
|
||||
h2.s.tick(context.Background())
|
||||
total += len(h2.rr.seen())
|
||||
}
|
||||
if total != 0 {
|
||||
t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
|
||||
func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
h.s.tick(context.Background()) // proves day0's archive
|
||||
h.s.tick(context.Background()) // nothing new has settled → nothing
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
|
||||
}
|
||||
|
||||
// Day 2, 03:00 — day1's archive has now settled.
|
||||
h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
|
||||
h.s.tick(context.Background())
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
|
||||
}
|
||||
if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
|
||||
t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
|
||||
//
|
||||
// - if rt.Pass && s.rtState != nil && target != "" {
|
||||
// - if s.rtState != nil && target != "" {
|
||||
//
|
||||
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
|
||||
//
|
||||
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
|
||||
// evaluations
|
||||
//
|
||||
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
|
||||
// freshly verified, which is the loudest signal this system produces going silent. Restored.
|
||||
func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
h.s.tick(context.Background())
|
||||
}
|
||||
|
||||
got := h.rr.seen()
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
|
||||
}
|
||||
if _, ok := h.st.ProvenArchive("local"); ok {
|
||||
t.Fatal("a FAILED restore-test must not record the archive as proven")
|
||||
}
|
||||
if _, ok := h.st.LastSuccess("local"); ok {
|
||||
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
|
||||
func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{
|
||||
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
|
||||
"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
|
||||
}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
|
||||
// Both tiers are due at this instant.
|
||||
due := h.s.EvaluateDue(context.Background())
|
||||
if len(due) != 2 || !due[0].Due || !due[1].Due {
|
||||
t.Fatalf("precondition: both tiers should be due; got %v", due)
|
||||
}
|
||||
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
|
||||
}
|
||||
|
||||
// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
|
||||
h.s.tick(context.Background())
|
||||
got := h.rr.seen()
|
||||
if len(got) != 2 || got[0] == got[1] {
|
||||
t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
|
||||
func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
gate := &InFlight{}
|
||||
h.s.inFlight = gate
|
||||
release, _, _ := gate.TryAcquire("backup:felhom-pbs")
|
||||
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
|
||||
}
|
||||
if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
|
||||
t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
|
||||
}
|
||||
release()
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 1 {
|
||||
t.Fatalf("must resume once the gate frees; got %d run(s)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
|
||||
func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
|
||||
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
|
||||
due := h.s.EvaluateDue(context.Background())
|
||||
if len(due) != 1 {
|
||||
t.Fatalf("want one verdict, got %v", due)
|
||||
}
|
||||
v := due[0]
|
||||
if v.Due || v.Err != nil || v.Archive != "" {
|
||||
t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
|
||||
}
|
||||
if v.Reason == "" {
|
||||
t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
|
||||
func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
|
||||
|
||||
if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
|
||||
t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
if n := len(h.rr.seen()); n != 0 {
|
||||
t.Fatalf("nothing settled → no run; got %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
|
||||
// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
|
||||
// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
|
||||
func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
|
||||
boom := errors.New("storage unreachable")
|
||||
ts := &tierStorage{
|
||||
archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
|
||||
err: map[string]error{"felhom-pbs": boom},
|
||||
}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
|
||||
var pbs DueVerdict
|
||||
for _, v := range h.s.EvaluateDue(context.Background()) {
|
||||
if v.Target == "felhom-pbs" {
|
||||
pbs = v
|
||||
}
|
||||
}
|
||||
if pbs.Err == nil {
|
||||
t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
|
||||
}
|
||||
if pbs.Due {
|
||||
t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
|
||||
}
|
||||
if pbs.Reason == "" {
|
||||
t.Fatal("the failure must be explained, not merely flagged")
|
||||
}
|
||||
|
||||
// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
|
||||
// tier its proof.
|
||||
h.s.tick(context.Background())
|
||||
if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
|
||||
t.Fatalf("the readable tier must still be proved; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
|
||||
|
||||
// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
|
||||
// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
|
||||
// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
|
||||
// invented by a migration.
|
||||
func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
|
||||
if err := writeFileForTest(path, legacy); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
st := NewRestoreTestState(path)
|
||||
if _, ok := st.ProvenArchive("local"); ok {
|
||||
t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
|
||||
}
|
||||
at, ok := st.LastSuccess("local")
|
||||
if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
|
||||
}
|
||||
// Ordering still works off the legacy times.
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||
t.Fatalf("oldest-first must still order legacy records; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The new shape round-trips, archive and all.
|
||||
func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
st := NewRestoreTestState(path)
|
||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
re := NewRestoreTestState(path)
|
||||
got, ok := re.ProvenArchive("felhom-pbs")
|
||||
if !ok || got != "felhom-pbs:backup/ct/9201/x" {
|
||||
t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
|
||||
}
|
||||
at, ok := re.LastSuccess("felhom-pbs")
|
||||
if !ok || !at.Equal(now) {
|
||||
t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
|
||||
func writeFileForTest(path, content string) error {
|
||||
return os.WriteFile(path, []byte(content), 0o600)
|
||||
}
|
||||
|
||||
// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
|
||||
// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
|
||||
// quiet journal is equally consistent with a healthy loop and a dead goroutine.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
|
||||
// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
|
||||
// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
|
||||
func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{
|
||||
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
|
||||
"felhom-pbs": nil, // no archive at all
|
||||
}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
// Prove the local tier so NOTHING is due.
|
||||
if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
|
||||
// test would pass against a tick that never calls it.
|
||||
var logbuf strings.Builder
|
||||
h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
h.s.tick(context.Background())
|
||||
got := logbuf.String()
|
||||
for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
|
||||
// reads as "nothing due" is the silence this rule exists to prevent.
|
||||
func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
|
||||
ts := &tierStorage{
|
||||
archives: map[string][]archiveStub{"local": nil},
|
||||
err: map[string]error{"felhom-pbs": errors.New("storage unreachable")},
|
||||
}
|
||||
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||
got := h.s.verdictSummary(context.Background())
|
||||
if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
|
||||
t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
|
||||
//
|
||||
// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
|
||||
// agent will not repeat the work. So the persisted record has to be able to become a host-report
|
||||
// entry — without inventing anything it does not know.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
|
||||
// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
|
||||
//
|
||||
// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
|
||||
// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
|
||||
// per-tier proof on it); got [{... SourceTier: ...}]
|
||||
//
|
||||
// Restored.
|
||||
func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rt.json")
|
||||
// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
|
||||
// file has ever had, which is what a real box carries after two upgrades.
|
||||
legacy := `{
|
||||
"old-v1": "2026-07-30T02:11:07Z",
|
||||
"old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
|
||||
"felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
|
||||
}`
|
||||
if err := writeFileForTest(path, legacy); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
|
||||
}
|
||||
e := got[0]
|
||||
if e.SourceTier != "pbs" {
|
||||
t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
|
||||
}
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
|
||||
t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
|
||||
}
|
||||
if e.TestedAt != "2026-08-03T13:25:14Z" {
|
||||
t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
|
||||
}
|
||||
if e.Verified != "boot+running" {
|
||||
t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
|
||||
}
|
||||
// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
|
||||
if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
|
||||
t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
|
||||
e.DurationSeconds, e.ScratchVMID)
|
||||
}
|
||||
// The legacy records still serve the DUE-check, which is a separate question from reporting.
|
||||
if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
|
||||
t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
|
||||
}
|
||||
}
|
||||
|
||||
// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
|
||||
// the production path, not a hand-built fixture.
|
||||
func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||
// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
|
||||
h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
|
||||
}
|
||||
h.s.tick(context.Background())
|
||||
|
||||
got := h.st.ProvenRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
|
||||
}
|
||||
if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
|
||||
t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
|
||||
func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
|
||||
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
|
||||
h.s.tick(context.Background())
|
||||
if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
|
||||
"would outlive the fault; got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,15 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
|
||||
@@ -28,42 +31,131 @@ import (
|
||||
//
|
||||
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
|
||||
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
|
||||
// trap the backup due-check avoids.
|
||||
// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
|
||||
// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
|
||||
// ever becoming due again. The rule earns its keep twice now.
|
||||
//
|
||||
// R-86 (1.2) — WHICH ARCHIVE, not just when.
|
||||
//
|
||||
// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
|
||||
// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
|
||||
// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
|
||||
// a tier is due once it holds a settled archive that has not been proven — so the identity of the
|
||||
// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
|
||||
//
|
||||
// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
|
||||
// here it records a result, but not WHICH result, and that is just as unable to answer the question
|
||||
// being asked of it.
|
||||
type RestoreTestState struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last SUCCESSFUL restore-test (UTC)
|
||||
last map[string]provenTier // target id → what was last PROVEN on that tier
|
||||
}
|
||||
|
||||
// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
|
||||
// and when.
|
||||
//
|
||||
// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
|
||||
// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
|
||||
// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
|
||||
// hub can stay ignorant of a real success until the next archive generation.
|
||||
//
|
||||
// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
|
||||
// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
|
||||
// at report-building time — a network call that can fail, on a path where failing means mis-labelling
|
||||
// a proof. Store what you knew when you knew it.
|
||||
type provenTier struct {
|
||||
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
|
||||
Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
|
||||
Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
|
||||
At time.Time // when that run passed (UTC)
|
||||
}
|
||||
|
||||
// reportable reports whether this record can be re-reported to the hub as a restore-test result.
|
||||
//
|
||||
// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
|
||||
// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
|
||||
// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
|
||||
// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
|
||||
func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
|
||||
|
||||
// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
|
||||
//
|
||||
// v1 (pre-R-86) "<target>": "<RFC3339>" — a time, no archive
|
||||
// v2 (R-86) "<target>": {archive, proven_at} — due-check usable, not reportable
|
||||
// v3 (R-189) "<target>": {archive, tier, verified, …} — both
|
||||
//
|
||||
// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
|
||||
// readers above test for — the migration needs no version number because the absence IS the answer.
|
||||
type provenTierJSON struct {
|
||||
Archive string `json:"archive"`
|
||||
Tier string `json:"tier,omitempty"`
|
||||
Verified string `json:"verified,omitempty"`
|
||||
ProvenAt string `json:"proven_at"`
|
||||
}
|
||||
|
||||
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
|
||||
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
|
||||
// corrupt file from wedging the daemon.
|
||||
//
|
||||
// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
|
||||
// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
|
||||
// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
|
||||
// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
|
||||
// read a legacy time as proof of whatever archive happens to be current, which would mark an
|
||||
// unproven archive proven — inventing a guarantee out of a migration.
|
||||
func NewRestoreTestState(path string) *RestoreTestState {
|
||||
s := &RestoreTestState{path: path, last: map[string]time.Time{}}
|
||||
s := &RestoreTestState{path: path, last: map[string]provenTier{}}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
var raw map[string]string
|
||||
var raw map[string]json.RawMessage
|
||||
if json.Unmarshal(data, &raw) != nil {
|
||||
return s
|
||||
}
|
||||
for target, ts := range raw {
|
||||
if t, perr := time.Parse(time.RFC3339, ts); perr == nil {
|
||||
s.last[target] = t.UTC()
|
||||
for target, msg := range raw {
|
||||
// Legacy shape: a bare RFC3339 string.
|
||||
var legacy string
|
||||
if json.Unmarshal(msg, &legacy) == nil {
|
||||
if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
|
||||
s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
|
||||
}
|
||||
continue
|
||||
}
|
||||
var cur provenTierJSON
|
||||
if json.Unmarshal(msg, &cur) != nil {
|
||||
continue // one unreadable entry must not lose the others
|
||||
}
|
||||
t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
|
||||
if perr != nil {
|
||||
continue
|
||||
}
|
||||
s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// RecordSuccess stamps a tier as proven at t. Only call this for a PASSING restore-test.
|
||||
func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
|
||||
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
|
||||
// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
|
||||
// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
|
||||
//
|
||||
// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
|
||||
// the next reader will notice failures are absent and try to "fix" it:
|
||||
//
|
||||
// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
|
||||
// the system quietly less tested than it believes. It must survive a restart.
|
||||
//
|
||||
// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
|
||||
// so a lost failure heals itself within one interval. Persisting it would do the opposite of
|
||||
// helping: a healed tier would keep reporting a failure that is no longer true.
|
||||
func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
|
||||
if target == "" {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.last[target] = t.UTC()
|
||||
s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
|
||||
return s.saveLocked()
|
||||
}
|
||||
|
||||
@@ -71,21 +163,75 @@ func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
|
||||
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
t, ok := s.last[target]
|
||||
return t, ok
|
||||
p, ok := s.last[target]
|
||||
return p.At, ok
|
||||
}
|
||||
|
||||
// Snapshot returns a copy of the whole map — for the host-report gauge.
|
||||
// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
|
||||
// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
|
||||
// not this one has not been proven.
|
||||
func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
p, ok := s.last[target]
|
||||
if !ok || p.Archive == "" {
|
||||
return "", false
|
||||
}
|
||||
return p.Archive, true
|
||||
}
|
||||
|
||||
// Snapshot returns a copy of the last-proven TIMES.
|
||||
//
|
||||
// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
|
||||
// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
|
||||
// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
|
||||
// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
|
||||
// only the times; if it acquires none, delete it rather than let it claim a purpose again.
|
||||
func (s *RestoreTestState) Snapshot() map[string]time.Time {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make(map[string]time.Time, len(s.last))
|
||||
for k, v := range s.last {
|
||||
out[k] = v
|
||||
out[k] = v.At
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
|
||||
//
|
||||
// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
|
||||
// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
|
||||
//
|
||||
// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
|
||||
// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
|
||||
// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
|
||||
// is not a claim; a fabricated one would be.
|
||||
//
|
||||
// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
|
||||
// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
|
||||
// worse defect than the one this fixes.
|
||||
func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := make([]hub.RestoreTest, 0, len(s.last))
|
||||
for _, p := range s.last {
|
||||
if !p.reportable() {
|
||||
continue
|
||||
}
|
||||
out = append(out, hub.RestoreTest{
|
||||
SourceArchive: p.Archive,
|
||||
SourceTier: p.Tier,
|
||||
Pass: true,
|
||||
Verified: p.Verified,
|
||||
TestedAt: p.At.UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
|
||||
// iteration is randomised.
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
|
||||
return out
|
||||
}
|
||||
|
||||
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
|
||||
//
|
||||
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
|
||||
@@ -100,8 +246,9 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
|
||||
defer s.mu.Unlock()
|
||||
out := append([]string(nil), targets...)
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
ti, oki := s.last[out[i]]
|
||||
tj, okj := s.last[out[j]]
|
||||
pi, oki := s.last[out[i]]
|
||||
pj, okj := s.last[out[j]]
|
||||
ti, tj := pi.At, pj.At
|
||||
switch {
|
||||
case !oki && !okj:
|
||||
return out[i] < out[j] // both never proven → deterministic
|
||||
@@ -119,9 +266,12 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
|
||||
}
|
||||
|
||||
func (s *RestoreTestState) saveLocked() error {
|
||||
raw := make(map[string]string, len(s.last))
|
||||
for target, t := range s.last {
|
||||
raw[target] = t.UTC().Format(time.RFC3339)
|
||||
raw := make(map[string]provenTierJSON, len(s.last))
|
||||
for target, p := range s.last {
|
||||
raw[target] = provenTierJSON{
|
||||
Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
|
||||
ProvenAt: p.At.UTC().Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
data, err := json.MarshalIndent(raw, "", " ")
|
||||
if err != nil {
|
||||
|
||||
@@ -39,10 +39,19 @@ func (r *rotRunner) seen() []string {
|
||||
return append([]string(nil), r.archives...)
|
||||
}
|
||||
|
||||
// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
|
||||
// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
|
||||
// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
|
||||
var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
|
||||
func archiveFor(m map[string]string) TierPicker {
|
||||
return func(_ context.Context, target string) (string, error) {
|
||||
return m[target], nil
|
||||
return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
|
||||
a := m[target]
|
||||
if a == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
return a, testLanded, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,14 +72,24 @@ func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPi
|
||||
})
|
||||
}
|
||||
|
||||
// ── SCENARIO A — both tiers get tested across consecutive cadences ───────────────────────────
|
||||
// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
|
||||
//
|
||||
// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
|
||||
// it was a faithful statement of the defect. It read:
|
||||
//
|
||||
// 4 ticks → 4 runs, and consecutive runs must hit different tiers
|
||||
//
|
||||
// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
|
||||
// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
|
||||
// newest settled archive is already proven is not re-tested just because time passed. So the
|
||||
// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
|
||||
// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
|
||||
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
|
||||
// "both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]",
|
||||
// i.e. the offsite tier never appears. That is today's behaviour, and it is why demo-hp's DR tier
|
||||
// went unproven for its entire existence.
|
||||
func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
|
||||
// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
|
||||
// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
|
||||
func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
|
||||
rr := &rotRunner{pass: true}
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
|
||||
@@ -94,14 +113,14 @@ func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if !sawLocal || !sawPBS {
|
||||
t.Fatalf("both tiers must be exercised across 4 cadences; got %v", got)
|
||||
t.Fatalf("both tiers must be exercised; got %v", got)
|
||||
}
|
||||
// Oldest-first must ALTERNATE, not clump — otherwise one tier is starved between visits.
|
||||
if len(got) != 4 {
|
||||
t.Fatalf("want 4 runs, got %d: %v", len(got), got)
|
||||
// Exactly one run per tier: the archives never changed, so nothing became due a second time.
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
|
||||
}
|
||||
if got[0] == got[1] {
|
||||
t.Fatalf("consecutive runs hit the same tier — oldest-first is not rotating: %v", got)
|
||||
t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -284,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) {
|
||||
t.Fatalf("unexpected: %v", got)
|
||||
}
|
||||
}
|
||||
_ = st.RecordSuccess("local", now)
|
||||
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
|
||||
}
|
||||
_ = st.RecordSuccess("felhom-pbs", now.Add(time.Hour))
|
||||
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
|
||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
|
||||
t.Fatalf("the least recently proven must sort first; got %v", got)
|
||||
}
|
||||
@@ -299,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) {
|
||||
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
|
||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||
now := time.Now().UTC()
|
||||
_ = st.RecordSuccess("b-tier", now)
|
||||
_ = st.RecordSuccess("a-tier", now)
|
||||
_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
|
||||
_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
|
||||
for i := 0; i < 20; i++ {
|
||||
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
|
||||
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
|
||||
@@ -315,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
|
||||
st := NewRestoreTestState(path)
|
||||
if err := st.RecordSuccess("felhom-pbs", now); err != nil {
|
||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reopened := NewRestoreTestState(path)
|
||||
|
||||
@@ -260,21 +260,59 @@ func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error)
|
||||
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
|
||||
// that into a failure would make every fresh box look broken for its first week.
|
||||
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
|
||||
archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
|
||||
return archive, err
|
||||
}
|
||||
|
||||
// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
|
||||
// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
|
||||
// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
|
||||
// one-line call into this and its contract is untouched (one scan, one owner).
|
||||
//
|
||||
// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
|
||||
// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
|
||||
// restore-testing the archive a backup is still writing proves nothing about the backup that
|
||||
// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
|
||||
// on that: the tier is due when a settled archive exists that has not been proven.
|
||||
//
|
||||
// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
|
||||
// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
|
||||
// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
|
||||
// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
|
||||
// the archive generation rather than by the evaluation interval.
|
||||
//
|
||||
// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
|
||||
// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
|
||||
// turning that into a failure would make every fresh box look broken for its first week.
|
||||
func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
|
||||
if target == "" {
|
||||
return "", nil
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
contents, err := r.api.StorageContent(ctx, target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return "", time.Time{}, err
|
||||
}
|
||||
var best string
|
||||
var bestCTime int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content == "backup" && e.CTime > bestCTime {
|
||||
if e.Content != "backup" {
|
||||
continue
|
||||
}
|
||||
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
|
||||
continue // not settled yet — a newer archive is not a reason to re-prove an older one
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
}
|
||||
if e.CTime > bestCTime {
|
||||
bestCTime, best = e.CTime, e.VolID
|
||||
}
|
||||
}
|
||||
return best, nil
|
||||
if best == "" {
|
||||
return "", time.Time{}, nil
|
||||
}
|
||||
return best, time.Unix(bestCTime, 0).UTC(), nil
|
||||
}
|
||||
|
||||
// latestArchive finds the newest backup archive volid + size for vmid on the target.
|
||||
|
||||
+119
-59
@@ -32,22 +32,31 @@ type CandidatePicker func(ctx context.Context) (string, error)
|
||||
// PBS archive was classified "local" and got the 10-minute local wait.
|
||||
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
|
||||
|
||||
// TierPicker resolves the newest archive on a NAMED tier, or "" when that tier holds none.
|
||||
// (*BackupRunner).PickRestoreCandidateOn satisfies it. "" must NOT be an error — a brand-new
|
||||
// offsite tier legitimately has nothing to restore yet.
|
||||
type TierPicker func(ctx context.Context, target string) (string, error)
|
||||
// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
|
||||
// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
|
||||
// satisfies it. A zero notAfter means "no settle requirement".
|
||||
//
|
||||
// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
|
||||
// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
|
||||
// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
|
||||
// brand-new offsite tier legitimately has nothing to restore yet.
|
||||
type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
|
||||
|
||||
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
|
||||
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
|
||||
// AND a valid scratch band is configured (validated by the caller before construction).
|
||||
type Scheduler struct {
|
||||
runner RestoreTestRunner
|
||||
pick CandidatePicker
|
||||
store *Store
|
||||
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
|
||||
runner RestoreTestRunner
|
||||
pick CandidatePicker
|
||||
store *Store
|
||||
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
|
||||
// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
|
||||
// longer the thing that decides a test happens; see restoretest_due.go.
|
||||
cadence time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// settle is how long an archive must have sat before it is a candidate (R-86).
|
||||
settle time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
|
||||
// (single tier via `pick`), which keeps every existing caller and test working untouched.
|
||||
@@ -64,9 +73,14 @@ type SchedulerOptions struct {
|
||||
Store *Store
|
||||
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
|
||||
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
|
||||
Spec SpecBuilder
|
||||
Cadence time.Duration // 0 → disabled
|
||||
Logger *slog.Logger
|
||||
Spec SpecBuilder
|
||||
// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
|
||||
// (R-86). 0 → disabled.
|
||||
Cadence time.Duration
|
||||
// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
|
||||
// 0 → no settle requirement (any archive is a candidate).
|
||||
Settle time.Duration
|
||||
Logger *slog.Logger
|
||||
|
||||
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
|
||||
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
|
||||
@@ -89,6 +103,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
|
||||
store: opts.Store,
|
||||
spec: opts.Spec,
|
||||
cadence: opts.Cadence,
|
||||
settle: opts.Settle,
|
||||
logger: logger,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
tiers: append([]string(nil), opts.Tiers...),
|
||||
@@ -98,9 +113,19 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
|
||||
}
|
||||
}
|
||||
|
||||
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
|
||||
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
|
||||
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
|
||||
// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
|
||||
// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
|
||||
//
|
||||
// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
|
||||
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
|
||||
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
|
||||
//
|
||||
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
|
||||
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
|
||||
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
|
||||
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
|
||||
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
|
||||
//
|
||||
// Returns nil on ctx cancellation.
|
||||
func (s *Scheduler) Run(ctx context.Context) error {
|
||||
if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
|
||||
@@ -108,7 +133,8 @@ func (s *Scheduler) Run(ctx context.Context) error {
|
||||
<-ctx.Done()
|
||||
return nil
|
||||
}
|
||||
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
|
||||
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
|
||||
"eval_interval", s.cadence, "settle", s.settle)
|
||||
t := time.NewTicker(s.cadence)
|
||||
defer t.Stop()
|
||||
for {
|
||||
@@ -122,8 +148,12 @@ func (s *Scheduler) Run(ctx context.Context) error {
|
||||
}
|
||||
}
|
||||
|
||||
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
|
||||
// no backup exists yet. Deterministic given s.now — tests call it directly.
|
||||
// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
|
||||
// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
|
||||
// is already in flight. Deterministic given s.now — tests call it directly.
|
||||
//
|
||||
// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
|
||||
// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
|
||||
func (s *Scheduler) tick(ctx context.Context) {
|
||||
if s.spec == nil {
|
||||
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
|
||||
@@ -132,27 +162,48 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
|
||||
return
|
||||
}
|
||||
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
|
||||
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
|
||||
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
|
||||
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
|
||||
// cancelled backup costs the backup.
|
||||
release, busy, ok := s.inFlight.TryAcquire("restore-test")
|
||||
if !ok {
|
||||
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight", "busy", busy)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
|
||||
// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
|
||||
//
|
||||
// It used to be the other way round, and correctly so: the gate was held for one heavy run a
|
||||
// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
|
||||
// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
|
||||
// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
|
||||
// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
|
||||
// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
|
||||
// not be able to manufacture that.
|
||||
//
|
||||
// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
|
||||
// before anything heavy begins.
|
||||
archive, target, err := s.pickForThisRun(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
|
||||
return
|
||||
}
|
||||
if archive == "" {
|
||||
s.logger.Info("backup: restore-test skipped; no backup available yet")
|
||||
// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
|
||||
//
|
||||
// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
|
||||
// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
|
||||
// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
|
||||
// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
|
||||
// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
|
||||
// nothing run last night?" is in the log rather than in a re-derivation.
|
||||
s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
|
||||
return
|
||||
}
|
||||
|
||||
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
|
||||
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
|
||||
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
|
||||
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
|
||||
// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
|
||||
release, busy, ok := s.inFlight.TryAcquire("restore-test")
|
||||
if !ok {
|
||||
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
|
||||
"busy", busy, "target", target, "archive", archive)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
|
||||
spec := s.spec(ctx, archive)
|
||||
spec.Archive = archive
|
||||
@@ -165,8 +216,14 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
|
||||
// that fails every time would look freshly proven and quietly stop being retried.
|
||||
if rt.Pass && s.rtState != nil && target != "" {
|
||||
if err := s.rtState.RecordSuccess(target, s.now()); err != nil {
|
||||
s.logger.Warn("backup: could not persist the restore-test rotation state", "target", target, "err", err)
|
||||
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
|
||||
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
|
||||
// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
|
||||
// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
|
||||
// this run was actually judged as, and deriving it later would need a storage lookup that
|
||||
// can fail on the one path where failing means mislabelling a proof.
|
||||
if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
|
||||
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
@@ -189,46 +246,49 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
// rotating reports whether multi-tier rotation is wired.
|
||||
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
|
||||
|
||||
// pickForThisRun chooses the tier and its newest archive.
|
||||
// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
|
||||
// order.
|
||||
//
|
||||
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1): the tier whose last SUCCESSFUL restore-test
|
||||
// is oldest goes first, never-proven first of all. Self-balancing, no config knob, and it naturally
|
||||
// prioritises a tier that has never been proven — which on this fleet was the offsite tier, unproven
|
||||
// for its entire existence while reporting `applied`.
|
||||
// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
|
||||
// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
|
||||
// due?", and "nothing" is a normal, frequent and correct answer.
|
||||
//
|
||||
// A tier with no archives is SKIPPED, not failed, and the next tier is tried. Skipping to a testable
|
||||
// tier is strictly better than burning the whole cadence: a brand-new offsite tier has nothing to
|
||||
// restore yet, and that is normal, not broken. It cannot starve the empty tier either — as soon as
|
||||
// it has an archive it still sorts first, because it is still the least recently proven.
|
||||
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
|
||||
// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
|
||||
// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
|
||||
// that a test happens at all.
|
||||
//
|
||||
// Returns ("", "", nil) when nothing anywhere is testable.
|
||||
// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
|
||||
// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
|
||||
// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
|
||||
// storage being unreadable must not cost the other tier its proof, and must not be silent either.
|
||||
//
|
||||
// Returns ("", "", nil) when nothing anywhere is due.
|
||||
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
|
||||
if !s.rotating() {
|
||||
// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
|
||||
// identity and no persisted proof here, so there is nothing to compare an archive against
|
||||
// and no due-check is possible. It runs on every evaluation, exactly as it always did.
|
||||
a, perr := s.pick(ctx)
|
||||
return a, "", perr // pre-R-85 single-tier path; no rotation credit to record
|
||||
}
|
||||
order := s.tiers
|
||||
if s.rtState != nil {
|
||||
order = s.rtState.OldestFirst(s.tiers)
|
||||
return a, "", perr
|
||||
}
|
||||
var firstErr error
|
||||
for _, t := range order {
|
||||
a, perr := s.tierPick(ctx, t)
|
||||
if perr != nil {
|
||||
// One tier's storage being unreadable must not block the others.
|
||||
for _, v := range s.EvaluateDue(ctx) {
|
||||
if v.Err != nil {
|
||||
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
|
||||
"target", t, "err", perr)
|
||||
"target", v.Target, "err", v.Err)
|
||||
if firstErr == nil {
|
||||
firstErr = perr
|
||||
firstErr = v.Err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if a == "" {
|
||||
s.logger.Debug("backup: restore-test tier has no archive yet; trying the next", "target", t)
|
||||
if !v.Due {
|
||||
s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
|
||||
continue
|
||||
}
|
||||
s.logger.Info("backup: restore-test tier selected (oldest-proven first)", "target", t, "archive", a)
|
||||
return a, t, nil
|
||||
s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
|
||||
"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
|
||||
return v.Archive, v.Target, nil
|
||||
}
|
||||
if firstErr != nil {
|
||||
return "", "", firstErr
|
||||
|
||||
@@ -10,8 +10,22 @@ import (
|
||||
// Store holds the agent's LATEST backup result per target and the latest restore-test
|
||||
// result — the point-in-time state the host-report surfaces. It is updated by the backup
|
||||
// runner + the restore-test scheduler/selftest and read by the collector via the hub
|
||||
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence
|
||||
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access.
|
||||
// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
|
||||
// collector vs scheduler access.
|
||||
//
|
||||
// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
|
||||
// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
|
||||
// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
|
||||
// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
|
||||
// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
|
||||
// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
|
||||
// consecutive host-reports carried `0 restore-tests`.
|
||||
//
|
||||
// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
|
||||
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
|
||||
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
|
||||
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
|
||||
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
|
||||
type Store struct {
|
||||
mu sync.Mutex
|
||||
byTarget map[string]hub.Backup // latest backup per target id
|
||||
|
||||
+95
-13
@@ -333,9 +333,23 @@ type BackupConfig struct {
|
||||
LocalBackupTarget string `json:"local_backup_target"`
|
||||
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
|
||||
RestoreStorage string `json:"restore_storage"`
|
||||
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
|
||||
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
|
||||
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
|
||||
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
|
||||
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
|
||||
// the interval trigger with a per-archive due-check — so a positive value now seeds
|
||||
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
|
||||
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
|
||||
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
|
||||
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
|
||||
// generation no matter how often it is asked. This interval sets two things — the latency
|
||||
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
|
||||
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
|
||||
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
|
||||
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
|
||||
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
|
||||
// Restore-testing an archive a backup is still writing proves nothing about the backup that
|
||||
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
|
||||
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
|
||||
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
|
||||
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
|
||||
// always excluded. Defaults to 990000–990009.
|
||||
@@ -545,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
|
||||
return defaultBackupTarget
|
||||
}
|
||||
|
||||
// Default scratch VMID band + restore-test cadence.
|
||||
// Default scratch VMID band + the two R-86 restore-test knobs.
|
||||
const (
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
defaultRestoreTestCadence = 24 * time.Hour
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
|
||||
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
|
||||
// and neither bound alone would have picked it:
|
||||
//
|
||||
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
|
||||
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
|
||||
// is one storage-content listing per tier:
|
||||
//
|
||||
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
|
||||
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
|
||||
// both tiers together .................. 430 ms
|
||||
//
|
||||
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
|
||||
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
|
||||
// number that would have forbidden a frequent poll is the one nobody measures.
|
||||
//
|
||||
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
|
||||
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
|
||||
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
|
||||
// timer retried a broken tier once a day.
|
||||
//
|
||||
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
|
||||
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
|
||||
// second rate limiter anywhere — the pacing remains one test per archive generation.
|
||||
defaultRestoreTestEvalInterval = 6 * time.Hour
|
||||
|
||||
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
|
||||
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
|
||||
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
|
||||
defaultRestoreTestSettle = 24 * time.Hour
|
||||
)
|
||||
|
||||
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
|
||||
// 0 → 24h default, negative → 0 (disabled).
|
||||
func (b BackupConfig) RestoreTestCadence() time.Duration {
|
||||
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
|
||||
// value as-is, 0 → the measured default, negative → 0 (disabled).
|
||||
//
|
||||
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
|
||||
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
|
||||
// sets the interval, because the interval no longer decides that a test happens.
|
||||
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
|
||||
if b.RestoreTestCadenceSeconds < 0 {
|
||||
return 0 // legacy DISABLE — preserved verbatim
|
||||
}
|
||||
switch {
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
|
||||
case b.RestoreTestCadenceSeconds < 0:
|
||||
case b.RestoreTestEvalIntervalSeconds > 0:
|
||||
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
|
||||
case b.RestoreTestEvalIntervalSeconds < 0:
|
||||
return 0 // disabled
|
||||
default:
|
||||
return defaultRestoreTestCadence
|
||||
return defaultRestoreTestEvalInterval
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
|
||||
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
|
||||
//
|
||||
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
|
||||
// was expressing "how long may pass between a backup and the confidence that it restores". That
|
||||
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
|
||||
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
|
||||
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
|
||||
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
|
||||
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
|
||||
func (b BackupConfig) RestoreTestSettle() time.Duration {
|
||||
switch {
|
||||
case b.RestoreTestSettleSeconds > 0:
|
||||
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
|
||||
case b.RestoreTestSettleSeconds < 0:
|
||||
return 0 // explicitly no settle requirement
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
|
||||
default:
|
||||
return defaultRestoreTestSettle
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
|
||||
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
|
||||
// without saying so is exactly the silent repurposing §8.3 forbids.
|
||||
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
|
||||
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
|
||||
}
|
||||
|
||||
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
|
||||
// negative → 0 (disabled).
|
||||
func (b BackupConfig) PBSVerifyCadence() time.Duration {
|
||||
@@ -810,6 +890,8 @@ func applyEnv(cfg *Config) {
|
||||
cfg.Backup.RestoreStorage = v
|
||||
}
|
||||
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
|
||||
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
|
||||
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
|
||||
}
|
||||
|
||||
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery
|
||||
// code R, and hand back EXACTLY ONE field: the offsite restic repository password.
|
||||
//
|
||||
// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and
|
||||
// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one
|
||||
// trust tier down — needs none of them, and returning them would widen the blast radius of a
|
||||
// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later.
|
||||
//
|
||||
// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a
|
||||
// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper
|
||||
// keeps that property: it takes R as an argument, passes it straight through, and holds no copy.
|
||||
// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent).
|
||||
//
|
||||
// The errors below are DISTINCT on purpose. "could not fetch", "no blob", "wrong code" and "the blob
|
||||
// predates the field" are FOUR different situations for the operator and only one of them is a fault.
|
||||
//
|
||||
// ⚠ THERE WERE THREE, AND THE FOURTH WAS THE DEFECT (R-224, 2026-08-06). This comment said "three"
|
||||
// and named "no blob", "wrong code" and "predates the field" — while a FAILED FETCH was wrapped as an
|
||||
// anonymous error and fell through the caller's `default` branch into the wrong-code message. So a
|
||||
// hub that could not be reached was reported to the customer as a bad recovery code.
|
||||
//
|
||||
// Measured live on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and
|
||||
// a CORRECT current recovery code, the customer was told the code did not open their package — in
|
||||
// 0.0556 s, when a real unseal costs ~1 s of scrypt. The agent's own log carried the truth the whole
|
||||
// time (`escrow: fetching the sealed bundle: hub: transport error: … no route to host`) and the HTTP
|
||||
// boundary threw it away.
|
||||
//
|
||||
// The discriminator therefore has to be a VALUE, not a log line — that is what ErrBundleFetch is.
|
||||
|
||||
var (
|
||||
// ErrBundleFetch — the sealed bundle could not be FETCHED (the hub refused, was unreachable, or
|
||||
// the transport failed). **The recovery code was never used**, so nothing about it is known and
|
||||
// nothing may be said about it. Wraps the underlying cause for the operator log; carries no secret.
|
||||
ErrBundleFetch = errors.New("escrow: the sealed bundle could not be fetched")
|
||||
// ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run.
|
||||
ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)")
|
||||
// ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected
|
||||
// for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be
|
||||
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
|
||||
// not sent hunting for a mistyped recovery code that was typed correctly.
|
||||
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
|
||||
)
|
||||
|
||||
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
|
||||
// An interface-free func field keeps this package free of any dependency on the hub client.
|
||||
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
|
||||
|
||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||
type OffsiteKeyRecoverer struct {
|
||||
Fetch BlobFetcher
|
||||
}
|
||||
|
||||
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
|
||||
//
|
||||
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
|
||||
// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere.
|
||||
// That property is the crypto's, not a check here, which is why this function has no "validate R"
|
||||
// step to get wrong.
|
||||
//
|
||||
// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes.
|
||||
func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) {
|
||||
if r.Fetch == nil {
|
||||
return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured")
|
||||
}
|
||||
if recoveryCode == "" {
|
||||
return "", fmt.Errorf("escrow: the recovery code is required")
|
||||
}
|
||||
blob, present, err := r.Fetch(ctx)
|
||||
if err != nil {
|
||||
// R-224: joined with ErrBundleFetch so the caller can classify by VALUE. The cause stays
|
||||
// wrapped for the operator log; neither carries a secret. Before this, the fetch failure was
|
||||
// an anonymous error and the local-api handler's `default` branch reported it to the customer
|
||||
// as a wrong recovery code.
|
||||
return "", fmt.Errorf("%w: %w", ErrBundleFetch, err)
|
||||
}
|
||||
if !present || len(blob) == 0 {
|
||||
return "", ErrNoEscrowBlob
|
||||
}
|
||||
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
|
||||
if err != nil {
|
||||
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||
}
|
||||
if bundle.ResticRepoPassword == "" {
|
||||
return "", ErrNoResticPassword
|
||||
}
|
||||
return bundle.ResticRepoPassword, nil
|
||||
}
|
||||
@@ -0,0 +1,267 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips
|
||||
// elsewhere). These are the unit half of the session's question — "is the repository password
|
||||
// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware.
|
||||
|
||||
const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
|
||||
|
||||
func sealBundle(t *testing.T, b IdentityBundle, r string) []byte {
|
||||
t.Helper()
|
||||
blob, err := WrapIdentityBundle(context.Background(), b, r)
|
||||
if err != nil {
|
||||
t.Fatalf("WrapIdentityBundle: %v", err)
|
||||
}
|
||||
return blob
|
||||
}
|
||||
|
||||
func fetcherFor(blob []byte) BlobFetcher {
|
||||
return func(context.Context) ([]byte, bool, error) { return blob, true, nil }
|
||||
}
|
||||
|
||||
// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it
|
||||
// is the REPOSITORY password rather than some other field of a bundle that also parses.
|
||||
//
|
||||
// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of
|
||||
// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS.
|
||||
// That mutation is the shape of the bug that would otherwise ship silently, because every one of
|
||||
// those fields is a non-empty string that looks like a secret.
|
||||
func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
blob := sealBundle(t, IdentityBundle{
|
||||
TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER",
|
||||
PBSToken: "PBS-TOKEN-NOT-THE-ANSWER",
|
||||
WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
|
||||
ResticRepoPassword: repoPW,
|
||||
}, testR)
|
||||
|
||||
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if err != nil {
|
||||
t.Fatalf("recover: %v", err)
|
||||
}
|
||||
if got != repoPW {
|
||||
t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+
|
||||
"field of the bundle was returned", len(got), len(repoPW))
|
||||
}
|
||||
// Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check
|
||||
// above by coincidence.
|
||||
for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} {
|
||||
if got == other {
|
||||
t.Fatalf("the recoverer returned the wrong bundle field")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is
|
||||
// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no
|
||||
// validation step here to get wrong — the test pins that it stays that way.
|
||||
func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||
|
||||
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all")
|
||||
if err == nil {
|
||||
t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids")
|
||||
}
|
||||
if got != "" {
|
||||
t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got))
|
||||
}
|
||||
// The error may name the step; it may never name a secret.
|
||||
for _, secret := range []string{repoPW, testR, "not the recovery code at all"} {
|
||||
if strings.Contains(err.Error(), secret) {
|
||||
t.Fatalf("the failure message leaked a secret: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before
|
||||
// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly
|
||||
// typed recovery code would be the wrong instruction.
|
||||
func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) {
|
||||
ensureAge(t)
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR)
|
||||
|
||||
_, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if !errors.Is(err, ErrNoResticPassword) {
|
||||
t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D at this layer — no blob is a clean, distinguishable answer.
|
||||
func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) {
|
||||
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
|
||||
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if !errors.Is(err, ErrNoEscrowBlob) {
|
||||
t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is
|
||||
// run for real, and the whole tree is then walked: no file may contain R (or the recovered password),
|
||||
// and the staging directory the unseal creates must be gone.
|
||||
//
|
||||
// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add
|
||||
// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before
|
||||
// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the
|
||||
// walk finds it → this FAILS.
|
||||
func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) {
|
||||
ensureAge(t)
|
||||
const repoPW = "1111111111111111111111111111111111111111111111111111111111111111"
|
||||
tmp := t.TempDir()
|
||||
t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk
|
||||
|
||||
const wrongR = "wrong code entirely"
|
||||
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||
if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil {
|
||||
t.Fatalf("recover: %v", err)
|
||||
}
|
||||
// A failed unseal must leave nothing either — exercise both paths before walking.
|
||||
_, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR)
|
||||
|
||||
// THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later
|
||||
// call OVERWRITING the leaked file with a different secret — which is exactly how the first
|
||||
// version of this test passed its own red-proof while R sat on disk. Nothing in this test writes
|
||||
// under TMPDIR, so after both calls the tree must contain no files at all.
|
||||
var survivors []string
|
||||
err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil || info == nil || info.IsDir() || path == tmp {
|
||||
return nil
|
||||
}
|
||||
survivors = append(survivors, strings.TrimPrefix(path, tmp))
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(survivors) > 0 {
|
||||
t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+
|
||||
"recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors)
|
||||
}
|
||||
// Defence in depth: any secret that DOES appear anywhere is named, for every code used.
|
||||
_ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil || info == nil || info.IsDir() {
|
||||
return nil
|
||||
}
|
||||
body, rerr := os.ReadFile(path)
|
||||
if rerr != nil {
|
||||
return nil
|
||||
}
|
||||
for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} {
|
||||
if strings.Contains(string(body), secret) {
|
||||
t.Errorf("%s survived on disk at %s", label, path)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
// And the staging directories are gone, not merely free of secrets.
|
||||
entries, _ := os.ReadDir(tmp)
|
||||
for _, e := range entries {
|
||||
if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") {
|
||||
t.Fatalf("an unseal staging directory survived: %s", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be
|
||||
// sent to re-read their recovery code because the hub was unreachable.
|
||||
//
|
||||
// ⚠ THIS TEST WAS GREEN THROUGHOUT THE DEFECT IT DESCRIBES (R-224, 2026-08-06). Its sentence is
|
||||
// exactly right and it did not prevent anything, for two reasons worth keeping:
|
||||
//
|
||||
// 1. **It asserted the MECHANISM, one layer below the consequence.** It checked this package's error
|
||||
// STRING. The merge happened one layer up, in the local-api handler's `default` branch, which
|
||||
// answered a fetch failure with "the recovery code did not open the sealed bundle". The customer
|
||||
// never sees this string; they see that one. The project's own rule — prefer the test that asserts
|
||||
// the CONSEQUENCE (does the customer get blamed?) over the one that asserts the MECHANISM (is the
|
||||
// error distinct here?) — names this case precisely.
|
||||
// 2. **It asserted on TEXT.** `strings.Contains(err.Error(), …)` cannot be consumed by a caller, so
|
||||
// it pinned something no production code could branch on. The distinction it checked was real and
|
||||
// unusable.
|
||||
//
|
||||
// It now asserts the SENTINEL, which is what the handler branches on, and its consequence-level twin
|
||||
// lives in `internal/localapi/escrow_recover_class_test.go` where the status is asserted.
|
||||
func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) {
|
||||
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) {
|
||||
return nil, false, errors.New("hub: connection refused")
|
||||
}}
|
||||
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if err == nil || !errors.Is(err, ErrBundleFetch) {
|
||||
t.Fatalf("a fetch failure must classify as ErrBundleFetch, got %v", err)
|
||||
}
|
||||
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
|
||||
t.Fatal("a transport failure must not masquerade as a content verdict")
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-224 — A FAILED FETCH IS NOT A WRONG CODE ──────────────────────────────────────────────────
|
||||
//
|
||||
// CAMPAIGN-11 F3 measured the consequence of these two being indistinguishable: with the hub
|
||||
// firewalled off and a CORRECT current recovery code, the customer was told the code did not open
|
||||
// their package, in 0.0556 s — no unseal was attempted at all.
|
||||
//
|
||||
// The pair below is the whole point. Asserting only the first would pass with a `return ErrBundleFetch`
|
||||
// stuck on every error path, which is the same defect pointing the other way.
|
||||
func TestRecoverOffsiteRepoPassword_FetchFailureIsClassifiedAsFetch(t *testing.T) {
|
||||
boom := errors.New("hub: transport error: dial tcp 37.191.56.193:443: connect: no route to host")
|
||||
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, boom }}
|
||||
|
||||
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if err == nil {
|
||||
t.Fatal("a failing fetch must return an error")
|
||||
}
|
||||
// RED-PROOF: drop the `%w: %w` join in RecoverOffsiteRepoPassword (return the bare wrapped cause,
|
||||
// as it was before R-224) → this FAILS, and the local-api handler falls back to the wrong-code
|
||||
// message exactly as it did on 2026-08-05.
|
||||
if !errors.Is(err, ErrBundleFetch) {
|
||||
t.Fatalf("a failed fetch must classify as ErrBundleFetch, got %v", err)
|
||||
}
|
||||
// The underlying cause survives for the operator log.
|
||||
if !errors.Is(err, boom) {
|
||||
t.Fatalf("the fetch cause must stay wrapped for the operator, got %v", err)
|
||||
}
|
||||
// And it must NOT be mistaken for either of the bundle-content situations.
|
||||
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
|
||||
t.Fatalf("a transport failure is neither of the bundle-content errors: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The other half: a genuinely wrong code must NOT classify as a fetch failure, or the fix trades one
|
||||
// misattribution for its mirror image and the customer is told the hub is down when they mistyped.
|
||||
func TestRecoverOffsiteRepoPassword_WrongCodeIsNotAFetchFailure(t *testing.T) {
|
||||
ensureAge(t)
|
||||
blob := sealBundle(t, IdentityBundle{ResticRepoPassword: "0123456789abcdef"}, testR)
|
||||
r := OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}
|
||||
|
||||
_, err := r.RecoverOffsiteRepoPassword(context.Background(),
|
||||
"wrong horse battery staple sedative anaconda wobbly kingdom placard yodel")
|
||||
if err == nil {
|
||||
t.Fatal("a wrong recovery code must fail closed")
|
||||
}
|
||||
if errors.Is(err, ErrBundleFetch) {
|
||||
t.Fatalf("a wrong code must NOT classify as a fetch failure, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A clean "the hub holds nothing" keeps its own identity too — it is not a fetch failure, and the
|
||||
// customer must not be told the hub was unreachable when it answered perfectly well.
|
||||
func TestRecoverOffsiteRepoPassword_AbsentBlobIsNotAFetchFailure(t *testing.T) {
|
||||
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
|
||||
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||
if !errors.Is(err, ErrNoEscrowBlob) {
|
||||
t.Fatalf("an absent blob must stay ErrNoEscrowBlob, got %v", err)
|
||||
}
|
||||
if errors.Is(err, ErrBundleFetch) {
|
||||
t.Fatalf("an absent blob is not a fetch FAILURE, got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -307,3 +307,50 @@ func tail(b []byte, max int) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199).
|
||||
// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet.
|
||||
type IdentityEscrowResponse struct {
|
||||
HostID string `json:"host_id"`
|
||||
Present bool `json:"present"`
|
||||
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||
}
|
||||
|
||||
// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the
|
||||
// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they
|
||||
// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds.
|
||||
//
|
||||
// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no
|
||||
// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different
|
||||
// endpoint with a different gate and is not reached from here.
|
||||
//
|
||||
// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged —
|
||||
// only its length.
|
||||
func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) {
|
||||
if c.hostID == "" {
|
||||
return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id")
|
||||
}
|
||||
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err)
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, &TransportError{Err: err}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||
}
|
||||
var out IdentityEscrowResponse
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err)
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
+96
-5
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
|
||||
RestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
|
||||
//
|
||||
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
|
||||
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
|
||||
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
|
||||
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
|
||||
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
|
||||
//
|
||||
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
|
||||
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
|
||||
//
|
||||
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
|
||||
type ProvenRestoreTestReporter interface {
|
||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||
type PBSReporter interface {
|
||||
@@ -79,6 +95,7 @@ type Collector struct {
|
||||
storage StorageObserver
|
||||
backups BackupReporter
|
||||
restoreTests RestoreTestReporter
|
||||
provenTests ProvenRestoreTestReporter
|
||||
pbs PBSReporter
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
@@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||
return []Backup{}
|
||||
}
|
||||
|
||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||
//
|
||||
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
|
||||
// than from a preference between them:
|
||||
//
|
||||
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
|
||||
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
|
||||
// record is short-lived by design;
|
||||
// - the persisted state holds the last SUCCESS per tier and survives a restart.
|
||||
//
|
||||
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
|
||||
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
|
||||
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
|
||||
// read at the hub as two tests.
|
||||
//
|
||||
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
|
||||
// would be a worse defect than the one this closes.
|
||||
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
|
||||
if c.restoreTests == nil {
|
||||
return []RestoreTest{}
|
||||
out := []RestoreTest{}
|
||||
if c.restoreTests != nil {
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
out = append(out, r...)
|
||||
}
|
||||
}
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
return r
|
||||
if c.provenTests == nil {
|
||||
return out
|
||||
}
|
||||
return []RestoreTest{}
|
||||
|
||||
// Index what we already have by tier, keeping the newest per tier.
|
||||
best := map[string]int{} // tier → index into out
|
||||
for i, rt := range out {
|
||||
if rt.SourceTier == "" {
|
||||
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
|
||||
}
|
||||
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
|
||||
best[rt.SourceTier] = i
|
||||
}
|
||||
}
|
||||
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
|
||||
if p.SourceTier == "" {
|
||||
continue // not usable as a per-tier proof; the state layer already filters these
|
||||
}
|
||||
i, seen := best[p.SourceTier]
|
||||
if !seen {
|
||||
out = append(out, p)
|
||||
best[p.SourceTier] = len(out) - 1
|
||||
continue
|
||||
}
|
||||
if newerRestoreTest(p, out[i]) {
|
||||
out[i] = p
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
|
||||
// treated as OLDER, so a malformed entry can never displace a good one.
|
||||
func newerRestoreTest(a, b RestoreTest) bool {
|
||||
ta, aok := parseRestoreTestedAt(a.TestedAt)
|
||||
tb, bok := parseRestoreTestedAt(b.TestedAt)
|
||||
if !aok {
|
||||
return false
|
||||
}
|
||||
if !bok {
|
||||
return true
|
||||
}
|
||||
return ta.After(tb)
|
||||
}
|
||||
|
||||
func parseRestoreTestedAt(s string) (time.Time, bool) {
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return t.UTC(), true
|
||||
}
|
||||
|
||||
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
|
||||
// argument because the persisted state is opened later in main() than the collector is built; the
|
||||
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
|
||||
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
|
||||
// all, and this fix must not become the next instance of that.
|
||||
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
|
||||
|
||||
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
|
||||
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
|
||||
if c.pbs == nil {
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-189 — a passing restore-test must survive an agent restart and reach the hub.
|
||||
//
|
||||
// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test
|
||||
// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged
|
||||
// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on
|
||||
// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86
|
||||
// made the agent refuse to re-test an archive it has already proven.
|
||||
//
|
||||
// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was
|
||||
// hollow because a perfectly regular series landed exactly on a threshold and passed under the
|
||||
// mutation it was meant to catch.
|
||||
|
||||
type fakeLatest struct{ tests []RestoreTest }
|
||||
|
||||
func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||
|
||||
type fakeProven struct{ tests []RestoreTest }
|
||||
|
||||
func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||
|
||||
func rt(tier, archive string, pass bool, at time.Time) RestoreTest {
|
||||
return RestoreTest{
|
||||
SourceArchive: archive, SourceTier: tier, Pass: pass,
|
||||
Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339),
|
||||
}
|
||||
}
|
||||
|
||||
// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what
|
||||
// is under test, not the rest of the collection.
|
||||
func mergeCollector(latest, proven []RestoreTest) *Collector {
|
||||
c := &Collector{}
|
||||
if latest != nil {
|
||||
c.restoreTests = &fakeLatest{tests: latest}
|
||||
}
|
||||
if proven != nil {
|
||||
c.provenTests = &fakeProven{tests: proven}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func findTier(got []RestoreTest, tier string) (RestoreTest, int) {
|
||||
var hit RestoreTest
|
||||
n := 0
|
||||
for _, e := range got {
|
||||
if e.SourceTier == tier {
|
||||
hit, n = e, n+1
|
||||
}
|
||||
}
|
||||
return hit, n
|
||||
}
|
||||
|
||||
// ── SCENARIO A — a proof survives a restart and reaches the hub ──────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from
|
||||
// collectRestoreTests (return the in-memory slice as it used to) →
|
||||
//
|
||||
// --- FAIL: TestMerge_ProofSurvivesARestart
|
||||
// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies)
|
||||
//
|
||||
// which is exactly the live observation: `0 restore-tests`. Restored.
|
||||
func TestMerge_ProofSurvivesARestart(t *testing.T) {
|
||||
provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp
|
||||
// After a restart the in-memory store is EMPTY — this is the whole point.
|
||||
c := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||
rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt),
|
||||
})
|
||||
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got)
|
||||
}
|
||||
e := got[0]
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" {
|
||||
t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive)
|
||||
}
|
||||
if e.SourceTier != "pbs" || !e.Pass {
|
||||
t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass)
|
||||
}
|
||||
if e.TestedAt != provenAt.Format(time.RFC3339) {
|
||||
t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO B — the report does not invent a pass ───────────────────────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the
|
||||
// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the
|
||||
// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which
|
||||
// this test injects directly and the assertion below rejects.
|
||||
func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) {
|
||||
// Nothing proven anywhere: no in-memory result, no persisted proof.
|
||||
c := mergeCollector([]RestoreTest{}, []RestoreTest{})
|
||||
if got := c.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+
|
||||
"worse than the defect being fixed; got %+v", got)
|
||||
}
|
||||
|
||||
// And an entry the state layer could not describe (no tier) is never promoted into a proof.
|
||||
c2 := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||
{SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true,
|
||||
TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)},
|
||||
})
|
||||
if got := c2.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||
t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ─────────────────────────
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry
|
||||
// unconditionally) →
|
||||
//
|
||||
// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier
|
||||
// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests
|
||||
//
|
||||
// Restored.
|
||||
func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) {
|
||||
lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box
|
||||
fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)
|
||||
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||
)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
e, n := findTier(got, "pbs")
|
||||
if n != 1 {
|
||||
t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got)
|
||||
}
|
||||
if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||
t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt)
|
||||
}
|
||||
|
||||
// ...and the older-in-memory / newer-persisted direction, which is the post-restart case.
|
||||
c2 := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||
)
|
||||
e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs")
|
||||
if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||
t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO D — a failure still reaches the hub ─────────────────────────────────────────────
|
||||
//
|
||||
// The merge must not mask a failure with an older stored success. A failing tier is retried at the
|
||||
// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest
|
||||
// DR signal this system produces.
|
||||
func TestMerge_AFailureIsStillReported(t *testing.T) {
|
||||
provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC)
|
||||
failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC)
|
||||
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)},
|
||||
)
|
||||
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||
if n != 1 {
|
||||
t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background()))
|
||||
}
|
||||
if e.Pass {
|
||||
t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+
|
||||
"silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// Two different tiers are both reported — the merge is per tier, not a single slot.
|
||||
func TestMerge_BothTiersSurvive(t *testing.T) {
|
||||
c := mergeCollector(
|
||||
[]RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true,
|
||||
time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))},
|
||||
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true,
|
||||
time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))},
|
||||
)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if _, n := findTier(got, "local"); n != 1 {
|
||||
t.Fatalf("the in-memory tier must survive the merge; got %+v", got)
|
||||
}
|
||||
if _, n := findTier(got, "pbs"); n != 1 {
|
||||
t.Fatalf("the persisted tier must survive the merge; got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A malformed timestamp must never displace a good entry — "unparseable" is not "newest".
|
||||
func TestMerge_MalformedTimestampNeverWins(t *testing.T) {
|
||||
good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC))
|
||||
bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"}
|
||||
|
||||
c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad})
|
||||
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||
if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" {
|
||||
t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil proven-source leaves the pre-R-189 behaviour exactly as it was.
|
||||
func TestMerge_NilProvenSourceIsANoOp(t *testing.T) {
|
||||
only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))
|
||||
c := mergeCollector([]RestoreTest{only}, nil)
|
||||
got := c.collectRestoreTests(context.Background())
|
||||
if len(got) != 1 || got[0].SourceArchive != only.SourceArchive {
|
||||
t.Fatalf("a nil durable source must not change anything; got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
)
|
||||
|
||||
// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository
|
||||
// password from the hub's sealed bundle, using the customer's recovery code R.
|
||||
//
|
||||
// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`)
|
||||
// is an agent runtime dependency and is deliberately absent from the controller image; the sealed
|
||||
// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is
|
||||
// that write's mirror; and the controller is a trust tier down — it should receive one field, not a
|
||||
// bundle it has no use for.
|
||||
//
|
||||
// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that
|
||||
// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it:
|
||||
// - arrives in the request body over the already-pinned local-API channel (the operator accepted
|
||||
// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end);
|
||||
// - is held in memory for the duration of one call and cleared on BOTH paths;
|
||||
// - is never written to disk, never an argument in a process list, and never logged at any level,
|
||||
// including inside an error;
|
||||
// - is never echoed: no response this endpoint can emit contains it.
|
||||
//
|
||||
// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see
|
||||
// `logRequests`. Do not add a body dump.
|
||||
//
|
||||
// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares
|
||||
// (compare by hash, never by value). The password itself is present because the next link — placing a
|
||||
// recovered password so the existing repository opens — needs it, and building a hash-only seam now
|
||||
// would have to be torn out to add it. The controller's diagnostic reads only the hash.
|
||||
|
||||
type recoverOffsitePasswordRequest struct {
|
||||
VMID int `json:"vmid"`
|
||||
// RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed.
|
||||
RecoveryCode string `json:"recovery_code"`
|
||||
}
|
||||
|
||||
// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only
|
||||
// the offsite repository password (plus its sha256, for hash-only comparison by the caller).
|
||||
func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
var req recoverOffsitePasswordRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||
return
|
||||
}
|
||||
R := strings.TrimSpace(req.RecoveryCode)
|
||||
req.RecoveryCode = "" // drop the decoded copy immediately
|
||||
if R == "" {
|
||||
writeErr(w, http.StatusBadRequest, "recovery_code is required")
|
||||
return
|
||||
}
|
||||
if s.escrowRecovery == nil {
|
||||
R = ""
|
||||
writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid)
|
||||
|
||||
pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R)
|
||||
R = "" // cleared on BOTH paths, before anything else can happen
|
||||
if err != nil {
|
||||
// Each situation gets its own status and its own words. None of them names a secret.
|
||||
switch {
|
||||
// ── R-224 (2026-08-06) — THE FETCH FAILURE IS NOT A WRONG CODE. ────────────────────────
|
||||
//
|
||||
// This case did not exist, and its absence is the defect. A failed fetch fell through to the
|
||||
// `default` below and was answered with "the recovery code did not open the sealed bundle" —
|
||||
// so a hub that could not be reached was reported to the customer as a bad recovery code, on
|
||||
// the one screen whose whole purpose is to be believed about their backups.
|
||||
//
|
||||
// Measured live 2026-08-05 (CAMPAIGN-11 F3 and F4): a CORRECT current code returned that
|
||||
// message in 0.0556 s with the hub firewalled off, and in 0.0299 s with this agent stopped —
|
||||
// against ~1.0 s for a genuine unseal. No unseal was attempted in either case.
|
||||
//
|
||||
// 502 rather than 400: 4xx says "your request was bad", and the request was not bad — an
|
||||
// upstream dependency failed. The status is the machine-readable half; the controller
|
||||
// classifies on it and must never parse this sentence.
|
||||
//
|
||||
// ⚠ THE CODE WAS NOT USED. Nothing may be said about it — not that it was wrong, and not
|
||||
// that it was right.
|
||||
case errors.Is(err, escrow.ErrBundleFetch):
|
||||
s.logger.Warn("local-api: offsite key recovery: the sealed bundle could not be FETCHED — the recovery code was never used", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusBadGateway, "the sealed recovery bundle could not be fetched from the hub — the recovery code was NOT used and nothing was written")
|
||||
case errors.Is(err, escrow.ErrNoEscrowBlob):
|
||||
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
|
||||
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
|
||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||
default:
|
||||
// The fail-closed WRONG-CODE case, and only it: the bundle was fetched and `age -d`
|
||||
// refused it. Every other situation above has its own status. The agent log records the
|
||||
// STEP, never the code.
|
||||
s.logger.Warn("local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle — nothing was written")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
|
||||
// §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this
|
||||
// concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing).
|
||||
s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+
|
||||
"(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)",
|
||||
"vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:]))
|
||||
writeOK(w, map[string]any{
|
||||
"restic_repo_password": pw,
|
||||
"restic_pw_sha256": hex.EncodeToString(sum[:]),
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
)
|
||||
|
||||
// R-224 — THE STATUS IS THE DISCRIMINATOR, and this test asserts the CONSEQUENCE (what the HTTP
|
||||
// boundary answers) rather than the mechanism (that the sentinel exists).
|
||||
//
|
||||
// The controller one trust tier down classifies on the STATUS and must never parse the sentence. So
|
||||
// the contract this pins is: four distinguishable situations, four distinct statuses, and the
|
||||
// wrong-code message reachable ONLY from a real refusal.
|
||||
//
|
||||
// Before R-224 the first and last rows both answered 400 with the same sentence — which is how
|
||||
// CAMPAIGN-11 F3 told a customer holding a CORRECT code that it did not open their package.
|
||||
|
||||
type fakeRecoverer struct{ err error }
|
||||
|
||||
func (f fakeRecoverer) RecoverOffsiteRepoPassword(context.Context, string) (string, error) {
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
return "0123456789abcdef0123456789abcdef", nil
|
||||
}
|
||||
|
||||
func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantStatus int
|
||||
// mustNotSay guards the specific misattribution each status exists to prevent.
|
||||
mustNotSay []string
|
||||
}{
|
||||
{
|
||||
name: "fetch failed — the code was NEVER used",
|
||||
err: errors.Join(escrow.ErrBundleFetch, errors.New("hub: transport error: no route to host")),
|
||||
wantStatus: 502,
|
||||
mustNotSay: []string{"did not open"},
|
||||
},
|
||||
{
|
||||
name: "wrong code — the bundle WAS fetched and refused it",
|
||||
err: errors.New("escrow: the recovery code did not unwrap the identity escrow"),
|
||||
wantStatus: 400,
|
||||
mustNotSay: []string{"could not be fetched"},
|
||||
},
|
||||
{
|
||||
name: "the hub holds no bundle",
|
||||
err: escrow.ErrNoEscrowBlob,
|
||||
wantStatus: 404,
|
||||
mustNotSay: []string{"did not open"},
|
||||
},
|
||||
{
|
||||
name: "the bundle predates the repository-password field",
|
||||
err: escrow.ErrNoResticPassword,
|
||||
wantStatus: 409,
|
||||
mustNotSay: []string{"could not be fetched"},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.escrowRecovery = fakeRecoverer{err: tc.err}
|
||||
w := do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
|
||||
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`)
|
||||
if w.Code != tc.wantStatus {
|
||||
t.Fatalf("status: got %d, want %d — body=%s", w.Code, tc.wantStatus, w.Body.String())
|
||||
}
|
||||
for _, phrase := range tc.mustNotSay {
|
||||
if strings.Contains(w.Body.String(), phrase) {
|
||||
t.Fatalf("the %d answer must not say %q — body=%s", tc.wantStatus, phrase, w.Body.String())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The pair that matters most, stated as its own assertion so a regression cannot hide inside a table:
|
||||
// a fetch failure and a wrong code must never answer with the SAME status. Collapsing them is the
|
||||
// whole of R-224.
|
||||
func TestRecoverOffsitePassword_FetchFailureAndWrongCodeDiffer(t *testing.T) {
|
||||
status := func(err error) int {
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.escrowRecovery = fakeRecoverer{err: err}
|
||||
return do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
|
||||
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`).Code
|
||||
}
|
||||
fetch := status(errors.Join(escrow.ErrBundleFetch, errors.New("no route to host")))
|
||||
wrong := status(errors.New("escrow: the recovery code did not unwrap the identity escrow"))
|
||||
// RED-PROOF: delete the ErrBundleFetch case from handleRecoverOffsitePassword → both become 400
|
||||
// → this FAILS. That is the exact pre-R-224 code, and the exact defect CAMPAIGN-11 measured.
|
||||
if fetch == wrong {
|
||||
t.Fatalf("a failed fetch and a wrong code must not share a status (both %d)", fetch)
|
||||
}
|
||||
}
|
||||
@@ -111,6 +111,14 @@ type HostMetricsProvider interface {
|
||||
}
|
||||
|
||||
// Options configures a Server.
|
||||
// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and
|
||||
// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the
|
||||
// localapi package needs no hub-client dependency and the route is testable without crypto.
|
||||
// R is an argument and is never retained by any implementation.
|
||||
type EscrowRecoverer interface {
|
||||
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error)
|
||||
}
|
||||
|
||||
type Options struct {
|
||||
ListenAddr string // bridge IP:port
|
||||
Cert tls.Certificate
|
||||
@@ -210,6 +218,11 @@ type Options struct {
|
||||
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
|
||||
LogRing *applog.Ring
|
||||
Logger *slog.Logger
|
||||
// EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind
|
||||
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
|
||||
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
|
||||
EscrowRecovery EscrowRecoverer
|
||||
|
||||
}
|
||||
|
||||
// defaultBackupCadence is the fallback /backup/due window when none is configured.
|
||||
@@ -273,6 +286,11 @@ type Server struct {
|
||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
|
||||
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
|
||||
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||
escrowRecovery EscrowRecoverer
|
||||
intent IntentRecorder // slice 10 P3 (optional)
|
||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
@@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
netMountRoot: storage.NetworkMountRoot,
|
||||
smbCredsDir: o.SmbCredsDir,
|
||||
escrowStagePath: o.EscrowStagePath,
|
||||
escrowRecovery: o.EscrowRecovery,
|
||||
intent: o.Intent,
|
||||
guestBinds: o.GuestBinds,
|
||||
formatJobs: o.FormatJobs,
|
||||
@@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
|
||||
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
|
||||
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
|
||||
// R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle,
|
||||
// using the customer recovery code supplied in the body. Returns that ONE field. See
|
||||
// escrow_recover.go for R's handling rules — they are the tightest in this package.
|
||||
mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
|
||||
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.
|
||||
|
||||
@@ -45,6 +45,35 @@ func (c *Client) Pool(ctx context.Context, name string) (PoolInfo, error) {
|
||||
return p, c.get(ctx, "/pools/"+url.PathEscape(name), &p)
|
||||
}
|
||||
|
||||
// Permissions returns the privileges this API TOKEN holds at an ACL path, as
|
||||
// GET /access/permissions?path=<path> answers it: privilege name → 1.
|
||||
//
|
||||
// R-185. It asks about the CALLER — the agent's own token — which is the only useful form of the
|
||||
// question. Asking as root answers a different question and always says yes.
|
||||
//
|
||||
// MEASURED SHAPE (demo-felhom, 2026-08-03), because the whole value of this call is reading the
|
||||
// answer correctly and the obvious reading is wrong:
|
||||
//
|
||||
// /storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
|
||||
// /storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
|
||||
//
|
||||
// The ungranted path does NOT answer empty, and does NOT 403. It answers with the privileges
|
||||
// INHERITED from the box-wide `/` grant — so "is this path present in the response" reports OK for a
|
||||
// storage the agent demonstrably cannot list. The caller must test for the SPECIFIC privilege.
|
||||
//
|
||||
// The response is keyed by path; an absent path yields no privileges, which is the same answer as
|
||||
// "none" and is treated as such by the caller.
|
||||
func (c *Client) Permissions(ctx context.Context, aclPath string) (map[string]int, error) {
|
||||
var raw map[string]map[string]int
|
||||
if err := c.get(ctx, "/access/permissions?path="+url.QueryEscape(aclPath), &raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if p, ok := raw[aclPath]; ok {
|
||||
return p, nil
|
||||
}
|
||||
return map[string]int{}, nil
|
||||
}
|
||||
|
||||
// GuestStatus returns GET /nodes/{node}/lxc/{vmid}/status/current. The API body
|
||||
// has no vmid field (it is in the path), so it is set from the argument.
|
||||
func (c *Client) GuestStatus(ctx context.Context, vmid int) (Guest, error) {
|
||||
|
||||
@@ -53,7 +53,11 @@ type claimFacts struct {
|
||||
nodes []claimNode // the whole disk + its children (partitions)
|
||||
lvmPV bool // pvs (authoritative): the disk / a partition is an LVM physical volume
|
||||
zfsMember bool // zpool (authoritative): the disk / a partition is a ZFS pool member
|
||||
gatherErr string // non-empty ⇒ a REQUIRED read failed ⇒ fail-safe CLAIMED
|
||||
// felhomOwnedMounts (R-220) — mountpoints OUTSIDE /mnt/felhom-drives that are nevertheless Felhom's
|
||||
// OWN, corroborated from the host mount table: the same device is also mounted at the managed path.
|
||||
// Empty means "nothing corroborated", which is the fail-safe direction.
|
||||
felhomOwnedMounts map[string]bool
|
||||
gatherErr string // non-empty ⇒ a REQUIRED read failed ⇒ fail-safe CLAIMED
|
||||
}
|
||||
|
||||
// classifyClaim is the pure guard verdict. unclaimed=true ONLY when the device is provably free for
|
||||
@@ -81,7 +85,20 @@ func classifyClaim(f claimFacts) (unclaimed bool, reason string) {
|
||||
if memberFSTypes[n.fstype] {
|
||||
return false, "device holds a " + n.fstype + " (" + n.name + ")"
|
||||
}
|
||||
if n.mountpoint != "" && !underFelhomDrives(n.mountpoint) {
|
||||
// ── R-220 — A MOUNT FELHOM ITSELF MADE IS NOT "SOMETHING ELSE". ───────────────────────
|
||||
//
|
||||
// Enrolment mounts a drive TWICE: at the managed path `/mnt/felhom-drives/<name>` and at the
|
||||
// raw `/mnt/<name>` it creates on the host. The host — and therefore that raw mount — survives
|
||||
// a guest rebuild, while the controller's registry does not. So after a rebuild the customer's
|
||||
// own drives looked foreign, `attach` returned an empty list, and the refusal told them to
|
||||
// choose from it. Measured live three times (CAMPAIGN-11 Phase 1, and the R-201 re-walk twice);
|
||||
// unmounting only the raw mounts flipped `attach: []` to both drives every time.
|
||||
//
|
||||
// The fence this must NOT breach: a disk genuinely in use by something else stays refused. So
|
||||
// the exemption is not "any /mnt/* path" — it is CORROBORATED: the same device must ALSO be
|
||||
// mounted at Felhom's managed path, which is a state only Felhom's own enrolment produces.
|
||||
// A foreign disk at /srv/data or /media/x has no such counterpart and is still refused.
|
||||
if n.mountpoint != "" && !underFelhomDrives(n.mountpoint) && !f.felhomOwnedMounts[n.mountpoint] {
|
||||
return false, "device is mounted at " + n.mountpoint + " (" + n.name + ")"
|
||||
}
|
||||
}
|
||||
@@ -154,6 +171,8 @@ func (h *SudoHostOps) gatherClaimFacts(ctx context.Context, device string) claim
|
||||
return f
|
||||
}
|
||||
f.nodes = nodes
|
||||
// R-220: corroborate which non-managed mountpoints are nevertheless Felhom's own.
|
||||
f.felhomOwnedMounts = felhomOwnedMounts(device, nodes, h.mountTable)
|
||||
|
||||
// LVM PV (authoritative). pvs installed but erroring ⇒ fail-safe claimed; absent ⇒ rely on lsblk's
|
||||
// LVM2_member FSTYPE (already in nodes).
|
||||
@@ -285,3 +304,71 @@ func (h *SudoHostOps) zfsMembers(ctx context.Context, nodes []claimNode, wholeDi
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// mountTableSource yields the host mount table as (device, mountpoint) pairs. A seam so the R-220
|
||||
// corroboration is unit-testable without a host. nil ⇒ the real /proc/mounts.
|
||||
type mountTableSource func() ([][2]string, error)
|
||||
|
||||
// procMounts reads /proc/mounts — WORLD-READABLE, so this needs no sudo and no allowlisted command.
|
||||
// That matters: the lsblk invocation is pinned verbatim in the sudoers file
|
||||
// (`lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/*`), so switching it to the plural MOUNTPOINTS
|
||||
// would have meant shipping a sudoers change with the binary — a far larger blast radius than this
|
||||
// finding warrants. Reading the mount table directly sidesteps that entirely.
|
||||
func procMounts() ([][2]string, error) {
|
||||
data, err := os.ReadFile("/proc/mounts")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out [][2]string
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
// /proc/mounts escapes spaces as \040; unescape so a path with a space still compares.
|
||||
out = append(out, [2]string{fields[0], strings.ReplaceAll(fields[1], `\040`, " ")})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// felhomOwnedMounts returns the mountpoints of `device` (and its children) that sit OUTSIDE
|
||||
// /mnt/felhom-drives but are still Felhom's own, corroborated by the same device also being mounted
|
||||
// UNDER /mnt/felhom-drives. That pairing is what enrolment produces and nothing else does.
|
||||
//
|
||||
// ⚠ FAIL-SAFE: an unreadable mount table returns an EMPTY set, never a permissive one. The device then
|
||||
// classifies exactly as it did before R-220 — refused — because "we could not corroborate" must never
|
||||
// read as "it is ours".
|
||||
func felhomOwnedMounts(device string, nodes []claimNode, src mountTableSource) map[string]bool {
|
||||
if src == nil {
|
||||
src = procMounts
|
||||
}
|
||||
table, err := src()
|
||||
if err != nil {
|
||||
return nil // unreadable ⇒ corroborate nothing
|
||||
}
|
||||
// Every device name this disk answers to: the whole disk and each child node.
|
||||
devs := map[string]bool{device: true}
|
||||
if wd, ok := wholeDiskOf(device); ok {
|
||||
devs[wd] = true
|
||||
}
|
||||
for _, n := range nodes {
|
||||
devs["/dev/"+n.name] = true
|
||||
}
|
||||
// A device is Felhom-managed only if it is mounted under the managed prefix.
|
||||
managed := map[string]bool{}
|
||||
for _, row := range table {
|
||||
if devs[row[0]] && underFelhomDrives(row[1]) {
|
||||
managed[row[0]] = true
|
||||
}
|
||||
}
|
||||
if len(managed) == 0 {
|
||||
return nil
|
||||
}
|
||||
owned := map[string]bool{}
|
||||
for _, row := range table {
|
||||
if managed[row[0]] && !underFelhomDrives(row[1]) {
|
||||
owned[path.Clean(row[1])] = true
|
||||
}
|
||||
}
|
||||
return owned
|
||||
}
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
package storage
|
||||
|
||||
import "testing"
|
||||
|
||||
// ── R-220 — A MOUNT FELHOM ITSELF MADE IS NOT "SOMETHING ELSE" ──────────────────────────────────
|
||||
//
|
||||
// Enrolment mounts a drive twice: at `/mnt/felhom-drives/<name>` and at the raw `/mnt/<name>` it
|
||||
// creates on the host. The host survives a guest rebuild; the controller's registry does not. So after
|
||||
// a rebuild the customer's own drives read as claimed-by-something-else, `attach` came back empty, and
|
||||
// the refusal told them to pick from the empty list. Measured three times live.
|
||||
//
|
||||
// The fence: a disk genuinely in use elsewhere must STILL be refused. These assert both directions.
|
||||
|
||||
// ── SCENARIO E — the customer's own drive is offered again after a rebuild ───────────────────────
|
||||
//
|
||||
// RED-PROOF: drop `&& !f.felhomOwnedMounts[n.mountpoint]` from classifyClaim — the pre-R-220 check —
|
||||
// and this FAILS with the drive refused and the list empty again.
|
||||
func TestClassifyClaim_R220_FelhomsOwnRawMountIsNotForeign(t *testing.T) {
|
||||
f := claimFacts{
|
||||
device: "/dev/sdb", wholeDisk: "/dev/sdb", wholeDiskOK: true,
|
||||
nodes: []claimNode{{name: "sdb", fstype: "ext4", mountpoint: "/mnt/adatok"}},
|
||||
// corroborated: the SAME device is also mounted at the managed path
|
||||
felhomOwnedMounts: map[string]bool{"/mnt/adatok": true},
|
||||
}
|
||||
unclaimed, reason := classifyClaim(f)
|
||||
if !unclaimed {
|
||||
t.Fatalf("R-220 RETURNED: the customer's own drive is refused after a rebuild — %q", reason)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — a genuinely foreign mount is STILL refused ──────────────────────────────────────
|
||||
//
|
||||
// RED-PROOF: over-widen the fix to exempt any /mnt/* path (or to skip the mountpoint check entirely)
|
||||
// and this FAILS — a disk another system is using would be offered for formatting.
|
||||
func TestClassifyClaim_R220_ForeignMountIsStillRefused(t *testing.T) {
|
||||
for _, mp := range []string{"/srv/data", "/media/photos", "/mnt/someone-elses-disk", "/var/lib/other"} {
|
||||
f := claimFacts{
|
||||
device: "/dev/sdb", wholeDisk: "/dev/sdb", wholeDiskOK: true,
|
||||
nodes: []claimNode{{name: "sdb", fstype: "ext4", mountpoint: mp}},
|
||||
felhomOwnedMounts: nil, // nothing corroborated it as ours
|
||||
}
|
||||
unclaimed, reason := classifyClaim(f)
|
||||
if unclaimed {
|
||||
t.Fatalf("THE FENCE BROKE: a disk mounted at %s was offered for formatting", mp)
|
||||
}
|
||||
if reason == "" {
|
||||
t.Fatalf("a refusal must carry a reason (%s)", mp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The corroboration itself: it must require BOTH mounts of the SAME device, and fail safe.
|
||||
func TestFelhomOwnedMounts_RequiresTheManagedCounterpart(t *testing.T) {
|
||||
nodes := []claimNode{{name: "sdb"}}
|
||||
|
||||
t.Run("both mounts present -> the raw one is ours", func(t *testing.T) {
|
||||
src := func() ([][2]string, error) {
|
||||
return [][2]string{
|
||||
{"/dev/sdb", "/mnt/adatok"},
|
||||
{"/dev/sdb", "/mnt/felhom-drives/adatok"},
|
||||
}, nil
|
||||
}
|
||||
got := felhomOwnedMounts("/dev/sdb", nodes, src)
|
||||
if !got["/mnt/adatok"] {
|
||||
t.Fatal("the raw enrolment mount was not recognised as Felhom's own")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("only the raw mount -> corroborates NOTHING", func(t *testing.T) {
|
||||
src := func() ([][2]string, error) {
|
||||
return [][2]string{{"/dev/sdb", "/mnt/adatok"}}, nil
|
||||
}
|
||||
if got := felhomOwnedMounts("/dev/sdb", nodes, src); len(got) != 0 {
|
||||
t.Fatalf("a lone /mnt/<name> mount must corroborate nothing, got %v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a DIFFERENT device under the managed path does not vouch for this one", func(t *testing.T) {
|
||||
src := func() ([][2]string, error) {
|
||||
return [][2]string{
|
||||
{"/dev/sdb", "/srv/data"},
|
||||
{"/dev/sdc", "/mnt/felhom-drives/mentes"}, // someone else's, not sdb's
|
||||
}, nil
|
||||
}
|
||||
if got := felhomOwnedMounts("/dev/sdb", nodes, src); got["/srv/data"] {
|
||||
t.Fatal("another device's managed mount vouched for a foreign one")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("an unreadable mount table corroborates NOTHING (fail-safe)", func(t *testing.T) {
|
||||
src := func() ([][2]string, error) { return nil, errRead }
|
||||
if got := felhomOwnedMounts("/dev/sdb", nodes, src); len(got) != 0 {
|
||||
t.Fatalf("an unreadable mount table must corroborate nothing, got %v", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
var errRead = errNoMountTable{}
|
||||
|
||||
type errNoMountTable struct{}
|
||||
|
||||
func (errNoMountTable) Error() string { return "mount table unreadable" }
|
||||
@@ -164,6 +164,9 @@ type SudoHostOps struct {
|
||||
// UNPRIVILEGED read (`systemctl is-failed`) — seam-injected so the reassert's F10 reset-failed path
|
||||
// is unit-testable without a real systemd. Default set in NewSudoHostOps.
|
||||
unitFailed func(ctx context.Context, unit string) bool
|
||||
// mountTable (R-220) yields the host mount table for the "is this mount Felhom's own?"
|
||||
// corroboration. nil ⇒ the real /proc/mounts; tests inject.
|
||||
mountTable mountTableSource
|
||||
}
|
||||
|
||||
// SudoHostOpsConfig configures a SudoHostOps.
|
||||
|
||||
@@ -9,6 +9,12 @@
|
||||
Gates (all must pass; **non-zero exit on any failure**):
|
||||
|
||||
1. reuse-refs every path cited by this repo's REUSE.md still resolves
|
||||
2. published every `v<semver>` tag has a downloadable package AND a tag tree that serves
|
||||
the agent's configs (R-115). NEEDS NETWORK, so it is **not** in `--fast` and
|
||||
the pre-push hook does not run it — a push must not fail because Gitea blinked
|
||||
or because someone is offline on a train. CI runs the FULL set for exactly this
|
||||
reason: it is the machine that can afford a network check, and it is the half
|
||||
that emails when something is wrong.
|
||||
|
||||
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
|
||||
|
||||
@@ -35,10 +41,14 @@ import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reuse_refs_check.py")
|
||||
SHARED_INSTRUCTIONS = os.path.join(
|
||||
os.path.dirname(ROOT), "felhom.eu", "scripts", "instructions_gate.py")
|
||||
|
||||
# (label, absolute script path, args, fast)
|
||||
GATES = [
|
||||
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
||||
("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
|
||||
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""check-published-versions.py — a released agent version must be INSTALLABLE, not merely built.
|
||||
|
||||
R-115. A box installs the agent from a Gitea generic package the hub vouches, never from git, and
|
||||
since R-110/R-183 it also fetches the agent's sixteen config files from `raw/tag/v<version>/`.
|
||||
Nothing in the build, deploy or session-end path checked that either existed, so "deployed" and
|
||||
"installable" were independent states that drifted silently. **Three instances in five days:**
|
||||
|
||||
* R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published — a new customer
|
||||
would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
* 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
* 0.120.0 (2026-08-03) built, committed, deployed to BOTH demo hosts, never published. A
|
||||
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
agent — and would have *succeeded* while doing it.
|
||||
|
||||
THE INVARIANTS — there are TWO now, and the second is R-188's price.
|
||||
|
||||
(1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
|
||||
and the tag must serve the agent's configs.
|
||||
|
||||
(2) No PUBLISHED version may be missing its tag.
|
||||
|
||||
Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
|
||||
AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
|
||||
package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
|
||||
needs a token this gate does not have. That reasoning was sound and the ordering was still wrong —
|
||||
the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
|
||||
seconds before its own package existed and mailing the operator a failure for a release that worked
|
||||
(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
|
||||
|
||||
Moving the push does not get to trade invariant (2) away, so it is asserted here instead — WITHOUT a
|
||||
token, and therefore as a BOUNDED PROBE rather than an enumeration:
|
||||
|
||||
* the FRONTIER — the versions immediately above the highest tag. This is the realistic failure the
|
||||
new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
|
||||
version beyond the newest tag.
|
||||
* the GAPS — patch versions that fall between two existing tags and have no tag of their own.
|
||||
|
||||
Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
|
||||
token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
|
||||
coverage is invisible reads as a guarantee it is not making.
|
||||
|
||||
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
|
||||
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
|
||||
and that was measured rather than assumed:** the hub's artifact manifest
|
||||
(`GET /api/v1/artifacts/<customer>`) answers **401** without a per-customer retrieval passphrase,
|
||||
and the Gitea package LISTING api answers **401** without a token, while the package DOWNLOAD url
|
||||
and the git TAGS api are both anonymous. So a credential-free gate can ask *"is this version
|
||||
installable"* but not *"which version is vouched"*. Putting an operator credential into CI to close
|
||||
that gap is the operator's call, not a gate author's — it is recorded as a limitation below and as a
|
||||
backlog row rather than quietly assumed away.
|
||||
|
||||
**What this invariant does catch: all three instances above.** `release-agent.sh` creates the tag and
|
||||
publishes in one act, so a release whose publish was skipped, failed, or was forgotten leaves a tag
|
||||
with no package — which is exactly what this refuses. It needs no version floor: tags begin at
|
||||
v0.120.0, which is published.
|
||||
|
||||
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
|
||||
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
|
||||
Nor does the converse probe prove that NO untagged package exists — only that none exists at the
|
||||
probed versions, which are printed. Closing that properly needs a read token in CI (→ R-184).
|
||||
|
||||
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
|
||||
INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
|
||||
standing rule this project earned twice, and a gate that green-lights on its own blindness is worse
|
||||
than no gate, because it looks like coverage.
|
||||
|
||||
Pure python3 + urllib, NO curl and no third-party module: the CI runner is a host-mode container
|
||||
carrying python3 and git and nothing else, and an earlier workflow step died on
|
||||
`curl: command not found`.
|
||||
|
||||
python3 scripts/check-published-versions.py
|
||||
|
||||
Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
|
||||
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
|
||||
OWNER = "admin"
|
||||
REPO = "felhom-agent"
|
||||
PKG = "felhom-agent"
|
||||
TIMEOUT = 25
|
||||
|
||||
# One config the installer fetches. Its presence proves the TAG's tree carries the configs the
|
||||
# sixteen `fetch_raw` calls will ask for — a tag that exists but predates them would 404 a box
|
||||
# mid-install, on a virgin machine, as root.
|
||||
PROBE_CONFIG = "configs/felhom-agent.service"
|
||||
|
||||
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
def _get(url, want_body=False):
|
||||
"""GET a URL. Returns (status, body_or_None). Network failure raises."""
|
||||
tried.append(url)
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
body = r.read() if want_body else None
|
||||
return r.status, body
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, None
|
||||
|
||||
|
||||
def inconclusive(msg):
|
||||
print("INCONCLUSIVE:", msg)
|
||||
print(" URLs tried (a 'no access' claim must name its attempts):")
|
||||
for u in tried:
|
||||
print(" ", u)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def _pkg_exists(version):
|
||||
"""True iff the generic package for `version` is downloadable anonymously."""
|
||||
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
|
||||
status, _ = _get(url)
|
||||
return status == 200, url
|
||||
|
||||
|
||||
def untagged_probe_set(versions):
|
||||
"""The versions to probe for invariant (2), as (version, why) pairs.
|
||||
|
||||
Bounded on purpose and printed by the caller: the package listing api needs a token (401,
|
||||
re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
|
||||
places an orphan would actually land.
|
||||
|
||||
FRONTIER — a publish that succeeded followed by a tag push that failed leaves the orphan
|
||||
exactly one version past the newest tag. This is the failure mode the R-188
|
||||
reordering makes possible, so it is the one that must not be guesswork.
|
||||
GAPS — a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
|
||||
tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
|
||||
"""
|
||||
parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
|
||||
have = set(parsed)
|
||||
out = []
|
||||
if not parsed:
|
||||
return out
|
||||
|
||||
hi = parsed[-1]
|
||||
for cand, why in (
|
||||
((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
|
||||
((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
|
||||
((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
|
||||
((hi[0] + 1, 0, 0), "next major after the newest tag"),
|
||||
):
|
||||
if cand not in have:
|
||||
out.append(("%d.%d.%d" % cand, why))
|
||||
|
||||
MAX_GAP_PROBES = 12
|
||||
for a, b in zip(parsed, parsed[1:]):
|
||||
if a[0] != b[0] or a[1] != b[1]:
|
||||
continue # a minor/major step is not a patch gap
|
||||
for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
|
||||
cand = (a[0], a[1], patch)
|
||||
if cand not in have:
|
||||
out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
print("check-published-versions — every released agent version must be INSTALLABLE")
|
||||
print(" gitea:", GITEA_BASE)
|
||||
|
||||
tags_url = "%s/api/v1/repos/%s/%s/tags?limit=200" % (GITEA_BASE, OWNER, REPO)
|
||||
try:
|
||||
status, body = _get(tags_url, want_body=True)
|
||||
except Exception as e:
|
||||
inconclusive("cannot reach Gitea to list tags: %s" % e)
|
||||
if status != 200 or not body:
|
||||
inconclusive("tags api returned HTTP %s — cannot enumerate releases" % status)
|
||||
try:
|
||||
tags = [t["name"] for t in json.loads(body.decode("utf-8"))]
|
||||
except Exception as e:
|
||||
inconclusive("tags api response is not the expected JSON: %s" % e)
|
||||
|
||||
versions = sorted(m.group(1) for m in (TAG_RE.match(t) for t in tags) if m)
|
||||
if not versions:
|
||||
# Not a failure: a repo legitimately has no version tags before its first release. Say so
|
||||
# loudly rather than reporting a vacuous pass — an empty check that prints OK is how an
|
||||
# unexamined thing becomes a documented-clean one.
|
||||
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
||||
print("\ncheck-published-versions: NOTHING TO CHECK")
|
||||
return 0
|
||||
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
|
||||
|
||||
bad = []
|
||||
for v in versions:
|
||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||
try:
|
||||
pkg_status, _ = _get(pkg_url)
|
||||
raw_status, _ = _get(raw_url)
|
||||
except Exception as e:
|
||||
inconclusive("network failure while checking v%s: %s" % (v, e))
|
||||
|
||||
problems = []
|
||||
if pkg_status != 200:
|
||||
problems.append("binary NOT downloadable (HTTP %s at %s)" % (pkg_status, pkg_url))
|
||||
if raw_status != 200:
|
||||
problems.append("tag does not serve %s (HTTP %s) — a box would 404 mid-install"
|
||||
% (PROBE_CONFIG, raw_status))
|
||||
if problems:
|
||||
bad.append((v, problems))
|
||||
print(" FAIL v%s:" % v)
|
||||
for p in problems:
|
||||
print(" -", p)
|
||||
else:
|
||||
print(" ok v%s: binary downloadable + tag serves its configs" % v)
|
||||
|
||||
# ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
|
||||
probes = untagged_probe_set(versions)
|
||||
orphans = []
|
||||
print()
|
||||
print(" converse probe — a published version with no tag (bounded; the package listing api")
|
||||
print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
|
||||
for v, why in probes:
|
||||
try:
|
||||
exists, url = _pkg_exists(v)
|
||||
except Exception as e:
|
||||
inconclusive("network failure while probing v%s: %s" % (v, e))
|
||||
mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
|
||||
print(" %-10s %-42s %s" % (v, why, mark))
|
||||
if exists:
|
||||
orphans.append((v, url))
|
||||
|
||||
print()
|
||||
if bad or orphans:
|
||||
if orphans:
|
||||
print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
|
||||
for v, url in orphans:
|
||||
print(" v%s is downloadable at %s but has no git tag." % (v, url))
|
||||
print(" A release publishes and then pushes its tag; a package with no tag means the")
|
||||
print(" push failed or was skipped. The local tag is probably still in the release")
|
||||
print(" clone — finish it with:")
|
||||
for v, _ in orphans:
|
||||
print(" git push origin v%s" % v)
|
||||
print(" (and if the tag is gone, re-create it on the released commit before pushing.)")
|
||||
if bad:
|
||||
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
|
||||
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
|
||||
print(" the R-115 defect, three times in five days. Publish it with:")
|
||||
print(" scripts/release-agent.sh <version>")
|
||||
if bad or orphans:
|
||||
return 1
|
||||
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Regular → Executable
+5
-1
@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
|
||||
BIN="$(mktemp -t felhom-agent.XXXXXX)"
|
||||
CLEANUP_BIN="$BIN"
|
||||
log "building felhom-agent $VERSION from $REPO_ROOT …"
|
||||
( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
||||
# These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
|
||||
# They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
|
||||
# the one the release path built for the same version. One version name must mean one binary
|
||||
# whichever entry point produced it.
|
||||
( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
||||
fi
|
||||
[[ -f "$BIN" ]] || die "binary not found: $BIN"
|
||||
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
|
||||
|
||||
Executable
+212
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env bash
|
||||
# release-agent.sh — THE way to release a felhom-agent version. One act: build → tag → publish →
|
||||
# verify by independent download.
|
||||
#
|
||||
# WHY THIS EXISTS (R-115). Publishing used to be a step someone had to remember, and it was
|
||||
# forgotten THREE TIMES IN FIVE DAYS:
|
||||
#
|
||||
# * R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published, so a new customer
|
||||
# would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
# * 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
# * 0.120.0 (2026-08-03) built, committed and deployed to BOTH demo hosts, never published. A
|
||||
# documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
# agent — and would have *succeeded* while doing it, because the current `step_grows`
|
||||
# sets SYSDATA_GROW=0 so the older agent's fatal mp1 resize never fires.
|
||||
#
|
||||
# R-111's own closing line said publishing should join the release train rather than stay a
|
||||
# remembered step. It closed SHIPPED without that leg, and the leg recurred the same afternoon —
|
||||
# which is the evidence that a note is not a mechanism. This file is the mechanism. The
|
||||
# documentation now points here instead of at a raw `go build` line, so there is ONE documented way
|
||||
# to release and it cannot complete without publishing.
|
||||
#
|
||||
# WHY IT TAGS (R-183). Since felhom-host-install.sh pins its sixteen agent-config fetches to
|
||||
# `raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root, on a
|
||||
# virgin machine. The tag and the package are two halves of one release and are created together.
|
||||
#
|
||||
# WHY IT DOES NOT VOUCH. Vouching is what points machines at a version, and it stays the operator's
|
||||
# deliberate act — the same prove-then-vouch principle that governed the golden two sessions ago.
|
||||
# This script prints the version and sha to vouch; a human decides when.
|
||||
#
|
||||
# Usage:
|
||||
# GITEA_USER=admin GITEA_TOKEN=<token> ./scripts/release-agent.sh <version>
|
||||
#
|
||||
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
||||
# GITEA_BASE / GITEA_OWNER override the defaults.
|
||||
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
||||
set -euo pipefail
|
||||
|
||||
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
||||
GITEA_OWNER="${GITEA_OWNER:-admin}"
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
die() { echo "[release-agent] ERROR: $*" >&2; exit 1; }
|
||||
log() { echo "[release-agent] $*" >&2; }
|
||||
|
||||
VERSION="${1:-}"
|
||||
[[ -n "$VERSION" ]] || die "version required (usage: GITEA_USER=.. GITEA_TOKEN=.. $0 <version>)"
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "version must be bare semver X.Y.Z (got '$VERSION')"
|
||||
TAG="v$VERSION"
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# ── 1. Clean-tree gate ──────────────────────────────────────────────────────────────────────────
|
||||
# An unpushed change does not exist. Releasing a dirty tree publishes a binary whose source nobody
|
||||
# else can obtain, and tags a commit that does not contain what was built.
|
||||
if [[ "${RELEASE_ALLOW_DIRTY:-0}" != "1" ]]; then
|
||||
[[ -z "$(git status --porcelain)" ]] || die "working tree is dirty — commit and push first"
|
||||
local_head="$(git rev-parse HEAD)"
|
||||
git fetch -q origin main
|
||||
[[ "$local_head" == "$(git rev-parse origin/main)" ]] \
|
||||
|| die "HEAD != origin/main — push first (an unpushed change does not exist)"
|
||||
fi
|
||||
|
||||
# ── 2. Refuse to re-release a version that already exists ───────────────────────────────────────
|
||||
# Silently overwriting a published artifact is how "the same version" comes to mean two different
|
||||
# binaries on two different boxes.
|
||||
if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
|
||||
die "tag $TAG already exists — releasing over it would make one version name two binaries"
|
||||
fi
|
||||
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
|
||||
|
||||
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
|
||||
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
||||
trap 'rm -f "$BIN"' EXIT
|
||||
log "building $VERSION …"
|
||||
# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
|
||||
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
|
||||
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
|
||||
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
|
||||
#
|
||||
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
||||
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
|
||||
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
||||
#
|
||||
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
|
||||
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
|
||||
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
|
||||
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
|
||||
#
|
||||
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
|
||||
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
|
||||
# version — one version name, two binaries, by whichever entry point was used. Both now build the
|
||||
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
|
||||
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
||||
|| die "go build failed"
|
||||
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
||||
[[ "$built_ver" == "$VERSION" ]] \
|
||||
|| die "the built binary reports '$built_ver', not '$VERSION' — the ldflag did not take"
|
||||
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
||||
log "built ok: sha256 $BUILT_SHA"
|
||||
|
||||
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
|
||||
#
|
||||
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
|
||||
#
|
||||
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
|
||||
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
|
||||
# because the Gitea package LISTING api needs a token the gate does not have.
|
||||
#
|
||||
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
|
||||
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
|
||||
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
|
||||
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
|
||||
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
|
||||
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
|
||||
#
|
||||
# So the tag is still created HERE, before anything is published — the build and the tag still
|
||||
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
|
||||
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
|
||||
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
|
||||
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
|
||||
# now checked rather than one being arranged for.
|
||||
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
||||
git tag -a "$TAG" -m "agent $TAG
|
||||
|
||||
Released by scripts/release-agent.sh.
|
||||
sha256 of the published binary: $BUILT_SHA
|
||||
|
||||
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
||||
so this tag is part of the released artifact, not a bookmark (R-183)."
|
||||
|
||||
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
||||
log "publishing …"
|
||||
# Invoked through `bash` DELIBERATELY, not as an executable. On 2026-08-03 the first real release
|
||||
# through this script died here — `publish-agent.sh` has been mode 0644 since it was created on
|
||||
# 2026-06-28, because every earlier caller ran it as `bash scripts/publish-agent.sh`. So the one leg
|
||||
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
|
||||
# the same commit; this line makes the release independent of it, because a file mode is exactly the
|
||||
# kind of thing that is lost again by a checkout, an archive, or a copy.
|
||||
if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
|
||||
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
|
||||
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
|
||||
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
|
||||
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
|
||||
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
if [[ "$now_published" == "200" ]]; then
|
||||
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
|
||||
else
|
||||
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
|
||||
fi
|
||||
die "publish failed"
|
||||
fi
|
||||
|
||||
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
|
||||
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
|
||||
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
|
||||
#
|
||||
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
|
||||
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
|
||||
# The recovery is one line and it is printed rather than described.
|
||||
log "pushing $TAG …"
|
||||
if ! git push origin "$TAG"; then
|
||||
cat >&2 <<EOF
|
||||
|
||||
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
|
||||
|
||||
version : $VERSION
|
||||
sha256 : $BUILT_SHA
|
||||
|
||||
The tag exists in this clone only. Nothing installs from an untagged version (the installer
|
||||
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
|
||||
FAIL on it as a published version with no tag. Finish the release with:
|
||||
|
||||
git push origin $TAG
|
||||
|
||||
EOF
|
||||
die "tag push failed after a successful publish — see above"
|
||||
fi
|
||||
|
||||
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
||||
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
||||
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
||||
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
||||
# returns the right sha are different claims.
|
||||
log "verifying by independent download …"
|
||||
DL="$(mktemp -t felhom-agent-dl-XXXXXX)"
|
||||
trap 'rm -f "$BIN" "$DL"' EXIT
|
||||
curl -fsS -o "$DL" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" \
|
||||
|| die "round-trip GET failed — the version is NOT installable"
|
||||
DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
|
||||
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|
||||
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
|
||||
|
||||
# The tag must also serve the configs the installer will fetch from it.
|
||||
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
|
||||
[[ "$cfg_code" == "200" ]] \
|
||||
|| die "tag $TAG does not serve configs/felhom-agent.service (HTTP $cfg_code) — a box would 404 mid-install"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
|
||||
version : $VERSION
|
||||
tag : $TAG
|
||||
sha256 : $BUILT_SHA
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
EOF
|
||||
Reference in New Issue
Block a user