Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6d7904786c | |||
| 856a127cd6 | |||
| 257c4d85c0 | |||
| 72161f6cf0 | |||
| 03b58cec0a | |||
| fe14bc62c0 | |||
| 0b28eae7bb | |||
| 7581f8140a | |||
| 3d0a1d615d | |||
| 77e2cc4583 | |||
| cd1b087db7 | |||
| 53d0c6bfc4 |
+260
@@ -1,3 +1,263 @@
|
|||||||
|
## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 7–8)
|
||||||
|
|
||||||
|
**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed
|
||||||
|
since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the
|
||||||
|
recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest
|
||||||
|
writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token` —
|
||||||
|
an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||||
|
offsite repository password into the same bundle. Anyone reading that output would conclude the
|
||||||
|
repository password was not there. It now names what THIS bundle actually carried and what it did not.
|
||||||
|
|
||||||
|
**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the
|
||||||
|
customer's recovery code, the agent fetches this host's own sealed blob from the hub
|
||||||
|
(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and
|
||||||
|
returns **only the offsite restic repository password** plus its sha256.
|
||||||
|
|
||||||
|
**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG
|
||||||
|
private key. The controller is a trust tier down and needs none of them; returning them would widen
|
||||||
|
the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not
|
||||||
|
recoverable later.
|
||||||
|
|
||||||
|
**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately
|
||||||
|
absent from the controller image; the blob is a host-scoped object whose only writer is this agent
|
||||||
|
under the per-host key, so the read is that write's mirror.
|
||||||
|
|
||||||
|
**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned
|
||||||
|
channel, is held in memory for one call, is cleared on the success path AND every failure path, is
|
||||||
|
never written to disk, never an argument in a process list, never logged at any level including
|
||||||
|
inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered
|
||||||
|
plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty
|
||||||
|
afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later
|
||||||
|
call overwriting the leaked file, which is exactly how the first version of that test passed its own
|
||||||
|
red-proof while R sat on disk.
|
||||||
|
|
||||||
|
Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a
|
||||||
|
bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and
|
||||||
|
a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to
|
||||||
|
re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids.
|
||||||
|
|
||||||
|
**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main` → `runDaemon` →
|
||||||
|
`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as
|
||||||
|
`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping
|
||||||
|
that makes cross-host retrieval impossible is a property of which key is used). This project's
|
||||||
|
built-but-never-wired count is six and links 6–7 were two of them; the fix must not become the seventh.
|
||||||
|
|
||||||
|
## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190)
|
||||||
|
|
||||||
|
**v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The
|
||||||
|
capability reported degraded for "one cycle" — meaning the probe call that performed the repair. But
|
||||||
|
`probeAll` is invoked **independently** by the periodic self-check log and by the collector building a
|
||||||
|
host-report. On the demo box the repairing call was the log's (`09:39:34`, journal shows
|
||||||
|
`GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED` and `degraded=1`), and the host-report built three
|
||||||
|
seconds later found the grant present and sent **`ok`**. The agent's journal had the record; the hub
|
||||||
|
had nothing; the operator would have learned nothing.
|
||||||
|
|
||||||
|
That is the exact silence R-190 is about, re-created inside its own mitigation — and every unit test
|
||||||
|
passed while it was true.
|
||||||
|
|
||||||
|
**The fix is a latch on TIME rather than on call count.** A confirmed repair is reported for
|
||||||
|
`storeGrantRepairReportWindow` (20 minutes), which comfortably exceeds the 900 s host-report interval,
|
||||||
|
so at least one report must carry the transition. It clears on its own — a permanently degraded
|
||||||
|
capability would be its own false alarm — and it is per tier.
|
||||||
|
|
||||||
|
**Two hollow tests were caught and fixed on the way**, both the same shape this repo keeps finding: a
|
||||||
|
test asserting a value it constructed itself, and a test asserting the latch HELPER rather than the
|
||||||
|
path that consumes it — whose red-proof duly passed. The decisions now live in
|
||||||
|
`storeGrantHealthyVerdict` and `storeGrantRepairedVerdict`, and the tests call those.
|
||||||
|
|
||||||
|
## v0.124.0 — a lost storage grant repairs itself, and says that it was lost (2026-08-04, R-190)
|
||||||
|
|
||||||
|
**R-190 is a grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24** — with a
|
||||||
|
host reinstall, logged `pveum` activity and cluster-log entries all ruled out by measurement. The
|
||||||
|
cause is still open. The resilience does not have to wait for it.
|
||||||
|
|
||||||
|
**Everything needed already existed and had only ever been called once.** The root wrapper
|
||||||
|
(`felhom-backup-target-apply grant <id>`), its sudoers vector (`grant *`, any storage id, confirmed
|
||||||
|
not assumed), and the exact command were all in place — and the `grant` verb had only ever run at
|
||||||
|
storage CREATION. That is the *built but never wired* shape, in a verb rather than a seam, and it is
|
||||||
|
this project's seventh instance.
|
||||||
|
|
||||||
|
**What v0.124.0 does:** when the store-grant probe finds the grant absent on a tier the box depends
|
||||||
|
on, it runs that wrapper and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including
|
||||||
|
its restraint: one attempt, one confirmation, and anything still wrong stays loudly wrong.
|
||||||
|
|
||||||
|
**THE RECORD IS THE POINT, AND IT IS THE HALF R-190 IS ACTUALLY ABOUT.** A repair that leaves only
|
||||||
|
`ok` behind destroys the only evidence a permission vanished, so a recurring loss becomes undetectable
|
||||||
|
forever — strictly worse than the fault it fixes. So a confirmed repair reports **DEGRADED for exactly
|
||||||
|
one cycle**, with the explanation in `Feature`:
|
||||||
|
|
||||||
|
```
|
||||||
|
backup tier felhom-backup: the agent's storage grant was MISSING and has been AUTOMATICALLY
|
||||||
|
RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Nothing new was built to carry it.** The hub's existing ok→degraded→ok edge is the channel — it
|
||||||
|
alerts and e-mails on the first edge and logs the recovery on the next cycle, so one loss produces
|
||||||
|
exactly one alert pair. No wire change, no hub change, no new event type. `Feature` carries the text
|
||||||
|
because that is the field the hub interpolates into the operator's e-mail; `Reason` does not travel.
|
||||||
|
|
||||||
|
**Bounded (Scenario F):** one attempt per tier per hour, in memory. A storage can be unreadable for
|
||||||
|
reasons an ACL cannot fix, and a re-grant on every report cycle is a repair loop wearing a fix's
|
||||||
|
clothes. An agent restart re-arms it, which is correct — a restart is exactly when a box should
|
||||||
|
re-check what it depends on.
|
||||||
|
|
||||||
|
**A failed repair never masks the fault:** the capability stays degraded with the failure in its
|
||||||
|
reason, and a repair that "succeeded" but did not survive the re-read is reported as needing a human.
|
||||||
|
|
||||||
|
## v0.123.0 — a tier the box cannot READ now says so (2026-08-03, R-185)
|
||||||
|
|
||||||
|
**The missing permission is one command. The silence was the defect.** On demo-felhom the agent's PVE
|
||||||
|
token held `FelhomAgentStore` on `local`, `local-lvm` and `felhom-pbs` — and **not** on
|
||||||
|
`felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked for
|
||||||
|
that storage's content the API answers `{"data":[]}` while root sees three archives (6.1–6.3 GB,
|
||||||
|
08-01/02/03).
|
||||||
|
|
||||||
|
**An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return.** `pickForThisRun` skips
|
||||||
|
an empty tier — correctly, because a fresh offsite tier legitimately has nothing yet — and reports
|
||||||
|
*"no settled archive yet"*. So that tier was never restore-testable on that box and nothing ever
|
||||||
|
mentioned it. This project's own rule, in a new place: an empty answer is not evidence that there is
|
||||||
|
nothing there.
|
||||||
|
|
||||||
|
**The permission question, unlike the listing, has a definite answer — so it is asked directly.**
|
||||||
|
`Client.Permissions` reads `GET /access/permissions?path=/storage/<target>` **as the agent's own
|
||||||
|
token** (asking as root answers a different question and always says yes), and one
|
||||||
|
`capability.Status` per configured tier reports the result. It composes *around* the sudo prober, the
|
||||||
|
way the pool-read check already does — an API read does not belong inside a sudo-policy probe.
|
||||||
|
|
||||||
|
**MEASURED FIRST, and the obvious reading is wrong.** The ungranted path does not answer empty and
|
||||||
|
does not 403:
|
||||||
|
|
||||||
|
```
|
||||||
|
/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
|
||||||
|
/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
|
||||||
|
```
|
||||||
|
|
||||||
|
It answers with the privileges **inherited** from the box-wide `/` grant. A probe asking *"is the
|
||||||
|
path present?"* or *"does it have `Datastore.Audit`?"* would report the blinded storage healthy — so
|
||||||
|
the probe tests for `Datastore.AllocateSpace` specifically, and a red-proof pins that.
|
||||||
|
|
||||||
|
**Decisions, each weighed once:**
|
||||||
|
|
||||||
|
- **The probed set comes from the box's own config** (`BackupTiers()`), not a fixed list. A hardcoded
|
||||||
|
probe list is exactly the defect being fixed, reproduced inside the fix.
|
||||||
|
- **CRITICAL**, because the hub alerts only on critical and a non-critical entry would ride the
|
||||||
|
report and alert nobody — the same silence with extra steps. **Except** the `local` fallback
|
||||||
|
target, which host-install's own comment calls the DEGRADED configuration: it is still probed and
|
||||||
|
still reported, but it does not page, because turning an ordinary documented setup into an alert
|
||||||
|
is how a signal becomes something an operator archives unread.
|
||||||
|
- **It never looks at content**, so it cannot alarm on a newborn tier by construction.
|
||||||
|
- **It never reports ok when it could not ask.** An unreachable PVE is degraded: a self-check that
|
||||||
|
fails open converts *"I do not know"* into *"fine"*.
|
||||||
|
|
||||||
|
The wire shape (`capability.Status`) is unchanged, so the hub's existing critical-degraded alert
|
||||||
|
applies with no hub change and no hub bump.
|
||||||
|
|
||||||
|
## v0.122.0 — three ways the signals lied about themselves (2026-08-03, R-189 · R-188 · R-186)
|
||||||
|
|
||||||
|
All three are the reporting and release path misreporting its own work. **No customer machine, no
|
||||||
|
backup, no restore, no disk layout, no data.** The restore-test itself and when it runs are unchanged
|
||||||
|
from v0.121.1.
|
||||||
|
|
||||||
|
### R-189 — a passing restore-test no longer vanishes on a restart
|
||||||
|
|
||||||
|
`restore_tests[]` came only from the in-memory `backup.Store`, whose own comment read *"lost on
|
||||||
|
restart; the cadence re-populates"*. That was true under a timer. It stopped being true when R-86 made
|
||||||
|
the agent refuse to re-test an archive it has already proven: a proof lost to a restart is not
|
||||||
|
repeated for a whole archive generation — **a week on the offsite tier** — and the hub calls the tier
|
||||||
|
unproven for all of it.
|
||||||
|
|
||||||
|
**Observed, not predicted (2026-08-03):** a real 14.5 GB offsite restore-test PASSED at 15:25:14, the
|
||||||
|
agent was restarted 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two
|
||||||
|
host-reports.
|
||||||
|
|
||||||
|
The durable proof already existed — `RestoreTestState`, on disk, per tier, with the archive since
|
||||||
|
R-86 — and `Snapshot()` had carried the doc comment *"for the host-report gauge"* since the day it was
|
||||||
|
written **with no caller at all**: a seam built, documented, and never connected. It now carries the
|
||||||
|
`tier` and what was `verified` as well (stored at proof time, when they are known for certain, rather
|
||||||
|
than derived later by a storage lookup that can fail), and `ProvenRestoreTests` renders them as report
|
||||||
|
entries which the collector merges.
|
||||||
|
|
||||||
|
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** A fresh failure beats a stored
|
||||||
|
success — the failure is the news and lives nowhere else; a stored success beats a stale in-memory
|
||||||
|
entry after a restart; a tier never appears twice, which the hub would read as two tests. An
|
||||||
|
unparseable timestamp counts as older, so a malformed entry cannot displace a good one.
|
||||||
|
- **It refuses to lie.** A record missing the archive or the tier produces NO entry, and run mechanics
|
||||||
|
(scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one
|
||||||
|
would be. An unproven tier reading as proven would be worse than the defect being fixed.
|
||||||
|
- **Only successes are persisted, and that asymmetry is now written down where it will be read:** a
|
||||||
|
success suppresses future work, so losing it leaves the system quietly less tested than it believes;
|
||||||
|
a failure causes future work and heals itself at the next evaluation.
|
||||||
|
- The `Store` comment that stopped being true is corrected in place rather than left to mislead.
|
||||||
|
|
||||||
|
### R-188 — a correct release no longer emails a failure
|
||||||
|
|
||||||
|
`on: [push]` fires the gates workflow on the **tag** push, and the release pushed its tag *before*
|
||||||
|
publishing, so CI ran the published-versions gate in the seconds before the package existed and
|
||||||
|
correctly reported it missing. Measured across two releases in one session: runs 12/13 and 17/18, same
|
||||||
|
sha each time, opposite results — a race, not a rule. R-168 made that mail the thing that cannot be
|
||||||
|
missed; one that is wrong half the time is one you stop reading.
|
||||||
|
|
||||||
|
**Only the tag PUSH moved** (build → tag locally → publish → push tag). The tag is still created before
|
||||||
|
anything is published, so the build and the tag still describe the same commit; it simply becomes
|
||||||
|
*visible* — to CI, and to any `raw/tag/…` fetch — once the package is downloadable.
|
||||||
|
|
||||||
|
The invariant the old order protected is **not traded away**: `check-published-versions.py` now asserts
|
||||||
|
the converse directly — **no published version may be missing its tag** — as a bounded probe of the
|
||||||
|
frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set every
|
||||||
|
run because a check whose coverage is invisible reads as a guarantee it is not making. The package
|
||||||
|
listing api still answers **401** without a token (re-measured), so absence cannot be enumerated, and
|
||||||
|
the script says so.
|
||||||
|
|
||||||
|
A publish that succeeds and a tag push that then fails now **dies loudly**, printing the one-line
|
||||||
|
recovery; and a publish that *fails* removes the local-only tag so the release can simply be retried
|
||||||
|
instead of colliding with itself.
|
||||||
|
|
||||||
|
### R-186 — a released binary can now be verified by rebuilding it
|
||||||
|
|
||||||
|
`go build` stamps a module version derived from VCS state, so a build made before the tag existed and
|
||||||
|
a rebuild made after it were different binaries. Measured at one commit, same source, same toolchain:
|
||||||
|
|
||||||
|
```
|
||||||
|
default flags, no tag yet ... 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
||||||
|
default flags, tagged ....... 4a38f394… 14 085 440 B (mod v0.121.99)
|
||||||
|
-trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
||||||
|
```
|
||||||
|
|
||||||
|
The stamp is removed rather than sequenced around — nothing in this repo reads it (no `ReadBuildInfo`
|
||||||
|
caller) and the version comes from the explicit `-X main.version` ldflag. `-trimpath` additionally
|
||||||
|
makes a rebuild from a different checkout directory match.
|
||||||
|
|
||||||
|
**A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and
|
||||||
|
therefore produced a binary **74 KB smaller** than the release path built for the same version — one
|
||||||
|
version name, two binaries, decided by which entry point was used. Both now build identically.
|
||||||
|
|
||||||
|
`CLAUDE.md` records the exact command an operator can run to verify a published binary independently.
|
||||||
|
|
||||||
|
## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3)
|
||||||
|
|
||||||
|
**Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that
|
||||||
|
had just shipped.** Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
|
||||||
|
construction. After it, *"nothing is due"* is the NORMAL outcome — and it was logged at **DEBUG**,
|
||||||
|
which journald drops. An empty journal would then have been equally consistent with a healthy loop
|
||||||
|
and with a dead goroutine: the exact shape the R-88 watcher was retired for, re-created in a new
|
||||||
|
place by making the quiet path the common one.
|
||||||
|
|
||||||
|
A not-due evaluation now logs one **INFO** line naming every tier's verdict:
|
||||||
|
|
||||||
|
```
|
||||||
|
backup: restore-test evaluated — nothing due
|
||||||
|
verdicts="felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven;
|
||||||
|
felhom-backup: no settled archive yet — nothing to prove (newborn or still settling)"
|
||||||
|
```
|
||||||
|
|
||||||
|
Four lines a day at the 6 h default, and the answer to *"why did nothing run last night?"* is in the
|
||||||
|
log instead of being re-derived. A tier whose storage cannot be listed reads `UNKNOWN` with its error
|
||||||
|
in the same line, so a lookup failure can never present as "nothing due".
|
||||||
|
|
||||||
|
Red-proved by reverting to the bare `Debug` line: the test asserts what the SCHEDULER emits on a real
|
||||||
|
`tick`, not what the helper returns — a helper-level test would have passed against a tick that never
|
||||||
|
called it.
|
||||||
|
|
||||||
## v0.121.0 — a restore-test proves each BACKUP, not the clock (2026-08-03, R-86)
|
## v0.121.0 — a restore-test proves each BACKUP, not the clock (2026-08-03, R-86)
|
||||||
|
|
||||||
**The trigger changed; the restore-test did not.** `Scheduler.Run` still has a ticker, but it is now
|
**The trigger changed; the restore-test did not.** `Scheduler.Run` still has a ticker, but it is now
|
||||||
|
|||||||
@@ -72,6 +72,32 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
|
|||||||
> verifies by an **independent download** rather than trusting the publish step's own output.
|
> verifies by an **independent download** rather than trusting the publish step's own output.
|
||||||
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
|
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
|
||||||
> than reimplementing it.
|
> than reimplementing it.
|
||||||
|
>
|
||||||
|
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
|
||||||
|
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
|
||||||
|
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
||||||
|
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
|
||||||
|
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
|
||||||
|
> protected is asserted directly instead: the gate now also refuses a **published version with no
|
||||||
|
> tag**. If the push fails after a successful publish the script says so loudly and prints the
|
||||||
|
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
|
||||||
|
>
|
||||||
|
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
|
||||||
|
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
|
||||||
|
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
|
||||||
|
> version yourself:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> V=0.122.0
|
||||||
|
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
|
||||||
|
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
|
||||||
|
> sha256sum /tmp/felhom-agent-check
|
||||||
|
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
|
||||||
|
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
|
||||||
|
> ever changes, change both or one version name means two binaries again.
|
||||||
|
|
||||||
| Step | Where | One-liner |
|
| Step | Where | One-liner |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@@ -79,6 +105,7 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
|
|||||||
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
||||||
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
||||||
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
||||||
|
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
|
||||||
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
|
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
|
||||||
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
||||||
|
|
||||||
|
|||||||
+61
@@ -5,6 +5,59 @@
|
|||||||
|
|
||||||
## Current
|
## Current
|
||||||
|
|
||||||
|
- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
|
||||||
|
`FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
|
||||||
|
storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
|
||||||
|
through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
|
||||||
|
archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
|
||||||
|
restore-testable and nothing said so.
|
||||||
|
- **The permission question is asked directly**, because unlike the listing it has a definite
|
||||||
|
answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
|
||||||
|
own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
|
||||||
|
composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
|
||||||
|
belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
|
||||||
|
degraded alert applies with **no hub change**.
|
||||||
|
- **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
|
||||||
|
403 — it carries the privileges INHERITED from the box-wide `/` grant
|
||||||
|
(`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
|
||||||
|
blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
|
||||||
|
changing that constant (`storeGrantRequiredPriv`, red-proved).
|
||||||
|
- **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
|
||||||
|
defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
|
||||||
|
does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
|
||||||
|
ok when it could not ask.
|
||||||
|
- **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
|
||||||
|
grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
|
||||||
|
and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
|
||||||
|
correctly picking the 08-02 archive (08-03 had not settled 24 h).
|
||||||
|
- **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
|
||||||
|
grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.
|
||||||
|
|
||||||
|
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
|
||||||
|
None touches data; all three cost attention, which every other signal depends on.
|
||||||
|
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
|
||||||
|
from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
|
||||||
|
timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
|
||||||
|
a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
|
||||||
|
twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
|
||||||
|
still read, and a record missing archive-or-tier is NOT reported), exposes
|
||||||
|
`ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
|
||||||
|
newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
|
||||||
|
twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
|
||||||
|
"host-report gauge" in its doc comment and had **no caller** for weeks.
|
||||||
|
- **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
|
||||||
|
future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
|
||||||
|
tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
|
||||||
|
- **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
|
||||||
|
locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
|
||||||
|
made the gate correctly fail a correct release ~half the time. The old order's invariant is now
|
||||||
|
asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
|
||||||
|
a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
|
||||||
|
- **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
|
||||||
|
bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
|
||||||
|
`CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
|
||||||
|
identical. The verification command is in `CLAUDE.md`.
|
||||||
|
|
||||||
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
|
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
|
||||||
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
|
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
|
||||||
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
|
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
|
||||||
@@ -26,6 +79,14 @@
|
|||||||
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
|
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
|
||||||
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
|
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
|
||||||
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
|
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
|
||||||
|
- **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
|
||||||
|
DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
|
||||||
|
a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
|
||||||
|
evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
|
||||||
|
with its error in that same line.
|
||||||
|
- **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
|
||||||
|
encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
|
||||||
|
named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
|
||||||
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
|
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
|
||||||
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
|
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
|
||||||
host tier has never been restore-testable there, and the due-check cannot distinguish that from
|
host tier has never been restore-testable there, and the due-check cannot distinguish that from
|
||||||
|
|||||||
@@ -1,100 +1,205 @@
|
|||||||
# REPORT — releasing publishes, and an unreleasable version fails CI (R-115, R-183)
|
# REPORT — R-185: a tier the box cannot READ must say so
|
||||||
|
|
||||||
**Date:** 2026-08-03 · **Repo:** `felhom-agent` · **NO VERSION BUMP** — the agent stays **v0.120.0**,
|
**Date:** 2026-08-03 · **Repos:** `felhom-agent` **v0.122.0 → v0.123.0** (`fe14bc6`) · `felhom.eu`
|
||||||
no Go code changed, nothing was built or deployed.
|
installer **1.23.0 → 1.24.0** (`688470c`, tag `installer-v1.24.0`, manifest bump `311dc06`) ·
|
||||||
|
**no hub change and no hub bump** — the hub already alerts on a degraded critical capability, which is
|
||||||
|
why that mechanism was chosen.
|
||||||
|
|
||||||
## What changed
|
---
|
||||||
|
|
||||||
| File | |
|
## 1. Baselines, re-read on arrival
|
||||||
|
|
||||||
|
| Repo | `main` @ commit | Version | Matched §1? |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `felhom-agent` | `0b28eae7bb14` | `v0.122.0` | **yes** |
|
||||||
|
| `felhom.eu` | `7a5694341d59` | installer `1.23.0`, both `--ref=installer-v1.23.0` (lines 327, 372) | **yes** |
|
||||||
|
|
||||||
|
Highest register ID in use **R-189**; R-190+ confirmed free by grep, and none was needed.
|
||||||
|
|
||||||
|
## 2. Part 0 — the measurements, before anything was designed against them
|
||||||
|
|
||||||
|
**The row's three-way observation, reproduced unchanged:**
|
||||||
|
|
||||||
|
| leg | result |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `scripts/release-agent.sh` | **new** — THE release path: build → tag → publish → verify by independent download |
|
| root, `pvesh … /storage/felhom-backup/content` | **3 archives** — 6.1 / 6.2 / 6.3 GB, dated 08-01, 08-02, 08-03 |
|
||||||
| `scripts/check-published-versions.py` | **new** — the R-115 gate |
|
| the **agent's token**, same endpoint | `{"data":[]}` |
|
||||||
| `scripts/agent_gates.py` | registers the gate as **not `--fast`** (it needs network) |
|
| the agent's token, `/storage/local/content` | **8 entries** — the token works where it is granted |
|
||||||
| `.gitea/workflows/gates.yml` | CI now runs the **full** gate set, not `--fast` |
|
|
||||||
| `CLAUDE.md` | the raw `go build` line is replaced by the release script; a **Vouch** row replaces the old Publish row |
|
|
||||||
|
|
||||||
## Why
|
So the token is the variable, not the storage. Two further checks removed the obvious alternative
|
||||||
|
explanation: guest **9201 IS in the `felhom` pool** (so `VM.Backup` is not the discriminator), and
|
||||||
|
`pveum acl list` showed ACL rows for `/storage/{local,local-lvm,felhom-pbs}` and **none** for
|
||||||
|
`/storage/felhom-backup`.
|
||||||
|
|
||||||
Publishing was a step someone had to remember and was **forgotten three times in five days** —
|
**The permission query, asked by the token itself — and the obvious reading is wrong:**
|
||||||
R-111's seventeen stranded releases, 0.114.0, and 0.120.0, which sat deployed on both demo hosts and
|
|
||||||
undownloadable, so a documented-path reinstall would have silently downgraded them to the pre-merge
|
|
||||||
agent **while reporting success**. R-111's own closing line named this leg and closed SHIPPED without
|
|
||||||
it; it recurred the same afternoon. A note is not a mechanism.
|
|
||||||
|
|
||||||
The script also **tags**, because `felhom-host-install.sh` now fetches the agent's sixteen config
|
```
|
||||||
files from `raw/tag/v<version>/` (R-183). A released version with no tag 404s a box mid-install, as
|
/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
|
||||||
root, on a virgin machine. Tag and package are two halves of one release.
|
/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
|
||||||
|
```
|
||||||
|
|
||||||
It **verifies by downloading what it just published** and comparing the sha to what it built. The
|
The ungranted path answers **neither empty nor 403**. It answers with the privileges **inherited**
|
||||||
publish step's own success is a report on its own write; a fetch returning the right bytes is a
|
from the box-wide `/` grant. A probe asking *"did the path come back?"* — or *"does it hold
|
||||||
different claim, and it is the one that matters.
|
`Datastore.Audit`?"* — would have reported the blinded storage **healthy**. This is exactly what §3
|
||||||
|
required to be measured rather than assumed, and it changed the design: the probe tests
|
||||||
|
`Datastore.AllocateSpace` specifically, and a red-proof pins that choice.
|
||||||
|
|
||||||
It **does not vouch** — that points machines at a version and stays the operator's act.
|
## 3. The probe
|
||||||
|
|
||||||
## The gate's invariant — not the one specified, and the reason was measured
|
`Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's own token**
|
||||||
|
(asking as root answers a different question and always says yes). `storeGrantStatuses` emits one
|
||||||
|
`capability.Status` per configured tier.
|
||||||
|
|
||||||
The task's §8.4 asked for *"the version the hub tells machines to install must be downloadable"*.
|
**Deviation from §5/§8.1, stated because a recommendation not followed gets a line:** the spec asked
|
||||||
**CI cannot see that**, measured rather than assumed (P-C):
|
for the sudo `Prober` to be minimally generalised. This repo already has the better-established
|
||||||
|
pattern for exactly this — `poolReadStatus`, composed **around** the prober, with the comment *"an API
|
||||||
|
read does not belong inside the sudo-policy probe"* (v0.62.0, audit A1). The probe follows that
|
||||||
|
precedent instead. `capability.Status` is untouched either way, which is the constraint that mattered.
|
||||||
|
|
||||||
| Endpoint | Anonymous |
|
**Decisions:**
|
||||||
|
|
||||||
|
- **The probed set comes from the box's own `BackupTiers()`**, never a fixed list — a hardcoded probe
|
||||||
|
list is the defect reproduced inside the fix.
|
||||||
|
- **Critical** (§8.3): the hub alerts only on critical, so a non-critical entry would ride the report
|
||||||
|
and alert nobody — the same silence with extra steps. **Except** the `local` fallback target, which
|
||||||
|
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||||
|
does not page. Turning an ordinary documented setup into an alert is how a signal becomes something
|
||||||
|
an operator archives unread.
|
||||||
|
- **It never consults content**, so it cannot alarm on a newborn tier by construction — a stronger
|
||||||
|
guarantee than gating on emptiness would be.
|
||||||
|
- **It never reports ok when it could not ask.** Unreachable PVE is degraded: a self-check that fails
|
||||||
|
open converts *"I do not know"* into *"fine"*.
|
||||||
|
|
||||||
|
## 4. The installer — the root cause was not where the row or the task expected
|
||||||
|
|
||||||
|
Both assumed `PVE_STORAGES` (the fixed grant list) was the culprit. **It is not.**
|
||||||
|
`configure_backup_target` has two arms:
|
||||||
|
|
||||||
|
- **Case A** creates the storage and calls `felhom-backup-target-apply grant` in the same breath — a
|
||||||
|
box that builds its own target has always been correct.
|
||||||
|
- **The Scenario-F arm** — *"the target already exists, leave it exactly as it is"* — **returned
|
||||||
|
without granting**.
|
||||||
|
|
||||||
|
So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
|
||||||
|
surviving a reinstall — which is both demo boxes) pointed `local_backup_target` at a storage its own
|
||||||
|
token could not read. The reuse arm now ensures the ACL through the same guarded wrapper.
|
||||||
|
|
||||||
|
**Scenario F is unviolated:** the storage DEFINITION is still untouched. Granting the role the agent is
|
||||||
|
supposed to have on the target this same script is about to write into `agent.json` is finishing the
|
||||||
|
job, not retargeting the box; `pveum acl modify` is idempotent, so a box that already has it is
|
||||||
|
unchanged and a box whose token was rotated gets it back.
|
||||||
|
|
||||||
|
**`$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES`,** and the comment now says why: that
|
||||||
|
list is granted in step 4/5, *before* `configure_backup_target` runs in step 6, and `--acl-storages`
|
||||||
|
entries are preflight-checked for existence. Adding it there would grant on a storage that may not yet
|
||||||
|
exist and would split ownership of the decision across two places.
|
||||||
|
|
||||||
|
**A gate now asserts it:** every arm of `configure_backup_target` that resolves the target must also
|
||||||
|
grant on it — the check that would have caught this.
|
||||||
|
|
||||||
|
## 5. Live validation, in order
|
||||||
|
|
||||||
|
| # | evidence |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Gitea package **download** | **200** (and **404** for a fake version — it discriminates) |
|
| 1 | Part 0's measurements above, taken **before** any change |
|
||||||
| Gitea **tags** api | **200** |
|
| 2 | **The signal that has never existed**, on the still-blind box: `capability DEGRADED … capability=pve:store-grant:felhom-backup … reason="the agent token lacks Datastore.AllocateSpace on /storage/felhom-backup (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested" critical=true`, with `ok=69 total=70 degraded=1`. The hub: `Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)` and **`Operator email sent`** |
|
||||||
| Gitea package **listing** api | **401** — token required |
|
| 3 | Grant applied (user **and** token — a privsep token's rights are the intersection); the token then lists **3 archives** where it listed none, and the permission answer becomes `{"Datastore.AllocateSpace":1,"Datastore.Allocate":1}` |
|
||||||
| Hub `/api/v1/artifacts/<customer>` | **401** — per-customer passphrase required |
|
| 4 | `capabilities self-check ok=70 total=70 degraded=0`; the hub: `degraded → ok (agent_capability_recovered)` |
|
||||||
|
| 5 | **The host tier is a due-check candidate for the first time on that box**: `tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" proven=""` — and the settle rule applies to it exactly as to the others, selecting the **08-02** archive because the 08-03 one has not settled 24 h |
|
||||||
|
| 6 | The served installer over HTTPS: `SCRIPT_VERSION="1.24.0"`, and the served bytes carry the fix itself, not merely the version |
|
||||||
|
|
||||||
So a credential-free gate can ask *"is this version installable"* but not *"which version is
|
## 6. The other machines
|
||||||
vouched"*. Adding an operator credential to CI to close that is the operator's call, not a gate
|
|
||||||
author's. The implemented invariant — **every `v<semver>` tag must have a downloadable package and a
|
|
||||||
tag tree that serves the agent's configs** — needs no credential and **catches all three recorded
|
|
||||||
instances**, because the release script creates the tag and publishes in one act.
|
|
||||||
|
|
||||||
**What it does not catch, stated rather than assumed away:** the hub vouching a version that was
|
- **demo-hp CARRIES THE SAME DRIFT — and was fixed.** `local_backup_target=felhom-backup`, ACL rows for
|
||||||
never released at all. Nothing here can see that; it belongs at vouch time in the hub. → **R-184**.
|
`local`, `local-lvm`, `felhom-pbs` only. §8.6 assumed a single affected box; the same one-line,
|
||||||
|
additive, path-scoped, idempotent grant applies to the other, and leaving a known-blind backup tier
|
||||||
|
on a Tier-0 box after finding it would be this row happening twice. Granted (user + token); its
|
||||||
|
token now lists **4 archives**. It still runs agent `0.120.0`, so it has no probe yet — that arrives
|
||||||
|
when you vouch.
|
||||||
|
- **The tester's box was NOT touched** (Tier 2, protected). **What is known without connecting to it:**
|
||||||
|
it very likely carries the same drift — the mechanism is the Scenario-F reuse arm, which fires on
|
||||||
|
any box whose target pre-dated its install, and its target was moved by the very runbook that
|
||||||
|
creates that condition. It is due for reinstall, and installer 1.24.0 fixes it on the way in.
|
||||||
|
|
||||||
## Proof
|
## 7. Tests and red-proofs
|
||||||
|
|
||||||
| Check | Result |
|
Green gate: `go build ./... && go vet ./... && go test ./...` — rc=0, plus `agent_gates.py` and
|
||||||
|---|---|
|
`repo_gates.py` all OK. Test runs and commits were always separate commands.
|
||||||
| `go build ./... && go vet ./...` | OK |
|
|
||||||
| `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
|
|
||||||
| `agent_gates.py --fast` | `published` correctly **SKIPPED** — the pre-push hook must not fail because Gitea blinked |
|
|
||||||
| `agent_gates.py` (full) | `reuse-refs` OK, `published` OK |
|
|
||||||
| release script: re-release guard | `ERROR: tag v0.120.0 already exists — releasing over it would make one version name two binaries`, rc=1 |
|
|
||||||
| release script: clean-tree guard | `ERROR: working tree is dirty — commit and push first`, rc=1 |
|
|
||||||
|
|
||||||
### Red-proof F — both directions
|
| # | Test | Asserts | Mutation | Observed |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| A | `TestStoreGrant_ForbiddenStorageIsDegradedAndNamed` | degraded, critical, naming storage **and** role | probe removed from `probeAll` | **FAIL** — `main.go never calls storeGrantStatuses` (via the seam test); with the wrong-privilege mutation: `must be DEGRADED, not "ok"` |
|
||||||
|
| A′ | `TestStoreGrant_InheritedPrivilegesAreNotAGrant` | the measured trap: inherited ≠ granted | probe `Datastore.Audit` instead | **FAIL** — `checking for the wrong privilege reports a blinded storage healthy; got "ok"` |
|
||||||
|
| B | `TestStoreGrant_GrantedButEmptyIsHealthy` | a readable-but-empty tier is healthy | — (it never reads content, so emptiness cannot reach it) | pass |
|
||||||
|
| B′ | `TestStoreGrant_TheFallbackTargetIsNotCritical` | `local` is reported but does not page | gating removed (`return true`) | **FAIL** — `must not page the operator about an ordinary, documented configuration` |
|
||||||
|
| C | `TestStoreGrant_ForbiddenAndNewbornAreDistinguishable` | different status **and** different capability id | — | pass |
|
||||||
|
| — | `TestStoreGrant_UnreachablePVEIsDegradedNotOK` | unknown ≠ ok | — | pass |
|
||||||
|
| F | `hostinstall_gates.py` backup-target assertion | every resolving arm also grants | reuse arm reverted | **FAIL** — `resolves the backup target in 2 place(s) but grants in only 1` |
|
||||||
|
| H | `TestMainWiresTheStoreGrantProbe` | **AST** of `main.go` | call commented out | **FAIL** — a `strings.Contains` check would have passed |
|
||||||
|
|
||||||
- **A tagged-but-unpublished version** (`v9.9.9` created for the purpose): gate **rc=1**,
|
**A hollow test caught and fixed before it shipped:** the first draft of `storegrant_test.go`
|
||||||
`binary NOT downloadable (HTTP 404 …)`. This is the R-115 shape exactly.
|
re-implemented the verdict branch inside the test. It passed, and would have kept passing while
|
||||||
- **The gate deregistered from the entry point**, same bad state: `agent_gates.py` → **rc=0, "all
|
production diverged. The decision was extracted into `storeGrantVerdict` and the tests now call it.
|
||||||
agent gates OK"**. Restored → **rc=1, CONVICTED: published**. The guard is what catches it, not
|
|
||||||
something else.
|
|
||||||
|
|
||||||
### Scenario F measured on REAL CI, not inferred
|
**Scenario B's red-proof, honestly:** the spec asked for "degrade on an empty content listing" as the
|
||||||
|
mutation. That is not a mutation of this code — the probe never looks at content, which is a stronger
|
||||||
|
guarantee than gating on emptiness. The gating red-proof above (`storeGrantCritical`) is the one that
|
||||||
|
exercises the guard that does exist, and it fails as required.
|
||||||
|
|
||||||
Runs **69** and **70** are on the **same commit** `0db7766`:
|
## 8. Files, commits, tag
|
||||||
|
|
||||||
| run | state of the repo | CI |
|
`internal/proxmox/query.go` (`Permissions`), `cmd/felhom-agent/main.go` (`storeGrantStatuses`,
|
||||||
|---|---|---|
|
`storeGrantVerdict`, `storeGrantCritical`, `storeGrantRequiredPriv`, wiring),
|
||||||
| 69 | no `v9.9.9` | **success** |
|
`cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `REPORT.md`.
|
||||||
| 70 | `v9.9.9` tagged, not published | **failure** |
|
`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`,
|
||||||
|
`manifests/webpage.yaml`, `CONTEXT.md`, `STATUS.md`, `documentation/architecture/00-capability-map.md`,
|
||||||
|
`documentation/backlog/OPEN-ITEMS.md`, `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.
|
||||||
|
|
||||||
Same code, same workflow, one variable — so the gate demonstrably RUNS in CI and fails for exactly
|
**Commits** — `felhom-agent`: `fe14bc6` (v0.123.0). `felhom.eu`: `688470c` (installer 1.24.0), `311dc06`
|
||||||
the R-115 condition. This also retrospectively explains runs 67/68, which were red in the window when
|
(manifest refs), `e3187c8` (docs). **Installer tag:** `installer-v1.24.0`.
|
||||||
`v9.9.9` first existed. **One deliberate CI failure e-mail reached the operator — that was this
|
|
||||||
proof, not an incident.**
|
|
||||||
|
|
||||||
I could not read CI's own step log to attribute those runs directly: the Gitea jobs endpoint requires
|
## 9. Deployment
|
||||||
an API token, and the only credential available on this host (`~/.docker/config.json`) is a registry
|
|
||||||
password, which the API rejects. The controlled before/after above replaced that log rather than an
|
|
||||||
assumption standing in for it.
|
|
||||||
|
|
||||||
`v9.9.9` was deleted afterwards; `git ls-remote --tags` shows only `v0.120.0`.
|
Agent released through `release-agent.sh` — tag `v0.123.0`, sha256
|
||||||
|
`74910135ac4feb1b7f0ad4dbd1541d965cbc0fe70d4f47b62ebf7e4bfb962453`, round-trip verified. The
|
||||||
|
**published bytes** were downloaded and deployed: the running binary's sha matches the published one.
|
||||||
|
`felhom-agent --version` → **0.123.0**, `systemctl is-active` → active, prior kept as `.bak-0.122.0`.
|
||||||
|
**NOT VOUCHED** — that stays the operator's act.
|
||||||
|
|
||||||
## Tag convention
|
## 10. Registers
|
||||||
|
|
||||||
`v<semver>`, at the commit the binary was built from. `v0.120.0` was created retroactively at
|
- **R-185 → CLOSED** (shipped + proven live on both demo boxes), with the corrected root cause
|
||||||
`cd6e267` — the commit that produced the published binary (sha `a7763d31b55b5ce7…`). `configs/` is
|
recorded on the row.
|
||||||
byte-identical between that commit and `main`, so nothing about the sixteen fetched files depends on
|
- No new IDs minted; `ROADMAP.md` contains no R-185 row, so there was nothing to collapse.
|
||||||
the choice; `cd6e267` is tagged because it is the honest one.
|
- **The capability map's whole-guest row was OPTIMISTIC and now says so:** every live restore-test it
|
||||||
|
cited is on the OFFSITE tier, and the HOST tier was not merely unproven but *unprovable* on both
|
||||||
|
demo boxes. It now records that, the closure, and that it will carry a host-tier live proof when one
|
||||||
|
runs.
|
||||||
|
- The vzdump-target-move runbook's item 5 **predicted this** and is annotated, not rewritten: it
|
||||||
|
expected a 403 on backup, and the reason it did not surface that way is that `vzdump` writes through
|
||||||
|
a root path, so backups kept landing while the agent's *read* stayed blind.
|
||||||
|
- `CONTEXT.md`: agent-side entry, plus `felhom.eu` **S-21** (empty ≠ forbidden; the measured trap) and
|
||||||
|
**S-22** (the Scenario-F arm must finish the job).
|
||||||
|
|
||||||
|
## 11. Teardown
|
||||||
|
|
||||||
|
**Nothing was provisioned.** No scratch storage, no fixture grant, no probe tag, no scratch package
|
||||||
|
version. The two ACL grants are the intended durable change; the only other mutation was the
|
||||||
|
installer label, which is reversible by moving the tag.
|
||||||
|
|
||||||
|
## 12. Observations — noticed, recorded, NOT acted on
|
||||||
|
|
||||||
|
- **Both demo boxes are now due for a host-tier restore-test**, which has never run on either. The
|
||||||
|
scheduler will pick it up within 6 h unattended (a ~6 GB local restore — fast, and cheaper than the
|
||||||
|
offsite ones). Expected, not a defect, and the first host-tier proof this fleet will have.
|
||||||
|
- **`--acl-storages` semantics are unchanged and the automatic grant does not consult it.** If an
|
||||||
|
operator passes `--acl-storages` deliberately excluding the backup target, the target is still
|
||||||
|
granted by the resolution path. That is the correct precedence — a box cannot function with an
|
||||||
|
unreadable backup target — but it is a place where an override is not absolute, and it is written
|
||||||
|
here rather than left to be discovered.
|
||||||
|
- **`storeGrantRequiredPriv` is a single privilege**, chosen from measurement. If PVE ever changes
|
||||||
|
which privilege gates content listing, the probe would report healthy while the tier is blind. The
|
||||||
|
test asserts the constant's value so a change forces a re-measurement, but nothing detects a change
|
||||||
|
on PVE's side.
|
||||||
|
- **Ten pre-existing `gofmt`-unclean files** remain in the agent repo (unchanged from yesterday's
|
||||||
|
observation); every file touched here is clean.
|
||||||
|
|||||||
@@ -148,7 +148,9 @@
|
|||||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,t)` / `ProvenArchive(target)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. |
|
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||||
|
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||||
|
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
|
||||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||||
|
|||||||
@@ -0,0 +1,188 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"go/ast"
|
||||||
|
"go/parser"
|
||||||
|
"go/token"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
|
||||||
|
// grepping for a string.
|
||||||
|
//
|
||||||
|
// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
|
||||||
|
// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
|
||||||
|
// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
|
||||||
|
// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
|
||||||
|
// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
|
||||||
|
// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
|
||||||
|
// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
|
||||||
|
|
||||||
|
func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
|
||||||
|
t.Helper()
|
||||||
|
fset := token.NewFileSet()
|
||||||
|
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parsing main.go: %v", err)
|
||||||
|
}
|
||||||
|
return fset, f
|
||||||
|
}
|
||||||
|
|
||||||
|
// callsWithin returns the set of function names called (directly, by identifier or selector) inside
|
||||||
|
// the named top-level function.
|
||||||
|
func callsWithin(f *ast.File, fnName string) map[string]bool {
|
||||||
|
out := map[string]bool{}
|
||||||
|
for _, d := range f.Decls {
|
||||||
|
fd, ok := d.(*ast.FuncDecl)
|
||||||
|
if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||||
|
ce, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
switch fn := ce.Fun.(type) {
|
||||||
|
case *ast.Ident:
|
||||||
|
out[fn.Name] = true
|
||||||
|
case *ast.SelectorExpr:
|
||||||
|
if x, ok := fn.X.(*ast.Ident); ok {
|
||||||
|
out[x.Name+"."+fn.Sel.Name] = true
|
||||||
|
}
|
||||||
|
out[fn.Sel.Name] = true
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
|
||||||
|
func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
|
||||||
|
_, f := parseMain(t)
|
||||||
|
|
||||||
|
// 1. main() reaches runDaemon.
|
||||||
|
if !callsWithin(f, "main")["runDaemon"] {
|
||||||
|
t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
|
||||||
|
}
|
||||||
|
// 2. runDaemon reaches buildLocalAPIServer.
|
||||||
|
if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||||
|
t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
|
||||||
|
// an escrow.OffsiteKeyRecoverer is constructed there.
|
||||||
|
var optionsHasField, recovererConstructed bool
|
||||||
|
for _, d := range f.Decls {
|
||||||
|
fd, ok := d.(*ast.FuncDecl)
|
||||||
|
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||||
|
cl, ok := n.(*ast.CompositeLit)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
sel, ok := cl.Type.(*ast.SelectorExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
pkg, _ := sel.X.(*ast.Ident)
|
||||||
|
if pkg == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
switch pkg.Name + "." + sel.Sel.Name {
|
||||||
|
case "localapi.Options":
|
||||||
|
for _, el := range cl.Elts {
|
||||||
|
kv, ok := el.(*ast.KeyValueExpr)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
|
||||||
|
optionsHasField = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "escrow.OffsiteKeyRecoverer":
|
||||||
|
recovererConstructed = true
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if !recovererConstructed {
|
||||||
|
t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
|
||||||
|
"production assembly point (the built-but-never-wired shape, seventh instance)")
|
||||||
|
}
|
||||||
|
if !optionsHasField {
|
||||||
|
t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
|
||||||
|
"exists and the route would answer 503 forever")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
|
||||||
|
// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
|
||||||
|
// key is used.
|
||||||
|
func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
|
||||||
|
fset, f := parseMain(t)
|
||||||
|
var fetchUsesHubClient bool
|
||||||
|
for _, d := range f.Decls {
|
||||||
|
fd, ok := d.(*ast.FuncDecl)
|
||||||
|
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||||
|
ce, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||||
|
if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
|
||||||
|
fetchUsesHubClient = true
|
||||||
|
} else {
|
||||||
|
t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
|
||||||
|
fset.Position(ce.Pos()))
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if !fetchUsesHubClient {
|
||||||
|
t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
|
||||||
|
"not wired, or it uses a client whose credentials are not this host's")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The route itself must be registered on the local API. A handler with no route is the same defect
|
||||||
|
// one layer down, and it has shipped here before.
|
||||||
|
func TestRecoverRouteIsRegistered(t *testing.T) {
|
||||||
|
fset := token.NewFileSet()
|
||||||
|
f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parsing localapi/server.go: %v", err)
|
||||||
|
}
|
||||||
|
var registered bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
ce, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok || len(ce.Args) < 2 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
sel, ok := ce.Fun.(*ast.SelectorExpr)
|
||||||
|
if !ok || sel.Sel.Name != "HandleFunc" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
lit, ok := ce.Args[0].(*ast.BasicLit)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
|
||||||
|
registered = true
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !registered {
|
||||||
|
t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
|
||||||
|
"exists and nothing can reach it")
|
||||||
|
}
|
||||||
|
}
|
||||||
+377
-4
@@ -24,6 +24,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -411,6 +412,318 @@ func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
|
|||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// storeGrantStatuses probes whether the agent's OWN TOKEN may read the storages this box depends
|
||||||
|
// on — one capability.Status per configured backup tier (R-185).
|
||||||
|
//
|
||||||
|
// ── WHY THIS EXISTS, AND WHY IT IS NOT A CONTENT LISTING ─────────────────────────────────────
|
||||||
|
//
|
||||||
|
// On demo-felhom the token had FelhomAgentStore on local, local-lvm and felhom-pbs — and NOT on
|
||||||
|
// `felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked
|
||||||
|
// for that storage's content the API answers `{"data":[]}` while root sees three archives.
|
||||||
|
//
|
||||||
|
// **An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return**, and no care at that
|
||||||
|
// call site can separate them: `pickForThisRun` skips an empty tier (correctly — a fresh offsite
|
||||||
|
// tier legitimately has nothing) and says "no settled archive yet". So the host tier on that box was
|
||||||
|
// never restore-testable and nothing ever mentioned it. That is this project's own rule failing in a
|
||||||
|
// new place: an empty answer is not evidence that there is nothing there.
|
||||||
|
//
|
||||||
|
// The permission question, unlike the listing, has a DEFINITE answer — so it is asked directly.
|
||||||
|
//
|
||||||
|
// ── WHAT IS PROBED, AND WHY NOT A FIXED LIST ─────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// The tiers come from this box's own config (`BackupTiers()`), because a hardcoded probe list is
|
||||||
|
// precisely the defect being fixed — the installer's hardcoded ACL set is what drifted from the
|
||||||
|
// target it went on to configure. Probing what the box says it depends on cannot drift from it.
|
||||||
|
//
|
||||||
|
// CRITICAL, deliberately: a tier the agent cannot read is a tier whose backups are invisible to it
|
||||||
|
// and which is never restore-tested. The hub alerts only on Critical, and a non-critical entry here
|
||||||
|
// would ride the report and alert nobody — the same silence with extra steps.
|
||||||
|
//
|
||||||
|
// One exception, so an ordinary configuration is not turned into an alarm: a box with no dedicated
|
||||||
|
// target (`local_backup_target: "local"`, which host-install's own comment calls the DEGRADED
|
||||||
|
// fallback) is not treated as critical for that tier — see storeGrantCritical.
|
||||||
|
func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Config, repair *storeGrantRepairer) []capability.Status {
|
||||||
|
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
|
||||||
|
out := make([]capability.Status, 0, len(tiers))
|
||||||
|
for _, t := range tiers {
|
||||||
|
out = append(out, storeGrantStatus(ctx, px, t.TargetID, storeGrantCritical(t.TargetID), repair))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantRepairReportWindow is how long after a repair the capability keeps reporting the
|
||||||
|
// transition. It MUST exceed the hub report interval, or the record never reaches the operator.
|
||||||
|
//
|
||||||
|
// FOUND BY THE LIVE RUN, NOT BY THE TESTS (2026-08-04). The first implementation reported degraded
|
||||||
|
// for exactly "one cycle" — the probe call that did the repair. But `probeAll` is invoked
|
||||||
|
// INDEPENDENTLY by the startup/periodic self-check log and by the collector building a host report,
|
||||||
|
// so the repairing call was the LOG's, and the report built three seconds later found the grant
|
||||||
|
// present and reported `ok`. The agent's journal had the record; the hub had nothing; the operator
|
||||||
|
// would have learned nothing. That is precisely the silence R-190 is about, re-created inside its own
|
||||||
|
// mitigation.
|
||||||
|
//
|
||||||
|
// A latch on TIME rather than on call count fixes it: 20 minutes comfortably exceeds the 900 s report
|
||||||
|
// interval, so at least one host-report must carry the transition, and it still clears on its own.
|
||||||
|
const storeGrantRepairReportWindow = 20 * time.Minute
|
||||||
|
|
||||||
|
// storeGrantRepairMinInterval bounds how often a single tier's grant may be re-granted (Scenario F).
|
||||||
|
//
|
||||||
|
// A storage can be unreadable for reasons an ACL cannot fix — the storage is gone, PVE is wedged,
|
||||||
|
// the wrapper is missing. Without a bound the probe would re-grant on every report cycle forever: a
|
||||||
|
// repair loop is a new defect wearing a fix's clothes. One attempt per tier per hour is frequent
|
||||||
|
// enough that a real loss is repaired within one backup window, and rare enough that a permanent
|
||||||
|
// fault produces attempts you can count on one hand per day.
|
||||||
|
const storeGrantRepairMinInterval = time.Hour
|
||||||
|
|
||||||
|
// storeGrantRepairer bounds and records the self-repair. It is deliberately in-memory: an agent
|
||||||
|
// restart re-arms the repair, which is correct — a restart is exactly when a box should re-check
|
||||||
|
// everything it depends on.
|
||||||
|
type storeGrantRepairer struct {
|
||||||
|
run func(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
|
||||||
|
log *slog.Logger
|
||||||
|
mu sync.Mutex
|
||||||
|
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||||
|
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||||
|
func (r *storeGrantRepairer) noteRepaired(target string, now time.Time) {
|
||||||
|
if r == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
if r.repaired == nil {
|
||||||
|
r.repaired = map[string]time.Time{}
|
||||||
|
}
|
||||||
|
r.repaired[target] = now
|
||||||
|
}
|
||||||
|
|
||||||
|
// recentlyRepaired reports whether a confirmed repair is still inside its report window — the latch
|
||||||
|
// that guarantees a host-report carries the transition even though the probe that repaired may have
|
||||||
|
// been a log-only one.
|
||||||
|
func (r *storeGrantRepairer) recentlyRepaired(target string, now time.Time) bool {
|
||||||
|
if r == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
t, ok := r.repaired[target]
|
||||||
|
return ok && now.Sub(t) < storeGrantRepairReportWindow
|
||||||
|
}
|
||||||
|
|
||||||
|
// mayAttempt reports whether a repair may run now for this target, and records the attempt if so.
|
||||||
|
func (r *storeGrantRepairer) mayAttempt(target string, now time.Time) bool {
|
||||||
|
if r == nil || r.run == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
if r.last == nil {
|
||||||
|
r.last = map[string]time.Time{}
|
||||||
|
}
|
||||||
|
if t, ok := r.last[target]; ok && now.Sub(t) < storeGrantRepairMinInterval {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
r.last[target] = now
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// repair runs the EXISTING root wrapper's `grant` verb for this storage. It adds no privileged
|
||||||
|
// surface: `felhom-backup-target-apply grant *` is already in the sudoers allowlist for any storage
|
||||||
|
// id (configs/felhom-agent.sudoers), and the verb already grants BOTH the user and the token — a
|
||||||
|
// privsep token's rights are the intersection, so granting one of the two grants nothing usable.
|
||||||
|
//
|
||||||
|
// This is the pbsdr shape (internal/pbsdr/manager.go, the R-22 self-grant): on a refusal, run the
|
||||||
|
// root wrapper and RE-READ ONCE rather than dead-locking. Its restraint is copied too — one attempt,
|
||||||
|
// one confirmation, and anything still wrong stays loudly wrong.
|
||||||
|
func (r *storeGrantRepairer) repair(ctx context.Context, target string) error {
|
||||||
|
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
_, errOut, err := r.run(rctx, localapi.BackupTargetWrapperPath, "grant", target)
|
||||||
|
if err != nil {
|
||||||
|
r.log.Error("store-grant: SELF-REPAIR FAILED — the tier stays unreadable",
|
||||||
|
"target", target, "err", err, "stderr", strings.TrimSpace(string(errOut)))
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantRequiredPriv is the privilege whose ABSENCE was measured to blind the content listing.
|
||||||
|
//
|
||||||
|
// Measured on demo-felhom 2026-08-03: the two storages that list through the token hold
|
||||||
|
// Datastore.Allocate + Datastore.AllocateSpace (the FelhomAgentStore role); the one that answers
|
||||||
|
// empty holds only what the box-wide grant propagates (Sys.Audit, SDN.Use, Datastore.Audit). It is
|
||||||
|
// NOT Datastore.Audit that is missing — checking for that would report the blinded storage healthy.
|
||||||
|
const storeGrantRequiredPriv = "Datastore.AllocateSpace"
|
||||||
|
|
||||||
|
// storeGrantCritical decides whether a missing grant on this target is Critical (operator-paged).
|
||||||
|
//
|
||||||
|
// "local" is host-install's DEGRADED fallback target — a box with no dedicated backup storage is a
|
||||||
|
// known, ordinary configuration, and turning it into a critical alert is how a signal becomes
|
||||||
|
// something an operator archives unread. It is still probed and still reported; only the paging
|
||||||
|
// differs.
|
||||||
|
func storeGrantCritical(targetID string) bool { return targetID != "local" }
|
||||||
|
|
||||||
|
// storeGrantStatus is one tier's grant probe. It NEVER reports ok when it could not ask: a
|
||||||
|
// self-check that fails open is worse than none, because it converts "I do not know" into "fine".
|
||||||
|
func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string, critical bool, repair *storeGrantRepairer) capability.Status {
|
||||||
|
s := capability.Status{
|
||||||
|
Name: "pve:store-grant:" + targetID,
|
||||||
|
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
|
||||||
|
Critical: critical,
|
||||||
|
Status: capability.StatusOK,
|
||||||
|
}
|
||||||
|
if px == nil {
|
||||||
|
s.Status, s.Reason = capability.StatusDegraded, "not configured"
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
if targetID == "" {
|
||||||
|
s.Status, s.Reason = capability.StatusDegraded, "tier has no target id"
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
pctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
privs, err := px.Permissions(pctx, "/storage/"+targetID)
|
||||||
|
s = storeGrantVerdict(targetID, critical, privs, err)
|
||||||
|
if err != nil {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
if s.Status != capability.StatusDegraded {
|
||||||
|
// Healthy — but if this tier was repaired moments ago, keep REPORTING the transition until a
|
||||||
|
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||||
|
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||||
|
// comment).
|
||||||
|
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── R-190 mitigation: the grant is missing — repair it, and SAY that it was missing ──────────
|
||||||
|
//
|
||||||
|
// R-190 is a grant that demonstrably worked at 04:44 and was gone by 09:24, with a reinstall,
|
||||||
|
// logged pveum activity and cluster-log entries all ruled out. The cause is still open; the
|
||||||
|
// resilience does not have to wait for it. Everything needed already exists — the root wrapper,
|
||||||
|
// its sudoers vector for any storage id, and the exact command — and until now the `grant` verb
|
||||||
|
// had only ever been called at CREATION. That is the "built but never wired" shape, in a verb
|
||||||
|
// rather than a seam.
|
||||||
|
if !repair.mayAttempt(targetID, time.Now()) {
|
||||||
|
// Bounded (Scenario F): an earlier attempt did not hold and it is too soon to try again. Stay
|
||||||
|
// degraded and say why — a quiet "we already tried" is how a permanent fault becomes silence.
|
||||||
|
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||||
|
" and a self-repair was attempted within the last " + storeGrantRepairMinInterval.String() +
|
||||||
|
" without holding — NOT retrying yet; this needs a human"
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
if rerr := repair.repair(ctx, targetID); rerr != nil {
|
||||||
|
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||||
|
" and the self-repair FAILED (" + rerr.Error() + ") — this tier's archives are INVISIBLE to the agent"
|
||||||
|
return s // Scenario E: a failed repair must never mask the degraded state.
|
||||||
|
}
|
||||||
|
// Re-read ONCE to confirm, exactly as pbsdr does — the wrapper reporting success is a claim about
|
||||||
|
// its own write; the grant being readable is a different claim, and it is the one that matters.
|
||||||
|
cctx, ccancel := context.WithTimeout(ctx, 10*time.Second)
|
||||||
|
defer ccancel()
|
||||||
|
privs2, err2 := px.Permissions(cctx, "/storage/"+targetID)
|
||||||
|
if err2 != nil || privs2[storeGrantRequiredPriv] != 1 {
|
||||||
|
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
|
||||||
|
" and the self-repair did not take (re-read says it is still missing) — this needs a human"
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// REPAIRED — and reported as DEGRADED for exactly this one cycle, deliberately.
|
||||||
|
//
|
||||||
|
// The tier works again, so "ok" would be true of this instant and would throw away the only
|
||||||
|
// evidence that anything happened. R-190's own words: the probe sees the STATE, nothing sees the
|
||||||
|
// TRANSITION. A silent self-repair makes a recurring loss undetectable forever, which is strictly
|
||||||
|
// worse than the fault it fixes.
|
||||||
|
//
|
||||||
|
// §8.5 asked whether the hub's existing degraded↔ok edge suffices before building anything new.
|
||||||
|
// It does — as a CHANNEL — but only if the agent deliberately reports one degraded cycle: the hub
|
||||||
|
// alerts and e-mails on the ok→degraded edge and logs the degraded→ok recovery, so one loss
|
||||||
|
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||||
|
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||||
|
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||||
|
repair.noteRepaired(targetID, time.Now())
|
||||||
|
s = storeGrantRepairedVerdict(targetID, critical)
|
||||||
|
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||||
|
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||||
|
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantHealthyVerdict decides what a HEALTHY probe reports — which is not always "ok".
|
||||||
|
//
|
||||||
|
// Split out so the tests exercise this decision rather than a copy of it. An earlier version of this
|
||||||
|
// guard lived inline and its red-proof PASSED, because the test asserted the latch helper instead of
|
||||||
|
// the path that consumes it — the same hollow shape this file has now caught twice.
|
||||||
|
//
|
||||||
|
// If the tier was repaired inside the report window, the transition is reported even though the grant
|
||||||
|
// is present: the probe that repaired may have been a log-only one, and without this the host-report
|
||||||
|
// carries `ok` and the operator never learns the permission vanished (measured live 2026-08-04).
|
||||||
|
func storeGrantHealthyVerdict(targetID string, critical bool, healthy capability.Status, repairedRecently bool) capability.Status {
|
||||||
|
if repairedRecently {
|
||||||
|
return storeGrantRepairedVerdict(targetID, critical)
|
||||||
|
}
|
||||||
|
return healthy
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantRepairedVerdict is the post-repair verdict — the RECORD half of R-190, split out so the
|
||||||
|
// tests exercise the real thing rather than a copy of it (yesterday's hollow-test lesson).
|
||||||
|
//
|
||||||
|
// It reports DEGRADED although the tier now works, and that is the whole point: "ok" would be true of
|
||||||
|
// this instant and would throw away the only evidence that a permission vanished. The hub raises its
|
||||||
|
// ok→degraded edge (an operator e-mail) and logs the degraded→ok recovery on the next cycle, so one
|
||||||
|
// loss produces exactly one alert pair. Nothing new was built for this — no wire change, no hub
|
||||||
|
// change, no new event type.
|
||||||
|
//
|
||||||
|
// The explanation lives in FEATURE because that is the field the hub interpolates into the operator's
|
||||||
|
// e-mail (`monitor/host_capability.go` emitTransition builds its message from the capability names
|
||||||
|
// and features; Reason does not travel). Putting it in Reason alone would be a record nobody reads.
|
||||||
|
func storeGrantRepairedVerdict(targetID string, critical bool) capability.Status {
|
||||||
|
return capability.Status{
|
||||||
|
Name: "pve:store-grant:" + targetID,
|
||||||
|
Critical: critical,
|
||||||
|
Status: capability.StatusDegraded,
|
||||||
|
Feature: "backup tier " + targetID + ": the agent's storage grant was MISSING and has been " +
|
||||||
|
"AUTOMATICALLY RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)",
|
||||||
|
Reason: "grant absent at probe time; `felhom-backup-target-apply grant " + targetID +
|
||||||
|
"` re-applied it and a re-read confirms " + storeGrantRequiredPriv + " is present again",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// repairLogger returns the repairer's logger, or the default — the record must survive a nil.
|
||||||
|
func repairLogger(r *storeGrantRepairer) *slog.Logger {
|
||||||
|
if r != nil && r.log != nil {
|
||||||
|
return r.log
|
||||||
|
}
|
||||||
|
return slog.Default()
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantVerdict is the DECISION, split out from the API call so the tests exercise the real
|
||||||
|
// thing rather than a copy of it. A test that re-implements this branch would pass while production
|
||||||
|
// diverged — which is the hollow shape this project keeps finding in its own tests.
|
||||||
|
func storeGrantVerdict(targetID string, critical bool, privs map[string]int, err error) capability.Status {
|
||||||
|
s := capability.Status{
|
||||||
|
Name: "pve:store-grant:" + targetID,
|
||||||
|
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
|
||||||
|
Critical: critical,
|
||||||
|
Status: capability.StatusOK,
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
// Unreachable PVE is UNKNOWN, and unknown is reported as degraded rather than ok: a
|
||||||
|
// self-check that fails open converts "I do not know" into "fine".
|
||||||
|
s.Status, s.Reason = capability.StatusDegraded, "could not read own permissions: "+err.Error()
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
if privs[storeGrantRequiredPriv] != 1 {
|
||||||
|
// Name the storage AND the missing role: "a storage grant is missing" without saying which
|
||||||
|
// one costs a diagnosis at 07:00.
|
||||||
|
s.Status, s.Reason = capability.StatusDegraded,
|
||||||
|
"the agent token lacks "+storeGrantRequiredPriv+" on /storage/"+targetID+
|
||||||
|
" (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested"
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
// logCapabilities logs the privileged-capability self-check at startup: one INFO summary, plus an
|
// logCapabilities logs the privileged-capability self-check at startup: one INFO summary, plus an
|
||||||
// ERROR per degraded capability naming the gated feature (so a missing grant is loud at cutover,
|
// ERROR per degraded capability naming the gated feature (so a missing grant is loud at cutover,
|
||||||
// not days later). Inactive (config-gated off, plumbing healthy — v0.86.0) is counted in the
|
// not days later). Inactive (config-gated off, plumbing healthy — v0.86.0) is counted in the
|
||||||
@@ -492,8 +805,19 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
// A1 (v0.62.0): compose the PVE pool-read check AROUND the sudo prober (an API read does not
|
// A1 (v0.62.0): compose the PVE pool-read check AROUND the sudo prober (an API read does not
|
||||||
// belong inside the sudo-policy probe). Non-critical: a degraded pool read means the stale-lock
|
// belong inside the sudo-policy probe). Non-critical: a degraded pool read means the stale-lock
|
||||||
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
|
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
|
||||||
|
// R-185: the store-grant probes compose around the sudo prober the same way the pool read does
|
||||||
|
// (an API read does not belong inside the sudo-policy probe — the v0.62.0 A1 precedent).
|
||||||
|
// R-190: the store-grant probe also REPAIRS a missing grant, through the root wrapper that
|
||||||
|
// already exists and is already sudoers-permitted for any storage id — and reports the loss.
|
||||||
|
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||||
|
// invoked through the privileged path, which prepends sudo itself.
|
||||||
|
grantRepairer := &storeGrantRepairer{
|
||||||
|
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||||
|
log: logger,
|
||||||
|
}
|
||||||
probeAll := func(ctx context.Context) []capability.Status {
|
probeAll := func(ctx context.Context) []capability.Status {
|
||||||
return append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
||||||
|
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
|
||||||
}
|
}
|
||||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||||
// already carries the gated view — v0.86.0.)
|
// already carries the gated view — v0.86.0.)
|
||||||
@@ -662,6 +986,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||||
heavyOps := &backup.InFlight{}
|
heavyOps := &backup.InFlight{}
|
||||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
||||||
|
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
||||||
|
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
||||||
|
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
||||||
|
// for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB
|
||||||
|
// offsite restore-test reached no host-report at all.
|
||||||
|
collector.SetProvenRestoreTests(rtState)
|
||||||
|
|
||||||
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
|
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
|
||||||
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
|
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
|
||||||
@@ -769,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
return false
|
return false
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||||
if localTokens != nil {
|
if localTokens != nil {
|
||||||
defer localTokens.Close()
|
defer localTokens.Close()
|
||||||
}
|
}
|
||||||
@@ -1347,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
|||||||
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
|
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
|
||||||
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
|
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
|
||||||
// fixed. The opened token store is returned via outTokens so the caller can Close it.
|
// fixed. The opened token store is returned via outTokens so the caller can Close it.
|
||||||
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
|
||||||
if !cfg.LocalAPI.Enabled() {
|
if !cfg.LocalAPI.Enabled() {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -1419,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
gaMode = proxmox.RunnerSudo
|
gaMode = proxmox.RunnerSudo
|
||||||
}
|
}
|
||||||
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
|
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
|
||||||
|
// R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's
|
||||||
|
// own hub client (per-host key, self-scoped server-side), so the recoverer can never read another
|
||||||
|
// host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config
|
||||||
|
// cannot reach this line (the daemon exits above), so the seam is always live in production —
|
||||||
|
// which is the point: links 6 and 7 spent months existing without a caller.
|
||||||
|
escrowRecoverer := escrow.OffsiteKeyRecoverer{
|
||||||
|
Fetch: func(ctx context.Context) ([]byte, bool, error) {
|
||||||
|
resp, ferr := hubClient.FetchIdentityEscrow(ctx)
|
||||||
|
if ferr != nil {
|
||||||
|
return nil, false, ferr
|
||||||
|
}
|
||||||
|
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||||
|
if derr != nil {
|
||||||
|
return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)")
|
||||||
|
}
|
||||||
|
return blob, true, nil
|
||||||
|
},
|
||||||
|
}
|
||||||
srv, err := localapi.NewServer(localapi.Options{
|
srv, err := localapi.NewServer(localapi.Options{
|
||||||
|
EscrowRecovery: escrowRecoverer,
|
||||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||||
Cert: cert,
|
Cert: cert,
|
||||||
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
|
||||||
@@ -2541,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger *
|
|||||||
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
|
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest)
|
// R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was
|
||||||
|
// accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the
|
||||||
|
// offsite repository password into the same bundle. Anyone reading the old output would conclude the
|
||||||
|
// repository password was not there, and that is part of how the chain's extraction link came to be
|
||||||
|
// described as missing for a month. Name what was recovered from THIS bundle, and name what is
|
||||||
|
// absent, rather than reciting a fixed list.
|
||||||
|
recovered := []string{"tunnel_token", "pbs_token"}
|
||||||
|
var absent []string
|
||||||
|
if bundle.WGPrivateKey != "" {
|
||||||
|
recovered = append(recovered, "wg_private_key")
|
||||||
|
} else {
|
||||||
|
absent = append(absent, "wg_private_key")
|
||||||
|
}
|
||||||
|
if bundle.ResticRepoPassword != "" {
|
||||||
|
recovered = append(recovered, "restic_repo_password")
|
||||||
|
} else {
|
||||||
|
absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)")
|
||||||
|
}
|
||||||
|
fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest)
|
||||||
|
if len(absent) > 0 {
|
||||||
|
fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; "))
|
||||||
|
}
|
||||||
|
|
||||||
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
|
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
|
||||||
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
|
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
|
||||||
|
|||||||
@@ -111,3 +111,46 @@ func parseMainForWiring(t *testing.T) *ast.File {
|
|||||||
}
|
}
|
||||||
return f
|
return f
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
|
||||||
|
//
|
||||||
|
// This test exists because the method it feeds is the project's own cautionary tale:
|
||||||
|
// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
|
||||||
|
// was written and **had no caller at all** — a seam built, documented and never connected, found
|
||||||
|
// only when a live restore-test's PASS reached no host-report. The fix must not become the next
|
||||||
|
// instance, so the wiring is asserted rather than trusted.
|
||||||
|
//
|
||||||
|
// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
|
||||||
|
// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
|
||||||
|
func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
|
||||||
|
f := parseMainForWiring(t)
|
||||||
|
|
||||||
|
var wired, feedsState bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
call, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||||
|
if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
wired = true
|
||||||
|
// ...and it must be fed the PERSISTED state, not the in-memory store.
|
||||||
|
if len(call.Args) == 1 {
|
||||||
|
if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
|
||||||
|
feedsState = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
|
||||||
|
if !wired {
|
||||||
|
t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
|
||||||
|
"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
|
||||||
|
}
|
||||||
|
if wired && !feedsState {
|
||||||
|
t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
|
||||||
|
"that does NOT survive a restart, so wiring it here would fix nothing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,417 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"go/ast"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-185 — a tier the box cannot READ must say so.
|
||||||
|
//
|
||||||
|
// THE OBSERVATION (demo-felhom, 2026-08-03, reproduced at the start of this session): root lists
|
||||||
|
// three archives on `felhom-backup`; the agent's own token gets `{"data":[]}` from the same
|
||||||
|
// endpoint; and `local`, which has the grant, lists through that same token. The token is the
|
||||||
|
// variable, not the storage.
|
||||||
|
//
|
||||||
|
// The defect is NOT the missing grant — that is one command. It is that an empty content listing is
|
||||||
|
// what a FORBIDDEN tier and a NEWBORN tier both return, so the box could not tell them apart and
|
||||||
|
// said nothing. These tests pin the distinction.
|
||||||
|
|
||||||
|
// permAnswer is the shape /access/permissions really returns, taken from the live measurement:
|
||||||
|
// an UNGRANTED path answers with the privileges inherited from the box-wide grant — NOT empty, and
|
||||||
|
// NOT a 403.
|
||||||
|
var (
|
||||||
|
permGranted = map[string]int{"Datastore.Allocate": 1, "Datastore.AllocateSpace": 1}
|
||||||
|
permUngranted = map[string]int{"Sys.Audit": 1, "SDN.Use": 1, "Datastore.Audit": 1}
|
||||||
|
)
|
||||||
|
|
||||||
|
// probeWith calls the PRODUCTION decision with a permissions answer. **Naming the seam:** everything
|
||||||
|
// below is true up to `storeGrantVerdict`; that the live call feeds it the real API answer is what
|
||||||
|
// Part 0's measurement established and what the live run on the box demonstrates. An earlier draft
|
||||||
|
// of this file re-implemented the branch here — it passed, and would have kept passing while
|
||||||
|
// production diverged, which is the hollow shape this project keeps catching in its own tests.
|
||||||
|
func probeWith(privs map[string]int, targetID string, critical bool) capability.Status {
|
||||||
|
return storeGrantVerdict(targetID, critical, privs, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO A — a forbidden storage is REPORTED, not passed over ────────────────────────────
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-03): delete the store-grant probes from `probeAll` in
|
||||||
|
// main.go — i.e. restore `append(capProber.Probe(ctx), poolReadStatus(ctx, px))` — and
|
||||||
|
// TestMainWiresTheStoreGrantProbe fails with "main.go never calls storeGrantStatuses". That is
|
||||||
|
// today's behaviour on the live box: complete silence about a tier it cannot read.
|
||||||
|
func TestStoreGrant_ForbiddenStorageIsDegradedAndNamed(t *testing.T) {
|
||||||
|
s := probeWith(permUngranted, "felhom-backup", true)
|
||||||
|
|
||||||
|
if s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("a storage the agent may not read must be DEGRADED, not %q — silence is the defect", s.Status)
|
||||||
|
}
|
||||||
|
if !s.Critical {
|
||||||
|
t.Fatal("it must be CRITICAL: the hub alerts only on critical, so a non-critical entry is the same silence with extra steps")
|
||||||
|
}
|
||||||
|
if !strings.Contains(s.Reason, "felhom-backup") {
|
||||||
|
t.Fatalf("the reason must NAME the storage — 'a grant is missing' costs a diagnosis at 07:00; got %q", s.Reason)
|
||||||
|
}
|
||||||
|
if !strings.Contains(s.Reason, "FelhomAgentStore") {
|
||||||
|
t.Fatalf("the reason must name the ROLE to grant, so the fix is in the alert; got %q", s.Reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE TRAP THE LIVE MEASUREMENT CAUGHT, pinned so it cannot be re-introduced: the ungranted answer
|
||||||
|
// is not empty and not a 403 — it carries the INHERITED box-wide privileges. A probe that asked
|
||||||
|
// "did the path come back?" or "does it have Datastore.Audit?" would report the blinded storage
|
||||||
|
// healthy.
|
||||||
|
func TestStoreGrant_InheritedPrivilegesAreNotAGrant(t *testing.T) {
|
||||||
|
if len(permUngranted) == 0 {
|
||||||
|
t.Fatal("fixture wrong: the ungranted answer is NOT empty — that is the whole trap")
|
||||||
|
}
|
||||||
|
if permUngranted["Datastore.Audit"] != 1 {
|
||||||
|
t.Fatal("fixture wrong: the ungranted path DOES carry Datastore.Audit, inherited box-wide")
|
||||||
|
}
|
||||||
|
if s := probeWith(permUngranted, "felhom-backup", true); s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("checking for the wrong privilege reports a blinded storage healthy; got %q", s.Status)
|
||||||
|
}
|
||||||
|
// ...and the privilege actually checked is the one whose absence was measured to blind listing.
|
||||||
|
if storeGrantRequiredPriv != "Datastore.AllocateSpace" {
|
||||||
|
t.Fatalf("the probed privilege changed to %q — re-measure before trusting it", storeGrantRequiredPriv)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO B — a newborn tier is still silent ──────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// A storage the agent IS allowed to read but which simply holds no archives yet is HEALTHY. The
|
||||||
|
// probe must not look at content at all, or every freshly provisioned box alarms and the signal dies.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): make the probe degrade on an empty content listing instead of on
|
||||||
|
// the permission — a granted-but-empty storage then reports degraded, i.e. every newborn box alarms.
|
||||||
|
func TestStoreGrant_GrantedButEmptyIsHealthy(t *testing.T) {
|
||||||
|
s := probeWith(permGranted, "felhom-pbs", true)
|
||||||
|
if s.Status != capability.StatusOK {
|
||||||
|
t.Fatalf("a readable tier is healthy whether or not it holds archives yet; got %q (%s)", s.Status, s.Reason)
|
||||||
|
}
|
||||||
|
if s.Reason != "" {
|
||||||
|
t.Fatalf("a healthy probe carries no reason; got %q", s.Reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO C — the two states are distinguishable at a glance ──────────────────────────────
|
||||||
|
func TestStoreGrant_ForbiddenAndNewbornAreDistinguishable(t *testing.T) {
|
||||||
|
forbidden := probeWith(permUngranted, "felhom-backup", true)
|
||||||
|
newborn := probeWith(permGranted, "felhom-pbs", true)
|
||||||
|
|
||||||
|
if forbidden.Status == newborn.Status {
|
||||||
|
t.Fatalf("the two states must differ — today both read as 'no settled archive yet'; got %q for both", forbidden.Status)
|
||||||
|
}
|
||||||
|
if forbidden.Name == newborn.Name {
|
||||||
|
t.Fatalf("each tier needs its own capability id, or one tier's fault hides another's; got %q twice", forbidden.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// §8.3, weighed once and pinned: a box with NO dedicated target ("local" — host-install's own
|
||||||
|
// DEGRADED fallback) must not turn an ordinary configuration into an operator page. It is still
|
||||||
|
// probed and still reported; only the paging differs.
|
||||||
|
func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
|
||||||
|
if storeGrantCritical("local") {
|
||||||
|
t.Fatal("a box whose backup target is the 'local' fallback must not page the operator about " +
|
||||||
|
"an ordinary, documented configuration")
|
||||||
|
}
|
||||||
|
for _, dedicated := range []string{"felhom-backup", "felhom-pbs", "some-nvme"} {
|
||||||
|
if !storeGrantCritical(dedicated) {
|
||||||
|
t.Fatalf("a DEDICATED target that cannot be read is user-facing and must be critical; %q was not", dedicated)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The fallback is still reported — silence for it would be the original defect, scoped smaller.
|
||||||
|
if s := probeWith(permUngranted, "local", storeGrantCritical("local")); s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("the fallback target must still report degraded when unreadable; got %q", s.Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
|
||||||
|
// probe, because it looks like coverage.
|
||||||
|
func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
|
||||||
|
s := storeGrantStatus(context.Background(), nil, "felhom-backup", true, nil)
|
||||||
|
if s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
|
||||||
|
}
|
||||||
|
if s.Reason == "" {
|
||||||
|
t.Fatal("it must say why it could not ask")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// This project's "built but never wired" count reached six last week. The fix for a SILENCE must not
|
||||||
|
// itself be silent. AST, not grep: a commented-out call still contains the string.
|
||||||
|
func TestMainWiresTheStoreGrantProbe(t *testing.T) {
|
||||||
|
f := parseMainForWiring(t)
|
||||||
|
|
||||||
|
var wired bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
call, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" {
|
||||||
|
wired = true
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !wired {
|
||||||
|
t.Error("main.go never calls storeGrantStatuses — the probe would exist and report to nobody, " +
|
||||||
|
"which is precisely the silence R-185 is about")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── R-190 — the grant repairs itself, and the repair is VISIBLE ──────────────────────────────
|
||||||
|
//
|
||||||
|
// R-190 is a storage grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24,
|
||||||
|
// with a host reinstall, logged `pveum` activity and cluster-log entries all ruled out. The cause is
|
||||||
|
// open; the resilience is not conditional on it.
|
||||||
|
//
|
||||||
|
// The half that matters is the RECORD. R-190's own words: the probe sees the state, nothing sees the
|
||||||
|
// transition. A self-repair that leaves only "ok" behind destroys the only evidence a loss happened,
|
||||||
|
// so a recurring loss becomes undetectable forever — strictly worse than the fault it fixes.
|
||||||
|
|
||||||
|
// fakeRepairRunner records wrapper invocations and can be made to fail.
|
||||||
|
type fakeRepairRunner struct {
|
||||||
|
calls [][]string
|
||||||
|
fail bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeRepairRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
f.calls = append(f.calls, append([]string{name}, args...))
|
||||||
|
if f.fail {
|
||||||
|
return nil, []byte("pveum: refused"), errors.New("exit status 2")
|
||||||
|
}
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRepairer(f *fakeRepairRunner) *storeGrantRepairer {
|
||||||
|
return &storeGrantRepairer{run: f.Run, log: slog.New(slog.NewTextHandler(io.Discard, nil))}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO F — the repair is BOUNDED ───────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-04): make mayAttempt always return true (drop the
|
||||||
|
// storeGrantRepairMinInterval check) →
|
||||||
|
//
|
||||||
|
// --- FAIL: TestGrantRepair_IsBounded
|
||||||
|
// storegrant_test.go: a repair must not run on every cycle; 5 cycles produced 5 attempt(s)
|
||||||
|
//
|
||||||
|
// which is a re-grant every report cycle, forever, against a fault an ACL cannot fix. Restored.
|
||||||
|
func TestGrantRepair_IsBounded(t *testing.T) {
|
||||||
|
f := &fakeRepairRunner{}
|
||||||
|
r := newRepairer(f)
|
||||||
|
// Jittered, so the series never lands exactly on the interval boundary — a perfectly regular
|
||||||
|
// series is how a threshold test passes its own mutation, which has happened here before.
|
||||||
|
base := time.Date(2026, 8, 4, 9, 17, 43, 0, time.UTC)
|
||||||
|
offsets := []time.Duration{0, 13*time.Minute + 7*time.Second, 27*time.Minute + 51*time.Second,
|
||||||
|
41*time.Minute + 19*time.Second, 55*time.Minute + 3*time.Second}
|
||||||
|
attempts := 0
|
||||||
|
for _, off := range offsets {
|
||||||
|
if r.mayAttempt("felhom-backup", base.Add(off)) {
|
||||||
|
attempts++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if attempts != 1 {
|
||||||
|
t.Fatalf("a repair must not run on every cycle; %d cycles produced %d attempt(s) within %s",
|
||||||
|
len(offsets), attempts, storeGrantRepairMinInterval)
|
||||||
|
}
|
||||||
|
// ...and once the interval has genuinely passed, it may try again — a bound is not a ban.
|
||||||
|
if !r.mayAttempt("felhom-backup", base.Add(storeGrantRepairMinInterval+2*time.Minute+11*time.Second)) {
|
||||||
|
t.Fatal("after the interval a repair must be allowed again — otherwise one failure disables the repair forever")
|
||||||
|
}
|
||||||
|
// A DIFFERENT tier is not throttled by this one's attempt.
|
||||||
|
if !r.mayAttempt("felhom-pbs", base.Add(time.Minute)) {
|
||||||
|
t.Fatal("the bound must be per tier — one tier's attempt must not suppress another's")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A nil repairer (or one with no runner) never attempts, and never panics.
|
||||||
|
func TestGrantRepair_NilIsSafe(t *testing.T) {
|
||||||
|
var r *storeGrantRepairer
|
||||||
|
if r.mayAttempt("felhom-backup", time.Now()) {
|
||||||
|
t.Fatal("a nil repairer must never claim an attempt")
|
||||||
|
}
|
||||||
|
if (&storeGrantRepairer{}).mayAttempt("felhom-backup", time.Now()) {
|
||||||
|
t.Fatal("a repairer with no runner must never claim an attempt")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The repair calls the EXISTING wrapper verb, with the storage id — no new privileged surface.
|
||||||
|
func TestGrantRepair_CallsTheExistingWrapperVerb(t *testing.T) {
|
||||||
|
f := &fakeRepairRunner{}
|
||||||
|
r := newRepairer(f)
|
||||||
|
if err := r.repair(context.Background(), "felhom-backup"); err != nil {
|
||||||
|
t.Fatalf("repair should succeed with a healthy runner: %v", err)
|
||||||
|
}
|
||||||
|
if len(f.calls) != 1 {
|
||||||
|
t.Fatalf("exactly one wrapper invocation expected; got %d", len(f.calls))
|
||||||
|
}
|
||||||
|
got := f.calls[0]
|
||||||
|
want := []string{"/usr/local/sbin/felhom-backup-target-apply", "grant", "felhom-backup"}
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("wrapper argv = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
t.Fatalf("wrapper argv = %v, want %v — the sudoers vector is `grant *`; anything else is a policy change", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A repair that FAILS must surface the failure, not swallow it (Scenario E's precondition).
|
||||||
|
func TestGrantRepair_FailureIsReturned(t *testing.T) {
|
||||||
|
f := &fakeRepairRunner{fail: true}
|
||||||
|
if err := newRepairer(f).repair(context.Background(), "felhom-backup"); err == nil {
|
||||||
|
t.Fatal("a failed wrapper run must return its error — a repair that cannot run must never read as done")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO D (the half that matters) — the REPAIR MUST BE VISIBLE ──────────────────────────
|
||||||
|
//
|
||||||
|
// A repair that leaves only "ok" behind is worse than the fault: the tier works, and the fact that a
|
||||||
|
// permission vanished is gone with it. R-190 exists because nothing saw the transition.
|
||||||
|
//
|
||||||
|
// The channel is the hub's EXISTING ok→degraded→ok edge (§8.5) — nothing new was built. That only
|
||||||
|
// works if the agent deliberately reports ONE degraded cycle after repairing, and if the explanation
|
||||||
|
// rides the field the hub actually puts in the operator's e-mail. The hub's message is built from the
|
||||||
|
// capability NAME and FEATURE (`internal/monitor/host_capability.go` emitTransition) — **not** from
|
||||||
|
// Reason — so the Feature must carry it.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-04): after a successful repair, report ok instead —
|
||||||
|
//
|
||||||
|
// s.Status = capability.StatusOK; s.Feature unchanged
|
||||||
|
//
|
||||||
|
// → --- FAIL: TestGrantRepair_ARepairedGrantIsReportedAsATransition
|
||||||
|
//
|
||||||
|
// storegrant_test.go: a self-repair must still report DEGRADED for one cycle so the hub raises
|
||||||
|
// its edge; got "ok" — the loss would be invisible
|
||||||
|
//
|
||||||
|
// i.e. exactly the silence R-190 is about. Restored.
|
||||||
|
func TestGrantRepair_ARepairedGrantIsReportedAsATransition(t *testing.T) {
|
||||||
|
// THE PRODUCTION verdict, not a copy of it. An earlier draft of this test built the Status
|
||||||
|
// itself and asserted its own construction — it would have passed while production reported ok,
|
||||||
|
// which is precisely the silence being guarded against.
|
||||||
|
if pre := probeWith(permUngranted, "felhom-backup", true); pre.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("precondition: a missing grant is degraded; got %q", pre.Status)
|
||||||
|
}
|
||||||
|
s := storeGrantRepairedVerdict("felhom-backup", true)
|
||||||
|
|
||||||
|
if s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("a self-repair must still report DEGRADED for one cycle so the hub raises its edge; "+
|
||||||
|
"got %q — the loss would be invisible", s.Status)
|
||||||
|
}
|
||||||
|
// The hub e-mails the FEATURE text. If the explanation is not there, the operator is told a
|
||||||
|
// capability was degraded and never learns it repaired itself or that anything vanished.
|
||||||
|
for _, want := range []string{"MISSING", "RESTORED", "felhom-backup", "R-190"} {
|
||||||
|
if !strings.Contains(s.Feature, want) {
|
||||||
|
t.Fatalf("the Feature text is what the hub puts in the operator's e-mail; it must contain %q. Got: %s", want, s.Feature)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !s.Critical {
|
||||||
|
t.Fatal("the transition must be CRITICAL or the hub does not alert on it at all")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// The wrapper's `grant` verb is itself a "built but never wired" example: it exists, is
|
||||||
|
// sudoers-permitted for any id, and had only ever been called at storage CREATION. The repair must
|
||||||
|
// not become the seventh instance. AST, not grep — a commented-out call still contains the string.
|
||||||
|
func TestMainWiresTheGrantRepair(t *testing.T) {
|
||||||
|
f := parseMainForWiring(t)
|
||||||
|
|
||||||
|
var built, passed bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
switch node := n.(type) {
|
||||||
|
case *ast.CompositeLit:
|
||||||
|
if id, ok := node.Type.(*ast.Ident); ok && id.Name == "storeGrantRepairer" {
|
||||||
|
built = true
|
||||||
|
}
|
||||||
|
case *ast.CallExpr:
|
||||||
|
if id, ok := node.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" && len(node.Args) == 4 {
|
||||||
|
if a, ok := node.Args[3].(*ast.Ident); ok && a.Name == "grantRepairer" {
|
||||||
|
passed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !built {
|
||||||
|
t.Error("main.go never constructs a storeGrantRepairer — nothing would ever repair a lost grant")
|
||||||
|
}
|
||||||
|
if !passed {
|
||||||
|
t.Error("storeGrantStatuses is not passed the repairer — the probe would detect the loss and " +
|
||||||
|
"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The transition must survive a probe that is NOT the one feeding the hub.
|
||||||
|
//
|
||||||
|
// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
|
||||||
|
// production while every unit test passed: `probeAll` is called independently by the self-check LOG
|
||||||
|
// and by the collector building a host-report. The repairing call was the log's; the report three
|
||||||
|
// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
|
||||||
|
// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
|
||||||
|
//
|
||||||
|
// --- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
|
||||||
|
// storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
|
||||||
|
// the host-report would carry ok and the operator would never learn the grant vanished
|
||||||
|
//
|
||||||
|
// Restored.
|
||||||
|
func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||||
|
r := newRepairer(&fakeRepairRunner{})
|
||||||
|
// Jittered, never landing on the window boundary.
|
||||||
|
repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
|
||||||
|
r.noteRepaired("felhom-backup", repairedAt)
|
||||||
|
|
||||||
|
// The DECISION a later probe makes — the production function, not the helper it calls. An
|
||||||
|
// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
|
||||||
|
// the latch's USE left the helper untouched.
|
||||||
|
healthy := probeWith(permGranted, "felhom-backup", true)
|
||||||
|
if healthy.Status != capability.StatusOK {
|
||||||
|
t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
|
||||||
|
}
|
||||||
|
got := storeGrantHealthyVerdict("felhom-backup", true,
|
||||||
|
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
|
||||||
|
if got.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
|
||||||
|
"would carry ok and the operator would never learn the grant vanished", got.Status)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got.Feature, "RESTORED") {
|
||||||
|
t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
|
||||||
|
}
|
||||||
|
// Outside the window it reports plain ok again.
|
||||||
|
late := storeGrantHealthyVerdict("felhom-backup", true,
|
||||||
|
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
|
||||||
|
if late.Status != capability.StatusOK {
|
||||||
|
t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
|
||||||
|
"would be its own false alarm", late.Status)
|
||||||
|
}
|
||||||
|
if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
|
||||||
|
t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
|
||||||
|
}
|
||||||
|
// ...and it clears on its own rather than latching a box degraded forever.
|
||||||
|
if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
|
||||||
|
t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
|
||||||
|
}
|
||||||
|
// It is per tier.
|
||||||
|
if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
|
||||||
|
t.Fatal("one tier's repair must not latch another tier's status")
|
||||||
|
}
|
||||||
|
// The window MUST exceed the report interval — the property, asserted rather than assumed.
|
||||||
|
if storeGrantRepairReportWindow <= 15*time.Minute {
|
||||||
|
t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
|
||||||
|
"be missed entirely", storeGrantRepairReportWindow)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -140,3 +140,29 @@ func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time
|
|||||||
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
|
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
|
||||||
return s.evaluateTier(ctx, target, s.settleCutoff())
|
return s.evaluateTier(ctx, target, s.settleCutoff())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
|
||||||
|
//
|
||||||
|
// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
|
||||||
|
// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
|
||||||
|
// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
|
||||||
|
// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
|
||||||
|
// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
|
||||||
|
func (s *Scheduler) verdictSummary(ctx context.Context) string {
|
||||||
|
out := ""
|
||||||
|
for _, v := range s.EvaluateDue(ctx) {
|
||||||
|
if out != "" {
|
||||||
|
out += "; "
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case v.Err != nil:
|
||||||
|
out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
|
||||||
|
default:
|
||||||
|
out += v.Target + ": " + v.Reason
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if out == "" {
|
||||||
|
return "no tiers configured"
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,8 +4,10 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -124,8 +126,8 @@ func dailyArchives(tier string, n int) []archiveStub {
|
|||||||
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
|
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
|
||||||
// was replaced by the naive age rule:
|
// was replaced by the naive age rule:
|
||||||
//
|
//
|
||||||
// - if ok && proven == archive { … not due … }
|
// - if ok && proven == archive { … not due … }
|
||||||
// + if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
|
// - if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
|
||||||
//
|
//
|
||||||
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
|
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
|
||||||
// old enough". Result:
|
// old enough". Result:
|
||||||
@@ -194,12 +196,13 @@ func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
|
|||||||
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
|
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
|
||||||
// ProvenArchive ignore the stored archive —
|
// ProvenArchive ignore the stored archive —
|
||||||
//
|
//
|
||||||
// - if !ok || p.Archive == "" { return "", false }
|
// - if !ok || p.Archive == "" { return "", false }
|
||||||
// + return "", false // per-tier time only, the pre-R-86 state
|
// - return "", false // per-tier time only, the pre-R-86 state
|
||||||
//
|
//
|
||||||
// → --- FAIL: TestDue_RestartRunsNothing
|
// → --- FAIL: TestDue_RestartRunsNothing
|
||||||
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
|
//
|
||||||
// produced 4 run(s)
|
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
|
||||||
|
// produced 4 run(s)
|
||||||
//
|
//
|
||||||
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
|
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
|
||||||
// today's behaviour with the ticker's phase reset by the deploy. Restored.
|
// today's behaviour with the ticker's phase reset by the deploy. Restored.
|
||||||
@@ -256,12 +259,13 @@ func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
|
|||||||
//
|
//
|
||||||
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
|
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
|
||||||
//
|
//
|
||||||
// - if rt.Pass && s.rtState != nil && target != "" {
|
// - if rt.Pass && s.rtState != nil && target != "" {
|
||||||
// + if s.rtState != nil && target != "" {
|
// - if s.rtState != nil && target != "" {
|
||||||
//
|
//
|
||||||
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
|
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
|
||||||
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
|
//
|
||||||
// evaluations
|
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
|
||||||
|
// evaluations
|
||||||
//
|
//
|
||||||
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
|
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
|
||||||
// freshly verified, which is the loudest signal this system produces going silent. Restored.
|
// freshly verified, which is the loudest signal this system produces going silent. Restored.
|
||||||
@@ -438,7 +442,7 @@ func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
|
|||||||
path := filepath.Join(t.TempDir(), "rt.json")
|
path := filepath.Join(t.TempDir(), "rt.json")
|
||||||
now := time.Now().UTC().Truncate(time.Second)
|
now := time.Now().UTC().Truncate(time.Second)
|
||||||
st := NewRestoreTestState(path)
|
st := NewRestoreTestState(path)
|
||||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
|
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
re := NewRestoreTestState(path)
|
re := NewRestoreTestState(path)
|
||||||
@@ -456,3 +460,134 @@ func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
|
|||||||
func writeFileForTest(path, content string) error {
|
func writeFileForTest(path, content string) error {
|
||||||
return os.WriteFile(path, []byte(content), 0o600)
|
return os.WriteFile(path, []byte(content), 0o600)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
|
||||||
|
// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
|
||||||
|
// quiet journal is equally consistent with a healthy loop and a dead goroutine.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
|
||||||
|
// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
|
||||||
|
// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
|
||||||
|
func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
|
||||||
|
ts := &tierStorage{archives: map[string][]archiveStub{
|
||||||
|
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
|
||||||
|
"felhom-pbs": nil, // no archive at all
|
||||||
|
}}
|
||||||
|
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||||
|
// Prove the local tier so NOTHING is due.
|
||||||
|
if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
|
||||||
|
// test would pass against a tick that never calls it.
|
||||||
|
var logbuf strings.Builder
|
||||||
|
h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||||
|
h.s.tick(context.Background())
|
||||||
|
got := logbuf.String()
|
||||||
|
for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
|
||||||
|
if !strings.Contains(got, want) {
|
||||||
|
t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
|
||||||
|
// reads as "nothing due" is the silence this rule exists to prevent.
|
||||||
|
func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
|
||||||
|
ts := &tierStorage{
|
||||||
|
archives: map[string][]archiveStub{"local": nil},
|
||||||
|
err: map[string]error{"felhom-pbs": errors.New("storage unreachable")},
|
||||||
|
}
|
||||||
|
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
|
||||||
|
got := h.s.verdictSummary(context.Background())
|
||||||
|
if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
|
||||||
|
t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
|
||||||
|
//
|
||||||
|
// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
|
||||||
|
// agent will not repeat the work. So the persisted record has to be able to become a host-report
|
||||||
|
// entry — without inventing anything it does not know.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
|
||||||
|
// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
|
||||||
|
//
|
||||||
|
// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
|
||||||
|
// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
|
||||||
|
// per-tier proof on it); got [{... SourceTier: ...}]
|
||||||
|
//
|
||||||
|
// Restored.
|
||||||
|
func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "rt.json")
|
||||||
|
// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
|
||||||
|
// file has ever had, which is what a real box carries after two upgrades.
|
||||||
|
legacy := `{
|
||||||
|
"old-v1": "2026-07-30T02:11:07Z",
|
||||||
|
"old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
|
||||||
|
"felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
|
||||||
|
}`
|
||||||
|
if err := writeFileForTest(path, legacy); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
|
||||||
|
}
|
||||||
|
e := got[0]
|
||||||
|
if e.SourceTier != "pbs" {
|
||||||
|
t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
|
||||||
|
}
|
||||||
|
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
|
||||||
|
t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
|
||||||
|
}
|
||||||
|
if e.TestedAt != "2026-08-03T13:25:14Z" {
|
||||||
|
t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
|
||||||
|
}
|
||||||
|
if e.Verified != "boot+running" {
|
||||||
|
t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
|
||||||
|
}
|
||||||
|
// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
|
||||||
|
if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
|
||||||
|
t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
|
||||||
|
e.DurationSeconds, e.ScratchVMID)
|
||||||
|
}
|
||||||
|
// The legacy records still serve the DUE-check, which is a separate question from reporting.
|
||||||
|
if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
|
||||||
|
t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
|
||||||
|
// the production path, not a hand-built fixture.
|
||||||
|
func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
|
||||||
|
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||||
|
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
|
||||||
|
// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
|
||||||
|
h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
|
||||||
|
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
|
||||||
|
}
|
||||||
|
h.s.tick(context.Background())
|
||||||
|
|
||||||
|
got := h.st.ProvenRestoreTests(context.Background())
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
|
||||||
|
}
|
||||||
|
if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
|
||||||
|
t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
|
||||||
|
func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
|
||||||
|
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
|
||||||
|
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
|
||||||
|
h.s.tick(context.Background())
|
||||||
|
if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
|
||||||
|
t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
|
||||||
|
"would outlive the fault; got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,12 +1,15 @@
|
|||||||
package backup
|
package backup
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
|
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
|
||||||
@@ -49,16 +52,45 @@ type RestoreTestState struct {
|
|||||||
last map[string]provenTier // target id → what was last PROVEN on that tier
|
last map[string]provenTier // target id → what was last PROVEN on that tier
|
||||||
}
|
}
|
||||||
|
|
||||||
// provenTier is one tier's proof: the archive that passed, and when it passed.
|
// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
|
||||||
|
// and when.
|
||||||
|
//
|
||||||
|
// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
|
||||||
|
// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
|
||||||
|
// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
|
||||||
|
// hub can stay ignorant of a real success until the next archive generation.
|
||||||
|
//
|
||||||
|
// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
|
||||||
|
// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
|
||||||
|
// at report-building time — a network call that can fail, on a path where failing means mis-labelling
|
||||||
|
// a proof. Store what you knew when you knew it.
|
||||||
type provenTier struct {
|
type provenTier struct {
|
||||||
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
|
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
|
||||||
At time.Time // when that run passed (UTC)
|
Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
|
||||||
|
Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
|
||||||
|
At time.Time // when that run passed (UTC)
|
||||||
}
|
}
|
||||||
|
|
||||||
// provenTierJSON is the on-disk shape (R-86). The legacy shape was a bare RFC3339 STRING per
|
// reportable reports whether this record can be re-reported to the hub as a restore-test result.
|
||||||
// target; both are read, only this one is written — see NewRestoreTestState.
|
//
|
||||||
|
// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
|
||||||
|
// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
|
||||||
|
// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
|
||||||
|
// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
|
||||||
|
func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
|
||||||
|
|
||||||
|
// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
|
||||||
|
//
|
||||||
|
// v1 (pre-R-86) "<target>": "<RFC3339>" — a time, no archive
|
||||||
|
// v2 (R-86) "<target>": {archive, proven_at} — due-check usable, not reportable
|
||||||
|
// v3 (R-189) "<target>": {archive, tier, verified, …} — both
|
||||||
|
//
|
||||||
|
// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
|
||||||
|
// readers above test for — the migration needs no version number because the absence IS the answer.
|
||||||
type provenTierJSON struct {
|
type provenTierJSON struct {
|
||||||
Archive string `json:"archive"`
|
Archive string `json:"archive"`
|
||||||
|
Tier string `json:"tier,omitempty"`
|
||||||
|
Verified string `json:"verified,omitempty"`
|
||||||
ProvenAt string `json:"proven_at"`
|
ProvenAt string `json:"proven_at"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -99,21 +131,31 @@ func NewRestoreTestState(path string) *RestoreTestState {
|
|||||||
if perr != nil {
|
if perr != nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
s.last[target] = provenTier{Archive: cur.Archive, At: t.UTC()}
|
s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
|
||||||
}
|
}
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed. Only call this for a
|
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
|
||||||
// PASSING restore-test — the archive is what makes the tier not-due, so recording one for a failed
|
// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
|
||||||
// run would retire the archive unproven.
|
// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
|
||||||
func (s *RestoreTestState) RecordSuccess(target, archive string, t time.Time) error {
|
//
|
||||||
|
// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
|
||||||
|
// the next reader will notice failures are absent and try to "fix" it:
|
||||||
|
//
|
||||||
|
// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
|
||||||
|
// the system quietly less tested than it believes. It must survive a restart.
|
||||||
|
//
|
||||||
|
// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
|
||||||
|
// so a lost failure heals itself within one interval. Persisting it would do the opposite of
|
||||||
|
// helping: a healed tier would keep reporting a failure that is no longer true.
|
||||||
|
func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
|
||||||
if target == "" {
|
if target == "" {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
s.last[target] = provenTier{Archive: archive, At: t.UTC()}
|
s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
|
||||||
return s.saveLocked()
|
return s.saveLocked()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -138,7 +180,13 @@ func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
|
|||||||
return p.Archive, true
|
return p.Archive, true
|
||||||
}
|
}
|
||||||
|
|
||||||
// Snapshot returns a copy of the last-proven TIMES — for the host-report gauge.
|
// Snapshot returns a copy of the last-proven TIMES.
|
||||||
|
//
|
||||||
|
// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
|
||||||
|
// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
|
||||||
|
// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
|
||||||
|
// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
|
||||||
|
// only the times; if it acquires none, delete it rather than let it claim a purpose again.
|
||||||
func (s *RestoreTestState) Snapshot() map[string]time.Time {
|
func (s *RestoreTestState) Snapshot() map[string]time.Time {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
@@ -149,6 +197,41 @@ func (s *RestoreTestState) Snapshot() map[string]time.Time {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
|
||||||
|
//
|
||||||
|
// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
|
||||||
|
// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
|
||||||
|
//
|
||||||
|
// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
|
||||||
|
// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
|
||||||
|
// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
|
||||||
|
// is not a claim; a fabricated one would be.
|
||||||
|
//
|
||||||
|
// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
|
||||||
|
// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
|
||||||
|
// worse defect than the one this fixes.
|
||||||
|
func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
out := make([]hub.RestoreTest, 0, len(s.last))
|
||||||
|
for _, p := range s.last {
|
||||||
|
if !p.reportable() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, hub.RestoreTest{
|
||||||
|
SourceArchive: p.Archive,
|
||||||
|
SourceTier: p.Tier,
|
||||||
|
Pass: true,
|
||||||
|
Verified: p.Verified,
|
||||||
|
TestedAt: p.At.UTC().Format(time.RFC3339),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
|
||||||
|
// iteration is randomised.
|
||||||
|
sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
|
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
|
||||||
//
|
//
|
||||||
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
|
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
|
||||||
@@ -185,7 +268,10 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
|
|||||||
func (s *RestoreTestState) saveLocked() error {
|
func (s *RestoreTestState) saveLocked() error {
|
||||||
raw := make(map[string]provenTierJSON, len(s.last))
|
raw := make(map[string]provenTierJSON, len(s.last))
|
||||||
for target, p := range s.last {
|
for target, p := range s.last {
|
||||||
raw[target] = provenTierJSON{Archive: p.Archive, ProvenAt: p.At.UTC().Format(time.RFC3339)}
|
raw[target] = provenTierJSON{
|
||||||
|
Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
|
||||||
|
ProvenAt: p.At.UTC().Format(time.RFC3339),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
data, err := json.MarshalIndent(raw, "", " ")
|
data, err := json.MarshalIndent(raw, "", " ")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -303,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) {
|
|||||||
t.Fatalf("unexpected: %v", got)
|
t.Fatalf("unexpected: %v", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", now)
|
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
|
||||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
|
||||||
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
|
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
|
||||||
}
|
}
|
||||||
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", now.Add(time.Hour))
|
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
|
||||||
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
|
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
|
||||||
t.Fatalf("the least recently proven must sort first; got %v", got)
|
t.Fatalf("the least recently proven must sort first; got %v", got)
|
||||||
}
|
}
|
||||||
@@ -318,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) {
|
|||||||
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
|
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
|
||||||
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
_ = st.RecordSuccess("b-tier", "b:archive", now)
|
_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
|
||||||
_ = st.RecordSuccess("a-tier", "a:archive", now)
|
_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
|
||||||
for i := 0; i < 20; i++ {
|
for i := 0; i < 20; i++ {
|
||||||
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
|
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
|
||||||
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
|
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
|
||||||
@@ -334,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
|
|||||||
now := time.Now().UTC().Truncate(time.Second)
|
now := time.Now().UTC().Truncate(time.Second)
|
||||||
|
|
||||||
st := NewRestoreTestState(path)
|
st := NewRestoreTestState(path)
|
||||||
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
|
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
reopened := NewRestoreTestState(path)
|
reopened := NewRestoreTestState(path)
|
||||||
|
|||||||
@@ -180,7 +180,15 @@ func (s *Scheduler) tick(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
if archive == "" {
|
if archive == "" {
|
||||||
s.logger.Debug("backup: restore-test not due this evaluation")
|
// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
|
||||||
|
//
|
||||||
|
// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
|
||||||
|
// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
|
||||||
|
// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
|
||||||
|
// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
|
||||||
|
// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
|
||||||
|
// nothing run last night?" is in the log rather than in a re-derivation.
|
||||||
|
s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -210,7 +218,11 @@ func (s *Scheduler) tick(ctx context.Context) {
|
|||||||
if rt.Pass && s.rtState != nil && target != "" {
|
if rt.Pass && s.rtState != nil && target != "" {
|
||||||
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
|
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
|
||||||
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
|
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
|
||||||
if err := s.rtState.RecordSuccess(target, archive, s.now()); err != nil {
|
// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
|
||||||
|
// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
|
||||||
|
// this run was actually judged as, and deriving it later would need a storage lookup that
|
||||||
|
// can fail on the one path where failing means mislabelling a proof.
|
||||||
|
if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
|
||||||
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
|
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,8 +10,22 @@ import (
|
|||||||
// Store holds the agent's LATEST backup result per target and the latest restore-test
|
// Store holds the agent's LATEST backup result per target and the latest restore-test
|
||||||
// result — the point-in-time state the host-report surfaces. It is updated by the backup
|
// result — the point-in-time state the host-report surfaces. It is updated by the backup
|
||||||
// runner + the restore-test scheduler/selftest and read by the collector via the hub
|
// runner + the restore-test scheduler/selftest and read by the collector via the hub
|
||||||
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence
|
// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
|
||||||
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access.
|
// collector vs scheduler access.
|
||||||
|
//
|
||||||
|
// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
|
||||||
|
// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
|
||||||
|
// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
|
||||||
|
// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
|
||||||
|
// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
|
||||||
|
// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
|
||||||
|
// consecutive host-reports carried `0 restore-tests`.
|
||||||
|
//
|
||||||
|
// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
|
||||||
|
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
|
||||||
|
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
|
||||||
|
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
|
||||||
|
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
|
||||||
type Store struct {
|
type Store struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
byTarget map[string]hub.Backup // latest backup per target id
|
byTarget map[string]hub.Backup // latest backup per target id
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
package escrow
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery
|
||||||
|
// code R, and hand back EXACTLY ONE field: the offsite restic repository password.
|
||||||
|
//
|
||||||
|
// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and
|
||||||
|
// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one
|
||||||
|
// trust tier down — needs none of them, and returning them would widen the blast radius of a
|
||||||
|
// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later.
|
||||||
|
//
|
||||||
|
// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a
|
||||||
|
// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper
|
||||||
|
// keeps that property: it takes R as an argument, passes it straight through, and holds no copy.
|
||||||
|
// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent).
|
||||||
|
//
|
||||||
|
// The errors below are DISTINCT on purpose. "no blob", "wrong code" and "the blob predates the field"
|
||||||
|
// are three different situations for the operator and only one of them is a fault.
|
||||||
|
|
||||||
|
var (
|
||||||
|
// ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run.
|
||||||
|
ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)")
|
||||||
|
// ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected
|
||||||
|
// for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be
|
||||||
|
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
|
||||||
|
// not sent hunting for a mistyped recovery code that was typed correctly.
|
||||||
|
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
|
||||||
|
)
|
||||||
|
|
||||||
|
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
|
||||||
|
// An interface-free func field keeps this package free of any dependency on the hub client.
|
||||||
|
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
|
||||||
|
|
||||||
|
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||||
|
type OffsiteKeyRecoverer struct {
|
||||||
|
Fetch BlobFetcher
|
||||||
|
}
|
||||||
|
|
||||||
|
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
|
||||||
|
//
|
||||||
|
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
|
||||||
|
// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere.
|
||||||
|
// That property is the crypto's, not a check here, which is why this function has no "validate R"
|
||||||
|
// step to get wrong.
|
||||||
|
//
|
||||||
|
// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes.
|
||||||
|
func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) {
|
||||||
|
if r.Fetch == nil {
|
||||||
|
return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured")
|
||||||
|
}
|
||||||
|
if recoveryCode == "" {
|
||||||
|
return "", fmt.Errorf("escrow: the recovery code is required")
|
||||||
|
}
|
||||||
|
blob, present, err := r.Fetch(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("escrow: fetching the sealed bundle: %w", err) // carries no secret
|
||||||
|
}
|
||||||
|
if !present || len(blob) == 0 {
|
||||||
|
return "", ErrNoEscrowBlob
|
||||||
|
}
|
||||||
|
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
|
||||||
|
if err != nil {
|
||||||
|
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||||
|
}
|
||||||
|
if bundle.ResticRepoPassword == "" {
|
||||||
|
return "", ErrNoResticPassword
|
||||||
|
}
|
||||||
|
return bundle.ResticRepoPassword, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
package escrow
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips
|
||||||
|
// elsewhere). These are the unit half of the session's question — "is the repository password
|
||||||
|
// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware.
|
||||||
|
|
||||||
|
const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
|
||||||
|
|
||||||
|
func sealBundle(t *testing.T, b IdentityBundle, r string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
blob, err := WrapIdentityBundle(context.Background(), b, r)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("WrapIdentityBundle: %v", err)
|
||||||
|
}
|
||||||
|
return blob
|
||||||
|
}
|
||||||
|
|
||||||
|
func fetcherFor(blob []byte) BlobFetcher {
|
||||||
|
return func(context.Context) ([]byte, bool, error) { return blob, true, nil }
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it
|
||||||
|
// is the REPOSITORY password rather than some other field of a bundle that also parses.
|
||||||
|
//
|
||||||
|
// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of
|
||||||
|
// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS.
|
||||||
|
// That mutation is the shape of the bug that would otherwise ship silently, because every one of
|
||||||
|
// those fields is a non-empty string that looks like a secret.
|
||||||
|
func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||||
|
blob := sealBundle(t, IdentityBundle{
|
||||||
|
TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER",
|
||||||
|
PBSToken: "PBS-TOKEN-NOT-THE-ANSWER",
|
||||||
|
WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
|
||||||
|
ResticRepoPassword: repoPW,
|
||||||
|
}, testR)
|
||||||
|
|
||||||
|
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("recover: %v", err)
|
||||||
|
}
|
||||||
|
if got != repoPW {
|
||||||
|
t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+
|
||||||
|
"field of the bundle was returned", len(got), len(repoPW))
|
||||||
|
}
|
||||||
|
// Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check
|
||||||
|
// above by coincidence.
|
||||||
|
for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} {
|
||||||
|
if got == other {
|
||||||
|
t.Fatalf("the recoverer returned the wrong bundle field")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is
|
||||||
|
// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no
|
||||||
|
// validation step here to get wrong — the test pins that it stays that way.
|
||||||
|
func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
|
||||||
|
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||||
|
|
||||||
|
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids")
|
||||||
|
}
|
||||||
|
if got != "" {
|
||||||
|
t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got))
|
||||||
|
}
|
||||||
|
// The error may name the step; it may never name a secret.
|
||||||
|
for _, secret := range []string{repoPW, testR, "not the recovery code at all"} {
|
||||||
|
if strings.Contains(err.Error(), secret) {
|
||||||
|
t.Fatalf("the failure message leaked a secret: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before
|
||||||
|
// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly
|
||||||
|
// typed recovery code would be the wrong instruction.
|
||||||
|
func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR)
|
||||||
|
|
||||||
|
_, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||||
|
if !errors.Is(err, ErrNoResticPassword) {
|
||||||
|
t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D at this layer — no blob is a clean, distinguishable answer.
|
||||||
|
func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) {
|
||||||
|
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
|
||||||
|
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||||
|
if !errors.Is(err, ErrNoEscrowBlob) {
|
||||||
|
t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is
|
||||||
|
// run for real, and the whole tree is then walked: no file may contain R (or the recovered password),
|
||||||
|
// and the staging directory the unseal creates must be gone.
|
||||||
|
//
|
||||||
|
// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add
|
||||||
|
// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before
|
||||||
|
// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the
|
||||||
|
// walk finds it → this FAILS.
|
||||||
|
func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
const repoPW = "1111111111111111111111111111111111111111111111111111111111111111"
|
||||||
|
tmp := t.TempDir()
|
||||||
|
t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk
|
||||||
|
|
||||||
|
const wrongR = "wrong code entirely"
|
||||||
|
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
|
||||||
|
if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil {
|
||||||
|
t.Fatalf("recover: %v", err)
|
||||||
|
}
|
||||||
|
// A failed unseal must leave nothing either — exercise both paths before walking.
|
||||||
|
_, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR)
|
||||||
|
|
||||||
|
// THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later
|
||||||
|
// call OVERWRITING the leaked file with a different secret — which is exactly how the first
|
||||||
|
// version of this test passed its own red-proof while R sat on disk. Nothing in this test writes
|
||||||
|
// under TMPDIR, so after both calls the tree must contain no files at all.
|
||||||
|
var survivors []string
|
||||||
|
err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||||
|
if err != nil || info == nil || info.IsDir() || path == tmp {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
survivors = append(survivors, strings.TrimPrefix(path, tmp))
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(survivors) > 0 {
|
||||||
|
t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+
|
||||||
|
"recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors)
|
||||||
|
}
|
||||||
|
// Defence in depth: any secret that DOES appear anywhere is named, for every code used.
|
||||||
|
_ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
|
||||||
|
if err != nil || info == nil || info.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
body, rerr := os.ReadFile(path)
|
||||||
|
if rerr != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} {
|
||||||
|
if strings.Contains(string(body), secret) {
|
||||||
|
t.Errorf("%s survived on disk at %s", label, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
// And the staging directories are gone, not merely free of secrets.
|
||||||
|
entries, _ := os.ReadDir(tmp)
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") {
|
||||||
|
t.Fatalf("an unseal staging directory survived: %s", e.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be
|
||||||
|
// sent to re-read their recovery code because the hub was unreachable.
|
||||||
|
func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) {
|
||||||
|
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) {
|
||||||
|
return nil, false, errors.New("hub: connection refused")
|
||||||
|
}}
|
||||||
|
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "fetching the sealed bundle") {
|
||||||
|
t.Fatalf("a fetch failure must say so, got %v", err)
|
||||||
|
}
|
||||||
|
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
|
||||||
|
t.Fatal("a transport failure must not masquerade as a content verdict")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -307,3 +307,50 @@ func tail(b []byte, max int) string {
|
|||||||
}
|
}
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199).
|
||||||
|
// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet.
|
||||||
|
type IdentityEscrowResponse struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
Present bool `json:"present"`
|
||||||
|
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the
|
||||||
|
// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they
|
||||||
|
// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds.
|
||||||
|
//
|
||||||
|
// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no
|
||||||
|
// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different
|
||||||
|
// endpoint with a different gate and is not reached from here.
|
||||||
|
//
|
||||||
|
// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged —
|
||||||
|
// only its length.
|
||||||
|
func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) {
|
||||||
|
if c.hostID == "" {
|
||||||
|
return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id")
|
||||||
|
}
|
||||||
|
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow"
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err)
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", "Bearer "+c.apiKey)
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
|
||||||
|
resp, err := c.hc.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &TransportError{Err: err}
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
|
||||||
|
}
|
||||||
|
var out IdentityEscrowResponse
|
||||||
|
if err := json.Unmarshal(raw, &out); err != nil {
|
||||||
|
return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err)
|
||||||
|
}
|
||||||
|
return &out, nil
|
||||||
|
}
|
||||||
|
|||||||
+96
-5
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
|
|||||||
RestoreTests(ctx context.Context) []RestoreTest
|
RestoreTests(ctx context.Context) []RestoreTest
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
|
||||||
|
//
|
||||||
|
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
|
||||||
|
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
|
||||||
|
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
|
||||||
|
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
|
||||||
|
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
|
||||||
|
//
|
||||||
|
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
|
||||||
|
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
|
||||||
|
//
|
||||||
|
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
|
||||||
|
type ProvenRestoreTestReporter interface {
|
||||||
|
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||||
|
}
|
||||||
|
|
||||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||||
type PBSReporter interface {
|
type PBSReporter interface {
|
||||||
@@ -79,6 +95,7 @@ type Collector struct {
|
|||||||
storage StorageObserver
|
storage StorageObserver
|
||||||
backups BackupReporter
|
backups BackupReporter
|
||||||
restoreTests RestoreTestReporter
|
restoreTests RestoreTestReporter
|
||||||
|
provenTests ProvenRestoreTestReporter
|
||||||
pbs PBSReporter
|
pbs PBSReporter
|
||||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||||
@@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
|||||||
return []Backup{}
|
return []Backup{}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||||
|
//
|
||||||
|
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
|
||||||
|
// than from a preference between them:
|
||||||
|
//
|
||||||
|
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
|
||||||
|
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
|
||||||
|
// record is short-lived by design;
|
||||||
|
// - the persisted state holds the last SUCCESS per tier and survives a restart.
|
||||||
|
//
|
||||||
|
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
|
||||||
|
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
|
||||||
|
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
|
||||||
|
// read at the hub as two tests.
|
||||||
|
//
|
||||||
|
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
|
||||||
|
// would be a worse defect than the one this closes.
|
||||||
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
|
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
|
||||||
if c.restoreTests == nil {
|
out := []RestoreTest{}
|
||||||
return []RestoreTest{}
|
if c.restoreTests != nil {
|
||||||
|
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||||
|
out = append(out, r...)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
if c.provenTests == nil {
|
||||||
return r
|
return out
|
||||||
}
|
}
|
||||||
return []RestoreTest{}
|
|
||||||
|
// Index what we already have by tier, keeping the newest per tier.
|
||||||
|
best := map[string]int{} // tier → index into out
|
||||||
|
for i, rt := range out {
|
||||||
|
if rt.SourceTier == "" {
|
||||||
|
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
|
||||||
|
}
|
||||||
|
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
|
||||||
|
best[rt.SourceTier] = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
|
||||||
|
if p.SourceTier == "" {
|
||||||
|
continue // not usable as a per-tier proof; the state layer already filters these
|
||||||
|
}
|
||||||
|
i, seen := best[p.SourceTier]
|
||||||
|
if !seen {
|
||||||
|
out = append(out, p)
|
||||||
|
best[p.SourceTier] = len(out) - 1
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if newerRestoreTest(p, out[i]) {
|
||||||
|
out[i] = p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
|
||||||
|
// treated as OLDER, so a malformed entry can never displace a good one.
|
||||||
|
func newerRestoreTest(a, b RestoreTest) bool {
|
||||||
|
ta, aok := parseRestoreTestedAt(a.TestedAt)
|
||||||
|
tb, bok := parseRestoreTestedAt(b.TestedAt)
|
||||||
|
if !aok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if !bok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return ta.After(tb)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseRestoreTestedAt(s string) (time.Time, bool) {
|
||||||
|
t, err := time.Parse(time.RFC3339, s)
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}, false
|
||||||
|
}
|
||||||
|
return t.UTC(), true
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
|
||||||
|
// argument because the persisted state is opened later in main() than the collector is built; the
|
||||||
|
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
|
||||||
|
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
|
||||||
|
// all, and this fix must not become the next instance of that.
|
||||||
|
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
|
||||||
|
|
||||||
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
|
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
|
||||||
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
|
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
|
||||||
if c.pbs == nil {
|
if c.pbs == nil {
|
||||||
|
|||||||
@@ -0,0 +1,213 @@
|
|||||||
|
package hub
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-189 — a passing restore-test must survive an agent restart and reach the hub.
|
||||||
|
//
|
||||||
|
// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test
|
||||||
|
// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged
|
||||||
|
// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on
|
||||||
|
// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86
|
||||||
|
// made the agent refuse to re-test an archive it has already proven.
|
||||||
|
//
|
||||||
|
// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was
|
||||||
|
// hollow because a perfectly regular series landed exactly on a threshold and passed under the
|
||||||
|
// mutation it was meant to catch.
|
||||||
|
|
||||||
|
type fakeLatest struct{ tests []RestoreTest }
|
||||||
|
|
||||||
|
func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||||
|
|
||||||
|
type fakeProven struct{ tests []RestoreTest }
|
||||||
|
|
||||||
|
func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests }
|
||||||
|
|
||||||
|
func rt(tier, archive string, pass bool, at time.Time) RestoreTest {
|
||||||
|
return RestoreTest{
|
||||||
|
SourceArchive: archive, SourceTier: tier, Pass: pass,
|
||||||
|
Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what
|
||||||
|
// is under test, not the rest of the collection.
|
||||||
|
func mergeCollector(latest, proven []RestoreTest) *Collector {
|
||||||
|
c := &Collector{}
|
||||||
|
if latest != nil {
|
||||||
|
c.restoreTests = &fakeLatest{tests: latest}
|
||||||
|
}
|
||||||
|
if proven != nil {
|
||||||
|
c.provenTests = &fakeProven{tests: proven}
|
||||||
|
}
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func findTier(got []RestoreTest, tier string) (RestoreTest, int) {
|
||||||
|
var hit RestoreTest
|
||||||
|
n := 0
|
||||||
|
for _, e := range got {
|
||||||
|
if e.SourceTier == tier {
|
||||||
|
hit, n = e, n+1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return hit, n
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO A — a proof survives a restart and reaches the hub ──────────────────────────────
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from
|
||||||
|
// collectRestoreTests (return the in-memory slice as it used to) →
|
||||||
|
//
|
||||||
|
// --- FAIL: TestMerge_ProofSurvivesARestart
|
||||||
|
// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies)
|
||||||
|
//
|
||||||
|
// which is exactly the live observation: `0 restore-tests`. Restored.
|
||||||
|
func TestMerge_ProofSurvivesARestart(t *testing.T) {
|
||||||
|
provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp
|
||||||
|
// After a restart the in-memory store is EMPTY — this is the whole point.
|
||||||
|
c := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||||
|
rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt),
|
||||||
|
})
|
||||||
|
|
||||||
|
got := c.collectRestoreTests(context.Background())
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got)
|
||||||
|
}
|
||||||
|
e := got[0]
|
||||||
|
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" {
|
||||||
|
t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive)
|
||||||
|
}
|
||||||
|
if e.SourceTier != "pbs" || !e.Pass {
|
||||||
|
t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass)
|
||||||
|
}
|
||||||
|
if e.TestedAt != provenAt.Format(time.RFC3339) {
|
||||||
|
t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO B — the report does not invent a pass ───────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the
|
||||||
|
// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the
|
||||||
|
// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which
|
||||||
|
// this test injects directly and the assertion below rejects.
|
||||||
|
func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) {
|
||||||
|
// Nothing proven anywhere: no in-memory result, no persisted proof.
|
||||||
|
c := mergeCollector([]RestoreTest{}, []RestoreTest{})
|
||||||
|
if got := c.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||||
|
t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+
|
||||||
|
"worse than the defect being fixed; got %+v", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// And an entry the state layer could not describe (no tier) is never promoted into a proof.
|
||||||
|
c2 := mergeCollector([]RestoreTest{}, []RestoreTest{
|
||||||
|
{SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true,
|
||||||
|
TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)},
|
||||||
|
})
|
||||||
|
if got := c2.collectRestoreTests(context.Background()); len(got) != 0 {
|
||||||
|
t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ─────────────────────────
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry
|
||||||
|
// unconditionally) →
|
||||||
|
//
|
||||||
|
// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier
|
||||||
|
// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests
|
||||||
|
//
|
||||||
|
// Restored.
|
||||||
|
func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) {
|
||||||
|
lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box
|
||||||
|
fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)
|
||||||
|
|
||||||
|
c := mergeCollector(
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||||
|
)
|
||||||
|
got := c.collectRestoreTests(context.Background())
|
||||||
|
e, n := findTier(got, "pbs")
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got)
|
||||||
|
}
|
||||||
|
if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||||
|
t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ...and the older-in-memory / newer-persisted direction, which is the post-restart case.
|
||||||
|
c2 := mergeCollector(
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
|
||||||
|
)
|
||||||
|
e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs")
|
||||||
|
if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
|
||||||
|
t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── SCENARIO D — a failure still reaches the hub ─────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// The merge must not mask a failure with an older stored success. A failing tier is retried at the
|
||||||
|
// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest
|
||||||
|
// DR signal this system produces.
|
||||||
|
func TestMerge_AFailureIsStillReported(t *testing.T) {
|
||||||
|
provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC)
|
||||||
|
failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC)
|
||||||
|
|
||||||
|
c := mergeCollector(
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)},
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)},
|
||||||
|
)
|
||||||
|
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background()))
|
||||||
|
}
|
||||||
|
if e.Pass {
|
||||||
|
t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+
|
||||||
|
"silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Two different tiers are both reported — the merge is per tier, not a single slot.
|
||||||
|
func TestMerge_BothTiersSurvive(t *testing.T) {
|
||||||
|
c := mergeCollector(
|
||||||
|
[]RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true,
|
||||||
|
time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))},
|
||||||
|
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true,
|
||||||
|
time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))},
|
||||||
|
)
|
||||||
|
got := c.collectRestoreTests(context.Background())
|
||||||
|
if _, n := findTier(got, "local"); n != 1 {
|
||||||
|
t.Fatalf("the in-memory tier must survive the merge; got %+v", got)
|
||||||
|
}
|
||||||
|
if _, n := findTier(got, "pbs"); n != 1 {
|
||||||
|
t.Fatalf("the persisted tier must survive the merge; got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A malformed timestamp must never displace a good entry — "unparseable" is not "newest".
|
||||||
|
func TestMerge_MalformedTimestampNeverWins(t *testing.T) {
|
||||||
|
good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC))
|
||||||
|
bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"}
|
||||||
|
|
||||||
|
c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad})
|
||||||
|
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
|
||||||
|
if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" {
|
||||||
|
t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A nil proven-source leaves the pre-R-189 behaviour exactly as it was.
|
||||||
|
func TestMerge_NilProvenSourceIsANoOp(t *testing.T) {
|
||||||
|
only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))
|
||||||
|
c := mergeCollector([]RestoreTest{only}, nil)
|
||||||
|
got := c.collectRestoreTests(context.Background())
|
||||||
|
if len(got) != 1 || got[0].SourceArchive != only.SourceArchive {
|
||||||
|
t.Fatalf("a nil durable source must not change anything; got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository
|
||||||
|
// password from the hub's sealed bundle, using the customer's recovery code R.
|
||||||
|
//
|
||||||
|
// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`)
|
||||||
|
// is an agent runtime dependency and is deliberately absent from the controller image; the sealed
|
||||||
|
// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is
|
||||||
|
// that write's mirror; and the controller is a trust tier down — it should receive one field, not a
|
||||||
|
// bundle it has no use for.
|
||||||
|
//
|
||||||
|
// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that
|
||||||
|
// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it:
|
||||||
|
// - arrives in the request body over the already-pinned local-API channel (the operator accepted
|
||||||
|
// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end);
|
||||||
|
// - is held in memory for the duration of one call and cleared on BOTH paths;
|
||||||
|
// - is never written to disk, never an argument in a process list, and never logged at any level,
|
||||||
|
// including inside an error;
|
||||||
|
// - is never echoed: no response this endpoint can emit contains it.
|
||||||
|
//
|
||||||
|
// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see
|
||||||
|
// `logRequests`. Do not add a body dump.
|
||||||
|
//
|
||||||
|
// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares
|
||||||
|
// (compare by hash, never by value). The password itself is present because the next link — placing a
|
||||||
|
// recovered password so the existing repository opens — needs it, and building a hash-only seam now
|
||||||
|
// would have to be torn out to add it. The controller's diagnostic reads only the hash.
|
||||||
|
|
||||||
|
type recoverOffsitePasswordRequest struct {
|
||||||
|
VMID int `json:"vmid"`
|
||||||
|
// RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed.
|
||||||
|
RecoveryCode string `json:"recovery_code"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only
|
||||||
|
// the offsite repository password (plus its sha256, for hash-only comparison by the caller).
|
||||||
|
func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||||
|
var req recoverOffsitePasswordRequest
|
||||||
|
if !decodeBody(w, r, &req) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
R := strings.TrimSpace(req.RecoveryCode)
|
||||||
|
req.RecoveryCode = "" // drop the decoded copy immediately
|
||||||
|
if R == "" {
|
||||||
|
writeErr(w, http.StatusBadRequest, "recovery_code is required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.escrowRecovery == nil {
|
||||||
|
R = ""
|
||||||
|
writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid)
|
||||||
|
|
||||||
|
pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R)
|
||||||
|
R = "" // cleared on BOTH paths, before anything else can happen
|
||||||
|
if err != nil {
|
||||||
|
// Each situation gets its own status and its own words. None of them names a secret.
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, escrow.ErrNoEscrowBlob):
|
||||||
|
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
|
||||||
|
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
|
||||||
|
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||||
|
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||||
|
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||||
|
default:
|
||||||
|
// Includes the fail-closed wrong-code case. The agent log records the STEP, never the code.
|
||||||
|
s.logger.Warn("local-api: offsite key recovery FAILED (wrong recovery code, or the blob could not be fetched)", "vmid", vmid, "err", err)
|
||||||
|
writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle, or the bundle could not be fetched — nothing was written")
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
|
||||||
|
// §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this
|
||||||
|
// concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing).
|
||||||
|
s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+
|
||||||
|
"(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)",
|
||||||
|
"vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:]))
|
||||||
|
writeOK(w, map[string]any{
|
||||||
|
"restic_repo_password": pw,
|
||||||
|
"restic_pw_sha256": hex.EncodeToString(sum[:]),
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -111,6 +111,14 @@ type HostMetricsProvider interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Options configures a Server.
|
// Options configures a Server.
|
||||||
|
// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and
|
||||||
|
// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the
|
||||||
|
// localapi package needs no hub-client dependency and the route is testable without crypto.
|
||||||
|
// R is an argument and is never retained by any implementation.
|
||||||
|
type EscrowRecoverer interface {
|
||||||
|
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error)
|
||||||
|
}
|
||||||
|
|
||||||
type Options struct {
|
type Options struct {
|
||||||
ListenAddr string // bridge IP:port
|
ListenAddr string // bridge IP:port
|
||||||
Cert tls.Certificate
|
Cert tls.Certificate
|
||||||
@@ -210,6 +218,11 @@ type Options struct {
|
|||||||
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
|
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
|
||||||
LogRing *applog.Ring
|
LogRing *applog.Ring
|
||||||
Logger *slog.Logger
|
Logger *slog.Logger
|
||||||
|
// EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind
|
||||||
|
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
|
||||||
|
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
|
||||||
|
EscrowRecovery EscrowRecoverer
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// defaultBackupCadence is the fallback /backup/due window when none is configured.
|
// defaultBackupCadence is the fallback /backup/due window when none is configured.
|
||||||
@@ -273,6 +286,11 @@ type Server struct {
|
|||||||
netMountRoot string // the user-data namespace root for the network-mount role gate
|
netMountRoot string // the user-data namespace root for the network-mount role gate
|
||||||
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
|
||||||
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
|
||||||
|
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
|
||||||
|
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
|
||||||
|
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
|
||||||
|
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
|
||||||
|
escrowRecovery EscrowRecoverer
|
||||||
intent IntentRecorder // slice 10 P3 (optional)
|
intent IntentRecorder // slice 10 P3 (optional)
|
||||||
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
|
||||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||||
@@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) {
|
|||||||
netMountRoot: storage.NetworkMountRoot,
|
netMountRoot: storage.NetworkMountRoot,
|
||||||
smbCredsDir: o.SmbCredsDir,
|
smbCredsDir: o.SmbCredsDir,
|
||||||
escrowStagePath: o.EscrowStagePath,
|
escrowStagePath: o.EscrowStagePath,
|
||||||
|
escrowRecovery: o.EscrowRecovery,
|
||||||
intent: o.Intent,
|
intent: o.Intent,
|
||||||
guestBinds: o.GuestBinds,
|
guestBinds: o.GuestBinds,
|
||||||
formatJobs: o.FormatJobs,
|
formatJobs: o.FormatJobs,
|
||||||
@@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler {
|
|||||||
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
|
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
|
||||||
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
|
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
|
||||||
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
|
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
|
||||||
|
// R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle,
|
||||||
|
// using the customer recovery code supplied in the body. Returns that ONE field. See
|
||||||
|
// escrow_recover.go for R's handling rules — they are the tightest in this package.
|
||||||
|
mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword))
|
||||||
|
|
||||||
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
|
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
|
||||||
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.
|
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.
|
||||||
|
|||||||
@@ -45,6 +45,35 @@ func (c *Client) Pool(ctx context.Context, name string) (PoolInfo, error) {
|
|||||||
return p, c.get(ctx, "/pools/"+url.PathEscape(name), &p)
|
return p, c.get(ctx, "/pools/"+url.PathEscape(name), &p)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Permissions returns the privileges this API TOKEN holds at an ACL path, as
|
||||||
|
// GET /access/permissions?path=<path> answers it: privilege name → 1.
|
||||||
|
//
|
||||||
|
// R-185. It asks about the CALLER — the agent's own token — which is the only useful form of the
|
||||||
|
// question. Asking as root answers a different question and always says yes.
|
||||||
|
//
|
||||||
|
// MEASURED SHAPE (demo-felhom, 2026-08-03), because the whole value of this call is reading the
|
||||||
|
// answer correctly and the obvious reading is wrong:
|
||||||
|
//
|
||||||
|
// /storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
|
||||||
|
// /storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
|
||||||
|
//
|
||||||
|
// The ungranted path does NOT answer empty, and does NOT 403. It answers with the privileges
|
||||||
|
// INHERITED from the box-wide `/` grant — so "is this path present in the response" reports OK for a
|
||||||
|
// storage the agent demonstrably cannot list. The caller must test for the SPECIFIC privilege.
|
||||||
|
//
|
||||||
|
// The response is keyed by path; an absent path yields no privileges, which is the same answer as
|
||||||
|
// "none" and is treated as such by the caller.
|
||||||
|
func (c *Client) Permissions(ctx context.Context, aclPath string) (map[string]int, error) {
|
||||||
|
var raw map[string]map[string]int
|
||||||
|
if err := c.get(ctx, "/access/permissions?path="+url.QueryEscape(aclPath), &raw); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if p, ok := raw[aclPath]; ok {
|
||||||
|
return p, nil
|
||||||
|
}
|
||||||
|
return map[string]int{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
// GuestStatus returns GET /nodes/{node}/lxc/{vmid}/status/current. The API body
|
// GuestStatus returns GET /nodes/{node}/lxc/{vmid}/status/current. The API body
|
||||||
// has no vmid field (it is in the path), so it is set from the argument.
|
// has no vmid field (it is in the path), so it is set from the argument.
|
||||||
func (c *Client) GuestStatus(ctx context.Context, vmid int) (Guest, error) {
|
func (c *Client) GuestStatus(ctx context.Context, vmid int) (Guest, error) {
|
||||||
|
|||||||
@@ -14,10 +14,32 @@ Nothing in the build, deploy or session-end path checked that either existed, so
|
|||||||
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||||
agent — and would have *succeeded* while doing it.
|
agent — and would have *succeeded* while doing it.
|
||||||
|
|
||||||
THE INVARIANT, AND WHY IT IS THIS ONE.
|
THE INVARIANTS — there are TWO now, and the second is R-188's price.
|
||||||
|
|
||||||
For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
|
(1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
|
||||||
and the tag must serve the agent's configs.
|
and the tag must serve the agent's configs.
|
||||||
|
|
||||||
|
(2) No PUBLISHED version may be missing its tag.
|
||||||
|
|
||||||
|
Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
|
||||||
|
AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
|
||||||
|
package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
|
||||||
|
needs a token this gate does not have. That reasoning was sound and the ordering was still wrong —
|
||||||
|
the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
|
||||||
|
seconds before its own package existed and mailing the operator a failure for a release that worked
|
||||||
|
(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
|
||||||
|
|
||||||
|
Moving the push does not get to trade invariant (2) away, so it is asserted here instead — WITHOUT a
|
||||||
|
token, and therefore as a BOUNDED PROBE rather than an enumeration:
|
||||||
|
|
||||||
|
* the FRONTIER — the versions immediately above the highest tag. This is the realistic failure the
|
||||||
|
new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
|
||||||
|
version beyond the newest tag.
|
||||||
|
* the GAPS — patch versions that fall between two existing tags and have no tag of their own.
|
||||||
|
|
||||||
|
Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
|
||||||
|
token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
|
||||||
|
coverage is invisible reads as a guarantee it is not making.
|
||||||
|
|
||||||
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
|
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
|
||||||
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
|
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
|
||||||
@@ -36,6 +58,8 @@ v0.120.0, which is published.
|
|||||||
|
|
||||||
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
|
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
|
||||||
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
|
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
|
||||||
|
Nor does the converse probe prove that NO untagged package exists — only that none exists at the
|
||||||
|
probed versions, which are printed. Closing that properly needs a read token in CI (→ R-184).
|
||||||
|
|
||||||
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
|
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
|
||||||
INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
|
INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
|
||||||
@@ -48,7 +72,7 @@ carrying python3 and git and nothing else, and an earlier workflow step died on
|
|||||||
|
|
||||||
python3 scripts/check-published-versions.py
|
python3 scripts/check-published-versions.py
|
||||||
|
|
||||||
Exit: 0 every tag installable · 1 at least one is not · 2 could not be determined.
|
Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
|
||||||
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
|
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
@@ -94,6 +118,53 @@ def inconclusive(msg):
|
|||||||
sys.exit(2)
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def _pkg_exists(version):
|
||||||
|
"""True iff the generic package for `version` is downloadable anonymously."""
|
||||||
|
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
|
||||||
|
status, _ = _get(url)
|
||||||
|
return status == 200, url
|
||||||
|
|
||||||
|
|
||||||
|
def untagged_probe_set(versions):
|
||||||
|
"""The versions to probe for invariant (2), as (version, why) pairs.
|
||||||
|
|
||||||
|
Bounded on purpose and printed by the caller: the package listing api needs a token (401,
|
||||||
|
re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
|
||||||
|
places an orphan would actually land.
|
||||||
|
|
||||||
|
FRONTIER — a publish that succeeded followed by a tag push that failed leaves the orphan
|
||||||
|
exactly one version past the newest tag. This is the failure mode the R-188
|
||||||
|
reordering makes possible, so it is the one that must not be guesswork.
|
||||||
|
GAPS — a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
|
||||||
|
tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
|
||||||
|
"""
|
||||||
|
parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
|
||||||
|
have = set(parsed)
|
||||||
|
out = []
|
||||||
|
if not parsed:
|
||||||
|
return out
|
||||||
|
|
||||||
|
hi = parsed[-1]
|
||||||
|
for cand, why in (
|
||||||
|
((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
|
||||||
|
((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
|
||||||
|
((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
|
||||||
|
((hi[0] + 1, 0, 0), "next major after the newest tag"),
|
||||||
|
):
|
||||||
|
if cand not in have:
|
||||||
|
out.append(("%d.%d.%d" % cand, why))
|
||||||
|
|
||||||
|
MAX_GAP_PROBES = 12
|
||||||
|
for a, b in zip(parsed, parsed[1:]):
|
||||||
|
if a[0] != b[0] or a[1] != b[1]:
|
||||||
|
continue # a minor/major step is not a patch gap
|
||||||
|
for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
|
||||||
|
cand = (a[0], a[1], patch)
|
||||||
|
if cand not in have:
|
||||||
|
out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
print("check-published-versions — every released agent version must be INSTALLABLE")
|
print("check-published-versions — every released agent version must be INSTALLABLE")
|
||||||
print(" gitea:", GITEA_BASE)
|
print(" gitea:", GITEA_BASE)
|
||||||
@@ -144,14 +215,42 @@ def main():
|
|||||||
else:
|
else:
|
||||||
print(" ok v%s: binary downloadable + tag serves its configs" % v)
|
print(" ok v%s: binary downloadable + tag serves its configs" % v)
|
||||||
|
|
||||||
|
# ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
|
||||||
|
probes = untagged_probe_set(versions)
|
||||||
|
orphans = []
|
||||||
print()
|
print()
|
||||||
|
print(" converse probe — a published version with no tag (bounded; the package listing api")
|
||||||
|
print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
|
||||||
|
for v, why in probes:
|
||||||
|
try:
|
||||||
|
exists, url = _pkg_exists(v)
|
||||||
|
except Exception as e:
|
||||||
|
inconclusive("network failure while probing v%s: %s" % (v, e))
|
||||||
|
mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
|
||||||
|
print(" %-10s %-42s %s" % (v, why, mark))
|
||||||
|
if exists:
|
||||||
|
orphans.append((v, url))
|
||||||
|
|
||||||
|
print()
|
||||||
|
if bad or orphans:
|
||||||
|
if orphans:
|
||||||
|
print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
|
||||||
|
for v, url in orphans:
|
||||||
|
print(" v%s is downloadable at %s but has no git tag." % (v, url))
|
||||||
|
print(" A release publishes and then pushes its tag; a package with no tag means the")
|
||||||
|
print(" push failed or was skipped. The local tag is probably still in the release")
|
||||||
|
print(" clone — finish it with:")
|
||||||
|
for v, _ in orphans:
|
||||||
|
print(" git push origin v%s" % v)
|
||||||
|
print(" (and if the tag is gone, re-create it on the released commit before pushing.)")
|
||||||
if bad:
|
if bad:
|
||||||
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
|
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
|
||||||
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
|
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
|
||||||
print(" the R-115 defect, three times in five days. Publish it with:")
|
print(" the R-115 defect, three times in five days. Publish it with:")
|
||||||
print(" scripts/release-agent.sh <version>")
|
print(" scripts/release-agent.sh <version>")
|
||||||
|
if bad or orphans:
|
||||||
return 1
|
return 1
|
||||||
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE")
|
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
|
|||||||
BIN="$(mktemp -t felhom-agent.XXXXXX)"
|
BIN="$(mktemp -t felhom-agent.XXXXXX)"
|
||||||
CLEANUP_BIN="$BIN"
|
CLEANUP_BIN="$BIN"
|
||||||
log "building felhom-agent $VERSION from $REPO_ROOT …"
|
log "building felhom-agent $VERSION from $REPO_ROOT …"
|
||||||
( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
# These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
|
||||||
|
# They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
|
||||||
|
# the one the release path built for the same version. One version name must mean one binary
|
||||||
|
# whichever entry point produced it.
|
||||||
|
( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
|
||||||
fi
|
fi
|
||||||
[[ -f "$BIN" ]] || die "binary not found: $BIN"
|
[[ -f "$BIN" ]] || die "binary not found: $BIN"
|
||||||
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
|
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
|
||||||
|
|||||||
@@ -74,7 +74,25 @@ existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
|||||||
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
||||||
trap 'rm -f "$BIN"' EXIT
|
trap 'rm -f "$BIN"' EXIT
|
||||||
log "building $VERSION …"
|
log "building $VERSION …"
|
||||||
go build -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
|
||||||
|
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
|
||||||
|
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
|
||||||
|
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
|
||||||
|
#
|
||||||
|
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
||||||
|
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
|
||||||
|
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
||||||
|
#
|
||||||
|
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
|
||||||
|
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
|
||||||
|
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
|
||||||
|
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
|
||||||
|
#
|
||||||
|
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
|
||||||
|
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
|
||||||
|
# version — one version name, two binaries, by whichever entry point was used. Both now build the
|
||||||
|
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
|
||||||
|
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
||||||
|| die "go build failed"
|
|| die "go build failed"
|
||||||
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
||||||
[[ "$built_ver" == "$VERSION" ]] \
|
[[ "$built_ver" == "$VERSION" ]] \
|
||||||
@@ -82,10 +100,27 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
|||||||
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
||||||
log "built ok: sha256 $BUILT_SHA"
|
log "built ok: sha256 $BUILT_SHA"
|
||||||
|
|
||||||
# ── 4. Tag (before publishing, so a published version always has a tag) ─────────────────────────
|
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
|
||||||
# Order matters in this direction only: a tag with no package is caught by
|
#
|
||||||
# scripts/check-published-versions.py on the next CI run; a package with no tag is invisible to it,
|
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
|
||||||
|
#
|
||||||
|
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
|
||||||
|
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
|
||||||
# because the Gitea package LISTING api needs a token the gate does not have.
|
# because the Gitea package LISTING api needs a token the gate does not have.
|
||||||
|
#
|
||||||
|
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
|
||||||
|
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
|
||||||
|
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
|
||||||
|
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
|
||||||
|
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
|
||||||
|
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
|
||||||
|
#
|
||||||
|
# So the tag is still created HERE, before anything is published — the build and the tag still
|
||||||
|
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
|
||||||
|
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
|
||||||
|
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
|
||||||
|
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
|
||||||
|
# now checked rather than one being arranged for.
|
||||||
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
||||||
git tag -a "$TAG" -m "agent $TAG
|
git tag -a "$TAG" -m "agent $TAG
|
||||||
|
|
||||||
@@ -94,7 +129,6 @@ sha256 of the published binary: $BUILT_SHA
|
|||||||
|
|
||||||
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
||||||
so this tag is part of the released artifact, not a bookmark (R-183)."
|
so this tag is part of the released artifact, not a bookmark (R-183)."
|
||||||
git push origin "$TAG" || die "tag push failed — refusing to publish an untagged version"
|
|
||||||
|
|
||||||
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
||||||
log "publishing …"
|
log "publishing …"
|
||||||
@@ -104,9 +138,48 @@ log "publishing …"
|
|||||||
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
|
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
|
||||||
# the same commit; this line makes the release independent of it, because a file mode is exactly the
|
# the same commit; this line makes the release independent of it, because a file mode is exactly the
|
||||||
# kind of thing that is lost again by a checkout, an archive, or a copy.
|
# kind of thing that is lost again by a checkout, an archive, or a copy.
|
||||||
bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN" || die "publish failed"
|
if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
|
||||||
|
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
|
||||||
|
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
|
||||||
|
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
|
||||||
|
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
|
||||||
|
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||||
|
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||||
|
if [[ "$now_published" == "200" ]]; then
|
||||||
|
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
|
||||||
|
else
|
||||||
|
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
|
||||||
|
fi
|
||||||
|
die "publish failed"
|
||||||
|
fi
|
||||||
|
|
||||||
# ── 6. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
|
||||||
|
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
|
||||||
|
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
|
||||||
|
#
|
||||||
|
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
|
||||||
|
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
|
||||||
|
# The recovery is one line and it is printed rather than described.
|
||||||
|
log "pushing $TAG …"
|
||||||
|
if ! git push origin "$TAG"; then
|
||||||
|
cat >&2 <<EOF
|
||||||
|
|
||||||
|
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
|
||||||
|
|
||||||
|
version : $VERSION
|
||||||
|
sha256 : $BUILT_SHA
|
||||||
|
|
||||||
|
The tag exists in this clone only. Nothing installs from an untagged version (the installer
|
||||||
|
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
|
||||||
|
FAIL on it as a published version with no tag. Finish the release with:
|
||||||
|
|
||||||
|
git push origin $TAG
|
||||||
|
|
||||||
|
EOF
|
||||||
|
die "tag push failed after a successful publish — see above"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
||||||
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
||||||
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
||||||
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
||||||
|
|||||||
Reference in New Issue
Block a user