12 Commits

Author SHA1 Message Date
admin 703db166e7 v0.127.0: a mount Felhom itself made is not 'something else' (R-220)
gates / gates (push) Successful in 8s
After a rebuild the customer's own drives could not be re-attached: candidates
returned initialize:[] attach:[] while both drives sat there, and the deploy
refused with 'choose an attached drive from the list' — a list that was empty.
Measured live three times.

Mechanism: enrolment mounts a drive TWICE, at /mnt/felhom-drives/<name> and at
the raw /mnt/<name> it creates on the host. The host survives a guest rebuild;
the controller's registry does not. So classifyClaim saw a mount outside the
managed prefix and concluded 'claimed by something else' — about our own mount.

The fix is CORROBORATED, not a widened prefix: a non-managed mountpoint is
forgiven only when the SAME device is also mounted under the managed path, a
pairing only our enrolment produces. A disk another system uses — /srv/data,
/media/x, even /mnt/someone-elses-disk — has no counterpart and is STILL
refused, with its own test and a red-proof showing an over-wide fix offering it
for formatting.

Read from /proc/mounts deliberately: the lsblk invocation is pinned verbatim in
configs/felhom-agent.sudoers, so using the plural MOUNTPOINTS would have coupled
this to a sudoers rollout. /proc/mounts is world-readable — no sudo, no new
allowlisted command, no config change.

Fail-safe: an unreadable mount table corroborates NOTHING, so the device
classifies exactly as before. 'Could not corroborate' must never read as 'ours'.

29 packages ok, vet clean, agent gates OK.
2026-08-06 12:55:28 +02:00
admin aa74294a7d docs: felhom-agent CLAUDE.md becomes a core plus path-scoped rules (R-229 leg b)
gates / gates (push) Successful in 8s
175 -> 99 effective lines. New .claude/rules/{proxmox,localapi,backup,storage}.md alongside the
existing health-checks.md. The release section points at the felhom-build-deploy skill rather than
restating a table that drifts from the script; the layout section's per-package annotations moved
into the rule file for their area instead of being deleted.

Kept in the core because it is the only part re-injected after /compact: the root-CLI fence and its
three exceptions, the destructive-op gate, prove-ownership (audit A1), the gate entry point, the F9
live-validation fence, and the checklist.

health-checks.md overlaps localapi.md and storage.md on three globs -- deliberate, both load,
stated in each file. Go build/vet/test green and unchanged.
2026-08-06 11:28:27 +02:00
admin 5b2666e3a2 docs: R-168 is CLOSED — the "CI is still owed" sentence was stale (R-229 part 2)
gates / gates (push) Successful in 11s
Corrected in all four instruction files across all four repos. Found while confirming this
session own push by run ID, which is precisely the check that catches it.

In felhom-agent/CLAUDE.md the sentence contradicted the same file release section, which
already said R-168 mails the failure -- a contradiction inside one instruction file, the exact
class the R-229 work exists to find.

REPORT.md deliberately NOT overwritten in the sibling repos: a one-line docs correction must not
destroy the record of their last real implementation.
2026-08-06 11:02:59 +02:00
admin 062a7027ab docs: remove expired and contradictory blocks from CLAUDE.md (R-229)
gates / gates (push) Successful in 8s
Surgical corrections only; the file is deliberately NOT restructured (deferred).

Deleted the expired TEMPORARY block. It read "felhom-pve is at a remote site
(until ~2026-08-02) ... Delete this block on return" and was still being read as
current fact on 2026-08-06, four days past its own deadline, while
felhom-controller/CLAUDE.md asserted the opposite. The location-independence fact
worth keeping (localapi binds 169.254.253.1:8443 on vmbr9 since the R-50 island
migration) moved to an HTML comment.

Every component version literal is gone from effective text, including the
--version reading and the go.mod Go directive. Versions change several times a
day; ask the hub's /hosts + /configs or the box.

The drill-VM claim and the host addresses now point at
documentation/operations/nodes.md, which already stated both correctly. This
file's drill-VM claim was the correct one -- confirmed by qm list on demo-hp.

The R-115/R-188/R-186 release narratives moved to an HTML comment and to the
felhom-build-deploy skill; the directives stayed (never hand-roll the build; the
build -> tag -> publish -> push order; reproducible -trimpath -buildvcs=false).

The health-check block-I/O rule became .claude/rules/health-checks.md, scoped to
the five packages where health checks are written. It had been duplicated from
felhom.eu/CLAUDE.md with a note explaining that that file does not load in an
agent-only session -- correct reasoning, made obsolete by path-scoped rules.

agent_gates.py registers the shared instructions gate.

Docs only -- no Go, no version bump, nothing built or deployed.
Ledger: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
2026-08-06 09:38:38 +02:00
admin a2e914f683 v0.126.0: a fetch failure is not a wrong recovery code (R-224)
gates / gates (push) Successful in 7s
A hub the agent could not reach was reported to the customer as a bad recovery
code. Measured live 2026-08-05 (CAMPAIGN-11 F3): hub firewalled off, a CORRECT
current code, and the customer told it did not open their package — in 0.0556s
against ~1.0s for a real unseal. No unseal was attempted.

The discriminator existed here and this boundary threw it away: recover.go
fails at four distinguishable points and the local-api handler had cases for
two, with a default answering 'the recovery code did not open the sealed
bundle, OR the bundle could not be fetched'.

escrow.ErrBundleFetch now joins the fetch leg and the handler routes it to 502
with its own words — the code was NOT used. 502 not 4xx: the request was not
bad, an upstream dependency failed. Four situations, four statuses: 502 fetch /
400 fetched-and-refused / 404 no bundle / 409 predates the field. The
controller classifies on the STATUS and never parses the sentence.

A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.
TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct has said since v0.125.0
that the operator must not be sent to re-read their code because the hub was
unreachable — and passed throughout, because it asserted this package's error
STRING one layer below the merge, and a string is not something a caller can
branch on. Re-pointed at the sentinel, with a consequence-level twin asserting
the status.

Red-proofs: removing the %w join fails the sentinel test; deleting the handler
case makes fetch and wrong-code both answer 400 with the wrong-code sentence.

29 packages ok, vet clean, agent gates OK.
2026-08-06 07:55:15 +02:00
admin 0404f60e6a pre-push: refuse a push from a clone outside the felhom workspace (R-204 rider)
gates / gates (push) Successful in 10s
The workspace root is already documented (workspace-CLAUDE.md, the workspace-root
CLAUDE.md 'stay inside it') and work drifted into a home directory anyway. A rule
that has failed once as a reminder is not fixed by writing it down again, so it is
now asserted where it can bite.

A push is the right trigger: throwaway clones under /tmp for probes and red-proofs
never push, so nothing legitimate breaks. Symlinks are resolved on both sides; an
absent workspace root SKIPS the check rather than failing it, so this cannot brick
a legitimate clone on another machine. The only bypass is the documented
--no-verify, whose use is already reportable.

Identical in all four repos.
2026-08-05 10:46:39 +02:00
admin 3f5f61b716 docs: R-199 links 6-8 — CONTEXT + REPORT (proven live on demo-felhom)
gates / gates (push) Successful in 7s
2026-08-04 13:56:38 +02:00
admin 6d7904786c agent v0.125.0: open the sealed bundle, return one field (R-199 links 7-8)
gates / gates (push) Successful in 7s
Link 7's only production caller was a --selftest reading R from an env var. Link 8 did not
exist: that selftest writes the whole bundle JSON and its success message named
"tunnel_token + pbs_token" -- accurate when written, a misstatement since v0.77.0 sealed the
offsite repository password into the same bundle. It now names what THIS bundle carried and
what it did not.

POST /escrow/recover-offsite-password: the controller supplies R, the agent fetches this
host's own blob from the hub (self-scoped by the per-host key), unseals it, and returns ONLY
the offsite restic repository password plus its sha256. Not the tunnel token, not the PBS
token, not the WG key -- the controller is a trust tier down and needs none of them.

R: in memory for one call, cleared on every path, never on disk, never in argv, never logged,
never echoed. A test redirects TMPDIR and asserts the tree is EMPTY afterwards -- emptiness
rather than a content scan, because a content scan is defeated by a later call overwriting the
leaked file, which is how the first version of that test passed its own red-proof while R sat
on disk.

Three distinct outcomes: no blob (404), a bundle that opens but predates the field (409), a
code that does not open it (400, fail-closed at the KDF, nothing written).

The wiring is asserted by an AST walk from func main() to the Options field, not by grep.
2026-08-04 13:41:12 +02:00
admin 856a127cd6 v0.124.1: the repair record must survive the probe that did NOT feed the hub (R-190)
gates / gates (push) Successful in 6s
v0.124.0's transition record never reached the hub, and only the live run showed
it. The capability reported degraded for "one cycle" — the probe call that did the
repair. But probeAll is invoked independently by the self-check log and by the
collector building a host report. On the demo box the repairing call was the log's
(09:39:34, journal shows the self-repair and degraded=1) and the report three
seconds later found the grant present and sent ok. The agent's journal had the
record; the hub had nothing. That is the silence R-190 is about, re-created inside
its own mitigation, with every unit test green.

Fixed with a latch on TIME, not call count: a confirmed repair reports for 20
minutes, which exceeds the 900s report interval, so at least one report must carry
it. It clears on its own and is per tier.

Two hollow tests caught and fixed on the way — one asserting a value it built
itself, one asserting the latch helper rather than the path consuming it (its
red-proof duly passed). The decisions now live in storeGrantHealthyVerdict and
storeGrantRepairedVerdict and the tests call those.
2026-08-04 09:44:56 +02:00
admin 257c4d85c0 v0.124.0: a lost storage grant repairs itself, and says that it was lost (R-190)
gates / gates (push) Successful in 7s
R-190 is a grant that worked at 04:44 on 2026-08-03 and was gone by 09:24, with a
reinstall, logged pveum activity and cluster-log entries all ruled out. The cause
is open; the resilience need not wait for it.

Everything needed already existed and had only ever been called once: the root
wrapper's `grant` verb, its sudoers vector (`grant *`, any storage id — confirmed,
not assumed), and the exact command. The verb had only ever run at storage
creation — the "built but never wired" shape in a verb rather than a seam.

The probe now runs that wrapper on a missing grant and re-reads ONCE to confirm,
the pbsdr R-22 shape including its restraint.

The record is the half that matters. A repair leaving only "ok" behind destroys
the only evidence a permission vanished, so a recurring loss becomes undetectable
— worse than the fault. A confirmed repair therefore reports DEGRADED for exactly
one cycle with the explanation in Feature, because that is the field the hub puts
in the operator's email (Reason does not travel). Nothing new was built: the hub's
existing ok->degraded->ok edge is the channel, so one loss produces one alert pair.
No wire change, no hub change, no new event type.

Bounded at one attempt per tier per hour: a storage can be unreadable for reasons
an ACL cannot fix, and re-granting every cycle is a repair loop wearing a fix's
clothes. A failed repair never masks the fault.
2026-08-04 09:38:27 +02:00
admin 72161f6cf0 REPORT: correct the manifest commit hash (311dc06)
gates / gates (push) Successful in 7s
2026-08-03 19:04:44 +02:00
admin 03b58cec0a REPORT + CONTEXT + REUSE: R-185 closed, with the corrected root cause
gates / gates (push) Successful in 7s
The installer defect was NOT PVE_STORAGES as the row and the task assumed: the
create arm of configure_backup_target grants, the Scenario-F reuse arm did not.
Also records the measured trap (an ungranted path answers with INHERITED
privileges, not empty and not 403), the deviation from the spec's suggested
Prober generalisation in favour of the existing poolReadStatus precedent, the
hollow test caught before it shipped, and that demo-hp carried the same drift and
was fixed.
2026-08-03 19:04:15 +02:00
24 changed files with 2220 additions and 474 deletions
+46
View File
@@ -0,0 +1,46 @@
---
paths: ["internal/backup/**", "internal/pbs/**", "internal/pbsdr/**", "internal/dr/**"]
---
# Backup, PBS and DR
`internal/backup/` is the vzdump runner, restore-test scheduler and report store. `internal/pbs/` is
the fingerprint-pinned PBS-API client plus the verify maintenance loop. `internal/pbsdr/` and
`internal/dr/` carry the DR tier and recipe halves.
## The three PBS laws
1. **Set-only.** `pvesm remove` **DELETES the encryption key**. Re-apply configuration; never remove
and re-add a PBS storage to change it.
2. **Secret on stdin.** A token secret is passed on stdin, never as an argv the process table shows.
3. **Verify the pin BEFORE consuming the secret.** A fingerprint check after the secret has been sent
protects nothing.
## Verify is server-side, and its default skips the work
The agent drives verification **remotely** via the PBS API; `proxmox-backup-client` has **no** verify
subcommand. `POST .../verify` defaults to **`ignore-verified=true`, which SKIPS already-verified
snapshots** — send `ignore-verified=false` to actually re-read and detect corruption. A verify that
skipped everything reports success.
## Presence is not success
A timestamp recording an **attempt** is not evidence of a **result**. Where a status field travels
beside a timestamp, the verdict must consult **both** — or the timestamp must record only successes.
Ask of any timestamp: *what exactly must have happened for this to be set?* If the answer is "we
tried", it cannot answer "did it work".
**Corollary:** when a verdict changes which field it counts from, the alarm text changes with it.
Leaving a message reading `last run 8h ago` while alarming on a six-day-old **success** turns a true
alarm into one the operator dismisses.
## Prune is server-side now
`DatastoreBackup` carries **no** `Datastore.Prune`. Boxes set `keep_last: 0` and the off-site endpoint
runs the prune jobs. **Box tokens stay write-only — never widen that grant** (R-89).
<!--
The ignore-verified default is the sharpest instance of the "absent log line" class in this repo: a
verify that silently skipped every snapshot completes fast, exits clean, and reports the same shape
as one that read every byte.
-->
+26
View File
@@ -0,0 +1,26 @@
---
paths: ["internal/capability/**", "internal/storage/**", "internal/localapi/**", "internal/hub/**", "internal/guesthook/**"]
---
# A health check issues no block I/O
No `statfs`, no `getdents`, no read, write or `fsync`**not even behind a timeout**.
A probe that touches a wedged device enters uninterruptible sleep, survives `SIGKILL`, and cannot be
recovered until the device returns or the host reboots — so `systemctl restart` hangs too. A timeout
protects the caller's control flow and nothing else: the blocked thread remains.
**Liveness is decided from `/proc` and the kernel's own state**, never by reading or writing the
filesystem.
<!--
Measured, R-117 spike §6.3 (felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md):
a probe stayed in D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT
needs journal access); and statfs/getdents returned HEALTHY on a namespace that EIOs every byte —
fast, and wrong.
This rule used to be duplicated verbatim in felhom-agent/CLAUDE.md with a note explaining that
felhom.eu/CLAUDE.md "does not load in an agent-only session". That reasoning was correct before
path-scoped rules existed. The single source is now felhom.eu/CLAUDE.md "Code quality rules"; this
file is the scoped copy that loads exactly where health checks are written. (2026-08-06)
-->
+44
View File
@@ -0,0 +1,44 @@
---
paths: ["internal/localapi/**", "internal/authz/**", "internal/guesthook/**"]
---
# Local API, authz and guest hooks — the per-guest blast radius
`internal/localapi/` is the narrow per-guest local API: token store, disks/format, guest binds,
controller swap, stale-lock recovery, pinned self-signed leaf. `internal/authz/` is the operator
signed-op verifier (SSHSIG) plus the durable nonce store. `internal/guesthook/` installs the
pre-start self-heal hookscript.
> **Overlap note:** `health-checks.md` also matches `internal/localapi/**` and
> `internal/guesthook/**`. That is deliberate — both rules apply there and both load. Neither
> supersedes the other.
## Scoping is the whole security property
This API is reachable **from inside a customer guest**. Every route must be scoped to the guest that
called it — a route that can name another guest's id has escaped its blast radius. Fail **safe to
protected**: an unrecognised or unresolvable caller gets less access, never more.
## Replay protection must survive a restart
**`authz.MemoryNonceStore` on a real host is a defect** — replay protection dies on restart. Use
`authz.FileNonceStore`. The memory store exists for tests.
## The token is a hash on disk, plaintext only at mint
The store keeps **hashes**. The plaintext token exists in exactly one place, `bootstrap.json` on the
PVE host — so a "read the token" step means reading that file, and a lost token is re-minted, never
recovered.
## Binds can brick guest boot
| Do not | Because | Use |
|---|---|---|
| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can **brick guest boot** (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
| `isHostMountpoint` to reconcile bind state | a boolean cannot converge stacked double-binds (the `/mnt` doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
<!--
Why fail-safe-to-protected rather than fail-closed: this API also carries the recovery paths. A hard
refusal on an unresolvable caller would make a half-broken guest unrecoverable through the very
interface built to recover it. Less access, never none.
-->
+44
View File
@@ -0,0 +1,44 @@
---
paths: ["internal/proxmox/**", "internal/reconcile/**", "internal/signedjobs/**"]
---
# Proxmox — the API contract, and how destructive work is gated
`internal/proxmox/` is the API-first `Client` plus the fenced root-CLI `Privileged`.
`internal/reconcile/` is the reconcile engine, reversibility gate, op journal and crash recovery.
`internal/signedjobs/` holds the operator-signed destructive executors (wipe, decommission).
## A 200 on the POST is not success
**Every mutating op is async**: it returns a **UPID**, and `WaitTask` must assert
`exitstatus == "OK"`. Authorization can fail at *task execution* long after the HTTP call returned
200. Treating the POST's status as the result is how a failed destroy reads as a successful one.
## The privsep token gotcha
A `--privsep 1` token's rights are the **intersection** of the backing user's permissions **and** the
token's own ACLs. The role must be granted on **both** or every call 403s. The same intersection rule
bites on PBS (`token ∩ user`).
## TLS
**SHA-256 leaf-cert pinning** against the self-signed host cert. **No insecure default**, ever. The
pin is the raw leaf-DER sha — the SAN is never checked, so a cert rotation changes the pin and the
agent must be re-pinned.
## The destructive path — never the direct call
| Do not | Because | Use |
|---|---|---|
| `Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
| add a method to `proxmox.Privileged` | breaks the 3-exception root-CLI fence (`routing_test.go`) | `proxmox.Runner` + a new sudoers `Cmnd_Alias` + `validate.go`-style checks |
| treat `ListLXC` output as "guests we own" | audit A1 — pre-v0.62.0 the stale-lock reaper did exactly this, contained only by the pool-scoped token | intersect with `Client.Pool` membership (`staleLockController.Guests()`); **fail safe on read failure** |
Full trap table: `REUSE.md` §3. Every guest joins the `felhom` pool — `VM.Audit` comes from the
`/pool` grant, not from a per-guest ACL.
<!--
The fence is not stylistic. It is what makes this component auditable: two types, one of which can
only speak HTTP and one of which can only shell out, with a test asserting neither crosses. A single
convenience method on Privileged that also makes an HTTP call would end that property silently.
-->
+49
View File
@@ -0,0 +1,49 @@
---
paths: ["internal/storage/**", "internal/escrow/**"]
---
# Storage and escrow — format safety and zero-knowledge recovery
`internal/storage/` is the storage observer, durable IDs, role/claim classifiers, `SudoHostOps` and
the watchdog. `internal/escrow/` is the PBS-key escrow with its zero-knowledge recovery code.
> **Overlap note:** `health-checks.md` also matches `internal/storage/**`. Deliberate — both rules
> apply there and both load.
## Never format the device you inspected
**AGENT-001 is a TOCTOU:** acting on the caller's `req.Device` (or any remembered `/dev` path) after
inspection lets `/dev` re-enumeration retarget the node to a **different physical disk**. Format the
**re-resolved** device — `Server.reresolveWipe` / `reresolveBlank`.
**Never exec raw `mkfs.*`** (including `Binaries.MkfsExt4`/`MkfsXfs`): sudoers no longer allowlists
raw mkfs, and going direct bypasses the claim filter and the wrapper's re-checks. Use
`SudoHostOps.Format`, which routes through `felhom-mkfs-guarded`.
## The two durable-ID schemes refuse each other
They are not interchangeable, and each returns a `binding_mismatch` for the other's scheme:
| Purpose | Scheme | Resolver |
|---|---|---|
| wipe confirmation | `byid:` / `byuuid:` | `ResolveDurableDevice`, `DiskInfo.WipeDurableID` |
| enrolled-storage remount | `uuid:` | `ResolveStorageDevice` |
Using `DiskInfo.DurableID` (a `uuid:`) as a wipe-confirmation id is F20-BUG2.
## Drive data is never taken by force
Plain `umount` only — **never `-l`, never `-f`**, and never any format operation under
`/mnt/felhom-drives`.
## Escrow is zero-knowledge, and a fetch failure is not a wrong code
The server holds no client key; a no-key restore fails with `missing key`. **A fetch failure must
never be reported as a wrong recovery code** — that told a customer their correct code was bad, in
hundredths of a second, when checking a code actually takes about one. Distinguish "we could not
reach the store" from "the code did not match", always.
<!--
The escrow recovery-code "flake" was a REAL defect, not a flake. "Known flake, re-run" needs evidence
before it is said out loud — that phrase cost this project a real finding once.
-->
+35
View File
@@ -29,6 +29,41 @@ root=$(git rev-parse --show-toplevel 2>/dev/null) || {
}
cd "$root" || exit 1
# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
# Refuse a push from a clone outside the felhom workspace.
#
# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
# fixed by writing it down again — it has to be asserted where it can bite.
#
# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
#
# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
# failed — this hook must not brick a legitimate clone on a different host.
#
# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
root_real=$(pwd -P) || root_real=""
if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
case "$root_real/" in
"$ws_real"/*) : ;; # inside the workspace — proceed
*)
echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
echo " clone: $root_real" >&2
echo " expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
echo " Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
echo " and state that you did in the session report." >&2
exit 1
;;
esac
fi
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
+243
View File
@@ -1,3 +1,246 @@
## v0.127.0 — a mount Felhom itself made is not "something else" (2026-08-06, R-220)
**After a rebuild the customer's own drives could not be re-attached, and the refusal named an action
they could not perform.** `GET /api/disks/candidates` returned `initialize: [], attach: []` while both
drives sat there, and the deploy refused with *"choose an attached drive from the list"* — a list that
was empty. Measured live **three times**: CAMPAIGN-11 Phase 1, and twice on the R-201 re-walk.
**The mechanism.** Enrolment mounts a drive **twice** — at the managed `/mnt/felhom-drives/<name>` and
at the raw `/mnt/<name>` it creates on the host. **The host survives a guest rebuild; the controller's
registry does not.** So `classifyClaim` saw a mount outside the managed prefix and correctly concluded
"claimed by something else" — about Felhom's own mount.
**The fix is CORROBORATED, not a widened prefix.** A mountpoint outside `/mnt/felhom-drives` is
forgiven **only when the same device is ALSO mounted under the managed path** — a pairing that only
Felhom's own enrolment produces. A disk another system is using, at `/srv/data` or `/media/x` or even
`/mnt/someone-elses-disk`, has no such counterpart and **is still refused**. That fence has its own
test, and its red-proof shows an over-wide fix offering `/mnt/someone-elses-disk` for formatting.
**Read from `/proc/mounts`, deliberately.** The lsblk invocation is pinned **verbatim** in
`configs/felhom-agent.sudoers` (`lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/*`), so switching it to
the plural `MOUNTPOINTS` would have meant shipping a sudoers change with the binary — a far larger
blast radius than this finding warrants. `/proc/mounts` is world-readable: **no sudo, no new allowlisted
command, no config change.**
**Fail-safe:** an unreadable mount table corroborates **nothing**, so the device classifies exactly as
it did before this change — refused. "We could not corroborate" must never read as "it is ours".
Tests: `claim_r220_test.go` — the own-drive case, the foreign-mount fence over four paths, and the
corroboration itself (both mounts required; a lone raw mount vouches for nothing; another device's
managed mount does not vouch for this one; an unreadable table corroborates nothing).
**Red-proofs:** removing the exemption refuses the customer's own drive again
(*"device is mounted at /mnt/adatok (sdb)"*); over-widening it to any `/mnt/*` path breaks the fence.
## docs — CLAUDE.md becomes a core plus path-scoped rules (2026-08-06, R-229 leg (b)) — no version bump
**Documentation only. No Go changed, nothing built, nothing deployed.** `go build`/`vet`/`test` green
and unchanged.
**175 -> 99 effective lines** (207 -> 103 raw, 14,093 -> 6,267 bytes). The release/publish-train
section was the largest block and the `felhom-build-deploy` skill already carries the procedure, so
the core points at it instead of restating a table that drifts from the script. The package layout
went the same way as the controller's: `REUSE.md` and the tree are its home, and the per-package
annotations that were doing real work moved into the rule file for the area they describe rather than
being deleted.
**New:** `.claude/rules/{proxmox,localapi,backup,storage}.md`, all `paths:`-scoped, all <=46 effective
lines, joining the existing `health-checks.md`.
**Kept in the core deliberately** — it is the only part re-injected after `/compact`: the root-CLI
fence and its three named exceptions (breaching it is how this component stops being auditable), the
destructive-op gate, the prove-ownership rule from audit A1, the gate entry point, the F9
live-validation fence, trunk-based with its revert-and-report escape hatch, and the end-of-session
checklist.
**Glob overlap, stated rather than silently resolved:** `health-checks.md` matches
`internal/{localapi,guesthook}/**` and `internal/storage/**`, which `localapi.md` and `storage.md`
also match. Both rules load in those directories and neither supersedes the other; each new file says
so in its own text so a reader who sees two rules fire is not left guessing which wins.
## docs — the "CI is still owed" claim was stale; corrected (2026-08-06, R-229 part 2) — no version bump
**One sentence, no code.** This file asserted that continuous integration was still owed
(`felhom.eu` `OPEN-ITEMS.md` R-168). **R-168 was CLOSED on 2026-08-02** — a Gitea Actions runner
re-runs each repo's gate entry point on every push and emails the operator on failure. Found while
confirming this session's own push by run ID, which is the check that caught it.
The same stale sentence was in four instruction files across all four repos and is corrected in all
four. In `felhom-agent/CLAUDE.md` it **contradicted the same file's release section**, which already
said R-168 mails the failure — a contradiction inside one instruction file, which is the exact class
the R-229 work exists to find.
## docs — expired and contradictory blocks removed from CLAUDE.md (2026-08-06, R-229) — no version bump
**Documentation and gate registration only. No Go changed, nothing built, nothing deployed.**
Surgical corrections; the file was deliberately **not** restructured (that is deferred, R-229).
- **Deleted the expired TEMPORARY block.** It read *"felhom-pve is at a remote site (until
~2026-08-02) … Delete this block on return"* and was still being read as current fact on
**2026-08-06**, four days past its own deadline — while `felhom-controller/CLAUDE.md` asserted the
opposite. The location-independence fact worth keeping (`localapi` binds `169.254.253.1:8443` on
`vmbr9` since the R-50 island migration) moved to an HTML comment.
- **Every component version literal is gone** from effective text, including
`felhom-agent --version → 0.115.0` and the `go.mod` Go directive. Versions change several times a
day; ask the hub's `/hosts` + `/configs` or the box.
- The drill-VM claim and the host addresses now point at `documentation/operations/nodes.md`, which
already stated both correctly. **This file's drill-VM claim was the correct one** — confirmed by
`qm list` on demo-hp.
- The R-115/R-188/R-186 release **narratives** moved to an HTML comment and to the
`felhom-build-deploy` skill; the **directives** stayed (never hand-roll the build; the
build → tag → publish → push order; reproducible `-trimpath -buildvcs=false`).
- The health-check block-I/O rule became `.claude/rules/health-checks.md`, scoped to the five
packages where health checks are written. It had been duplicated from `felhom.eu/CLAUDE.md` *with a
note explaining that that file does not load in an agent-only session* — correct reasoning, made
obsolete by path-scoped rules.
`agent_gates.py` now registers **`instructions`** (shared, `felhom.eu/scripts/`, never copied).
Full accounting: `felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md`.
## v0.126.0 — a fetch failure is not a wrong recovery code (2026-08-06, R-224)
**A hub the agent could not reach was reported to the customer as a bad recovery code.** Measured live
on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and a **correct,
current** recovery code, the customer was told their code did not open their package — **in 0.0556 s**,
against ~1.0 s for a genuine unseal. No unseal was attempted. F4 produced the same message in 0.0299 s
with this agent stopped.
**The discriminator existed here the whole time and this boundary threw it away.** `recover.go` fails
at four distinguishable points; the local-api handler had cases for two of them and a `default` that
answered *"the recovery code did not open the sealed bundle, or the bundle could not be fetched"*
one sentence for two situations, only one of which is the customer's doing.
**The fix is a value, not a log line.** `escrow.ErrBundleFetch` joins the fetch leg's error, and the
handler routes it to **502** with its own words: *"the sealed recovery bundle could not be fetched from
the hub — the recovery code was NOT used and nothing was written."* 502 rather than 4xx because the
request was not bad; an upstream dependency failed. The `default` now carries **only** the fail-closed
wrong-code case and says so without the "or".
Four situations, four statuses — **502** fetch failed · **400** the bundle was fetched and refused the
code · **404** the hub holds no bundle · **409** the bundle predates the repository-password field.
The controller classifies on the STATUS and must never parse these sentences.
⚠ **A GREEN TEST NAMED THIS DEFECT AND DID NOT PREVENT IT.**
`TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct` has said since v0.125.0 that *"the operator must
not be sent to re-read their recovery code because the hub was unreachable"* — and it passed
throughout, because it asserted this package's error **string** one layer below where the merge
happened, and a string is something no caller can branch on. It now asserts the sentinel, and its
consequence-level twin asserts the STATUS at the boundary the customer's message is derived from.
**Prefer the test that asserts the consequence over the one that asserts the mechanism.**
Tests: `recover_test.go` (fetch classifies as `ErrBundleFetch`; a wrong code does **not**; an absent
blob keeps its own identity) and `localapi/escrow_recover_class_test.go` (each situation's status, and
a standalone assertion that fetch-failure and wrong-code never share one). **Red-proofs:** removing the
`%w` join fails the sentinel test; deleting the handler case makes both answer `400` with the
wrong-code sentence — the exact pre-fix code, and the exact defect CAMPAIGN-11 measured.
## v0.125.0 — the agent opens the sealed bundle and returns one field (2026-08-04, R-199 links 78)
**Link 7 had one production caller and it was a `--selftest`.** `UnwrapIdentityBundle` has existed
since slice 10D.1 and the only thing that ever called it was `runSelftestIdentityConsume`, reading the
recovery code from an environment variable by hand. **Link 8 did not exist at all:** that selftest
writes the whole bundle JSON to a file, and its success message named `tunnel_token + pbs_token`
an enumeration that was accurate when written and became a MISSTATEMENT the moment v0.77.0 sealed the
offsite repository password into the same bundle. Anyone reading that output would conclude the
repository password was not there. It now names what THIS bundle actually carried and what it did not.
**`POST /escrow/recover-offsite-password`** on the pinned local API: the controller supplies the
customer's recovery code, the agent fetches this host's own sealed blob from the hub
(`hub.Client.FetchIdentityEscrow`, hub ≥ v0.94.0, self-scoped by the per-host key), unseals it, and
returns **only the offsite restic repository password** plus its sha256.
**Only that field, on purpose.** The bundle also carries the tunnel token, the PBS token and the WG
private key. The controller is a trust tier down and needs none of them; returning them would widen
the blast radius of a controller compromise for nothing. Narrowing costs nothing now and is not
recoverable later.
**Why the agent and not the controller:** `age` is an agent runtime dependency and is deliberately
absent from the controller image; the blob is a host-scoped object whose only writer is this agent
under the per-host key, so the read is that write's mirror.
**R's handling is the tightest rule in this release.** It arrives in the request body over the pinned
channel, is held in memory for one call, is cleared on the success path AND every failure path, is
never written to disk, never an argument in a process list, never logged at any level including
inside an error, and is never echoed. `UnwrapIdentity` already stages only the blob and the recovered
plaintext in a temp dir it removes; a test redirects TMPDIR and asserts **the tree is empty
afterwards** — emptiness rather than a content scan, because a content scan is defeated by a later
call overwriting the leaked file, which is exactly how the first version of that test passed its own
red-proof while R sat on disk.
Three outcomes are distinct rather than one generic failure: no blob (404 — no ceremony has run), a
bundle that opens but predates the field (409 — a pre-fork-4 blob, which cannot be retro-fitted), and
a code that does not open it (400 — fail-closed at the KDF, nothing written). Sending an operator to
re-check a correctly typed recovery code because the hub was unreachable is the mistake this avoids.
**The wiring is asserted by an AST walk**, not a `strings.Contains`: `main``runDaemon`
`buildLocalAPIServer`, where an `escrow.OffsiteKeyRecoverer` is constructed and passed as
`localapi.Options.EscrowRecovery`, and its fetcher calls the DAEMON's own hub client (the self-scoping
that makes cross-host retrieval impossible is a property of which key is used). This project's
built-but-never-wired count is six and links 67 were two of them; the fix must not become the seventh.
## v0.124.1 — the repair record must survive the probe that did NOT feed the hub (2026-08-04, R-190)
**v0.124.0's transition record did not reach the hub, and the live run is what showed it.** The
capability reported degraded for "one cycle" — meaning the probe call that performed the repair. But
`probeAll` is invoked **independently** by the periodic self-check log and by the collector building a
host-report. On the demo box the repairing call was the log's (`09:39:34`, journal shows
`GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED` and `degraded=1`), and the host-report built three
seconds later found the grant present and sent **`ok`**. The agent's journal had the record; the hub
had nothing; the operator would have learned nothing.
That is the exact silence R-190 is about, re-created inside its own mitigation — and every unit test
passed while it was true.
**The fix is a latch on TIME rather than on call count.** A confirmed repair is reported for
`storeGrantRepairReportWindow` (20 minutes), which comfortably exceeds the 900 s host-report interval,
so at least one report must carry the transition. It clears on its own — a permanently degraded
capability would be its own false alarm — and it is per tier.
**Two hollow tests were caught and fixed on the way**, both the same shape this repo keeps finding: a
test asserting a value it constructed itself, and a test asserting the latch HELPER rather than the
path that consumes it — whose red-proof duly passed. The decisions now live in
`storeGrantHealthyVerdict` and `storeGrantRepairedVerdict`, and the tests call those.
## v0.124.0 — a lost storage grant repairs itself, and says that it was lost (2026-08-04, R-190)
**R-190 is a grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24** — with a
host reinstall, logged `pveum` activity and cluster-log entries all ruled out by measurement. The
cause is still open. The resilience does not have to wait for it.
**Everything needed already existed and had only ever been called once.** The root wrapper
(`felhom-backup-target-apply grant <id>`), its sudoers vector (`grant *`, any storage id, confirmed
not assumed), and the exact command were all in place — and the `grant` verb had only ever run at
storage CREATION. That is the *built but never wired* shape, in a verb rather than a seam, and it is
this project's seventh instance.
**What v0.124.0 does:** when the store-grant probe finds the grant absent on a tier the box depends
on, it runs that wrapper and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including
its restraint: one attempt, one confirmation, and anything still wrong stays loudly wrong.
**THE RECORD IS THE POINT, AND IT IS THE HALF R-190 IS ACTUALLY ABOUT.** A repair that leaves only
`ok` behind destroys the only evidence a permission vanished, so a recurring loss becomes undetectable
forever — strictly worse than the fault it fixes. So a confirmed repair reports **DEGRADED for exactly
one cycle**, with the explanation in `Feature`:
```
backup tier felhom-backup: the agent's storage grant was MISSING and has been AUTOMATICALLY
RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)
```
**Nothing new was built to carry it.** The hub's existing ok→degraded→ok edge is the channel — it
alerts and e-mails on the first edge and logs the recovery on the next cycle, so one loss produces
exactly one alert pair. No wire change, no hub change, no new event type. `Feature` carries the text
because that is the field the hub interpolates into the operator's e-mail; `Reason` does not travel.
**Bounded (Scenario F):** one attempt per tier per hour, in memory. A storage can be unreadable for
reasons an ACL cannot fix, and a re-grant on every report cycle is a repair loop wearing a fix's
clothes. An agent restart re-arms it, which is correct — a restart is exactly when a box should
re-check what it depends on.
**A failed repair never masks the fault:** the capability stays degraded with the failure in its
reason, and a repair that "succeeded" but did not survive the re-read is reported as needing a human.
## v0.123.0 — a tier the box cannot READ now says so (2026-08-03, R-185)
**The missing permission is one command. The silence was the defect.** On demo-felhom the agent's PVE
+81 -194
View File
@@ -1,216 +1,103 @@
# CLAUDE.md — `felhom-agent`
> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
> Stable orientation only — **current state lives in `CONTEXT.md` and the top of `CHANGELOG.md`**,
> never here. Cross-repo conventions (artifact taxonomy, access, clean-tree gate, secrets,
> CHANGELOG/REPORT): workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`. Path-scoped detail:
> `.claude/rules/`.
## What this repo is
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
The operator-tier **host agent**, one per Proxmox host, owning **all** Proxmox interaction:
provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a
narrow per-guest local API. It is the **most privilege-sensitive component in the system**.
- Renamed former `proxmox-controller` repo.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
creds). Do not confuse them.
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
- Renamed from `proxmox-controller`.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller, Docker-only, holding no
Proxmox credentials. Do not confuse them.
- **Control plane, not data plane:** if the agent dies, apps keep serving; only management degrades.
- Pure Go stdlib + `golang.org/x/crypto`. No web frameworks.
## Read before writing code
## Doing X → read Y
- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
in the same commit that changes a shared helper or pattern.
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
`felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
| Doing | Read |
|---|---|
| writing any new code | `REUSE.md` — helpers, format-safety guards, traps, seams |
| needing current state / open threads | `CONTEXT.md` + the top `CHANGELOG.md` entry |
| Proxmox, reconcile or signed jobs | loads itself: `.claude/rules/proxmox.md` |
| local API, authz or guest hooks | loads itself: `.claude/rules/localapi.md` |
| backup, PBS or DR | loads itself: `.claude/rules/backup.md` |
| storage or escrow | loads itself: `.claude/rules/storage.md` |
| writing a health check | loads itself: `.claude/rules/health-checks.md` |
| **release, build, publish, deploy, verify a version** | the **`felhom-build-deploy`** skill — **never hand-roll it** |
| writing or reviewing a test, fixing a bug | the **`felhom-testing`** skill |
| host addresses, break-glass, node facts | `felhom.eu/documentation/operations/nodes.md` — never restate them |
| which box may I break | `felhom.eu/documentation/runbooks/target-selection.md` |
| what version is live anywhere | ask the hub (`/hosts`, `/configs`) or the box — **never a doc** |
| the authoritative design | `felhom.eu/documentation/architecture/03-host-agent.md` (locked) |
## Layout (verified against the tree)
## The root-CLI fence — API-first, exactly three exceptions
```
cmd/felhom-agent/ main + flags + --selftest modes + the daemon entry
cmd/felhom-opsign/ offline operator signing CLI (SSHSIG)
internal/authz/ operator signed-op verifier (SSHSIG) + durable FileNonceStore
internal/backup/ vzdump backup runner + restore-test scheduler + report store
internal/capability/ live sudo-policy capability probe (degradation visibility)
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/desired/ hub desired-state syncer (envelope observer)
internal/escrow/ PBS-key escrow (zero-knowledge recovery code)
internal/guesthook/ pre-start self-heal hookscript install
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
internal/localapi/ per-guest local API: token store, disks/format, guest binds, controller swap,
stale-lock recovery, pinned self-signed leaf
internal/log/ slog setup
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop
internal/provision/ guest bootstrap back-half (token mint → bootstrap.json → pct bind)
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
internal/storage/ storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
```
## Build / run
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
host — build and run live tests locally.
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).
> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
> and undownloadable, where a documented-path reinstall would have silently downgraded them while
> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
> verifies by an **independent download** rather than trusting the publish step's own output.
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
> than reimplementing it.
>
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
> protected is asserted directly instead: the gate now also refuses a **published version with no
> tag**. If the push fails after a successful publish the script says so loudly and prints the
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
>
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
> version yourself:
>
> ```bash
> V=0.122.0
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
> sha256sum /tmp/felhom-agent-check
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
> ```
>
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
> ever changes, change both or one version name means two binaries again.
| Step | Where | One-liner |
|---|---|---|
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
| Deploy | felhom-pve | backup `.bak-<old>``install -m0755``systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
## Proxmox model (the load-bearing rules)
This is in the core because breaching it is how this component stops being auditable.
- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
`Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
the POST is **not** success; authorization can fail at task execution.
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
AND the token's ACLs — the role must be granted on **both**, or every call 403s.
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
`Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors.
- **`Client` never shells out; `Privileged` never makes HTTP calls** — asserted by `routing_test.go`.
Adding a method to `proxmox.Privileged` breaks the fence; use `proxmox.Runner` plus a new sudoers
`Cmnd_Alias` and `validate.go`-style checks (`REUSE.md` §3).
- **Destructive ops go through the reconcile gate / signed-jobs path.** Never call
`Client.DestroyLXC` / `Vzdump` / `SetConfig` ad-hoc — that skips classification, signature,
per-guest serialization and crash recovery.
- **Ownership must be PROVEN, never assumed.** A raw `ListLXC` list is not "guests the agent owns";
intersect with `Client.Pool` membership and fail safe on a read failure (audit A1).
## Demo host (for live tests)
## Gates — ONE entry point
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
**Which box is safe to break, and what may be done to each:
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
**Run `python3 scripts/agent_gates.py` from the repo root after ANY change here.** It runs this
repo's gates — `reuse_refs_check` and `instructions_gate`, both the **shared** copies in
`felhom.eu/scripts/`, never copied into this repo (a copy recreates the drift they detect; an absent
sibling clone FAILS). `--fast` selects the gates touching no network and no container runtime; today
that is all of them. **A missing gate is a FAILURE, never a skip.**
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
>
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
**per-clone** — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the session
report when you use it** — CI re-runs the same entry point on every push and **emails the operator on
failure**, so a bypass is noticed even though it is not blocked (R-168, CLOSED 2026-08-02).
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
> (`cygpath -w` for the local scp path; CRLF hazard on config files).
<!--
WHY ONE ENTRY POINT (2026-08-02, R-29): a census of all gates across the four repos found every check
a CLAUDE.md names was passing, and two of the four nobody is told to run were failing. This repo was
the extreme case — nothing ran against it at all, and 90 cited paths were checked by no one.
-->
## Live validation — the fence
Exercise the **SERVER-SIDE PIPELINE** a real user triggers, end-to-end. **The forbidden shortcut is
BYPASSING it** — the F9 episode was a raw guest-attach with hand-set state, and it proved nothing.
`claude-in-chrome` is NOT available on DooPlex. Invoking the exact endpoint the UI invokes is an
acceptable proxy — **say which method was used**. Low-level mechanism tests where the direct call IS
the mechanism are exempt.
## Conventions
### Trunk-based — no branches
All shippable work commits **directly to `main`**; `main` equals what is deployed.
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — cumulative log, newest on top.
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
guess** when you'd otherwise invent input/output.
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
- **Trunk-based — no branches.** All shippable work commits directly to `main`; `main` equals what is
deployed. Report-only artifacts (audits, findings, fixspecs) go to `felhom.eu/documentation/`.
- **Unattended escape hatch:** if a fix cannot be cleanly verified and shipped, **revert and report**
— never park it on a branch.
- **Logging**: the slog logger fans out to journald (configured level) plus the always-DEBUG
`applog.Ring` (remote pulls). English, keys-never-values, durations on outcomes. Full rules:
`felhom.eu/documentation/runbooks/logging-conventions.md`.
- Update `REUSE.md` in the same commit that adds, changes or deprecates a shared helper or pattern.
### Live validation
## End-of-session checklist
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
mechanism tests where the direct call IS the mechanism are exempt.
## Workflow & artifacts
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
CHANGELOG + REPORT.md.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
real customer data — demo scratch guests are fair game.
- **`CHANGELOG.md`** (cumulative, newest on top) and **`REPORT.md`** (overwritten with this run only)
— in every repo touched.
- **`CONTEXT.md`** — decisions, state, what is next.
- **`REUSE.md`** — if a shared helper or pattern moved.
- **A finding goes in `felhom.eu/documentation/backlog/OPEN-ITEMS.md` first**, never only in a report
or an audit.
- **Confirm your own last push's CI run went green, by run ID** — CI mails on failure, which is a PUSH
signal; this is the PULL check that catches a lost or unread mail. An unchecked green is an
assumption, not an observation.
+62
View File
@@ -3,8 +3,70 @@
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
> entry = current); the end-of-task detail lives in `REPORT.md`.
## R-199 (v0.125.0) — links 68 of the recovery chain, assembled and walked
`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
(`hub.Client.FetchIdentityEscrow``GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
repository password plus its sha256.
**Rules that must not erode:**
- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
at age's KDF, nothing written).
- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
`main``runDaemon``buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
`Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
Links 6 and 7 were two of this project's six built-but-never-wired instances.
**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
recovered password, reopens a repository, or restores a file.
**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
into the same bundle — anyone reading it would conclude the password was not there. It now names what
THIS bundle carried and what it did not.
## Current
- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
`FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
restore-testable and nothing said so.
- **The permission question is asked directly**, because unlike the listing it has a definite
answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
degraded alert applies with **no hub change**.
- **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
403 — it carries the privileges INHERITED from the box-wide `/` grant
(`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
changing that constant (`storeGrantRequiredPriv`, red-proved).
- **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
ok when it could not ask.
- **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
correctly picking the 08-02 archive (08-03 had not settled 24 h).
- **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
None touches data; all three cost attention, which every other signal depends on.
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
+29 -269
View File
@@ -1,281 +1,41 @@
# REPORT — R-189 · R-188 · R-186: three ways the signals lied about themselves
# REPORT — felhom-agent v0.127.0: a mount Felhom made is not foreign (R-220)
**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.121.1 → v0.122.0** (`7581f81`) · released,
published, verified by an independent download **and by rebuilding it**, deployed to demo-felhom.
`felhom.eu`: register + docs only, **no hub change and no hub bump** — the hub already reads
`restore_tests[]`; the defect was that the agent stopped sending them.
**Scope: the host half of R-220.** The customer-facing refusal message is the controller's half and
ships as felhom-controller v0.203.0.
---
## What changed
## 1. Baselines, re-read on arrival
| File | Change |
|---|---|
| `internal/storage/claim.go` | `claimFacts.felhomOwnedMounts`; `classifyClaim` forgives a non-managed mountpoint **only when corroborated**; `felhomOwnedMounts()` + `procMounts()` |
| `internal/storage/hostops.go` | `mountTable` seam (nil ⇒ real `/proc/mounts`) |
| `internal/storage/claim_r220_test.go` | new — the own-drive case, the fence, and the corroboration's four edges |
| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-agent` | `3d0a1d615d11` | `v0.121.1` | **yes** |
| `felhom.eu` | `c9a3e48b2106` | hub `v0.91.1` | **yes** |
## The shape chosen, and why (§7.3)
Highest register ID in use was **R-189**; no new IDs were needed — all three rows already existed.
(Grep confirmed R-190+ free, in case one had been.)
**Candidate (b): the claimed check distinguishes a mount Felhom made from a foreign one** — the task
called it "nearer the truth" and it is, because the host and its knowledge survive the rebuild while
the guest's registry does not. Candidate (a) — having the rebuild path clear the raw mounts — would
have made correctness depend on a cleanup step running, and a cleanup that does not run leaves exactly
today's defect.
## 2. Scenario H — the reproducibility measurement (Part 3, done first on purpose)
**The discriminator is corroboration, not a path prefix**: the same device must ALSO be mounted under
`/mnt/felhom-drives`. Only enrolment produces that pairing.
**Before**, one commit, same source, same toolchain, same ldflags — the only difference is whether the
tag existed when the build ran:
**`/proc/mounts` rather than `lsblk MOUNTPOINTS`**, because the lsblk invocation is pinned verbatim in
the sudoers file; changing it would have coupled this fix to a config rollout. `/proc/mounts` is
world-readable and needs neither.
| build | sha256 | size | embedded module version |
|---|---|---|---|
| default flags, **no tag yet** | `18f4a495…` | 14 085 464 B | `v0.121.2-0.20260803133646-3d0a1d61` |
| default flags, **tagged** | `4a38f394…` | 14 085 440 B | `v0.121.99` |
| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* |
## Green gate
**After, on the real release (v0.122.0) — the three values §15.2 asks for:**
`go build` · `go vet` clean · `go test ./...`**29 packages ok** · `agent_gates.py --fast` → all OK.
| artifact | sha256 | size |
|---|---|---|
| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| Red-proof | Result |
|---|---|
| remove the `felhomOwnedMounts` exemption | **FAILS** — "device is mounted at /mnt/adatok (sdb)", the pre-fix refusal |
| over-widen the exemption to any `/mnt/*` | **FAILS** — "/mnt/someone-elses-disk was offered for formatting" |
**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops
a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from
the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different
checkout directory match.
## Not changed
**A second discrepancy fell out of the measurement and is fixed with it:** `publish-agent.sh`'s
fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's
**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which
entry point ran. Both paths now use identical flags, each commented with a pointer to the other.
## 3. Scenario E — a correct release no longer emails a failure (Part 2)
**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is
still created before anything is published, so the build and the tag describe the same commit; it
becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is
downloadable.
**The old ordering's invariant is asserted directly rather than arranged for.**
`check-published-versions.py` now carries two invariants: every tag has an installable package (as
before) **and no published version is missing its tag** (new). The second is a **bounded probe**
the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it
**prints its probe set on every run**, because a check whose coverage is invisible reads as a
guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a
token, so absence still cannot be enumerated, and the script says so in its own output.
**Live result — this release is the test:**
| release | CI runs on the release commit | outcome |
|---|---|---|
| v0.121.0 (yesterday) | #12 / #13 | one **green**, one **red** |
| v0.121.1 (yesterday) | #17 / #18 | one **red**, one **green** |
| **v0.122.0 (this one)** | **#21 (task id 96) / #22 (task id 97)** | **both green** |
**Failure modes made loud rather than tidy:** a publish that succeeds followed by a tag push that
fails now dies printing `git push origin v<ver>` — the local tag is already there, so recovery is one
line — and a publish that *fails* deletes the local-only tag so a retry is clean instead of colliding
with step 2's re-release guard.
## 4. Scenarios F and G — both directions demonstrated, then cleaned up
**G — a published version with no tag must FAIL.** A real fixture: version **0.121.2** (the frontier,
exactly where a failed tag push lands) published to the live registry with no tag.
```
0.121.2 next patch after the newest tag PUBLISHED — NO TAG
check-published-versions: 1 PUBLISHED VERSION(S) WITH NO TAG
v0.121.2 is downloadable at … but has no git tag.
git push origin v0.121.2
EXIT=1
```
**Red-proof (observed):** with the fixture still live, replacing the probe set with `[]` produced
`ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED`, **exit 0** — a green run over a published
orphan. Restored.
**Teardown:** fixture deleted (HTTP 204), absence independently re-verified (`GET → 404`), gate green
again (exit 0). No scratch tag was ever pushed.
**F — a tag with no package must still FAIL.** Demonstrated against a local stand-in serving two tags
where only one has a package:
```
ok v0.120.0: binary downloadable + tag serves its configs
FAIL v0.199.0:
- binary NOT downloadable (HTTP 404 …)
- tag does not serve configs/felhom-agent.service (HTTP 404) — a box would 404 mid-install
EXIT=1
```
**Why not a real pushed tag:** pushing one wakes CI and would have emailed the operator a **true**
alarm about a fixture — the same attention cost R-188 exists to remove. F is also already
demonstrated in the wild: CI runs **#13** and **#17** failed for exactly this reason yesterday.
## 5. R-189 — a proof that survives a restart reaches the hub (Part 1)
`restore_tests[]` came only from the in-memory `backup.Store`, whose comment read *"lost on restart;
the cadence re-populates"*. True under a timer; false since R-86, because the agent refuses to re-test
an archive it has already proven — so a lost proof is not repeated for a whole archive generation.
**What changed**
- `RestoreTestState` stores the **tier** and what was **verified** beside the archive (v3 shape).
Both are recorded **at proof time from the run's own result** — deriving them later would need a
storage-type lookup at report-building time, a network call that can fail on the one path where
failing means mislabelling a proof.
- `ProvenRestoreTests` renders the stored proofs as report entries; `Collector.SetProvenRestoreTests`
merges them with the in-memory result.
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** It falls out of what each source
means rather than from a preference: a fresh failure beats a stored success (the failure is the
news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a
tier never appears twice — the hub would read that as two tests. An unparseable timestamp counts as
**older**, so a malformed entry cannot displace a good one.
- **It refuses to lie.** A record missing the archive **or** the tier produces no entry, and run
mechanics (scratch VMID, duration) are not re-invented: an absent duration is not a claim, a
fabricated one would be.
- **The asymmetry is now in the code** (§8.1): a success *suppresses* future work so it must be
durable; a failure *causes* future work and heals itself, and persisting one would make a healed
tier keep reporting a fault.
- **`Store`'s comment is corrected in place** — leaving it is how the next reader concludes this is
handled.
**Migration, and it is visible on the live box:** a pre-R-189 record has an archive but no tier, so it
is **not** reportable. Upgrading does not retroactively make an old proof visible; the tier's next
real proof fills it in. Confirmed immediately after the deploy — still `0 restore-tests`, with the
v2 record sitting on disk.
## 6. Scenario A, live on demo-felhom — against the observation that filed R-189
**The observation being replaced (2026-08-03, 15:25):** a real offsite restore-test PASSED, the agent
was restarted 2 m 43 s later, and the hub logged `0 restore-tests` on the next two host-reports.
**The same sequence, on v0.122.0:**
```
16:44:06 restore-test tier is DUE target=felhom-pbs
archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z
reason="newest settled archive … has not been proven (last proven archive was a different one)"
16:55:21 restore-test: scratch guest torn down vmid=990000
16:55:21 backup: scheduled restore-test PASSED archive=felhom-pbs:…2026-07-27T19:55:41Z duration_s=675.1
16:55:32 systemctl restart felhom-agent ← INSIDE the 15-minute reporting window
16:55:36 hub: host-report from demo-felhom-8363b5 (… 1 restore-tests …) ← was 0
```
**The proof on disk (v3 — the tier is what the old shape lacked):**
```json
{"felhom-pbs": {"archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
"tier": "pbs", "verified": "boot+running", "proven_at": "2026-08-03T14:55:21Z"}}
```
**What the HUB stored** — read from its own database (copied with its `-wal`, freshness confirmed by
the newest row's `received_at` = `2026-08-03 14:55:36` UTC, matching the ingest line):
```json
{ "source_archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
"source_tier": "pbs", "pass": true, "verified": "boot+running",
"tested_at": "2026-08-03T14:55:21Z", "scratch_vmid": 0, "duration_seconds": 0 }
```
That report was built **after** the restart, when the in-memory store was empty — so the entry can
only have come from the persisted state. The archive, the tier, and the **original** test time
survived; the run mechanics are zero because they are deliberately not re-invented.
**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and
this time the proof outlived the process that produced it.
**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs`
(0), the validation drop-in removed and the daemon back on its defaults
(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained
deliberately** — it is the proof the staleness check reads, so deleting it would delete the result.
No `restore_test_*` event was raised, because nothing failed and nothing is stale.
## 7. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...`**29 packages ok, rc=0**, plus
`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit
were always separate commands.
| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL**`after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly |
| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass |
| B | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL**`got 3` entries, two with an empty `SourceTier`/`SourceArchive` |
| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL**`one entry per tier; got 2 for "pbs" — the hub would read two tests` |
| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier |
| — | `TestMerge_MalformedTimestampNeverWins` | unparseable ≠ newest | — | pass |
| — | `TestMerge_NilProvenSourceIsANoOp` | pre-R-189 behaviour unchanged when unwired | — | pass |
| — | `TestScheduler_ProofIsRecordedReportably` | a pass **through the scheduler** leaves a reportable proof | — | pass |
| — | `TestScheduler_AFailureLeavesNoPersistedProof` | §8.1's asymmetry, asserted not assumed | — | pass |
| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) |
| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL**`main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) |
| H | reproducibility | three identical sha256 | — (measurement, §2) | pass |
**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`,
`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test
was hollow because a perfectly regular series landed exactly on a threshold and survived its own
mutation.
## 8. Files changed
`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go`
(the comment that had become false), `internal/backup/schedule.go` (record tier + verified),
`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring),
`scripts/release-agent.sh` (ordering, reproducible build, loud half-done release),
`scripts/publish-agent.sh` (identical build flags), `scripts/check-published-versions.py` (the
converse invariant), plus `REUSE.md`, `CHANGELOG.md`, `CONTEXT.md`, `CLAUDE.md` and three test files.
**Commits** — `felhom-agent`: `7581f81` (v0.122.0). `felhom.eu`: see §10.
## 9. The independent-verification command (Part 3, recorded in `CLAUDE.md`)
```bash
V=0.122.0
git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
-o /tmp/felhom-agent-check ./cmd/felhom-agent
sha256sum /tmp/felhom-agent-check
curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
```
Both print `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` (§2).
## 10. Registers
- **R-189 → CLOSED** (shipped + proven live), **R-188 → CLOSED** (shipped), **R-186 → CLOSED**
(shipped + measured).
- **R-185 remains OPEN and untouched** — it is a missing `/storage/felhom-backup` ACL on demo-felhom,
a permission defect, not a reporting one. Nothing in this session changed it, and the priority list
says so explicitly.
- No new IDs minted. `ROADMAP.md` contains none of these three rows, so there was nothing to collapse.
- `00-capability-map.md`'s restore-proof row now records that the evidence path itself had a gap and
what closed it; `CONTEXT.md` gains **S-19** (the proof/failure asymmetry and the merge rule) and
**S-20** (the release ordering and what each step protects); `STATUS.md` rewritten for the operator
and trimmed to 83 lines.
## 11. Observations — noticed, recorded, NOT acted on
- **The proof state holds ONE record per tier, so proving an OLDER archive re-arms a newer one —
CONFIRMED after the validation, not merely predicted.** With the defaults restored, the due-check
reads: `tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" proven="…2026-07-27T19:55:41Z" —
newest settled archive has not been proven (last proven archive was a different one)`.
Surfaced by this session's own validation method: to get a fresh proof without waiting a week, the
settle lag was widened so the older, unproven offsite archive became the candidate. That overwrote
the record for the newer archive, so once the default 24 h settle returns, the newest settled
archive is no longer the recorded proof and the tier becomes due once more. **Consequence, stated
rather than left to surprise: demo-felhom will run one further unattended offsite restore-test
within 6 h, after which the newest archive is the recorded proof** — the correct steady state. In
normal operation this cannot arise, because the candidate only ever moves forward.
- **`RestoreTestState.Snapshot()` has no caller again.** The host report is now fed by
`ProvenRestoreTests`, which carries what a bare timestamp cannot. The method's doc comment says in
as many words that it should be deleted if it does not acquire one — deliberately not deleted in
this session, because removing an exported method is a change with no bearing on the three rows.
- **Ten files in this repo are not `gofmt`-clean and were already so on arrival**
(`internal/capability/probe.go`, `internal/escrow/consume.go`, `internal/mgmtplane/mgmtplane.go`,
`internal/reconcile/bringup.go`, `internal/signedjobs/runner.go`, `internal/storage/{candidates,
intent}.go` and three test files). Every file this session touched is clean; the others are
untouched, and no gate checks formatting.
- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for the restore-test
staleness check. Unchanged here and not a defect at this fleet size; it is the cost centre if the
fleet grows, and it is the reason the window read was deliberately left at 14 days yesterday.
- **`felhom.eu/CONTEXT.md` still carries duplicate standing-ruling IDs** (three `S-14`s, two `S-15`s)
from before yesterday. New rulings continue to be numbered above the collision (S-19, S-20) rather
than adding to it; renumbering the existing ones is a separate, purely editorial change.
No sudoers, no allowlisted command, no PVE surface, no format path. Every other claim signal
(system disk, read-only, LVM PV, ZFS member, member FSTYPEs, empty-topology backstop) is untouched.
+1
View File
@@ -148,6 +148,7 @@
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing**`{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
@@ -0,0 +1,188 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// Scenario H — THE SEAM IS WIRED IN THE PRODUCTION PATH, proven by walking the AST rather than by
// grepping for a string.
//
// WHY THIS TEST EXISTS AND WHY IT IS AN AST WALK. This project's built-but-never-wired count is six,
// and links 6 and 7 of the recovery chain were TWO of them: `UnwrapIdentityBundle` sat in the tree
// for two months with no caller but a `--selftest`, and the hub's blob-serving endpoints have no
// client to this day. The fix must not become the seventh. `strings.Contains` on the file would pass
// against a commented-out line, a line inside a test helper, or a line in dead code behind a flag
// nobody sets — so this resolves the call graph instead: `Options{EscrowRecovery: …}` must be
// constructed inside a function that `runDaemon` reaches, and `runDaemon` must be reached by `main`.
func parseMain(t *testing.T) (*token.FileSet, *ast.File) {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
if err != nil {
t.Fatalf("parsing main.go: %v", err)
}
return fset, f
}
// callsWithin returns the set of function names called (directly, by identifier or selector) inside
// the named top-level function.
func callsWithin(f *ast.File, fnName string) map[string]bool {
out := map[string]bool{}
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != fnName || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok {
return true
}
switch fn := ce.Fun.(type) {
case *ast.Ident:
out[fn.Name] = true
case *ast.SelectorExpr:
if x, ok := fn.X.(*ast.Ident); ok {
out[x.Name+"."+fn.Sel.Name] = true
}
out[fn.Sel.Name] = true
}
return true
})
}
return out
}
// TestEscrowRecoveryIsWiredIntoTheDaemon asserts the whole chain from func main() to the field.
func TestEscrowRecoveryIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
// 1. main() reaches runDaemon.
if !callsWithin(f, "main")["runDaemon"] {
t.Fatal("func main() does not call runDaemon — the daemon path this test asserts is not the live one")
}
// 2. runDaemon reaches buildLocalAPIServer.
if !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("runDaemon does not call buildLocalAPIServer — the local API is not built on the daemon path")
}
// 3. Inside buildLocalAPIServer, a localapi.Options composite literal carries EscrowRecovery, and
// an escrow.OffsiteKeyRecoverer is constructed there.
var optionsHasField, recovererConstructed bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
cl, ok := n.(*ast.CompositeLit)
if !ok {
return true
}
sel, ok := cl.Type.(*ast.SelectorExpr)
if !ok {
return true
}
pkg, _ := sel.X.(*ast.Ident)
if pkg == nil {
return true
}
switch pkg.Name + "." + sel.Sel.Name {
case "localapi.Options":
for _, el := range cl.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
if k, ok := kv.Key.(*ast.Ident); ok && k.Name == "EscrowRecovery" {
optionsHasField = true
}
}
case "escrow.OffsiteKeyRecoverer":
recovererConstructed = true
}
return true
})
}
if !recovererConstructed {
t.Error("no escrow.OffsiteKeyRecoverer is constructed in buildLocalAPIServer — links 6→8 have no " +
"production assembly point (the built-but-never-wired shape, seventh instance)")
}
if !optionsHasField {
t.Error("localapi.Options in buildLocalAPIServer carries no EscrowRecovery field — the recoverer " +
"exists and the route would answer 503 forever")
}
}
// The hub fetch must be the DAEMON's own hub client, not a freshly constructed one with different
// credentials — the self-scoping that makes cross-host retrieval impossible is a property of WHICH
// key is used.
func TestEscrowRecoveryUsesTheDaemonHubClient(t *testing.T) {
fset, f := parseMain(t)
var fetchUsesHubClient bool
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
continue
}
ast.Inspect(fd.Body, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := ce.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "FetchIdentityEscrow" {
return true
}
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "hubClient" {
fetchUsesHubClient = true
} else {
t.Errorf("FetchIdentityEscrow at %s is called on something other than the injected hub client",
fset.Position(ce.Pos()))
}
return true
})
}
if !fetchUsesHubClient {
t.Fatal("the recoverer's fetcher does not call hubClient.FetchIdentityEscrow — either the fetch is " +
"not wired, or it uses a client whose credentials are not this host's")
}
}
// The route itself must be registered on the local API. A handler with no route is the same defect
// one layer down, and it has shipped here before.
func TestRecoverRouteIsRegistered(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "../../internal/localapi/server.go", nil, 0)
if err != nil {
t.Fatalf("parsing localapi/server.go: %v", err)
}
var registered bool
ast.Inspect(f, func(n ast.Node) bool {
ce, ok := n.(*ast.CallExpr)
if !ok || len(ce.Args) < 2 {
return true
}
sel, ok := ce.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "HandleFunc" {
return true
}
lit, ok := ce.Args[0].(*ast.BasicLit)
if !ok {
return true
}
if strings.Contains(lit.Value, "/escrow/recover-offsite-password") {
registered = true
}
return true
})
if !registered {
t.Fatal("POST /escrow/recover-offsite-password is not registered on the local API mux — the handler " +
"exists and nothing can reach it")
}
}
+268 -8
View File
@@ -24,6 +24,7 @@ import (
"path/filepath"
"strconv"
"strings"
"sync"
"syscall"
"time"
@@ -441,15 +442,113 @@ func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
// One exception, so an ordinary configuration is not turned into an alarm: a box with no dedicated
// target (`local_backup_target: "local"`, which host-install's own comment calls the DEGRADED
// fallback) is not treated as critical for that tier — see storeGrantCritical.
func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Config) []capability.Status {
func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Config, repair *storeGrantRepairer) []capability.Status {
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
out := make([]capability.Status, 0, len(tiers))
for _, t := range tiers {
out = append(out, storeGrantStatus(ctx, px, t.TargetID, storeGrantCritical(t.TargetID)))
out = append(out, storeGrantStatus(ctx, px, t.TargetID, storeGrantCritical(t.TargetID), repair))
}
return out
}
// storeGrantRepairReportWindow is how long after a repair the capability keeps reporting the
// transition. It MUST exceed the hub report interval, or the record never reaches the operator.
//
// FOUND BY THE LIVE RUN, NOT BY THE TESTS (2026-08-04). The first implementation reported degraded
// for exactly "one cycle" — the probe call that did the repair. But `probeAll` is invoked
// INDEPENDENTLY by the startup/periodic self-check log and by the collector building a host report,
// so the repairing call was the LOG's, and the report built three seconds later found the grant
// present and reported `ok`. The agent's journal had the record; the hub had nothing; the operator
// would have learned nothing. That is precisely the silence R-190 is about, re-created inside its own
// mitigation.
//
// A latch on TIME rather than on call count fixes it: 20 minutes comfortably exceeds the 900 s report
// interval, so at least one host-report must carry the transition, and it still clears on its own.
const storeGrantRepairReportWindow = 20 * time.Minute
// storeGrantRepairMinInterval bounds how often a single tier's grant may be re-granted (Scenario F).
//
// A storage can be unreadable for reasons an ACL cannot fix — the storage is gone, PVE is wedged,
// the wrapper is missing. Without a bound the probe would re-grant on every report cycle forever: a
// repair loop is a new defect wearing a fix's clothes. One attempt per tier per hour is frequent
// enough that a real loss is repaired within one backup window, and rare enough that a permanent
// fault produces attempts you can count on one hand per day.
const storeGrantRepairMinInterval = time.Hour
// storeGrantRepairer bounds and records the self-repair. It is deliberately in-memory: an agent
// restart re-arms the repair, which is correct — a restart is exactly when a box should re-check
// everything it depends on.
type storeGrantRepairer struct {
run func(ctx context.Context, name string, args ...string) ([]byte, []byte, error)
log *slog.Logger
mu sync.Mutex
last map[string]time.Time // target id → last ATTEMPT (success or failure)
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
}
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
func (r *storeGrantRepairer) noteRepaired(target string, now time.Time) {
if r == nil {
return
}
r.mu.Lock()
defer r.mu.Unlock()
if r.repaired == nil {
r.repaired = map[string]time.Time{}
}
r.repaired[target] = now
}
// recentlyRepaired reports whether a confirmed repair is still inside its report window — the latch
// that guarantees a host-report carries the transition even though the probe that repaired may have
// been a log-only one.
func (r *storeGrantRepairer) recentlyRepaired(target string, now time.Time) bool {
if r == nil {
return false
}
r.mu.Lock()
defer r.mu.Unlock()
t, ok := r.repaired[target]
return ok && now.Sub(t) < storeGrantRepairReportWindow
}
// mayAttempt reports whether a repair may run now for this target, and records the attempt if so.
func (r *storeGrantRepairer) mayAttempt(target string, now time.Time) bool {
if r == nil || r.run == nil {
return false
}
r.mu.Lock()
defer r.mu.Unlock()
if r.last == nil {
r.last = map[string]time.Time{}
}
if t, ok := r.last[target]; ok && now.Sub(t) < storeGrantRepairMinInterval {
return false
}
r.last[target] = now
return true
}
// repair runs the EXISTING root wrapper's `grant` verb for this storage. It adds no privileged
// surface: `felhom-backup-target-apply grant *` is already in the sudoers allowlist for any storage
// id (configs/felhom-agent.sudoers), and the verb already grants BOTH the user and the token — a
// privsep token's rights are the intersection, so granting one of the two grants nothing usable.
//
// This is the pbsdr shape (internal/pbsdr/manager.go, the R-22 self-grant): on a refusal, run the
// root wrapper and RE-READ ONCE rather than dead-locking. Its restraint is copied too — one attempt,
// one confirmation, and anything still wrong stays loudly wrong.
func (r *storeGrantRepairer) repair(ctx context.Context, target string) error {
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
_, errOut, err := r.run(rctx, localapi.BackupTargetWrapperPath, "grant", target)
if err != nil {
r.log.Error("store-grant: SELF-REPAIR FAILED — the tier stays unreadable",
"target", target, "err", err, "stderr", strings.TrimSpace(string(errOut)))
return err
}
return nil
}
// storeGrantRequiredPriv is the privilege whose ABSENCE was measured to blind the content listing.
//
// Measured on demo-felhom 2026-08-03: the two storages that list through the token hold
@@ -468,7 +567,7 @@ func storeGrantCritical(targetID string) bool { return targetID != "local" }
// storeGrantStatus is one tier's grant probe. It NEVER reports ok when it could not ask: a
// self-check that fails open is worse than none, because it converts "I do not know" into "fine".
func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string, critical bool) capability.Status {
func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string, critical bool, repair *storeGrantRepairer) capability.Status {
s := capability.Status{
Name: "pve:store-grant:" + targetID,
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
@@ -486,7 +585,117 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
pctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
privs, err := px.Permissions(pctx, "/storage/"+targetID)
return storeGrantVerdict(targetID, critical, privs, err)
s = storeGrantVerdict(targetID, critical, privs, err)
if err != nil {
return s
}
if s.Status != capability.StatusDegraded {
// Healthy — but if this tier was repaired moments ago, keep REPORTING the transition until a
// host-report has certainly carried it. Without this latch the repairing probe may be a
// log-only one and the hub never learns anything happened (measured live, see the window's
// comment).
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
}
// ── R-190 mitigation: the grant is missing — repair it, and SAY that it was missing ──────────
//
// R-190 is a grant that demonstrably worked at 04:44 and was gone by 09:24, with a reinstall,
// logged pveum activity and cluster-log entries all ruled out. The cause is still open; the
// resilience does not have to wait for it. Everything needed already exists — the root wrapper,
// its sudoers vector for any storage id, and the exact command — and until now the `grant` verb
// had only ever been called at CREATION. That is the "built but never wired" shape, in a verb
// rather than a seam.
if !repair.mayAttempt(targetID, time.Now()) {
// Bounded (Scenario F): an earlier attempt did not hold and it is too soon to try again. Stay
// degraded and say why — a quiet "we already tried" is how a permanent fault becomes silence.
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
" and a self-repair was attempted within the last " + storeGrantRepairMinInterval.String() +
" without holding — NOT retrying yet; this needs a human"
return s
}
if rerr := repair.repair(ctx, targetID); rerr != nil {
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
" and the self-repair FAILED (" + rerr.Error() + ") — this tier's archives are INVISIBLE to the agent"
return s // Scenario E: a failed repair must never mask the degraded state.
}
// Re-read ONCE to confirm, exactly as pbsdr does — the wrapper reporting success is a claim about
// its own write; the grant being readable is a different claim, and it is the one that matters.
cctx, ccancel := context.WithTimeout(ctx, 10*time.Second)
defer ccancel()
privs2, err2 := px.Permissions(cctx, "/storage/"+targetID)
if err2 != nil || privs2[storeGrantRequiredPriv] != 1 {
s.Reason = "the agent token lacks " + storeGrantRequiredPriv + " on /storage/" + targetID +
" and the self-repair did not take (re-read says it is still missing) — this needs a human"
return s
}
// REPAIRED — and reported as DEGRADED for exactly this one cycle, deliberately.
//
// The tier works again, so "ok" would be true of this instant and would throw away the only
// evidence that anything happened. R-190's own words: the probe sees the STATE, nothing sees the
// TRANSITION. A silent self-repair makes a recurring loss undetectable forever, which is strictly
// worse than the fault it fixes.
//
// §8.5 asked whether the hub's existing degraded↔ok edge suffices before building anything new.
// It does — as a CHANNEL — but only if the agent deliberately reports one degraded cycle: the hub
// alerts and e-mails on the ok→degraded edge and logs the degraded→ok recovery, so one loss
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
// change, no hub change, no new event type. The `Feature` text carries the explanation because
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
repair.noteRepaired(targetID, time.Now())
s = storeGrantRepairedVerdict(targetID, critical)
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
"target", targetID, "privilege", storeGrantRequiredPriv,
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
return s
}
// storeGrantHealthyVerdict decides what a HEALTHY probe reports — which is not always "ok".
//
// Split out so the tests exercise this decision rather than a copy of it. An earlier version of this
// guard lived inline and its red-proof PASSED, because the test asserted the latch helper instead of
// the path that consumes it — the same hollow shape this file has now caught twice.
//
// If the tier was repaired inside the report window, the transition is reported even though the grant
// is present: the probe that repaired may have been a log-only one, and without this the host-report
// carries `ok` and the operator never learns the permission vanished (measured live 2026-08-04).
func storeGrantHealthyVerdict(targetID string, critical bool, healthy capability.Status, repairedRecently bool) capability.Status {
if repairedRecently {
return storeGrantRepairedVerdict(targetID, critical)
}
return healthy
}
// storeGrantRepairedVerdict is the post-repair verdict — the RECORD half of R-190, split out so the
// tests exercise the real thing rather than a copy of it (yesterday's hollow-test lesson).
//
// It reports DEGRADED although the tier now works, and that is the whole point: "ok" would be true of
// this instant and would throw away the only evidence that a permission vanished. The hub raises its
// ok→degraded edge (an operator e-mail) and logs the degraded→ok recovery on the next cycle, so one
// loss produces exactly one alert pair. Nothing new was built for this — no wire change, no hub
// change, no new event type.
//
// The explanation lives in FEATURE because that is the field the hub interpolates into the operator's
// e-mail (`monitor/host_capability.go` emitTransition builds its message from the capability names
// and features; Reason does not travel). Putting it in Reason alone would be a record nobody reads.
func storeGrantRepairedVerdict(targetID string, critical bool) capability.Status {
return capability.Status{
Name: "pve:store-grant:" + targetID,
Critical: critical,
Status: capability.StatusDegraded,
Feature: "backup tier " + targetID + ": the agent's storage grant was MISSING and has been " +
"AUTOMATICALLY RESTORED — the tier works now, but a permission that vanished on its own needs investigating (R-190)",
Reason: "grant absent at probe time; `felhom-backup-target-apply grant " + targetID +
"` re-applied it and a re-read confirms " + storeGrantRequiredPriv + " is present again",
}
}
// repairLogger returns the repairer's logger, or the default — the record must survive a nil.
func repairLogger(r *storeGrantRepairer) *slog.Logger {
if r != nil && r.log != nil {
return r.log
}
return slog.Default()
}
// storeGrantVerdict is the DECISION, split out from the API call so the tests exercise the real
@@ -598,9 +807,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
// R-185: the store-grant probes compose around the sudo prober the same way the pool read does
// (an API read does not belong inside the sudo-policy probe — the v0.62.0 A1 precedent).
// R-190: the store-grant probe also REPAIRS a missing grant, through the root wrapper that
// already exists and is already sudoers-permitted for any storage id — and reports the loss.
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
// invoked through the privileged path, which prepends sudo itself.
grantRepairer := &storeGrantRepairer{
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger,
}
probeAll := func(ctx context.Context) []capability.Status {
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
return append(out, storeGrantStatuses(ctx, px, cfg)...)
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
}
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
// already carries the gated view — v0.86.0.)
@@ -882,7 +1099,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
return false
},
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localTokens != nil {
defer localTokens.Close()
}
@@ -1460,7 +1677,7 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
// leaf (stable fingerprint). Any failure DISABLES the server (returns nil) WITHOUT crashing the
// daemon — the host still reports/reconciles; only the controller channel is unavailable until
// fixed. The opened token store is returned via outTokens so the caller can Close it.
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.Store, inFlight *backup.InFlight, observer *storage.Observer, driveTargets storage.KnownTargets, hostOps *storage.SudoHostOps, gate *reconcile.Gate, collector *hub.Collector, hubClient *hub.Client, intent localapi.IntentRecorder, guestBinds *localapi.GuestBindStore, formatJobs *localapi.FormatJobStore, logRing *applog.Ring, escrowCeremony *localapi.EscrowCeremonyConfig, logger *slog.Logger, outTokens **localapi.TokenStore) *localapi.Server {
if !cfg.LocalAPI.Enabled() {
return nil
}
@@ -1532,7 +1749,29 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
gaMode = proxmox.RunnerSudo
}
guestBinder := localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, logger)
// R-199 (v0.125.0) — chain links 6->8, assembled here and ONLY here. The fetcher is this daemon's
// own hub client (per-host key, self-scoped server-side), so the recoverer can never read another
// host's blob even if asked to. `client` is the same one the report loop uses; a nil hub config
// cannot reach this line (the daemon exits above), so the seam is always live in production —
// which is the point: links 6 and 7 spent months existing without a caller.
escrowRecoverer := escrow.OffsiteKeyRecoverer{
Fetch: func(ctx context.Context) ([]byte, bool, error) {
resp, ferr := hubClient.FetchIdentityEscrow(ctx)
if ferr != nil {
return nil, false, ferr
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return nil, false, nil
}
blob, derr := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if derr != nil {
return nil, false, fmt.Errorf("hub served a malformed escrow blob (not base64)")
}
return blob, true, nil
},
}
srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr,
Cert: cert,
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
@@ -2654,7 +2893,28 @@ func runSelftestIdentityConsume(ctx context.Context, cfg config.Config, logger *
fmt.Fprintln(os.Stderr, " [FAIL] writing recovered bundle:", err)
return 1
}
fmt.Printf(" [OK] identity recovered (tunnel_token + pbs_token) → %s (0600) — never printed\n", keyDest)
// R-199 / §8.6: this line used to read "(tunnel_token + pbs_token)" — an enumeration that was
// accurate when it was written (pre-fork-4) and became a MISSTATEMENT the moment v0.77.0 sealed the
// offsite repository password into the same bundle. Anyone reading the old output would conclude the
// repository password was not there, and that is part of how the chain's extraction link came to be
// described as missing for a month. Name what was recovered from THIS bundle, and name what is
// absent, rather than reciting a fixed list.
recovered := []string{"tunnel_token", "pbs_token"}
var absent []string
if bundle.WGPrivateKey != "" {
recovered = append(recovered, "wg_private_key")
} else {
absent = append(absent, "wg_private_key")
}
if bundle.ResticRepoPassword != "" {
recovered = append(recovered, "restic_repo_password")
} else {
absent = append(absent, "restic_repo_password (pre-fork-4 blob — the field did not exist when this was sealed)")
}
fmt.Printf(" [OK] identity recovered (%s) → %s (0600) — values never printed\n", strings.Join(recovered, " + "), keyDest)
if len(absent) > 0 {
fmt.Printf(" [NOTE] fields ABSENT from this bundle: %s\n", strings.Join(absent, "; "))
}
// S5 DR: install the recovered WG private key so the tunnel re-establishes with the SAME
// identity/pubkey (→ the same hub /32), no fresh keygen. Create-only (refuses to overwrite a
+252 -1
View File
@@ -2,9 +2,13 @@ package main
import (
"context"
"errors"
"go/ast"
"io"
"log/slog"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
@@ -132,7 +136,7 @@ func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
// probe, because it looks like coverage.
func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
s := storeGrantStatus(context.Background(), nil, "felhom-backup", true)
s := storeGrantStatus(context.Background(), nil, "felhom-backup", true, nil)
if s.Status != capability.StatusDegraded {
t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
}
@@ -164,3 +168,250 @@ func TestMainWiresTheStoreGrantProbe(t *testing.T) {
"which is precisely the silence R-185 is about")
}
}
// ── R-190 — the grant repairs itself, and the repair is VISIBLE ──────────────────────────────
//
// R-190 is a storage grant that demonstrably worked at 04:44 on 2026-08-03 and was gone by 09:24,
// with a host reinstall, logged `pveum` activity and cluster-log entries all ruled out. The cause is
// open; the resilience is not conditional on it.
//
// The half that matters is the RECORD. R-190's own words: the probe sees the state, nothing sees the
// transition. A self-repair that leaves only "ok" behind destroys the only evidence a loss happened,
// so a recurring loss becomes undetectable forever — strictly worse than the fault it fixes.
// fakeRepairRunner records wrapper invocations and can be made to fail.
type fakeRepairRunner struct {
calls [][]string
fail bool
}
func (f *fakeRepairRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
f.calls = append(f.calls, append([]string{name}, args...))
if f.fail {
return nil, []byte("pveum: refused"), errors.New("exit status 2")
}
return nil, nil, nil
}
func newRepairer(f *fakeRepairRunner) *storeGrantRepairer {
return &storeGrantRepairer{run: f.Run, log: slog.New(slog.NewTextHandler(io.Discard, nil))}
}
// ── SCENARIO F — the repair is BOUNDED ───────────────────────────────────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-04): make mayAttempt always return true (drop the
// storeGrantRepairMinInterval check) →
//
// --- FAIL: TestGrantRepair_IsBounded
// storegrant_test.go: a repair must not run on every cycle; 5 cycles produced 5 attempt(s)
//
// which is a re-grant every report cycle, forever, against a fault an ACL cannot fix. Restored.
func TestGrantRepair_IsBounded(t *testing.T) {
f := &fakeRepairRunner{}
r := newRepairer(f)
// Jittered, so the series never lands exactly on the interval boundary — a perfectly regular
// series is how a threshold test passes its own mutation, which has happened here before.
base := time.Date(2026, 8, 4, 9, 17, 43, 0, time.UTC)
offsets := []time.Duration{0, 13*time.Minute + 7*time.Second, 27*time.Minute + 51*time.Second,
41*time.Minute + 19*time.Second, 55*time.Minute + 3*time.Second}
attempts := 0
for _, off := range offsets {
if r.mayAttempt("felhom-backup", base.Add(off)) {
attempts++
}
}
if attempts != 1 {
t.Fatalf("a repair must not run on every cycle; %d cycles produced %d attempt(s) within %s",
len(offsets), attempts, storeGrantRepairMinInterval)
}
// ...and once the interval has genuinely passed, it may try again — a bound is not a ban.
if !r.mayAttempt("felhom-backup", base.Add(storeGrantRepairMinInterval+2*time.Minute+11*time.Second)) {
t.Fatal("after the interval a repair must be allowed again — otherwise one failure disables the repair forever")
}
// A DIFFERENT tier is not throttled by this one's attempt.
if !r.mayAttempt("felhom-pbs", base.Add(time.Minute)) {
t.Fatal("the bound must be per tier — one tier's attempt must not suppress another's")
}
}
// A nil repairer (or one with no runner) never attempts, and never panics.
func TestGrantRepair_NilIsSafe(t *testing.T) {
var r *storeGrantRepairer
if r.mayAttempt("felhom-backup", time.Now()) {
t.Fatal("a nil repairer must never claim an attempt")
}
if (&storeGrantRepairer{}).mayAttempt("felhom-backup", time.Now()) {
t.Fatal("a repairer with no runner must never claim an attempt")
}
}
// The repair calls the EXISTING wrapper verb, with the storage id — no new privileged surface.
func TestGrantRepair_CallsTheExistingWrapperVerb(t *testing.T) {
f := &fakeRepairRunner{}
r := newRepairer(f)
if err := r.repair(context.Background(), "felhom-backup"); err != nil {
t.Fatalf("repair should succeed with a healthy runner: %v", err)
}
if len(f.calls) != 1 {
t.Fatalf("exactly one wrapper invocation expected; got %d", len(f.calls))
}
got := f.calls[0]
want := []string{"/usr/local/sbin/felhom-backup-target-apply", "grant", "felhom-backup"}
if len(got) != len(want) {
t.Fatalf("wrapper argv = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("wrapper argv = %v, want %v — the sudoers vector is `grant *`; anything else is a policy change", got, want)
}
}
}
// A repair that FAILS must surface the failure, not swallow it (Scenario E's precondition).
func TestGrantRepair_FailureIsReturned(t *testing.T) {
f := &fakeRepairRunner{fail: true}
if err := newRepairer(f).repair(context.Background(), "felhom-backup"); err == nil {
t.Fatal("a failed wrapper run must return its error — a repair that cannot run must never read as done")
}
}
// ── SCENARIO D (the half that matters) — the REPAIR MUST BE VISIBLE ──────────────────────────
//
// A repair that leaves only "ok" behind is worse than the fault: the tier works, and the fact that a
// permission vanished is gone with it. R-190 exists because nothing saw the transition.
//
// The channel is the hub's EXISTING ok→degraded→ok edge (§8.5) — nothing new was built. That only
// works if the agent deliberately reports ONE degraded cycle after repairing, and if the explanation
// rides the field the hub actually puts in the operator's e-mail. The hub's message is built from the
// capability NAME and FEATURE (`internal/monitor/host_capability.go` emitTransition) — **not** from
// Reason — so the Feature must carry it.
//
// COMPANION RED-PROOF (observed 2026-08-04): after a successful repair, report ok instead —
//
// s.Status = capability.StatusOK; s.Feature unchanged
//
// → --- FAIL: TestGrantRepair_ARepairedGrantIsReportedAsATransition
//
// storegrant_test.go: a self-repair must still report DEGRADED for one cycle so the hub raises
// its edge; got "ok" — the loss would be invisible
//
// i.e. exactly the silence R-190 is about. Restored.
func TestGrantRepair_ARepairedGrantIsReportedAsATransition(t *testing.T) {
// THE PRODUCTION verdict, not a copy of it. An earlier draft of this test built the Status
// itself and asserted its own construction — it would have passed while production reported ok,
// which is precisely the silence being guarded against.
if pre := probeWith(permUngranted, "felhom-backup", true); pre.Status != capability.StatusDegraded {
t.Fatalf("precondition: a missing grant is degraded; got %q", pre.Status)
}
s := storeGrantRepairedVerdict("felhom-backup", true)
if s.Status != capability.StatusDegraded {
t.Fatalf("a self-repair must still report DEGRADED for one cycle so the hub raises its edge; "+
"got %q — the loss would be invisible", s.Status)
}
// The hub e-mails the FEATURE text. If the explanation is not there, the operator is told a
// capability was degraded and never learns it repaired itself or that anything vanished.
for _, want := range []string{"MISSING", "RESTORED", "felhom-backup", "R-190"} {
if !strings.Contains(s.Feature, want) {
t.Fatalf("the Feature text is what the hub puts in the operator's e-mail; it must contain %q. Got: %s", want, s.Feature)
}
}
if !s.Critical {
t.Fatal("the transition must be CRITICAL or the hub does not alert on it at all")
}
}
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
//
// The wrapper's `grant` verb is itself a "built but never wired" example: it exists, is
// sudoers-permitted for any id, and had only ever been called at storage CREATION. The repair must
// not become the seventh instance. AST, not grep — a commented-out call still contains the string.
func TestMainWiresTheGrantRepair(t *testing.T) {
f := parseMainForWiring(t)
var built, passed bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CompositeLit:
if id, ok := node.Type.(*ast.Ident); ok && id.Name == "storeGrantRepairer" {
built = true
}
case *ast.CallExpr:
if id, ok := node.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" && len(node.Args) == 4 {
if a, ok := node.Args[3].(*ast.Ident); ok && a.Name == "grantRepairer" {
passed = true
}
}
}
return true
})
if !built {
t.Error("main.go never constructs a storeGrantRepairer — nothing would ever repair a lost grant")
}
if !passed {
t.Error("storeGrantStatuses is not passed the repairer — the probe would detect the loss and " +
"leave it, which is v0.123.0's behaviour and not R-190's mitigation")
}
}
// The transition must survive a probe that is NOT the one feeding the hub.
//
// MEASURED LIVE 2026-08-04, and this test exists because the first implementation failed it in
// production while every unit test passed: `probeAll` is called independently by the self-check LOG
// and by the collector building a host-report. The repairing call was the log's; the report three
// seconds later found the grant present and reported `ok`. The agent's journal had the record and the
// hub had nothing — the exact silence R-190 is about, re-created inside its own mitigation.
//
// COMPANION RED-PROOF (observed): delete the `recentlyRepaired` branch from the healthy path →
//
// --- FAIL: TestGrantRepair_TransitionSurvivesALaterProbe
// storegrant_test.go: a probe AFTER the repair must still report the transition; got "ok" —
// the host-report would carry ok and the operator would never learn the grant vanished
//
// Restored.
func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
r := newRepairer(&fakeRepairRunner{})
// Jittered, never landing on the window boundary.
repairedAt := time.Date(2026, 8, 4, 9, 39, 34, 0, time.UTC)
r.noteRepaired("felhom-backup", repairedAt)
// The DECISION a later probe makes — the production function, not the helper it calls. An
// earlier draft asserted `recentlyRepaired` directly and its red-proof PASSED, because removing
// the latch's USE left the helper untouched.
healthy := probeWith(permGranted, "felhom-backup", true)
if healthy.Status != capability.StatusOK {
t.Fatalf("precondition: a granted tier is ok; got %q", healthy.Status)
}
got := storeGrantHealthyVerdict("felhom-backup", true,
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(3*time.Second)))
if got.Status != capability.StatusDegraded {
t.Fatalf("a probe AFTER the repair must still report the transition; got %q — the host-report "+
"would carry ok and the operator would never learn the grant vanished", got.Status)
}
if !strings.Contains(got.Feature, "RESTORED") {
t.Fatalf("the later probe must carry the explanation into the hub's e-mail; got: %s", got.Feature)
}
// Outside the window it reports plain ok again.
late := storeGrantHealthyVerdict("felhom-backup", true,
healthy, r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute)))
if late.Status != capability.StatusOK {
t.Fatalf("outside the window a healthy tier reports ok; got %q — a permanent degraded state "+
"would be its own false alarm", late.Status)
}
if !r.recentlyRepaired("felhom-backup", repairedAt.Add(14*time.Minute+37*time.Second)) {
t.Fatal("the latch must outlast the 900s hub report interval, or the record never reaches the hub")
}
// ...and it clears on its own rather than latching a box degraded forever.
if r.recentlyRepaired("felhom-backup", repairedAt.Add(storeGrantRepairReportWindow+time.Minute+7*time.Second)) {
t.Fatal("the latch must clear — a permanent degraded state would be its own false alarm")
}
// It is per tier.
if r.recentlyRepaired("felhom-pbs", repairedAt.Add(time.Second)) {
t.Fatal("one tier's repair must not latch another tier's status")
}
// The window MUST exceed the report interval — the property, asserted rather than assumed.
if storeGrantRepairReportWindow <= 15*time.Minute {
t.Fatalf("the report window (%s) must exceed the 900s hub report interval, or a transition can "+
"be missed entirely", storeGrantRepairReportWindow)
}
}
+95
View File
@@ -0,0 +1,95 @@
package escrow
import (
"context"
"errors"
"fmt"
)
// R-199 links 6→8 — fetch this host's own sealed identity blob, open it with the customer's recovery
// code R, and hand back EXACTLY ONE field: the offsite restic repository password.
//
// WHY ONLY ONE FIELD. The bundle also carries the Cloudflare tunnel token, the PBS access token and
// the WG private key (see IdentityBundle). The caller in this flow — the in-guest controller, one
// trust tier down — needs none of them, and returning them would widen the blast radius of a
// controller compromise for no gain. Narrowing costs nothing here and is not recoverable later.
//
// WHY R NEVER TOUCHES DISK. `UnwrapIdentity` stages the BLOB and the recovered plaintext in a
// `MkdirTemp` that it removes, and feeds R through the pty; R itself is never written. This wrapper
// keeps that property: it takes R as an argument, passes it straight through, and holds no copy.
// Callers must clear their own reference (the `R = ""` discipline in cmd/felhom-agent).
//
// The errors below are DISTINCT on purpose. "could not fetch", "no blob", "wrong code" and "the blob
// predates the field" are FOUR different situations for the operator and only one of them is a fault.
//
// ⚠ THERE WERE THREE, AND THE FOURTH WAS THE DEFECT (R-224, 2026-08-06). This comment said "three"
// and named "no blob", "wrong code" and "predates the field" — while a FAILED FETCH was wrapped as an
// anonymous error and fell through the caller's `default` branch into the wrong-code message. So a
// hub that could not be reached was reported to the customer as a bad recovery code.
//
// Measured live on 2026-08-05 (CAMPAIGN-11 F3): with the hub REJECTed at the appliance's firewall and
// a CORRECT current recovery code, the customer was told the code did not open their package — in
// 0.0556 s, when a real unseal costs ~1 s of scrypt. The agent's own log carried the truth the whole
// time (`escrow: fetching the sealed bundle: hub: transport error: … no route to host`) and the HTTP
// boundary threw it away.
//
// The discriminator therefore has to be a VALUE, not a log line — that is what ErrBundleFetch is.
var (
// ErrBundleFetch — the sealed bundle could not be FETCHED (the hub refused, was unreachable, or
// the transport failed). **The recovery code was never used**, so nothing about it is known and
// nothing may be said about it. Wraps the underlying cause for the operator log; carries no secret.
ErrBundleFetch = errors.New("escrow: the sealed bundle could not be fetched")
// ErrNoEscrowBlob — the hub holds no sealed bundle for this host. Not a fault: no ceremony has run.
ErrNoEscrowBlob = errors.New("escrow: the hub holds no sealed identity bundle for this host (no ceremony has run)")
// ErrNoResticPassword — the bundle opened, but carries no repository password. Real and expected
// for a pre-fork-4 blob (agent < v0.77.0, 2026-07-09): the field did not exist and CANNOT be
// retro-fitted, because R is never retained. Distinguished from a wrong code so the operator is
// not sent hunting for a mistyped recovery code that was typed correctly.
ErrNoResticPassword = errors.New("escrow: the recovered bundle carries NO offsite repository password (a pre-fork-4 blob — the field did not exist when it was sealed and cannot be retro-fitted)")
)
// BlobFetcher yields this host's own opaque identity-escrow blob. present=false is a clean "none".
// An interface-free func field keeps this package free of any dependency on the hub client.
type BlobFetcher func(ctx context.Context) (blob []byte, present bool, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct {
Fetch BlobFetcher
}
// RecoverOffsiteRepoPassword fetches, unseals and extracts. It returns ONLY the repository password.
//
// A WRONG RECOVERY CODE FAILS CLOSED at the scrypt KDF inside UnwrapIdentity — `age -d` exits
// non-zero and emits no plaintext, so there is no partial result and nothing is written anywhere.
// That property is the crypto's, not a check here, which is why this function has no "validate R"
// step to get wrong.
//
// NOTHING IS LOGGED BY THIS FUNCTION and no error it returns contains R, the password, or blob bytes.
func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error) {
if r.Fetch == nil {
return "", fmt.Errorf("escrow: recoverer has no blob fetcher configured")
}
if recoveryCode == "" {
return "", fmt.Errorf("escrow: the recovery code is required")
}
blob, present, err := r.Fetch(ctx)
if err != nil {
// R-224: joined with ErrBundleFetch so the caller can classify by VALUE. The cause stays
// wrapped for the operator log; neither carries a secret. Before this, the fetch failure was
// an anonymous error and the local-api handler's `default` branch reported it to the customer
// as a wrong recovery code.
return "", fmt.Errorf("%w: %w", ErrBundleFetch, err)
}
if !present || len(blob) == 0 {
return "", ErrNoEscrowBlob
}
bundle, err := UnwrapIdentityBundle(ctx, blob, recoveryCode)
if err != nil {
return "", err // already the fail-closed "the recovery code did not unwrap…" message; no secret in it
}
if bundle.ResticRepoPassword == "" {
return "", ErrNoResticPassword
}
return bundle.ResticRepoPassword, nil
}
+267
View File
@@ -0,0 +1,267 @@
package escrow
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
// R-199 links 6→8, with REAL crypto (age is present on the build/demo host; ensureAge skips
// elsewhere). These are the unit half of the session's question — "is the repository password
// actually recoverable from the sealed bundle" — and the live half is the same equality on hardware.
const testR = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
func sealBundle(t *testing.T, b IdentityBundle, r string) []byte {
t.Helper()
blob, err := WrapIdentityBundle(context.Background(), b, r)
if err != nil {
t.Fatalf("WrapIdentityBundle: %v", err)
}
return blob
}
func fetcherFor(blob []byte) BlobFetcher {
return func(context.Context) ([]byte, bool, error) { return blob, true, nil }
}
// Scenario A (unit) — the recovered repository password is BYTE-IDENTICAL to the sealed one, and it
// is the REPOSITORY password rather than some other field of a bundle that also parses.
//
// RED-PROOF: return bundle.PBSToken (or TunnelToken, or WGPrivateKey) instead of
// bundle.ResticRepoPassword → a plausible-looking bundle yields a non-matching key → this FAILS.
// That mutation is the shape of the bug that would otherwise ship silently, because every one of
// those fields is a non-empty string that looks like a secret.
func TestRecoverOffsiteRepoPassword_ReturnsTheRepositoryPassword(t *testing.T) {
ensureAge(t)
const repoPW = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
blob := sealBundle(t, IdentityBundle{
TunnelToken: "TUNNEL-TOKEN-NOT-THE-ANSWER",
PBSToken: "PBS-TOKEN-NOT-THE-ANSWER",
WGPrivateKey: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
ResticRepoPassword: repoPW,
}, testR)
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
if err != nil {
t.Fatalf("recover: %v", err)
}
if got != repoPW {
t.Fatalf("the recovered key is not the sealed repository password (len %d vs %d) — a different "+
"field of the bundle was returned", len(got), len(repoPW))
}
// Belt: it must not be any of the OTHER fields, so a future refactor cannot satisfy the check
// above by coincidence.
for _, other := range []string{"TUNNEL-TOKEN-NOT-THE-ANSWER", "PBS-TOKEN-NOT-THE-ANSWER", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="} {
if got == other {
t.Fatalf("the recoverer returned the wrong bundle field")
}
}
}
// Scenario B — a WRONG recovery code fails closed, the failure names no secret, and nothing is
// written. The fail-closed property is the crypto's (age's scrypt KDF), which is why there is no
// validation step here to get wrong — the test pins that it stays that way.
func TestRecoverOffsiteRepoPassword_WrongCodeFailsClosed(t *testing.T) {
ensureAge(t)
const repoPW = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
got, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), "not the recovery code at all")
if err == nil {
t.Fatal("a wrong recovery code MUST fail — a plausible-but-wrong bundle is the one outcome the design forbids")
}
if got != "" {
t.Fatalf("a failed unseal returned %d bytes — there must be no partial result", len(got))
}
// The error may name the step; it may never name a secret.
for _, secret := range []string{repoPW, testR, "not the recovery code at all"} {
if strings.Contains(err.Error(), secret) {
t.Fatalf("the failure message leaked a secret: %v", err)
}
}
}
// A bundle with no repository password is its OWN answer, not a wrong-code error. Sealed before
// fork-4 (agent < v0.77.0) the field did not exist; sending the operator to re-check a correctly
// typed recovery code would be the wrong instruction.
func TestRecoverOffsiteRepoPassword_PreForkFourBundle(t *testing.T) {
ensureAge(t)
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p"}, testR)
_, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoResticPassword) {
t.Fatalf("a pre-fork-4 bundle must report its own error, got %v", err)
}
}
// Scenario D at this layer — no blob is a clean, distinguishable answer.
func TestRecoverOffsiteRepoPassword_NoBlob(t *testing.T) {
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoEscrowBlob) {
t.Fatalf("absent blob must yield ErrNoEscrowBlob, got %v", err)
}
}
// Scenario F — R persists NOWHERE. TMPDIR is redirected into the test's own directory, the unseal is
// run for real, and the whole tree is then walked: no file may contain R (or the recovered password),
// and the staging directory the unseal creates must be gone.
//
// RED-PROOF: write R to a temp file anywhere in the flow (e.g. add
// `os.WriteFile(filepath.Join(work,"r"), []byte(recoveryCode), 0o600)` inside UnwrapIdentity before
// its defer removes the dir — or simply drop that defer and let the plaintext staging survive) → the
// walk finds it → this FAILS.
func TestRecoverOffsiteRepoPassword_RLeavesNoTrace(t *testing.T) {
ensureAge(t)
const repoPW = "1111111111111111111111111111111111111111111111111111111111111111"
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp) // os.MkdirTemp honours this — every staging dir lands under the walk
const wrongR = "wrong code entirely"
blob := sealBundle(t, IdentityBundle{TunnelToken: "t", PBSToken: "p", ResticRepoPassword: repoPW}, testR)
if _, err := (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), testR); err != nil {
t.Fatalf("recover: %v", err)
}
// A failed unseal must leave nothing either — exercise both paths before walking.
_, _ = (OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}).RecoverOffsiteRepoPassword(context.Background(), wrongR)
// THE PRIMARY ASSERTION IS EMPTINESS, not content. A content scan alone is defeatable by a later
// call OVERWRITING the leaked file with a different secret — which is exactly how the first
// version of this test passed its own red-proof while R sat on disk. Nothing in this test writes
// under TMPDIR, so after both calls the tree must contain no files at all.
var survivors []string
err := filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
if err != nil || info == nil || info.IsDir() || path == tmp {
return nil
}
survivors = append(survivors, strings.TrimPrefix(path, tmp))
return nil
})
if err != nil {
t.Fatal(err)
}
if len(survivors) > 0 {
t.Fatalf("the unseal left %d file(s) behind under TMPDIR: %v — R, the sealed blob and the "+
"recovered plaintext all pass through there and none of them may outlive the call", len(survivors), survivors)
}
// Defence in depth: any secret that DOES appear anywhere is named, for every code used.
_ = filepath.Walk(tmp, func(path string, info os.FileInfo, err error) error {
if err != nil || info == nil || info.IsDir() {
return nil
}
body, rerr := os.ReadFile(path)
if rerr != nil {
return nil
}
for label, secret := range map[string]string{"R": testR, "a wrong R": wrongR, "the repository password": repoPW} {
if strings.Contains(string(body), secret) {
t.Errorf("%s survived on disk at %s", label, path)
}
}
return nil
})
// And the staging directories are gone, not merely free of secrets.
entries, _ := os.ReadDir(tmp)
for _, e := range entries {
if e.IsDir() && strings.HasPrefix(e.Name(), "felhom-idesc-") {
t.Fatalf("an unseal staging directory survived: %s", e.Name())
}
}
}
// A fetch failure surfaces as a fetch failure, not as a wrong-code error — the operator must not be
// sent to re-read their recovery code because the hub was unreachable.
//
// ⚠ THIS TEST WAS GREEN THROUGHOUT THE DEFECT IT DESCRIBES (R-224, 2026-08-06). Its sentence is
// exactly right and it did not prevent anything, for two reasons worth keeping:
//
// 1. **It asserted the MECHANISM, one layer below the consequence.** It checked this package's error
// STRING. The merge happened one layer up, in the local-api handler's `default` branch, which
// answered a fetch failure with "the recovery code did not open the sealed bundle". The customer
// never sees this string; they see that one. The project's own rule — prefer the test that asserts
// the CONSEQUENCE (does the customer get blamed?) over the one that asserts the MECHANISM (is the
// error distinct here?) — names this case precisely.
// 2. **It asserted on TEXT.** `strings.Contains(err.Error(), …)` cannot be consumed by a caller, so
// it pinned something no production code could branch on. The distinction it checked was real and
// unusable.
//
// It now asserts the SENTINEL, which is what the handler branches on, and its consequence-level twin
// lives in `internal/localapi/escrow_recover_class_test.go` where the status is asserted.
func TestRecoverOffsiteRepoPassword_FetchErrorIsDistinct(t *testing.T) {
rec := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) {
return nil, false, errors.New("hub: connection refused")
}}
_, err := rec.RecoverOffsiteRepoPassword(context.Background(), testR)
if err == nil || !errors.Is(err, ErrBundleFetch) {
t.Fatalf("a fetch failure must classify as ErrBundleFetch, got %v", err)
}
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
t.Fatal("a transport failure must not masquerade as a content verdict")
}
}
// ── R-224 — A FAILED FETCH IS NOT A WRONG CODE ──────────────────────────────────────────────────
//
// CAMPAIGN-11 F3 measured the consequence of these two being indistinguishable: with the hub
// firewalled off and a CORRECT current recovery code, the customer was told the code did not open
// their package, in 0.0556 s — no unseal was attempted at all.
//
// The pair below is the whole point. Asserting only the first would pass with a `return ErrBundleFetch`
// stuck on every error path, which is the same defect pointing the other way.
func TestRecoverOffsiteRepoPassword_FetchFailureIsClassifiedAsFetch(t *testing.T) {
boom := errors.New("hub: transport error: dial tcp 37.191.56.193:443: connect: no route to host")
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, boom }}
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
if err == nil {
t.Fatal("a failing fetch must return an error")
}
// RED-PROOF: drop the `%w: %w` join in RecoverOffsiteRepoPassword (return the bare wrapped cause,
// as it was before R-224) → this FAILS, and the local-api handler falls back to the wrong-code
// message exactly as it did on 2026-08-05.
if !errors.Is(err, ErrBundleFetch) {
t.Fatalf("a failed fetch must classify as ErrBundleFetch, got %v", err)
}
// The underlying cause survives for the operator log.
if !errors.Is(err, boom) {
t.Fatalf("the fetch cause must stay wrapped for the operator, got %v", err)
}
// And it must NOT be mistaken for either of the bundle-content situations.
if errors.Is(err, ErrNoEscrowBlob) || errors.Is(err, ErrNoResticPassword) {
t.Fatalf("a transport failure is neither of the bundle-content errors: %v", err)
}
}
// The other half: a genuinely wrong code must NOT classify as a fetch failure, or the fix trades one
// misattribution for its mirror image and the customer is told the hub is down when they mistyped.
func TestRecoverOffsiteRepoPassword_WrongCodeIsNotAFetchFailure(t *testing.T) {
ensureAge(t)
blob := sealBundle(t, IdentityBundle{ResticRepoPassword: "0123456789abcdef"}, testR)
r := OffsiteKeyRecoverer{Fetch: fetcherFor(blob)}
_, err := r.RecoverOffsiteRepoPassword(context.Background(),
"wrong horse battery staple sedative anaconda wobbly kingdom placard yodel")
if err == nil {
t.Fatal("a wrong recovery code must fail closed")
}
if errors.Is(err, ErrBundleFetch) {
t.Fatalf("a wrong code must NOT classify as a fetch failure, got %v", err)
}
}
// A clean "the hub holds nothing" keeps its own identity too — it is not a fetch failure, and the
// customer must not be told the hub was unreachable when it answered perfectly well.
func TestRecoverOffsiteRepoPassword_AbsentBlobIsNotAFetchFailure(t *testing.T) {
r := OffsiteKeyRecoverer{Fetch: func(context.Context) ([]byte, bool, error) { return nil, false, nil }}
_, err := r.RecoverOffsiteRepoPassword(context.Background(), testR)
if !errors.Is(err, ErrNoEscrowBlob) {
t.Fatalf("an absent blob must stay ErrNoEscrowBlob, got %v", err)
}
if errors.Is(err, ErrBundleFetch) {
t.Fatalf("an absent blob is not a fetch FAILURE, got %v", err)
}
}
+47
View File
@@ -307,3 +307,50 @@ func tail(b []byte, max int) string {
}
return s
}
// IdentityEscrowResponse mirrors GET /api/v1/hosts/{host_id}/escrow (hub >= v0.94.0, R-199).
// Present=false is a CLEAN answer, not a fault: the host simply has no sealed bundle yet.
type IdentityEscrowResponse struct {
HostID string `json:"host_id"`
Present bool `json:"present"`
IdentityEscrowB64 string `json:"identity_escrow_b64"`
}
// FetchIdentityEscrow reads back THIS host's own opaque identity-escrow blob (R-199 link 6 — the
// mirror of UploadEscrow, self-scoped server-side by the per-host key). The bytes are ciphertext: they
// are useless without the customer's recovery code R, which neither the hub nor this agent ever holds.
//
// It is the ONLY retrieval this client performs, and it is deliberately narrow — no directive, no
// K-escrow, no key rotation. The operator-driven DR path (recovery-mode re-enroll) is a different
// endpoint with a different gate and is not reached from here.
//
// Errors are typed (transport vs HTTP) and never include the bearer token. The BLOB is never logged —
// only its length.
func (c *Client) FetchIdentityEscrow(ctx context.Context) (*IdentityEscrowResponse, error) {
if c.hostID == "" {
return nil, fmt.Errorf("hub: FetchIdentityEscrow requires a configured host_id")
}
url := c.baseURL + "/api/v1/hosts/" + c.hostID + "/escrow"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("hub: building escrow-fetch request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)
if err != nil {
return nil, &TransportError{Err: err}
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
}
var out IdentityEscrowResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("hub: decoding escrow fetch: %w", err)
}
return &out, nil
}
+124
View File
@@ -0,0 +1,124 @@
package localapi
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"net/http"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-199 (agent v0.125.0) — the in-guest controller asks the agent to recover the offsite repository
// password from the hub's sealed bundle, using the customer's recovery code R.
//
// WHY THE AGENT AND NOT THE CONTROLLER. Three reasons, all structural: the unsealing binary (`age`)
// is an agent runtime dependency and is deliberately absent from the controller image; the sealed
// blob is a HOST-scoped object whose only writer is this agent under the per-host key, so the read is
// that write's mirror; and the controller is a trust tier down — it should receive one field, not a
// bundle it has no use for.
//
// R'S HANDLING, WHICH IS THE TIGHTEST RULE IN THIS FLOW. R is the one secret in the system that
// cannot be rotated, re-issued or recovered — it exists only in the customer's hands. Here it:
// - arrives in the request body over the already-pinned local-API channel (the operator accepted
// that crossing on 2026-08-04; the acceptance covers the CHANNEL, not carelessness at either end);
// - is held in memory for the duration of one call and cleared on BOTH paths;
// - is never written to disk, never an argument in a process list, and never logged at any level,
// including inside an error;
// - is never echoed: no response this endpoint can emit contains it.
//
// The request-level DEBUG middleware logs method/path/status/duration and never bodies — see
// `logRequests`. Do not add a body dump.
//
// THE RESPONSE CARRIES THE PASSWORD AND ITS HASH. The hash is what this session's proof compares
// (compare by hash, never by value). The password itself is present because the next link — placing a
// recovered password so the existing repository opens — needs it, and building a hash-only seam now
// would have to be torn out to add it. The controller's diagnostic reads only the hash.
type recoverOffsitePasswordRequest struct {
VMID int `json:"vmid"`
// RecoveryCode is the customer's R. NEVER logged, never persisted, never echoed.
RecoveryCode string `json:"recovery_code"`
}
// handleRecoverOffsitePassword fetches this host's sealed bundle, unseals it with R and returns only
// the offsite repository password (plus its sha256, for hash-only comparison by the caller).
func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Request, vmid int) {
var req recoverOffsitePasswordRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
R := strings.TrimSpace(req.RecoveryCode)
req.RecoveryCode = "" // drop the decoded copy immediately
if R == "" {
writeErr(w, http.StatusBadRequest, "recovery_code is required")
return
}
if s.escrowRecovery == nil {
R = ""
writeErr(w, http.StatusServiceUnavailable, "offsite key recovery is not configured on this agent (no hub client)")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
s.logger.Info("local-api: recovering the offsite repository password from the sealed escrow (R via body, never logged/persisted)", "vmid", vmid)
pw, err := s.escrowRecovery.RecoverOffsiteRepoPassword(ctx, R)
R = "" // cleared on BOTH paths, before anything else can happen
if err != nil {
// Each situation gets its own status and its own words. None of them names a secret.
switch {
// ── R-224 (2026-08-06) — THE FETCH FAILURE IS NOT A WRONG CODE. ────────────────────────
//
// This case did not exist, and its absence is the defect. A failed fetch fell through to the
// `default` below and was answered with "the recovery code did not open the sealed bundle" —
// so a hub that could not be reached was reported to the customer as a bad recovery code, on
// the one screen whose whole purpose is to be believed about their backups.
//
// Measured live 2026-08-05 (CAMPAIGN-11 F3 and F4): a CORRECT current code returned that
// message in 0.0556 s with the hub firewalled off, and in 0.0299 s with this agent stopped —
// against ~1.0 s for a genuine unseal. No unseal was attempted in either case.
//
// 502 rather than 400: 4xx says "your request was bad", and the request was not bad — an
// upstream dependency failed. The status is the machine-readable half; the controller
// classifies on it and must never parse this sentence.
//
// ⚠ THE CODE WAS NOT USED. Nothing may be said about it — not that it was wrong, and not
// that it was right.
case errors.Is(err, escrow.ErrBundleFetch):
s.logger.Warn("local-api: offsite key recovery: the sealed bundle could not be FETCHED — the recovery code was never used", "vmid", vmid, "err", err)
writeErr(w, http.StatusBadGateway, "the sealed recovery bundle could not be fetched from the hub — the recovery code was NOT used and nothing was written")
case errors.Is(err, escrow.ErrNoEscrowBlob):
s.logger.Warn("local-api: offsite key recovery: the hub holds no sealed bundle for this host", "vmid", vmid)
writeErr(w, http.StatusNotFound, "the hub holds no sealed recovery bundle for this host — no escrow ceremony has run")
case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
default:
// The fail-closed WRONG-CODE case, and only it: the bundle was fetched and `age -d`
// refused it. Every other situation above has its own status. The agent log records the
// STEP, never the code.
s.logger.Warn("local-api: offsite key recovery: the fetched bundle did not open with the supplied recovery code", "vmid", vmid, "err", err)
writeErr(w, http.StatusBadRequest, "the recovery code did not open the sealed bundle — nothing was written")
}
return
}
sum := sha256.Sum256([]byte(strings.TrimSpace(pw)))
// §8.6's lesson, applied: say exactly WHAT was recovered and what was NOT, so nobody reading this
// concludes the wrong thing about the bundle's contents (which is how link 8 came to be missing).
s.logger.Info("local-api: offsite repository password RECOVERED from the sealed escrow — returning that field ONLY "+
"(the tunnel token, the PBS token and the WG key stay inside the agent and are not returned)",
"vmid", vmid, "restic_pw_sha256", hex.EncodeToString(sum[:]))
writeOK(w, map[string]any{
"restic_repo_password": pw,
"restic_pw_sha256": hex.EncodeToString(sum[:]),
})
}
@@ -0,0 +1,99 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
)
// R-224 — THE STATUS IS THE DISCRIMINATOR, and this test asserts the CONSEQUENCE (what the HTTP
// boundary answers) rather than the mechanism (that the sentinel exists).
//
// The controller one trust tier down classifies on the STATUS and must never parse the sentence. So
// the contract this pins is: four distinguishable situations, four distinct statuses, and the
// wrong-code message reachable ONLY from a real refusal.
//
// Before R-224 the first and last rows both answered 400 with the same sentence — which is how
// CAMPAIGN-11 F3 told a customer holding a CORRECT code that it did not open their package.
type fakeRecoverer struct{ err error }
func (f fakeRecoverer) RecoverOffsiteRepoPassword(context.Context, string) (string, error) {
if f.err != nil {
return "", f.err
}
return "0123456789abcdef0123456789abcdef", nil
}
func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
cases := []struct {
name string
err error
wantStatus int
// mustNotSay guards the specific misattribution each status exists to prevent.
mustNotSay []string
}{
{
name: "fetch failed — the code was NEVER used",
err: errors.Join(escrow.ErrBundleFetch, errors.New("hub: transport error: no route to host")),
wantStatus: 502,
mustNotSay: []string{"did not open"},
},
{
name: "wrong code — the bundle WAS fetched and refused it",
err: errors.New("escrow: the recovery code did not unwrap the identity escrow"),
wantStatus: 400,
mustNotSay: []string{"could not be fetched"},
},
{
name: "the hub holds no bundle",
err: escrow.ErrNoEscrowBlob,
wantStatus: 404,
mustNotSay: []string{"did not open"},
},
{
name: "the bundle predates the repository-password field",
err: escrow.ErrNoResticPassword,
wantStatus: 409,
mustNotSay: []string{"could not be fetched"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowRecovery = fakeRecoverer{err: tc.err}
w := do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`)
if w.Code != tc.wantStatus {
t.Fatalf("status: got %d, want %d — body=%s", w.Code, tc.wantStatus, w.Body.String())
}
for _, phrase := range tc.mustNotSay {
if strings.Contains(w.Body.String(), phrase) {
t.Fatalf("the %d answer must not say %q — body=%s", tc.wantStatus, phrase, w.Body.String())
}
}
})
}
}
// The pair that matters most, stated as its own assertion so a regression cannot hide inside a table:
// a fetch failure and a wrong code must never answer with the SAME status. Collapsing them is the
// whole of R-224.
func TestRecoverOffsitePassword_FetchFailureAndWrongCodeDiffer(t *testing.T) {
status := func(err error) int {
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.escrowRecovery = fakeRecoverer{err: err}
return do(t, srv.Handler(), "POST", "/escrow/recover-offsite-password", "A",
`{"vmid":8200,"recovery_code":"correct horse battery staple sedative anaconda wobbly kingdom placard yodel"}`).Code
}
fetch := status(errors.Join(escrow.ErrBundleFetch, errors.New("no route to host")))
wrong := status(errors.New("escrow: the recovery code did not unwrap the identity escrow"))
// RED-PROOF: delete the ErrBundleFetch case from handleRecoverOffsitePassword → both become 400
// → this FAILS. That is the exact pre-R-224 code, and the exact defect CAMPAIGN-11 measured.
if fetch == wrong {
t.Fatalf("a failed fetch and a wrong code must not share a status (both %d)", fetch)
}
}
+23
View File
@@ -111,6 +111,14 @@ type HostMetricsProvider interface {
}
// Options configures a Server.
// EscrowRecoverer opens this host's sealed identity bundle with the customer recovery code and
// returns ONLY the offsite restic repository password (R-199 links 6-8). An interface so the
// localapi package needs no hub-client dependency and the route is testable without crypto.
// R is an argument and is never retained by any implementation.
type EscrowRecoverer interface {
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (string, error)
}
type Options struct {
ListenAddr string // bridge IP:port
Cert tls.Certificate
@@ -210,6 +218,11 @@ type Options struct {
// GET /debug/logs. OPTIONAL — when nil the endpoint reports "not configured".
LogRing *applog.Ring
Logger *slog.Logger
// EscrowRecovery (R-199, v0.125.0) is the offsite-key recovery seam behind
// POST /escrow/recover-offsite-password. OPTIONAL — nil → that route reports "not configured"
// (503) instead of failing obscurely. Satisfied by escrow.OffsiteKeyRecoverer.
EscrowRecovery EscrowRecoverer
}
// defaultBackupCadence is the fallback /backup/due window when none is configured.
@@ -273,6 +286,11 @@ type Server struct {
netMountRoot string // the user-data namespace root for the network-mount role gate
smbCredsDir string // where SMB creds files are written (out-of-band, 0600)
escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password
// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
// offsite repository password. OPTIONAL — nil (no hub client configured) makes
// POST /escrow/recover-offsite-password answer 503 rather than pretending.
escrowRecovery EscrowRecoverer
intent IntentRecorder // slice 10 P3 (optional)
guestBinds *GuestBindStore // F9 startup bind re-assert record (optional)
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
@@ -423,6 +441,7 @@ func NewServer(o Options) (*Server, error) {
netMountRoot: storage.NetworkMountRoot,
smbCredsDir: o.SmbCredsDir,
escrowStagePath: o.EscrowStagePath,
escrowRecovery: o.EscrowRecovery,
intent: o.Intent,
guestBinds: o.GuestBinds,
formatJobs: o.FormatJobs,
@@ -518,6 +537,10 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /escrow/stage-secret", s.withGuest(s.handleStageEscrowSecret))
// fork-4 hygiene: wipe the staged secret once escrowed (controller calls this on confirm). Idempotent.
mux.HandleFunc("DELETE /escrow/stage-secret", s.withGuest(s.handleWipeStagedEscrowSecret))
// R-199 (v0.125.0): recover the offsite repository password from the hub-held sealed bundle,
// using the customer recovery code supplied in the body. Returns that ONE field. See
// escrow_recover.go for R's handling rules — they are the tightest in this package.
mux.HandleFunc("POST /escrow/recover-offsite-password", s.withGuest(s.handleRecoverOffsitePassword))
// Controller-driven escrow ceremony (v0.88.0): preflight checklist, the detached root ceremony
// job (fixed-argv sudo self-invocation), its status, and the ONE-SHOT in-memory R claim.
+89 -2
View File
@@ -53,7 +53,11 @@ type claimFacts struct {
nodes []claimNode // the whole disk + its children (partitions)
lvmPV bool // pvs (authoritative): the disk / a partition is an LVM physical volume
zfsMember bool // zpool (authoritative): the disk / a partition is a ZFS pool member
gatherErr string // non-empty ⇒ a REQUIRED read failed ⇒ fail-safe CLAIMED
// felhomOwnedMounts (R-220) — mountpoints OUTSIDE /mnt/felhom-drives that are nevertheless Felhom's
// OWN, corroborated from the host mount table: the same device is also mounted at the managed path.
// Empty means "nothing corroborated", which is the fail-safe direction.
felhomOwnedMounts map[string]bool
gatherErr string // non-empty ⇒ a REQUIRED read failed ⇒ fail-safe CLAIMED
}
// classifyClaim is the pure guard verdict. unclaimed=true ONLY when the device is provably free for
@@ -81,7 +85,20 @@ func classifyClaim(f claimFacts) (unclaimed bool, reason string) {
if memberFSTypes[n.fstype] {
return false, "device holds a " + n.fstype + " (" + n.name + ")"
}
if n.mountpoint != "" && !underFelhomDrives(n.mountpoint) {
// ── R-220 — A MOUNT FELHOM ITSELF MADE IS NOT "SOMETHING ELSE". ───────────────────────
//
// Enrolment mounts a drive TWICE: at the managed path `/mnt/felhom-drives/<name>` and at the
// raw `/mnt/<name>` it creates on the host. The host — and therefore that raw mount — survives
// a guest rebuild, while the controller's registry does not. So after a rebuild the customer's
// own drives looked foreign, `attach` returned an empty list, and the refusal told them to
// choose from it. Measured live three times (CAMPAIGN-11 Phase 1, and the R-201 re-walk twice);
// unmounting only the raw mounts flipped `attach: []` to both drives every time.
//
// The fence this must NOT breach: a disk genuinely in use by something else stays refused. So
// the exemption is not "any /mnt/* path" — it is CORROBORATED: the same device must ALSO be
// mounted at Felhom's managed path, which is a state only Felhom's own enrolment produces.
// A foreign disk at /srv/data or /media/x has no such counterpart and is still refused.
if n.mountpoint != "" && !underFelhomDrives(n.mountpoint) && !f.felhomOwnedMounts[n.mountpoint] {
return false, "device is mounted at " + n.mountpoint + " (" + n.name + ")"
}
}
@@ -154,6 +171,8 @@ func (h *SudoHostOps) gatherClaimFacts(ctx context.Context, device string) claim
return f
}
f.nodes = nodes
// R-220: corroborate which non-managed mountpoints are nevertheless Felhom's own.
f.felhomOwnedMounts = felhomOwnedMounts(device, nodes, h.mountTable)
// LVM PV (authoritative). pvs installed but erroring ⇒ fail-safe claimed; absent ⇒ rely on lsblk's
// LVM2_member FSTYPE (already in nodes).
@@ -285,3 +304,71 @@ func (h *SudoHostOps) zfsMembers(ctx context.Context, nodes []claimNode, wholeDi
}
return false, nil
}
// mountTableSource yields the host mount table as (device, mountpoint) pairs. A seam so the R-220
// corroboration is unit-testable without a host. nil ⇒ the real /proc/mounts.
type mountTableSource func() ([][2]string, error)
// procMounts reads /proc/mounts — WORLD-READABLE, so this needs no sudo and no allowlisted command.
// That matters: the lsblk invocation is pinned verbatim in the sudoers file
// (`lsblk -J -o NAME,FSTYPE,PTTYPE,MOUNTPOINT /dev/*`), so switching it to the plural MOUNTPOINTS
// would have meant shipping a sudoers change with the binary — a far larger blast radius than this
// finding warrants. Reading the mount table directly sidesteps that entirely.
func procMounts() ([][2]string, error) {
data, err := os.ReadFile("/proc/mounts")
if err != nil {
return nil, err
}
var out [][2]string
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
// /proc/mounts escapes spaces as \040; unescape so a path with a space still compares.
out = append(out, [2]string{fields[0], strings.ReplaceAll(fields[1], `\040`, " ")})
}
return out, nil
}
// felhomOwnedMounts returns the mountpoints of `device` (and its children) that sit OUTSIDE
// /mnt/felhom-drives but are still Felhom's own, corroborated by the same device also being mounted
// UNDER /mnt/felhom-drives. That pairing is what enrolment produces and nothing else does.
//
// ⚠ FAIL-SAFE: an unreadable mount table returns an EMPTY set, never a permissive one. The device then
// classifies exactly as it did before R-220 — refused — because "we could not corroborate" must never
// read as "it is ours".
func felhomOwnedMounts(device string, nodes []claimNode, src mountTableSource) map[string]bool {
if src == nil {
src = procMounts
}
table, err := src()
if err != nil {
return nil // unreadable ⇒ corroborate nothing
}
// Every device name this disk answers to: the whole disk and each child node.
devs := map[string]bool{device: true}
if wd, ok := wholeDiskOf(device); ok {
devs[wd] = true
}
for _, n := range nodes {
devs["/dev/"+n.name] = true
}
// A device is Felhom-managed only if it is mounted under the managed prefix.
managed := map[string]bool{}
for _, row := range table {
if devs[row[0]] && underFelhomDrives(row[1]) {
managed[row[0]] = true
}
}
if len(managed) == 0 {
return nil
}
owned := map[string]bool{}
for _, row := range table {
if managed[row[0]] && !underFelhomDrives(row[1]) {
owned[path.Clean(row[1])] = true
}
}
return owned
}
+102
View File
@@ -0,0 +1,102 @@
package storage
import "testing"
// ── R-220 — A MOUNT FELHOM ITSELF MADE IS NOT "SOMETHING ELSE" ──────────────────────────────────
//
// Enrolment mounts a drive twice: at `/mnt/felhom-drives/<name>` and at the raw `/mnt/<name>` it
// creates on the host. The host survives a guest rebuild; the controller's registry does not. So after
// a rebuild the customer's own drives read as claimed-by-something-else, `attach` came back empty, and
// the refusal told them to pick from the empty list. Measured three times live.
//
// The fence: a disk genuinely in use elsewhere must STILL be refused. These assert both directions.
// ── SCENARIO E — the customer's own drive is offered again after a rebuild ───────────────────────
//
// RED-PROOF: drop `&& !f.felhomOwnedMounts[n.mountpoint]` from classifyClaim — the pre-R-220 check —
// and this FAILS with the drive refused and the list empty again.
func TestClassifyClaim_R220_FelhomsOwnRawMountIsNotForeign(t *testing.T) {
f := claimFacts{
device: "/dev/sdb", wholeDisk: "/dev/sdb", wholeDiskOK: true,
nodes: []claimNode{{name: "sdb", fstype: "ext4", mountpoint: "/mnt/adatok"}},
// corroborated: the SAME device is also mounted at the managed path
felhomOwnedMounts: map[string]bool{"/mnt/adatok": true},
}
unclaimed, reason := classifyClaim(f)
if !unclaimed {
t.Fatalf("R-220 RETURNED: the customer's own drive is refused after a rebuild — %q", reason)
}
}
// ── SCENARIO F — a genuinely foreign mount is STILL refused ──────────────────────────────────────
//
// RED-PROOF: over-widen the fix to exempt any /mnt/* path (or to skip the mountpoint check entirely)
// and this FAILS — a disk another system is using would be offered for formatting.
func TestClassifyClaim_R220_ForeignMountIsStillRefused(t *testing.T) {
for _, mp := range []string{"/srv/data", "/media/photos", "/mnt/someone-elses-disk", "/var/lib/other"} {
f := claimFacts{
device: "/dev/sdb", wholeDisk: "/dev/sdb", wholeDiskOK: true,
nodes: []claimNode{{name: "sdb", fstype: "ext4", mountpoint: mp}},
felhomOwnedMounts: nil, // nothing corroborated it as ours
}
unclaimed, reason := classifyClaim(f)
if unclaimed {
t.Fatalf("THE FENCE BROKE: a disk mounted at %s was offered for formatting", mp)
}
if reason == "" {
t.Fatalf("a refusal must carry a reason (%s)", mp)
}
}
}
// The corroboration itself: it must require BOTH mounts of the SAME device, and fail safe.
func TestFelhomOwnedMounts_RequiresTheManagedCounterpart(t *testing.T) {
nodes := []claimNode{{name: "sdb"}}
t.Run("both mounts present -> the raw one is ours", func(t *testing.T) {
src := func() ([][2]string, error) {
return [][2]string{
{"/dev/sdb", "/mnt/adatok"},
{"/dev/sdb", "/mnt/felhom-drives/adatok"},
}, nil
}
got := felhomOwnedMounts("/dev/sdb", nodes, src)
if !got["/mnt/adatok"] {
t.Fatal("the raw enrolment mount was not recognised as Felhom's own")
}
})
t.Run("only the raw mount -> corroborates NOTHING", func(t *testing.T) {
src := func() ([][2]string, error) {
return [][2]string{{"/dev/sdb", "/mnt/adatok"}}, nil
}
if got := felhomOwnedMounts("/dev/sdb", nodes, src); len(got) != 0 {
t.Fatalf("a lone /mnt/<name> mount must corroborate nothing, got %v", got)
}
})
t.Run("a DIFFERENT device under the managed path does not vouch for this one", func(t *testing.T) {
src := func() ([][2]string, error) {
return [][2]string{
{"/dev/sdb", "/srv/data"},
{"/dev/sdc", "/mnt/felhom-drives/mentes"}, // someone else's, not sdb's
}, nil
}
if got := felhomOwnedMounts("/dev/sdb", nodes, src); got["/srv/data"] {
t.Fatal("another device's managed mount vouched for a foreign one")
}
})
t.Run("an unreadable mount table corroborates NOTHING (fail-safe)", func(t *testing.T) {
src := func() ([][2]string, error) { return nil, errRead }
if got := felhomOwnedMounts("/dev/sdb", nodes, src); len(got) != 0 {
t.Fatalf("an unreadable mount table must corroborate nothing, got %v", got)
}
})
}
var errRead = errNoMountTable{}
type errNoMountTable struct{}
func (errNoMountTable) Error() string { return "mount table unreadable" }
+3
View File
@@ -164,6 +164,9 @@ type SudoHostOps struct {
// UNPRIVILEGED read (`systemctl is-failed`) — seam-injected so the reassert's F10 reset-failed path
// is unit-testable without a real systemd. Default set in NewSudoHostOps.
unitFailed func(ctx context.Context, unit string) bool
// mountTable (R-220) yields the host mount table for the "is this mount Felhom's own?"
// corroboration. nil ⇒ the real /proc/mounts; tests inject.
mountTable mountTableSource
}
// SudoHostOpsConfig configures a SudoHostOps.
+3
View File
@@ -41,10 +41,13 @@ import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reuse_refs_check.py")
SHARED_INSTRUCTIONS = os.path.join(
os.path.dirname(ROOT), "felhom.eu", "scripts", "instructions_gate.py")
# (label, absolute script path, args, fast)
GATES = [
("reuse-refs", SHARED_REUSE, [ROOT], True),
("instructions", SHARED_INSTRUCTIONS, [ROOT], True),
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
]