felhom-agent/REPORT.md

REPORT — R-185: a tier the box cannot READ must say so

Date: 2026-08-03 · Repos: felhom-agent v0.122.0 → v0.123.0 (fe14bc6) · felhom.eu installer 1.23.0 → 1.24.0 (688470c, tag installer-v1.24.0, manifest bump 311dc06) · no hub change and no hub bump — the hub already alerts on a degraded critical capability, which is why that mechanism was chosen.

---

1. Baselines, re-read on arrival

Repo	main @ commit	Version	Matched §1?
felhom-agent	0b28eae7bb14	v0.122.0	yes
felhom.eu	7a5694341d59	installer 1.23.0, both --ref=installer-v1.23.0 (lines 327, 372)	yes

Highest register ID in use R-189; R-190+ confirmed free by grep, and none was needed.

2. Part 0 — the measurements, before anything was designed against them

The row's three-way observation, reproduced unchanged:

leg	result
root, pvesh … /storage/felhom-backup/content	3 archives — 6.1 / 6.2 / 6.3 GB, dated 08-01, 08-02, 08-03
the agent's token, same endpoint	{"data":[]}
the agent's token, /storage/local/content	8 entries — the token works where it is granted

So the token is the variable, not the storage. Two further checks removed the obvious alternative explanation: guest 9201 IS in the felhom pool (so VM.Backup is not the discriminator), and pveum acl list showed ACL rows for /storage/{local,local-lvm,felhom-pbs} and none for /storage/felhom-backup.

The permission query, asked by the token itself — and the obvious reading is wrong:

/storage/felhom-pbs     → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
/storage/felhom-backup  → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}

The ungranted path answers neither empty nor 403. It answers with the privileges inherited from the box-wide / grant. A probe asking "did the path come back?" — or "does it hold Datastore.Audit?" — would have reported the blinded storage healthy. This is exactly what §3 required to be measured rather than assumed, and it changed the design: the probe tests Datastore.AllocateSpace specifically, and a red-proof pins that choice.

3. The probe

Client.Permissions reads /access/permissions?path=/storage/<target> as the agent's own token (asking as root answers a different question and always says yes). storeGrantStatuses emits one capability.Status per configured tier.

Deviation from §5/§8.1, stated because a recommendation not followed gets a line: the spec asked for the sudo Prober to be minimally generalised. This repo already has the better-established pattern for exactly this — poolReadStatus, composed around the prober, with the comment "an API read does not belong inside the sudo-policy probe" (v0.62.0, audit A1). The probe follows that precedent instead. capability.Status is untouched either way, which is the constraint that mattered.

Decisions:

• The probed set comes from the box's own BackupTiers(), never a fixed list — a hardcoded probe list is the defect reproduced inside the fix.
• Critical (§8.3): the hub alerts only on critical, so a non-critical entry would ride the report and alert nobody — the same silence with extra steps. Except the local fallback target, which host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it does not page. Turning an ordinary documented setup into an alert is how a signal becomes something an operator archives unread.
• It never consults content, so it cannot alarm on a newborn tier by construction — a stronger guarantee than gating on emptiness would be.
• It never reports ok when it could not ask. Unreachable PVE is degraded: a self-check that fails open converts "I do not know" into "fine".

4. The installer — the root cause was not where the row or the task expected

Both assumed PVE_STORAGES (the fixed grant list) was the culprit. It is not. configure_backup_target has two arms:

• Case A creates the storage and calls felhom-backup-target-apply grant in the same breath — a box that builds its own target has always been correct.
• The Scenario-F arm — "the target already exists, leave it exactly as it is" — returned without granting.

So a box whose felhom-backup pre-dated the install (created by the vzdump-target-move runbook, or surviving a reinstall — which is both demo boxes) pointed local_backup_target at a storage its own token could not read. The reuse arm now ensures the ACL through the same guarded wrapper.

Scenario F is unviolated: the storage DEFINITION is still untouched. Granting the role the agent is supposed to have on the target this same script is about to write into agent.json is finishing the job, not retargeting the box; pveum acl modify is idempotent, so a box that already has it is unchanged and a box whose token was rotated gets it back.

$BACKUP_TARGET_ID is deliberately still NOT in PVE_STORAGES, and the comment now says why: that list is granted in step 4/5, before configure_backup_target runs in step 6, and --acl-storages entries are preflight-checked for existence. Adding it there would grant on a storage that may not yet exist and would split ownership of the decision across two places.

A gate now asserts it: every arm of configure_backup_target that resolves the target must also grant on it — the check that would have caught this.

5. Live validation, in order

#	evidence
1	Part 0's measurements above, taken before any change
2	The signal that has never existed, on the still-blind box: capability DEGRADED … capability=pve:store-grant:felhom-backup … reason="the agent token lacks Datastore.AllocateSpace on /storage/felhom-backup (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested" critical=true, with ok=69 total=70 degraded=1. The hub: Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded) and Operator email sent
3	Grant applied (user and token — a privsep token's rights are the intersection); the token then lists 3 archives where it listed none, and the permission answer becomes {"Datastore.AllocateSpace":1,"Datastore.Allocate":1}
4	capabilities self-check ok=70 total=70 degraded=0; the hub: degraded → ok (agent_capability_recovered)
5	The host tier is a due-check candidate for the first time on that box: tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" proven="" — and the settle rule applies to it exactly as to the others, selecting the 08-02 archive because the 08-03 one has not settled 24 h
6	The served installer over HTTPS: SCRIPT_VERSION="1.24.0", and the served bytes carry the fix itself, not merely the version

6. The other machines

• demo-hp CARRIES THE SAME DRIFT — and was fixed. local_backup_target=felhom-backup, ACL rows for local, local-lvm, felhom-pbs only. §8.6 assumed a single affected box; the same one-line, additive, path-scoped, idempotent grant applies to the other, and leaving a known-blind backup tier on a Tier-0 box after finding it would be this row happening twice. Granted (user + token); its token now lists 4 archives. It still runs agent 0.120.0, so it has no probe yet — that arrives when you vouch.
• The tester's box was NOT touched (Tier 2, protected). What is known without connecting to it: it very likely carries the same drift — the mechanism is the Scenario-F reuse arm, which fires on any box whose target pre-dated its install, and its target was moved by the very runbook that creates that condition. It is due for reinstall, and installer 1.24.0 fixes it on the way in.

7. Tests and red-proofs

Green gate: go build ./... && go vet ./... && go test ./... — rc=0, plus agent_gates.py and repo_gates.py all OK. Test runs and commits were always separate commands.

#	Test	Asserts	Mutation	Observed
A	TestStoreGrant_ForbiddenStorageIsDegradedAndNamed	degraded, critical, naming storage and role	probe removed from probeAll	FAIL — main.go never calls storeGrantStatuses (via the seam test); with the wrong-privilege mutation: must be DEGRADED, not "ok"
A′	TestStoreGrant_InheritedPrivilegesAreNotAGrant	the measured trap: inherited ≠ granted	probe Datastore.Audit instead	FAIL — checking for the wrong privilege reports a blinded storage healthy; got "ok"
B	TestStoreGrant_GrantedButEmptyIsHealthy	a readable-but-empty tier is healthy	— (it never reads content, so emptiness cannot reach it)	pass
B′	TestStoreGrant_TheFallbackTargetIsNotCritical	local is reported but does not page	gating removed (return true)	FAIL — must not page the operator about an ordinary, documented configuration
C	TestStoreGrant_ForbiddenAndNewbornAreDistinguishable	different status and different capability id	—	pass
—	TestStoreGrant_UnreachablePVEIsDegradedNotOK	unknown ≠ ok	—	pass
F	hostinstall_gates.py backup-target assertion	every resolving arm also grants	reuse arm reverted	FAIL — resolves the backup target in 2 place(s) but grants in only 1
H	TestMainWiresTheStoreGrantProbe	AST of main.go	call commented out	FAIL — a strings.Contains check would have passed

A hollow test caught and fixed before it shipped: the first draft of storegrant_test.go re-implemented the verdict branch inside the test. It passed, and would have kept passing while production diverged. The decision was extracted into storeGrantVerdict and the tests now call it.

Scenario B's red-proof, honestly: the spec asked for "degrade on an empty content listing" as the mutation. That is not a mutation of this code — the probe never looks at content, which is a stronger guarantee than gating on emptiness. The gating red-proof above (storeGrantCritical) is the one that exercises the guard that does exist, and it fails as required.

8. Files, commits, tag

internal/proxmox/query.go (Permissions), cmd/felhom-agent/main.go (storeGrantStatuses, storeGrantVerdict, storeGrantCritical, storeGrantRequiredPriv, wiring), cmd/felhom-agent/storegrant_test.go, CHANGELOG.md, CONTEXT.md, REUSE.md, REPORT.md. felhom.eu: scripts/felhom-host-install.sh, scripts/hostinstall_gates.py, scripts/CHANGELOG.md, manifests/webpage.yaml, CONTEXT.md, STATUS.md, documentation/architecture/00-capability-map.md, documentation/backlog/OPEN-ITEMS.md, documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md.

Commits — felhom-agent: fe14bc6 (v0.123.0). felhom.eu: 688470c (installer 1.24.0), 311dc06 (manifest refs), e3187c8 (docs). Installer tag: installer-v1.24.0.

9. Deployment

Agent released through release-agent.sh — tag v0.123.0, sha256 74910135ac4feb1b7f0ad4dbd1541d965cbc0fe70d4f47b62ebf7e4bfb962453, round-trip verified. The published bytes were downloaded and deployed: the running binary's sha matches the published one. felhom-agent --version → 0.123.0, systemctl is-active → active, prior kept as .bak-0.122.0. NOT VOUCHED — that stays the operator's act.

10. Registers

• R-185 → CLOSED (shipped + proven live on both demo boxes), with the corrected root cause recorded on the row.
• No new IDs minted; ROADMAP.md contains no R-185 row, so there was nothing to collapse.
• The capability map's whole-guest row was OPTIMISTIC and now says so: every live restore-test it cited is on the OFFSITE tier, and the HOST tier was not merely unproven but unprovable on both demo boxes. It now records that, the closure, and that it will carry a host-tier live proof when one runs.
• The vzdump-target-move runbook's item 5 predicted this and is annotated, not rewritten: it expected a 403 on backup, and the reason it did not surface that way is that vzdump writes through a root path, so backups kept landing while the agent's read stayed blind.
• CONTEXT.md: agent-side entry, plus felhom.eu S-21 (empty ≠ forbidden; the measured trap) and S-22 (the Scenario-F arm must finish the job).

11. Teardown

Nothing was provisioned. No scratch storage, no fixture grant, no probe tag, no scratch package version. The two ACL grants are the intended durable change; the only other mutation was the installer label, which is reversible by moving the tag.

12. Observations — noticed, recorded, NOT acted on

• Both demo boxes are now due for a host-tier restore-test, which has never run on either. The scheduler will pick it up within 6 h unattended (a ~6 GB local restore — fast, and cheaper than the offsite ones). Expected, not a defect, and the first host-tier proof this fleet will have.
• --acl-storages semantics are unchanged and the automatic grant does not consult it. If an operator passes --acl-storages deliberately excluding the backup target, the target is still granted by the resolution path. That is the correct precedence — a box cannot function with an unreadable backup target — but it is a place where an override is not absolute, and it is written here rather than left to be discovered.
• storeGrantRequiredPriv is a single privilege, chosen from measurement. If PVE ever changes which privilege gates content listing, the probe would report healthy while the tier is blind. The test asserts the constant's value so a change forces a re-measurement, but nothing detects a change on PVE's side.
• Ten pre-existing gofmt-unclean files remain in the agent repo (unchanged from yesterday's observation); every file touched here is clean.