# REPORT — R-185: a tier the box cannot READ must say so

**Date:** 2026-08-03 · **Repos:** `felhom-agent` **v0.122.0 → v0.123.0** (`fe14bc6`) · `felhom.eu`
installer **1.23.0 → 1.24.0** (`688470c`, tag `installer-v1.24.0`, manifest bump `311dc06`) ·
**no hub change and no hub bump** — the hub already alerts on a degraded critical capability, which is
why that mechanism was chosen.

---

## 1. Baselines, re-read on arrival

| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-agent` | `0b28eae7bb14` | `v0.122.0` | **yes** |
| `felhom.eu` | `7a5694341d59` | installer `1.23.0`, both `--ref=installer-v1.23.0` (lines 327, 372) | **yes** |

Highest register ID in use **R-189**; R-190+ confirmed free by grep, and none was needed.

## 2. Part 0 — the measurements, before anything was designed against them

**The row's three-way observation, reproduced unchanged:**

| leg | result |
|---|---|
| root, `pvesh … /storage/felhom-backup/content` | **3 archives** — 6.1 / 6.2 / 6.3 GB, dated 08-01, 08-02, 08-03 |
| the **agent's token**, same endpoint | `{"data":[]}` |
| the agent's token, `/storage/local/content` | **8 entries** — the token works where it is granted |

So the token is the variable, not the storage. Two further checks removed the obvious alternative
explanation: guest **9201 IS in the `felhom` pool** (so `VM.Backup` is not the discriminator), and
`pveum acl list` showed ACL rows for `/storage/{local,local-lvm,felhom-pbs}` and **none** for
`/storage/felhom-backup`.

**The permission query, asked by the token itself — and the obvious reading is wrong:**

```
/storage/felhom-pbs     → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
/storage/felhom-backup  → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
```

The ungranted path answers **neither empty nor 403**. It answers with the privileges **inherited**
from the box-wide `/` grant. A probe asking *"did the path come back?"* — or *"does it hold
`Datastore.Audit`?"* — would have reported the blinded storage **healthy**. This is exactly what §3
required to be measured rather than assumed, and it changed the design: the probe tests
`Datastore.AllocateSpace` specifically, and a red-proof pins that choice.

## 3. The probe

`Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's own token**
(asking as root answers a different question and always says yes). `storeGrantStatuses` emits one
`capability.Status` per configured tier.

**Deviation from §5/§8.1, stated because a recommendation not followed gets a line:** the spec asked
for the sudo `Prober` to be minimally generalised. This repo already has the better-established
pattern for exactly this — `poolReadStatus`, composed **around** the prober, with the comment *"an API
read does not belong inside the sudo-policy probe"* (v0.62.0, audit A1). The probe follows that
precedent instead. `capability.Status` is untouched either way, which is the constraint that mattered.

**Decisions:**

- **The probed set comes from the box's own `BackupTiers()`**, never a fixed list — a hardcoded probe
  list is the defect reproduced inside the fix.
- **Critical** (§8.3): the hub alerts only on critical, so a non-critical entry would ride the report
  and alert nobody — the same silence with extra steps. **Except** the `local` fallback target, which
  host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
  does not page. Turning an ordinary documented setup into an alert is how a signal becomes something
  an operator archives unread.
- **It never consults content**, so it cannot alarm on a newborn tier by construction — a stronger
  guarantee than gating on emptiness would be.
- **It never reports ok when it could not ask.** Unreachable PVE is degraded: a self-check that fails
  open converts *"I do not know"* into *"fine"*.

## 4. The installer — the root cause was not where the row or the task expected

Both assumed `PVE_STORAGES` (the fixed grant list) was the culprit. **It is not.**
`configure_backup_target` has two arms:

- **Case A** creates the storage and calls `felhom-backup-target-apply grant` in the same breath — a
  box that builds its own target has always been correct.
- **The Scenario-F arm** — *"the target already exists, leave it exactly as it is"* — **returned
  without granting**.

So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
surviving a reinstall — which is both demo boxes) pointed `local_backup_target` at a storage its own
token could not read. The reuse arm now ensures the ACL through the same guarded wrapper.

**Scenario F is unviolated:** the storage DEFINITION is still untouched. Granting the role the agent is
supposed to have on the target this same script is about to write into `agent.json` is finishing the
job, not retargeting the box; `pveum acl modify` is idempotent, so a box that already has it is
unchanged and a box whose token was rotated gets it back.

**`$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES`,** and the comment now says why: that
list is granted in step 4/5, *before* `configure_backup_target` runs in step 6, and `--acl-storages`
entries are preflight-checked for existence. Adding it there would grant on a storage that may not yet
exist and would split ownership of the decision across two places.

**A gate now asserts it:** every arm of `configure_backup_target` that resolves the target must also
grant on it — the check that would have caught this.

## 5. Live validation, in order

| # | evidence |
|---|---|
| 1 | Part 0's measurements above, taken **before** any change |
| 2 | **The signal that has never existed**, on the still-blind box: `capability DEGRADED … capability=pve:store-grant:felhom-backup … reason="the agent token lacks Datastore.AllocateSpace on /storage/felhom-backup (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested" critical=true`, with `ok=69 total=70 degraded=1`. The hub: `Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)` and **`Operator email sent`** |
| 3 | Grant applied (user **and** token — a privsep token's rights are the intersection); the token then lists **3 archives** where it listed none, and the permission answer becomes `{"Datastore.AllocateSpace":1,"Datastore.Allocate":1}` |
| 4 | `capabilities self-check ok=70 total=70 degraded=0`; the hub: `degraded → ok (agent_capability_recovered)` |
| 5 | **The host tier is a due-check candidate for the first time on that box**: `tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst" proven=""` — and the settle rule applies to it exactly as to the others, selecting the **08-02** archive because the 08-03 one has not settled 24 h |
| 6 | The served installer over HTTPS: `SCRIPT_VERSION="1.24.0"`, and the served bytes carry the fix itself, not merely the version |

## 6. The other machines

- **demo-hp CARRIES THE SAME DRIFT — and was fixed.** `local_backup_target=felhom-backup`, ACL rows for
  `local`, `local-lvm`, `felhom-pbs` only. §8.6 assumed a single affected box; the same one-line,
  additive, path-scoped, idempotent grant applies to the other, and leaving a known-blind backup tier
  on a Tier-0 box after finding it would be this row happening twice. Granted (user + token); its
  token now lists **4 archives**. It still runs agent `0.120.0`, so it has no probe yet — that arrives
  when you vouch.
- **The tester's box was NOT touched** (Tier 2, protected). **What is known without connecting to it:**
  it very likely carries the same drift — the mechanism is the Scenario-F reuse arm, which fires on
  any box whose target pre-dated its install, and its target was moved by the very runbook that
  creates that condition. It is due for reinstall, and installer 1.24.0 fixes it on the way in.

## 7. Tests and red-proofs

Green gate: `go build ./... && go vet ./... && go test ./...` — rc=0, plus `agent_gates.py` and
`repo_gates.py` all OK. Test runs and commits were always separate commands.

| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestStoreGrant_ForbiddenStorageIsDegradedAndNamed` | degraded, critical, naming storage **and** role | probe removed from `probeAll` | **FAIL** — `main.go never calls storeGrantStatuses` (via the seam test); with the wrong-privilege mutation: `must be DEGRADED, not "ok"` |
| A′ | `TestStoreGrant_InheritedPrivilegesAreNotAGrant` | the measured trap: inherited ≠ granted | probe `Datastore.Audit` instead | **FAIL** — `checking for the wrong privilege reports a blinded storage healthy; got "ok"` |
| B | `TestStoreGrant_GrantedButEmptyIsHealthy` | a readable-but-empty tier is healthy | — (it never reads content, so emptiness cannot reach it) | pass |
| B′ | `TestStoreGrant_TheFallbackTargetIsNotCritical` | `local` is reported but does not page | gating removed (`return true`) | **FAIL** — `must not page the operator about an ordinary, documented configuration` |
| C | `TestStoreGrant_ForbiddenAndNewbornAreDistinguishable` | different status **and** different capability id | — | pass |
| — | `TestStoreGrant_UnreachablePVEIsDegradedNotOK` | unknown ≠ ok | — | pass |
| F | `hostinstall_gates.py` backup-target assertion | every resolving arm also grants | reuse arm reverted | **FAIL** — `resolves the backup target in 2 place(s) but grants in only 1` |
| H | `TestMainWiresTheStoreGrantProbe` | **AST** of `main.go` | call commented out | **FAIL** — a `strings.Contains` check would have passed |

**A hollow test caught and fixed before it shipped:** the first draft of `storegrant_test.go`
re-implemented the verdict branch inside the test. It passed, and would have kept passing while
production diverged. The decision was extracted into `storeGrantVerdict` and the tests now call it.

**Scenario B's red-proof, honestly:** the spec asked for "degrade on an empty content listing" as the
mutation. That is not a mutation of this code — the probe never looks at content, which is a stronger
guarantee than gating on emptiness. The gating red-proof above (`storeGrantCritical`) is the one that
exercises the guard that does exist, and it fails as required.

## 8. Files, commits, tag

`internal/proxmox/query.go` (`Permissions`), `cmd/felhom-agent/main.go` (`storeGrantStatuses`,
`storeGrantVerdict`, `storeGrantCritical`, `storeGrantRequiredPriv`, wiring),
`cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `REPORT.md`.
`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`,
`manifests/webpage.yaml`, `CONTEXT.md`, `STATUS.md`, `documentation/architecture/00-capability-map.md`,
`documentation/backlog/OPEN-ITEMS.md`, `documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md`.

**Commits** — `felhom-agent`: `fe14bc6` (v0.123.0). `felhom.eu`: `688470c` (installer 1.24.0), `311dc06`
(manifest refs), `e3187c8` (docs). **Installer tag:** `installer-v1.24.0`.

## 9. Deployment

Agent released through `release-agent.sh` — tag `v0.123.0`, sha256
`74910135ac4feb1b7f0ad4dbd1541d965cbc0fe70d4f47b62ebf7e4bfb962453`, round-trip verified. The
**published bytes** were downloaded and deployed: the running binary's sha matches the published one.
`felhom-agent --version` → **0.123.0**, `systemctl is-active` → active, prior kept as `.bak-0.122.0`.
**NOT VOUCHED** — that stays the operator's act.

## 10. Registers

- **R-185 → CLOSED** (shipped + proven live on both demo boxes), with the corrected root cause
  recorded on the row.
- No new IDs minted; `ROADMAP.md` contains no R-185 row, so there was nothing to collapse.
- **The capability map's whole-guest row was OPTIMISTIC and now says so:** every live restore-test it
  cited is on the OFFSITE tier, and the HOST tier was not merely unproven but *unprovable* on both
  demo boxes. It now records that, the closure, and that it will carry a host-tier live proof when one
  runs.
- The vzdump-target-move runbook's item 5 **predicted this** and is annotated, not rewritten: it
  expected a 403 on backup, and the reason it did not surface that way is that `vzdump` writes through
  a root path, so backups kept landing while the agent's *read* stayed blind.
- `CONTEXT.md`: agent-side entry, plus `felhom.eu` **S-21** (empty ≠ forbidden; the measured trap) and
  **S-22** (the Scenario-F arm must finish the job).

## 11. Teardown

**Nothing was provisioned.** No scratch storage, no fixture grant, no probe tag, no scratch package
version. The two ACL grants are the intended durable change; the only other mutation was the
installer label, which is reversible by moving the tag.

## 12. Observations — noticed, recorded, NOT acted on

- **Both demo boxes are now due for a host-tier restore-test**, which has never run on either. The
  scheduler will pick it up within 6 h unattended (a ~6 GB local restore — fast, and cheaper than the
  offsite ones). Expected, not a defect, and the first host-tier proof this fleet will have.
- **`--acl-storages` semantics are unchanged and the automatic grant does not consult it.** If an
  operator passes `--acl-storages` deliberately excluding the backup target, the target is still
  granted by the resolution path. That is the correct precedence — a box cannot function with an
  unreadable backup target — but it is a place where an override is not absolute, and it is written
  here rather than left to be discovered.
- **`storeGrantRequiredPriv` is a single privilege**, chosen from measurement. If PVE ever changes
  which privilege gates content listing, the probe would report healthy while the tier is blind. The
  test asserts the constant's value so a change forces a re-measurement, but nothing detects a change
  on PVE's side.
- **Ten pre-existing `gofmt`-unclean files** remain in the agent repo (unchanged from yesterday's
  observation); every file touched here is clean.