felhom.eu/REPORT.md

REPORT — R-191 (a backup that worked must not report failure) + R-190 (a permission must repair itself)

Date: 2026-08-04 · Repos: felhom-agent v0.123.0 → v0.124.0 → v0.124.1 (257c4d8, and the v0.124.1 fix) · felhom.eu installer 1.24.0 → 1.25.0 (tag installer-v1.25.0) · no hub change and no hub bump.

---

1. Baselines

Repo	cited in §1	found on arrival	matched?
felhom-agent	72161f6cf010 / v0.123.0	same	yes
felhom.eu	d692c46db08f / installer 1.24.0	33f0ab2f33f7 / installer 1.24.0	version yes, commit NO

felhom.eu had moved four commits forward since the brief was written — this morning's alert diagnosis (R-190/191/192/193 rows, the demo-hp offsite re-issue), all pushed. Recorded rather than glossed. Both --ref=installer-v1.24.0 lines confirmed at 327 and 372.

2. Part 0 — the gate, and the three broken instruments that nearly failed it

VERDICT: PASS. ep0 prunes both namespaces on schedule.

prune-demo-felhom   store=felhom-offsite  ns=demo-felhom  schedule=03:30  keep-last 2
prune-demo-hp       store=felhom-offsite  ns=demo-hp      schedule=03:30  keep-last 2

18 prunejob tasks, all status=OK, every day from 2026-07-27 to 2026-08-04, two per day, one per namespace. The newest task log, read in full:

prune job 'prune-demo-felhom' / task triggered by schedule '03:30'
retention options: --ns demo-felhom --max-depth 0 --keep-last 2
Pruning group demo-felhom:"ct/9201"
keep ct/9201/2026-07-27T19:55:41Z
keep ct/9201/2026-07-28T04:49:43Z
TASK OK

Read-only throughout: no job created, edited or run; nothing deleted. ep0 is Tier 2.

The part that matters more than the verdict

Three of my own queries said the endpoint prunes NOTHING, and all three were broken instruments:

query	why it was wrong
'prune' in t['worker-type'] → 0 prune tasks	the field is worker_type; worker-type does not exist, so every task read None
worker_type == 'prune' → 0	the worker type is prunejob, not prune
journalctl -u proxmox-backup | grep -c prune → 0	the unit is proxmox-backup-proxy

A fourth reading was mine to mis-frame: "3 snapshots under keep-last 2 proves nothing is pruning" — the third snapshot landed at 04:49, after that day's 03:30 window, so it proved nothing. I caught that one myself; the other three were caught only by checking the field names when the answer looked too categorical.

Had I acted on them, I would have disabled the only pruning ATTEMPT while filing a finding that nothing prunes anywhere — a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. The gate did its job precisely because it demanded evidence rather than a verdict.

3. R-191 — the off-site tier stops asking to prune

The defect: keep_last: 2 on the offsite tier, so vzdump uploaded and then failed the whole job on missing Datastore.Modify|Datastore.Prune → TASK ERROR: job errors → whole_guest_backup_failed. R-89 moved offsite pruning server-side; the 2026-07-26 "two weeks" ruling was not reversed — where it is enforced moved and this value did not follow.

• Installer 1.25.0: keep_last: 0, and the justifying paragraph rewritten to say where retention now lives, citing R-89 and the Part 0 evidence, plus the warning that if ep0 ever stops pruning this zero is unbounded growth. The agent's guard allowPBSPrune = !primary && keep_last > 0 already reads 0 as never prune — no agent change.
• A gate asserts it (hostinstall_gates.py), red-proved: pinning keep_last: 2 back → FAIL: the offsite backup_target arms a CLIENT-SIDE prune (keep_last=2).
• Both live boxes, quoted:

demo-felhom  backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
demo-felhom  backup tier armed target=felhom-pbs    … keep_last=0 … prune_pbs_allowed=false primary=false
felhom-host  backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
felhom-host  backup tier armed target=felhom-pbs    … keep_last=0 … prune_pbs_allowed=false primary=false

The local tier is untouched (Scenario C).

Not yet observed: a weekly offsite run completing OK end to end. The failing step is removed and the boxes are armed correctly; the schedule proving it is next week's event. The row says so rather than implying the run has happened.

4. R-190 — the grant repairs itself, and the repair is visible

Mitigation. On a missing grant the agent runs the existing root wrapper felhom-backup-target-apply grant <id> and re-reads once — the pbsdr R-22 shape and its restraint. No new privileged surface: grant * was already in the sudoers allowlist for any storage id (confirmed in configs/felhom-agent.sudoers, not assumed), and the verb already grants both user and token. It existed, was permitted, and had only ever been called at storage creation — the built but never wired shape in a verb rather than a seam. Bounded: one attempt per tier per hour.

The record — and how v0.124.0 got it wrong in production with a green test suite

§8.5 asked whether the hub's existing degraded↔ok edge sufficed before building anything new. It does, as a channel — but only if the agent deliberately reports one degraded cycle, because a silent repair produces no edge at all. So nothing new was built: no wire change, no hub change, no new event type. The explanation rides Feature, because that is the field the hub interpolates into the operator's e-mail (monitor/host_capability.go builds its message from names and features; Reason does not travel).

v0.124.0 implemented "one cycle" as one probe CALL, and that failed live:

09:39:34  agent: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED … confirmed_by=re-read
09:39:34  agent: capabilities self-check ok=69 total=70 degraded=1
09:39:37  hub:   host-report stored → pve:store-grant:felhom-backup = ok      ← the record never arrived

probeAll is invoked independently by the self-check log and by the collector building a report. The repairing call was the log's; the report three seconds later found the grant present and sent ok. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence R-190 exists for, re-created inside its own mitigation, while every unit test passed. v0.124.1 replaces it with a latch on time (20 min > the 900 s report interval).

Live proof, twice, on demo-felhom

Grant deleted by hand (both rows — user and token), which reproduces R-190 and the R-185 symptom ({"data":[]} through the token):

10:00:29  agent: store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)
          target=felhom-backup privilege=Datastore.AllocateSpace
          action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read
10:00:30  hub:   host-report → pve:store-grant:felhom-backup = DEGRADED
          feature: "…the agent's storage grant was MISSING and has been AUTOMATICALLY RESTORED…"
10:00:40  hub:   Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)
10:00:40  hub:   Operator email sent for demo-felhom/agent_capability_degraded
          pveum acl list | grep -c felhom-backup → 2   (both rows restored)

And the pair CLOSED, which is the other half of "exactly one alert per loss":

10:20:29  the 20-minute latch expires
10:30:40  hub: Host capability: demo-felhom-8363b5 degraded → ok (agent_capability_recovered)

So one lost grant produced one degraded event + one operator e-mail + one recovery, and nothing further. Final state on both boxes: agent 0.124.1, two ACL rows on /storage/felhom-backup.

Scenario E is unchanged from v0.123.0 — a failed repair, or one that does not survive the re-read, leaves the capability degraded with the failure in its reason.

5. Part 3 — one bounded pass at the mechanism, negatives named

The lead is real as a CLASS and is documented in our own installer: "pveum user token remove purges the token's ACL, so re-applying post-rotate is mandatory" — a rotation would take every grant and re-apply only the hardcoded PVE_STORAGES set, which is exactly the observed end state.

It does not fit this box. A rotation purges all the token's ACLs and mints a new secret; demo-felhom's token still authenticates with the same secret (--selftest OK), it kept its other three storage grants throughout, and only felhom-backup was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at the time).

Ruled out, by name: a host reinstall; any pveum/ACL/user.cfg activity in syslog 04:00–10:00; any cluster-log ACL entry; a token rotation on this box. Not ruled out fleet-wide — any installer run still purges and re-grants only PVE_STORAGES, though installer 1.24.0's reuse-arm fix now re-grants the backup target on that path.

The mechanism remains open, and the row says which half it reflects: mitigation shipped, cause unexplained. No reproduction was attempted and the session was not extended for it.

6. Tests and red-proofs

Green gate both repos: go build ./... && go vet ./... && go test ./... rc=0; agent_gates.py and repo_gates.py all OK. Test runs and commits always separate commands.

#	Test	Mutation	Observed
G	hostinstall_gates.py offsite assertion	pin keep_last: 2 back	FAIL — arms a CLIENT-SIDE prune (keep_last=2)
F	TestGrantRepair_IsBounded	drop the interval check	FAIL — 5 cycles produced 5 attempt(s)
D(record)	TestGrantRepair_ARepairedGrantIsReportedAsATransition	report ok after a repair	FAIL — got "ok" — the loss would be invisible
D(latch)	TestGrantRepair_TransitionSurvivesALaterProbe	ignore the latch in the healthy path	FAIL — got "ok" — the host-report would carry ok
H	TestMainWiresTheGrantRepair	pass nil instead of the repairer	FAIL — not passed the repairer
—	TestGrantRepair_CallsTheExistingWrapperVerb	—	pass (argv is exactly grant <id>; anything else is a policy change)
—	TestGrantRepair_FailureIsReturned, _NilIsSafe	—	pass

Two red-proofs did not fail on the first attempt, and both are reported rather than quietly fixed:

1. The seam mutation (grantRepairer → nil) failed to COMPILE (declared and not used), so the test never ran. The guard holds — you cannot ship it — but it was not demonstrated. The mutation that does fail it keeps the variable used (_ = grantRepairer) and passes nil.
2. The latch red-proof passed, because the test asserted the latch helper rather than the path consuming it. Fixed by extracting storeGrantHealthyVerdict and pointing the test at it — after which the mutation fails as it should. This is the second hollow test caught in this file; the first asserted a Status it had constructed itself. Both are now production functions under test.

Timestamps in the repair tests carry jitter (09:39:34, +13m07s, +27m51s, …) rather than round intervals, per §10.

7. Files, commits, tag

felhom-agent: cmd/felhom-agent/main.go (repairer, latch, storeGrantHealthyVerdict, storeGrantRepairedVerdict, wiring), cmd/felhom-agent/storegrant_test.go, CHANGELOG.md. Commits 257c4d8 (v0.124.0) and the v0.124.1 fix. Released shas: 5e417938… (0.124.0), 5c279bda… (0.124.1), both round-trip verified.

felhom.eu: scripts/felhom-host-install.sh, scripts/hostinstall_gates.py, scripts/CHANGELOG.md, manifests/webpage.yaml, documentation/backlog/OPEN-ITEMS.md, documentation/architecture/07-backup-architecture.md, CONTEXT.md, STATUS.md, REPORT.md. Installer tag: installer-v1.25.0.

8. The served installer

$ curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 SCRIPT_VERSION
SCRIPT_VERSION="1.25.0"
$ … | grep -o '"keep_last":[0-9]*'
"keep_last":0

Both --ref= lines bumped to installer-v1.25.0 (327, 372); ArgoCD Synced/Healthy; webpage rollout complete. The served bytes carry the change, not just the version.

9. Deployed versions

Both demo hosts run felhom-agent 0.124.1, deployed from the published artifact (downloaded from Gitea; deployed sha matches the published sha on both). systemctl is-active → active on both. NOT VOUCHED — that stays the operator's act.

10. Registers

• R-191 → CLOSED (installer 1.25.0 + both boxes), carrying the Part 0 evidence and the broken-instrument warning, because the latter is the more reusable lesson.
• R-190 → MITIGATION SHIPPED, MECHANISM STILL OPEN — stated in the status field itself so the row cannot be misread as solved.
• R-194 → filed (grep established it free): PVE caches permissions — ~40 s in one run and ~16 minutes in another between deleting both ACL rows and the permission read reflecting it. It puts a floor under detection speed, makes a single permission read a lagging indicator, and is a candidate explanation for when R-190's refusal surfaced (not for what removed the grant).
• 07-backup-architecture.md: the offsite retention row was already correct — the configuration disagreed with it. Annotated, because a doc that states a contract does not enforce it; the gate does.
• ROADMAP.md holds none of these rows.

11. Teardown

The hand-removed grant was restored by the agent, which is the test, and confirmed present (2 ACL rows). Nothing else was provisioned: no scratch storage, no fixture, no prune job, no probe tag. Nothing was deleted from ep0 or from any tier.

12. Observations — noticed, NOT acted on

• PVE's permission cache is the reason detection is not instant (R-194). Notably, the storage content listing reflected the loss immediately ({"data":[]}) while the permission read still said present — two signals disagreeing, and today only one is read. Worth deciding whether the probe should consult both.
• logCapabilities runs probeAll independently of the report path. The latch makes that safe for this capability, but any future probe with one-shot semantics will hit the same trap. The general fix would be one probe per report cycle, shared — deliberately not attempted here.
• demo-hp's offsite tier is still escrow_state: escrowed, snapshot_count: 0 after this morning's re-issue; whether the pre-rebuild 15 snapshots reattach is answered by its next offbox run (04:15), and belongs to R-193 rather than to this session.
• The felhom-pbs tier is armed on both boxes with cadence=168h, so the R-191 fix cannot be observed end-to-end until that cadence fires. Nothing was triggered manually to shortcut it.