# REPORT — R-191 (a backup that worked must not report failure) + R-190 (a permission must repair itself) **Date:** 2026-08-04 · **Repos:** `felhom-agent` **v0.123.0 → v0.124.0 → v0.124.1** (`257c4d8`, and the v0.124.1 fix) · `felhom.eu` installer **1.24.0 → 1.25.0** (tag `installer-v1.25.0`) · **no hub change and no hub bump**. --- ## 1. Baselines | Repo | cited in §1 | found on arrival | matched? | |---|---|---|---| | `felhom-agent` | `72161f6cf010` / `v0.123.0` | same | **yes** | | `felhom.eu` | `d692c46db08f` / installer `1.24.0` | **`33f0ab2f33f7`** / installer `1.24.0` | **version yes, commit NO** | `felhom.eu` had moved four commits forward since the brief was written — this morning's alert diagnosis (R-190/191/192/193 rows, the demo-hp offsite re-issue), all pushed. Recorded rather than glossed. Both `--ref=installer-v1.24.0` lines confirmed at 327 and 372. ## 2. Part 0 — the gate, and the three broken instruments that nearly failed it **VERDICT: PASS.** ep0 prunes **both** namespaces on schedule. ``` prune-demo-felhom store=felhom-offsite ns=demo-felhom schedule=03:30 keep-last 2 prune-demo-hp store=felhom-offsite ns=demo-hp schedule=03:30 keep-last 2 ``` **18 prunejob tasks, all `status=OK`, every day from 2026-07-27 to 2026-08-04**, two per day, one per namespace. The newest task log, read in full: ``` prune job 'prune-demo-felhom' / task triggered by schedule '03:30' retention options: --ns demo-felhom --max-depth 0 --keep-last 2 Pruning group demo-felhom:"ct/9201" keep ct/9201/2026-07-27T19:55:41Z keep ct/9201/2026-07-28T04:49:43Z TASK OK ``` Read-only throughout: no job created, edited or run; nothing deleted. ep0 is Tier 2. ### The part that matters more than the verdict **Three of my own queries said the endpoint prunes NOTHING, and all three were broken instruments:** | query | why it was wrong | |---|---| | `'prune' in t['worker-type']` → *0 prune tasks* | the field is **`worker_type`**; `worker-type` does not exist, so every task read `None` | | `worker_type == 'prune'` → *0* | the worker type is **`prunejob`**, not `prune` | | `journalctl -u proxmox-backup \| grep -c prune` → *0* | the unit is **`proxmox-backup-proxy`** | A fourth reading was mine to mis-frame: *"3 snapshots under keep-last 2 proves nothing is pruning"* — the third snapshot landed at 04:49, **after** that day's 03:30 window, so it proved nothing. I caught that one myself; the other three were caught only by checking the field names when the answer looked too categorical. **Had I acted on them, I would have disabled the only pruning ATTEMPT while filing a finding that nothing prunes anywhere** — a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. The gate did its job precisely because it demanded evidence rather than a verdict. ## 3. R-191 — the off-site tier stops asking to prune **The defect:** `keep_last: 2` on the offsite tier, so vzdump uploaded and then failed the whole job on `missing Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed`. R-89 moved offsite pruning server-side; the 2026-07-26 "two weeks" ruling was not reversed — **where it is enforced moved and this value did not follow.** - **Installer 1.25.0**: `keep_last: 0`, and the justifying paragraph rewritten to say where retention now lives, citing R-89 and the Part 0 evidence, plus the warning that if ep0 ever stops pruning this zero is unbounded growth. The agent's guard `allowPBSPrune = !primary && keep_last > 0` already reads 0 as *never prune* — **no agent change**. - **A gate asserts it** (`hostinstall_gates.py`), red-proved: pinning `keep_last: 2` back → `FAIL: the offsite backup_target arms a CLIENT-SIDE prune (keep_last=2)`. - **Both live boxes**, quoted: ``` demo-felhom backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true demo-felhom backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false felhom-host backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true felhom-host backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false ``` The local tier is untouched (Scenario C). **Not yet observed:** a weekly offsite run completing OK end to end. The failing step is removed and the boxes are armed correctly; the *schedule* proving it is next week's event. The row says so rather than implying the run has happened. ## 4. R-190 — the grant repairs itself, and the repair is visible **Mitigation.** On a missing grant the agent runs the existing root wrapper `felhom-backup-target-apply grant ` and **re-reads once** — the pbsdr R-22 shape and its restraint. **No new privileged surface:** `grant *` was already in the sudoers allowlist for any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants both user and token. It existed, was permitted, and had only ever been called at storage **creation** — the *built but never wired* shape in a verb rather than a seam. Bounded: one attempt per tier per hour. ### The record — and how v0.124.0 got it wrong in production with a green test suite §8.5 asked whether the hub's existing degraded↔ok edge sufficed **before building anything new**. It does, as a *channel* — but only if the agent deliberately reports one degraded cycle, because a silent repair produces no edge at all. So **nothing new was built**: no wire change, no hub change, no new event type. The explanation rides `Feature`, because that is the field the hub interpolates into the operator's e-mail (`monitor/host_capability.go` builds its message from names and features; `Reason` does not travel). **v0.124.0 implemented "one cycle" as one probe CALL, and that failed live:** ``` 09:39:34 agent: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED … confirmed_by=re-read 09:39:34 agent: capabilities self-check ok=69 total=70 degraded=1 09:39:37 hub: host-report stored → pve:store-grant:felhom-backup = ok ← the record never arrived ``` `probeAll` is invoked **independently** by the self-check log and by the collector building a report. The repairing call was the log's; the report three seconds later found the grant present and sent `ok`. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence R-190 exists for, re-created inside its own mitigation, **while every unit test passed**. **v0.124.1** replaces it with a latch on **time** (20 min > the 900 s report interval). ### Live proof, twice, on demo-felhom Grant deleted by hand (both rows — user and token), which reproduces R-190 and the R-185 symptom (`{"data":[]}` through the token): ``` 10:00:29 agent: store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read 10:00:30 hub: host-report → pve:store-grant:felhom-backup = DEGRADED feature: "…the agent's storage grant was MISSING and has been AUTOMATICALLY RESTORED…" 10:00:40 hub: Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded) 10:00:40 hub: Operator email sent for demo-felhom/agent_capability_degraded pveum acl list | grep -c felhom-backup → 2 (both rows restored) ``` **And the pair CLOSED**, which is the other half of "exactly one alert per loss": ``` 10:20:29 the 20-minute latch expires 10:30:40 hub: Host capability: demo-felhom-8363b5 degraded → ok (agent_capability_recovered) ``` So one lost grant produced **one degraded event + one operator e-mail + one recovery**, and nothing further. Final state on both boxes: agent `0.124.1`, two ACL rows on `/storage/felhom-backup`. Scenario E is unchanged from v0.123.0 — a failed repair, or one that does not survive the re-read, leaves the capability degraded with the failure in its reason. ## 5. Part 3 — one bounded pass at the mechanism, negatives named **The lead is real as a CLASS and is documented in our own installer:** *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"* — a rotation would take every grant and re-apply only the hardcoded `PVE_STORAGES` set, which is exactly the observed end state. **It does not fit this box.** A rotation purges **all** the token's ACLs and mints a **new secret**; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it kept its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at the time). **Ruled out, by name:** a host reinstall; any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00; any cluster-log ACL entry; a token rotation **on this box**. **Not ruled out fleet-wide** — any installer run still purges and re-grants only `PVE_STORAGES`, though installer 1.24.0's reuse-arm fix now re-grants the backup target on that path. **The mechanism remains open**, and the row says which half it reflects: mitigation shipped, cause unexplained. No reproduction was attempted and the session was not extended for it. ## 6. Tests and red-proofs Green gate both repos: `go build ./... && go vet ./... && go test ./...` rc=0; `agent_gates.py` and `repo_gates.py` all OK. Test runs and commits always separate commands. | # | Test | Mutation | Observed | |---|---|---|---| | G | `hostinstall_gates.py` offsite assertion | pin `keep_last: 2` back | **FAIL** — `arms a CLIENT-SIDE prune (keep_last=2)` | | F | `TestGrantRepair_IsBounded` | drop the interval check | **FAIL** — `5 cycles produced 5 attempt(s)` | | D(record) | `TestGrantRepair_ARepairedGrantIsReportedAsATransition` | report `ok` after a repair | **FAIL** — `got "ok" — the loss would be invisible` | | D(latch) | `TestGrantRepair_TransitionSurvivesALaterProbe` | ignore the latch in the healthy path | **FAIL** — `got "ok" — the host-report would carry ok` | | H | `TestMainWiresTheGrantRepair` | pass `nil` instead of the repairer | **FAIL** — `not passed the repairer` | | — | `TestGrantRepair_CallsTheExistingWrapperVerb` | — | pass (argv is exactly `grant `; anything else is a policy change) | | — | `TestGrantRepair_FailureIsReturned`, `_NilIsSafe` | — | pass | **Two red-proofs did not fail on the first attempt, and both are reported rather than quietly fixed:** 1. **The seam mutation (`grantRepairer` → `nil`) failed to COMPILE** (`declared and not used`), so the test never ran. The guard holds — you cannot ship it — but it was not demonstrated. The mutation that does fail it keeps the variable used (`_ = grantRepairer`) and passes `nil`. 2. **The latch red-proof passed**, because the test asserted the latch *helper* rather than the path consuming it. Fixed by extracting `storeGrantHealthyVerdict` and pointing the test at it — after which the mutation fails as it should. **This is the second hollow test caught in this file**; the first asserted a `Status` it had constructed itself. Both are now production functions under test. Timestamps in the repair tests carry jitter (`09:39:34`, `+13m07s`, `+27m51s`, …) rather than round intervals, per §10. ## 7. Files, commits, tag `felhom-agent`: `cmd/felhom-agent/main.go` (repairer, latch, `storeGrantHealthyVerdict`, `storeGrantRepairedVerdict`, wiring), `cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`. Commits `257c4d8` (v0.124.0) and the v0.124.1 fix. Released shas: **`5e417938…`** (0.124.0), **`5c279bda…`** (0.124.1), both round-trip verified. `felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`, `manifests/webpage.yaml`, `documentation/backlog/OPEN-ITEMS.md`, `documentation/architecture/07-backup-architecture.md`, `CONTEXT.md`, `STATUS.md`, `REPORT.md`. **Installer tag: `installer-v1.25.0`.** ## 8. The served installer ``` $ curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 SCRIPT_VERSION SCRIPT_VERSION="1.25.0" $ … | grep -o '"keep_last":[0-9]*' "keep_last":0 ``` Both `--ref=` lines bumped to `installer-v1.25.0` (327, 372); ArgoCD Synced/Healthy; webpage rollout complete. The served bytes carry the change, not just the version. ## 9. Deployed versions Both demo hosts run **`felhom-agent 0.124.1`**, deployed from the **published** artifact (downloaded from Gitea; deployed sha matches the published sha on both). `systemctl is-active` → active on both. **NOT VOUCHED** — that stays the operator's act. ## 10. Registers - **R-191 → CLOSED** (installer 1.25.0 + both boxes), carrying the Part 0 evidence *and* the broken-instrument warning, because the latter is the more reusable lesson. - **R-190 → MITIGATION SHIPPED, MECHANISM STILL OPEN** — stated in the status field itself so the row cannot be misread as solved. - **R-194 → filed** (grep established it free): PVE caches permissions — ~40 s in one run and ~16 minutes in another between deleting both ACL rows and the permission read reflecting it. It puts a floor under detection speed, makes a single permission read a lagging indicator, and is a candidate explanation for *when* R-190's refusal surfaced (not for what removed the grant). - `07-backup-architecture.md`: the offsite retention row was **already correct** — the configuration disagreed with it. Annotated, because a doc that states a contract does not enforce it; the gate does. - `ROADMAP.md` holds none of these rows. ## 11. Teardown The hand-removed grant was restored **by the agent**, which is the test, and confirmed present (2 ACL rows). Nothing else was provisioned: no scratch storage, no fixture, no prune job, no probe tag. Nothing was deleted from ep0 or from any tier. ## 12. Observations — noticed, NOT acted on - **PVE's permission cache is the reason detection is not instant** (R-194). Notably, the storage *content* listing reflected the loss immediately (`{"data":[]}`) while the *permission* read still said present — two signals disagreeing, and today only one is read. Worth deciding whether the probe should consult both. - **`logCapabilities` runs `probeAll` independently of the report path.** The latch makes that safe for this capability, but any future probe with one-shot semantics will hit the same trap. The general fix would be one probe per report cycle, shared — deliberately not attempted here. - **demo-hp's offsite tier is still `escrow_state: escrowed, snapshot_count: 0`** after this morning's re-issue; whether the pre-rebuild 15 snapshots reattach is answered by its next offbox run (04:15), and belongs to R-193 rather than to this session. - **The `felhom-pbs` tier is armed on both boxes with `cadence=168h`**, so the R-191 fix cannot be observed end-to-end until that cadence fires. Nothing was triggered manually to shortcut it.