R-190 filed; two corrections to yesterday's R-185 record
gates / gates (push) Successful in 8s

CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did
not surface as a 403 because writes go through a root path. That is WRONG.
demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST
on exactly that storage and privilege, and the hub raised whole_guest_backup_failed
at the first with edge-triggering suppressing the rest. The impact was not only an
unreadable tier: the agent's own whole-guest backups to it were failing.

CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the
agent's token to that storage completed OK at 04:44:50 the same morning; the first
403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any
pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster
log. Correlated but not established: guest 9201 was reprovisioned nine minutes
before the first failure.

R-190 files the unexplained disappearance, and notes that the new store-grant
probe detects the STATE but says nothing about the TRANSITION.
This commit is contained in:
2026-08-03 20:11:09 +02:00
parent e3187c86d5
commit b93ee06abc
2 changed files with 3 additions and 2 deletions
@@ -431,7 +431,7 @@ label. Filed under E-2.
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. It did not surface as a 403 on backup, which is what this item predicted: `vzdump` writes through a root path, so backups kept landing. It surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
| 8 | The honest **single-drive label**. |