diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 5594d3a..43ed23a 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -28,11 +28,12 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | | **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.121.0**, hub **v0.91.0**) | — | **The rule that shipped:** *let A be the newest archive on a tier that has settled ≥24 h; the tier is DUE when A exists and A has not already been proven.* Daily tier → proved daily on yesterday's archive; weekly tier → weekly on its own; newborn → UNKNOWN. The daemon-start ticker survives only as the **evaluation interval**. **THE TRAP, recorded because it is the version a reasonable person writes:** the row's own wording implemented literally — *"due when the newest archive is ≥24 h old"* — is NEVER true on a **daily** tier, because a new archive resets the newest-archive age to zero long before it reaches the lag; it would have silently switched restore-testing OFF for the tier that matters most. Red-proved at **0 runs over 5 simulated days**. **The state now records WHICH archive was proven**, not when a tier last passed — a time cannot answer *have we proven this archive*. A pre-R-86 state file keeps its time (rotation ordering survives) and yields no proven archive, so each tier is due exactly once after the upgrade: the safe direction. **Two knobs replace one and the old one is not silently repurposed:** `restore_test_eval_interval_seconds` (6 h) and `restore_test_settle_seconds` (24 h); the deprecated `restore_test_cadence_seconds` keeps its DISABLE meaning verbatim, now seeds the settle lag, and the daemon WARNs once at start-up naming both. **6 h is bounded from both ends, not picked:** MEASURED cost of one evaluation on demo-felhom — local dir storage **18 ms**, PBS tier over the WAN to ep0 **392 ms**, both **430 ms** — so cost is irrelevant; the CEILING is that a FAILING tier stays due, making the evaluation interval its retry interval for a multi-GB restore. **Part 2 shipped WITH it and was not optional** — see the hub half in this row's sibling text and `07-backup-architecture.md` §3: `restoreProvenStaleAfter` was a flat 7 days derived from the very cadence this removed, and a healthy weekly tier's proof age reaches **exactly** 168 h against a 168 h window — it sat ON the line, so any ordinary delay tipped it into a nightly alarm about a working system. The window is now per tier from that tier's observed archive interval, ×4 generations, floored at the old 7 days and capped at 12 days (strictly inside the two-week offsite retention), falling back to the tier's DECLARED rhythm (26 h host / 8 d offsite — the backup-freshness checker's own thresholds) when history is too short to observe one. **A hollow test caught by its own red-proof:** the first Scenario-G fixture had no jitter and PASSED under the flat-window mutation, because a perfectly regular weekly tier sits exactly ON the line rather than over it. The jitter is what makes it a test. **Also fixed in passing:** the candidate picker now skips archives failing `archivePlausiblyComplete` (under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof and leave the tier due at EVERY evaluation), and the due-check runs BEFORE the heavy-operation gate is taken (a frequent poll must not be able to make a starting backup record a failure — F-A1). **Live proof:** see `felhom-agent/REPORT.md` | CC | | **R-87** | The restic tier is never restore-tested | **READY — RE-RANKED UP 2026-08-03 (R-86 closed)** | — | Design a controller-side test (no scratch-guest analogue transfers). **Most of what this row needed now exists.** R-86 built the piece that was missing: a tier is proved **per archive generation**, on its own rhythm, with the proof recorded as *which archive* — which is exactly the shape a weekly-ish restic tier needs, and the reason this row could not simply reuse the whole-guest scheduler before. What remains is genuinely restic-specific and is NOT a scheduling problem: there is no scratch-guest analogue, so the test has to be a controller-side restore of a bounded sample into a throwaway path, with its own definition of "proved". **Two things to carry over rather than re-derive:** the proof must record the SNAPSHOT it proved (not a timestamp), and the hub's staleness window must learn this tier's rhythm the way `restoreProvenWindow` now does — a restic tier on a weekly cadence lands on the same false-alarm line the flat 7 days did. **And R-95 still applies:** that credential can delete, so a restic restore-test must never be able to write to the repo | CC | -| **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.123.0**, installer **1.24.0**) | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began **BOTH HALVES FIXED, AND THE ROOT CAUSE WAS NOT WHERE IT LOOKED.** **(1) The silence.** `Client.Permissions` asks `GET /access/permissions?path=/storage/` **as the agent's own token**, and one `capability.Status` per configured tier reports it — composed around the sudo prober the way the pool-read check already is, with the `Status` wire shape untouched so the hub's existing critical-degraded alert applies with **no hub change**. **MEASURED BEFORE IT WAS DESIGNED AGAINST, and the obvious reading is wrong:** an ungranted path answers neither empty nor 403 but with the privileges INHERITED from the box-wide `/` grant — `felhom-pbs` → `{Datastore.Allocate:1, Datastore.AllocateSpace:1}` vs `felhom-backup` → `{Sys.Audit:1, SDN.Use:1, Datastore.Audit:1}`. A probe asking *"is the path present?"* or *"does it have Datastore.Audit?"* would report the blinded storage HEALTHY; the probe tests `Datastore.AllocateSpace` and a red-proof pins that. The probed set comes from the box's own `BackupTiers()`, never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical (the hub alerts only on critical), **except** the `local` fallback target, which is reported but does not page: turning host-install's own documented DEGRADED configuration into an alert is how a signal becomes something an operator archives unread. It never looks at content, so it cannot alarm on a newborn tier by construction; it never reports ok when it could not ask. **(2) The installer.** The row and the task both assumed `PVE_STORAGES` was the culprit. It is not: `configure_backup_target`'s **Case A** arm creates the storage and grants in the same breath, so a box that builds its own target was always fine. The **Scenario-F** arm — *"the target already exists, leave it exactly as it is"* — returned **without granting**, which is why a box whose `felhom-backup` pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token could not read. That arm now ensures the ACL through the same guarded wrapper; the storage DEFINITION is still untouched, so Scenario F is unviolated, and `pveum acl modify` is idempotent. `$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES` — that list is granted in step 4/5, before the target is resolved in step 6, and `--acl-storages` entries are preflight-checked for existence. A **gate** now asserts every arm that resolves the target also grants on it (red-proved: *resolves in 2 place(s) but grants in only 1*). **LIVE, in order:** the three-way observation reproduced unchanged (root 3 archives / token `{"data":[]}` / `local` lists through the same token) → agent v0.123.0 deployed → **`capability DEGRADED … pve:store-grant:felhom-backup … lacks Datastore.AllocateSpace … INVISIBLE to the agent and it is never restore-tested`, critical=true**, and the hub raised `agent_capability_degraded` **and emailed the operator** → grant applied (user AND token) → the token lists **3 archives** → `ok=70 total=70 degraded=0` and the hub logged `degraded → ok (agent_capability_recovered)` → **the host tier is a due-check candidate for the first time on that box**, correctly picking the 08-02 archive because the 08-03 one has not settled 24 h. **demo-hp CARRIES THE SAME DRIFT AND WAS FIXED TOO** — `local_backup_target=felhom-backup`, no ACL row, now granted and listing 4 archives. §8.6 assumed one box was affected; the same one-line additive idempotent fix applies to the other, and leaving a known-blind backup tier on a Tier-0 box after finding it would be this row happening twice. The tester's box was NOT touched (Tier 2) — see the note in this row's outcome. | CC | +| **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.123.0**, installer **1.24.0**) | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began **BOTH HALVES FIXED, AND THE ROOT CAUSE WAS NOT WHERE IT LOOKED.** **(1) The silence.** `Client.Permissions` asks `GET /access/permissions?path=/storage/` **as the agent's own token**, and one `capability.Status` per configured tier reports it — composed around the sudo prober the way the pool-read check already is, with the `Status` wire shape untouched so the hub's existing critical-degraded alert applies with **no hub change**. **MEASURED BEFORE IT WAS DESIGNED AGAINST, and the obvious reading is wrong:** an ungranted path answers neither empty nor 403 but with the privileges INHERITED from the box-wide `/` grant — `felhom-pbs` → `{Datastore.Allocate:1, Datastore.AllocateSpace:1}` vs `felhom-backup` → `{Sys.Audit:1, SDN.Use:1, Datastore.Audit:1}`. A probe asking *"is the path present?"* or *"does it have Datastore.Audit?"* would report the blinded storage HEALTHY; the probe tests `Datastore.AllocateSpace` and a red-proof pins that. The probed set comes from the box's own `BackupTiers()`, never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical (the hub alerts only on critical), **except** the `local` fallback target, which is reported but does not page: turning host-install's own documented DEGRADED configuration into an alert is how a signal becomes something an operator archives unread. It never looks at content, so it cannot alarm on a newborn tier by construction; it never reports ok when it could not ask. **(2) The installer.** The row and the task both assumed `PVE_STORAGES` was the culprit. It is not: `configure_backup_target`'s **Case A** arm creates the storage and grants in the same breath, so a box that builds its own target was always fine. The **Scenario-F** arm — *"the target already exists, leave it exactly as it is"* — returned **without granting**, which is why a box whose `felhom-backup` pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token could not read. That arm now ensures the ACL through the same guarded wrapper; the storage DEFINITION is still untouched, so Scenario F is unviolated, and `pveum acl modify` is idempotent. `$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES` — that list is granted in step 4/5, before the target is resolved in step 6, and `--acl-storages` entries are preflight-checked for existence. A **gate** now asserts every arm that resolves the target also grants on it (red-proved: *resolves in 2 place(s) but grants in only 1*). **LIVE, in order:** the three-way observation reproduced unchanged (root 3 archives / token `{"data":[]}` / `local` lists through the same token) → agent v0.123.0 deployed → **`capability DEGRADED … pve:store-grant:felhom-backup … lacks Datastore.AllocateSpace … INVISIBLE to the agent and it is never restore-tested`, critical=true**, and the hub raised `agent_capability_degraded` **and emailed the operator** → grant applied (user AND token) → the token lists **3 archives** → `ok=70 total=70 degraded=0` and the hub logged `degraded → ok (agent_capability_recovered)` → **the host tier is a due-check candidate for the first time on that box**, correctly picking the 08-02 archive because the 08-03 one has not settled 24 h. **demo-hp CARRIES THE SAME DRIFT AND WAS FIXED TOO** — `local_backup_target=felhom-backup`, no ACL row, now granted and listing 4 archives. §8.6 assumed one box was affected; the same one-line additive idempotent fix applies to the other, and leaving a known-blind backup tier on a Tier-0 box after finding it would be this row happening twice. The tester's box was NOT touched (Tier 2) — see the note in this row's outcome. **CORRECTION + WIDER IMPACT, measured 2026-08-03 during the R-185 close-out run.** This row (and the runbook annotation written with it) said the drift was invisible because writes went through a root path. **That is wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST — `POST …/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)` — and the hub raised `whole_guest_backup_failed` at the first (*"retrying with backoff"*), edge-triggering suppressing the rest. So the impact was not only an unreadable tier: **the agent's own whole-guest backups to it were failing**, and the operator was told once. The failures stop at the grant. **AND THE GRANT WAS LOST, NOT MERELY NEVER ISSUED — on this box.** A vzdump by `felhom-agent@pve!agent` with `--storage felhom-backup` **succeeded at 04:44:50 CEST the same day**, and the first 403 is 09:24:56. Ruled out: a host reinstall (uptime 12 days), any logged `pveum`/ACL activity (none in syslog 04:00–10:00, none in the cluster log). Correlated but NOT established as causal: `host_leaf_changed` 09:15 and `controller_started` 09:19, i.e. guest 9201 was reprovisioned nine minutes before the first failure. **How the grant disappeared is unexplained → R-190.** | CC | | **R-186** | **A released agent binary's sha256 cannot be reproduced from its tag.** `release-agent.sh` builds at step 3 and tags at step 4, so Go's VCS stamp records a PSEUDO-version (`v0.120.1-0.20260803130452-4d825910…`) in the published bytes, while any rebuild after the tag exists stamps `v0.121.0` — a different binary. Measured 2026-08-03 on v0.121.0: published `b2128f3c…` (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source, identical toolchain, 4 096 bytes apart | **CLOSED — SHIPPED + MEASURED 2026-08-03** (agent **v0.122.0**) | — | **Why it matters:** the sha the operator vouches is the one thing tying a machine to a binary, and today nobody can independently rebuild it to check. **The build order is deliberate** (the script's own comment: a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it), so the fix is not to swap the steps blind. Candidates: `-buildvcs=false` or `-trimpath` for a version-stable stamp, or tag-then-build with the tag deleted on a failed publish. **Mitigation used this session:** the DEPLOYED binary is the PUBLISHED artifact, downloaded from Gitea — not a local rebuild — so the running bytes are the vouchable ones **FIXED by removing the cause, not by sequencing around it.** `-buildvcs=false` drops the VCS stamp (nothing in the repo reads it — no `ReadBuildInfo` caller) and `-trimpath` removes absolute build paths, so the same source produces the same bytes whether or not the tag exists yet and regardless of checkout directory. **Measured before the change**, same source, same toolchain, one commit: default flags with no tag `18f4a495…` 14 085 464 B (mod `v0.121.2-0.2026…`) vs default flags tagged `4a38f394…` 14 085 440 B (mod `v0.121.99`) — different bytes; with the new flags, `7ffcdf1d…` 14 064 574 B **identical both ways**. **Measured after the change on the real release:** published `d5f294e5…` 14 076 649 B = rebuild-at-tag #1 = rebuild-at-tag #2, **all three identical**. **A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and therefore produced a binary **74 KB smaller** (13 990 236 B vs 14 064 574 B) than the release path built for the same version — one version name, two binaries, decided by which entry point ran. Both paths now build identically, with a comment in each naming the other. **The operator can now check a published sha without being there:** the exact command is in `felhom-agent/CLAUDE.md`. | CC | | **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC | | **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC | | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC | +| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had | CC | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — | diff --git a/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md b/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md index 70f19c9..a87c41e 100644 --- a/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md +++ b/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md @@ -431,7 +431,7 @@ label. Filed under E-2. | 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). | | 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. | | 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. | -| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. It did not surface as a 403 on backup, which is what this item predicted: `vzdump` writes through a root path, so backups kept landing. It surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). | +| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). | | 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. | | 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. | | 8 | The honest **single-drive label**. |