REPORT: R-191 closed, R-190 mitigation shipped, and the three broken instruments
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Part 0's gate PASSED — ep0 prunes both namespaces daily since 2026-07-27 (18 tasks, all OK) — but three of my own queries said the opposite and all three were broken instruments. Acting on them would have disabled the only pruning attempt while filing a finding that nothing prunes. Also records that v0.124.0's transition record failed in production with a green test suite, that two red-proofs did not fail on the first attempt (one could not compile, one asserted a helper rather than the path), and that two hollow tests were caught in one file.
This commit is contained in:
@@ -1,173 +1,253 @@
|
|||||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
# REPORT — R-191 (a backup that worked must not report failure) + R-190 (a permission must repair itself)
|
||||||
|
|
||||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
**Date:** 2026-08-04 · **Repos:** `felhom-agent` **v0.123.0 → v0.124.0 → v0.124.1** (`257c4d8`, and
|
||||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
the v0.124.1 fix) · `felhom.eu` installer **1.24.0 → 1.25.0** (tag `installer-v1.25.0`) ·
|
||||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
**no hub change and no hub bump**.
|
||||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
|
||||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
|
||||||
|
|
||||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Preconditions
|
## 1. Baselines
|
||||||
|
|
||||||
| # | demo-felhom | demo-hp |
|
| Repo | cited in §1 | found on arrival | matched? |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
| `felhom-agent` | `72161f6cf010` / `v0.123.0` | same | **yes** |
|
||||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
| `felhom.eu` | `d692c46db08f` / installer `1.24.0` | **`33f0ab2f33f7`** / installer `1.24.0` | **version yes, commit NO** |
|
||||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
|
||||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
|
||||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
|
||||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
|
||||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
|
||||||
|
|
||||||
## 2. The due verdicts before the run, quoted
|
`felhom.eu` had moved four commits forward since the brief was written — this morning's alert
|
||||||
|
diagnosis (R-190/191/192/193 rows, the demo-hp offsite re-issue), all pushed. Recorded rather than
|
||||||
|
glossed. Both `--ref=installer-v1.24.0` lines confirmed at 327 and 372.
|
||||||
|
|
||||||
|
## 2. Part 0 — the gate, and the three broken instruments that nearly failed it
|
||||||
|
|
||||||
|
**VERDICT: PASS.** ep0 prunes **both** namespaces on schedule.
|
||||||
|
|
||||||
```
|
```
|
||||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
prune-demo-felhom store=felhom-offsite ns=demo-felhom schedule=03:30 keep-last 2
|
||||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
prune-demo-hp store=felhom-offsite ns=demo-hp schedule=03:30 keep-last 2
|
||||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
|
||||||
reason: … has not been proven (last proven archive was a different one)
|
|
||||||
|
|
||||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
|
||||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
**18 prunejob tasks, all `status=OK`, every day from 2026-07-27 to 2026-08-04**, two per day, one per
|
||||||
|
namespace. The newest task log, read in full:
|
||||||
## 3. The runs — all four SCHEDULED, none triggered
|
|
||||||
|
|
||||||
| box | tier | due at | archive | result |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
|
||||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
|
||||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
|
||||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
|
||||||
|
|
||||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
|
||||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
|
||||||
which start nothing.
|
|
||||||
|
|
||||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
|
||||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
|
||||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
|
||||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
|
||||||
|
|
||||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
|
||||||
tier an ordinary recovery uses is also the cheapest to prove.
|
|
||||||
|
|
||||||
## 4. What the runs left behind
|
|
||||||
|
|
||||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
|
||||||
|
|
||||||
```json
|
|
||||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
|
||||||
proven_at 2026-08-03T22:56:45Z}
|
|
||||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
|
||||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
|
||||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
|
||||||
```
|
|
||||||
|
|
||||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
|
||||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
|
||||||
persisted state, because the in-memory store held only that morning's offsite run:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
prune job 'prune-demo-felhom' / task triggered by schedule '03:30'
|
||||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
retention options: --ns demo-felhom --max-depth 0 --keep-last 2
|
||||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
Pruning group demo-felhom:"ct/9201"
|
||||||
|
keep ct/9201/2026-07-27T19:55:41Z
|
||||||
|
keep ct/9201/2026-07-28T04:49:43Z
|
||||||
|
TASK OK
|
||||||
```
|
```
|
||||||
|
|
||||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
Read-only throughout: no job created, edited or run; nothing deleted. ep0 is Tier 2.
|
||||||
|
|
||||||
|
### The part that matters more than the verdict
|
||||||
|
|
||||||
|
**Three of my own queries said the endpoint prunes NOTHING, and all three were broken instruments:**
|
||||||
|
|
||||||
|
| query | why it was wrong |
|
||||||
|
|---|---|
|
||||||
|
| `'prune' in t['worker-type']` → *0 prune tasks* | the field is **`worker_type`**; `worker-type` does not exist, so every task read `None` |
|
||||||
|
| `worker_type == 'prune'` → *0* | the worker type is **`prunejob`**, not `prune` |
|
||||||
|
| `journalctl -u proxmox-backup \| grep -c prune` → *0* | the unit is **`proxmox-backup-proxy`** |
|
||||||
|
|
||||||
|
A fourth reading was mine to mis-frame: *"3 snapshots under keep-last 2 proves nothing is pruning"* —
|
||||||
|
the third snapshot landed at 04:49, **after** that day's 03:30 window, so it proved nothing. I caught
|
||||||
|
that one myself; the other three were caught only by checking the field names when the answer looked
|
||||||
|
too categorical.
|
||||||
|
|
||||||
|
**Had I acted on them, I would have disabled the only pruning ATTEMPT while filing a finding that
|
||||||
|
nothing prunes anywhere** — a weekly false alarm traded for unbounded growth on the protected
|
||||||
|
endpoint, invisible for months. The gate did its job precisely because it demanded evidence rather
|
||||||
|
than a verdict.
|
||||||
|
|
||||||
|
## 3. R-191 — the off-site tier stops asking to prune
|
||||||
|
|
||||||
|
**The defect:** `keep_last: 2` on the offsite tier, so vzdump uploaded and then failed the whole job
|
||||||
|
on `missing Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` →
|
||||||
|
`whole_guest_backup_failed`. R-89 moved offsite pruning server-side; the 2026-07-26 "two weeks" ruling
|
||||||
|
was not reversed — **where it is enforced moved and this value did not follow.**
|
||||||
|
|
||||||
|
- **Installer 1.25.0**: `keep_last: 0`, and the justifying paragraph rewritten to say where retention
|
||||||
|
now lives, citing R-89 and the Part 0 evidence, plus the warning that if ep0 ever stops pruning this
|
||||||
|
zero is unbounded growth. The agent's guard `allowPBSPrune = !primary && keep_last > 0` already
|
||||||
|
reads 0 as *never prune* — **no agent change**.
|
||||||
|
- **A gate asserts it** (`hostinstall_gates.py`), red-proved: pinning `keep_last: 2` back →
|
||||||
|
`FAIL: the offsite backup_target arms a CLIENT-SIDE prune (keep_last=2)`.
|
||||||
|
- **Both live boxes**, quoted:
|
||||||
|
|
||||||
```
|
```
|
||||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
demo-felhom backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
|
||||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
demo-felhom backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false
|
||||||
|
felhom-host backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
|
||||||
|
felhom-host backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false primary=false
|
||||||
```
|
```
|
||||||
|
|
||||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
The local tier is untouched (Scenario C).
|
||||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
|
||||||
|
|
||||||
## 5. Teardown — three layers, per box
|
**Not yet observed:** a weekly offsite run completing OK end to end. The failing step is removed and
|
||||||
|
the boxes are armed correctly; the *schedule* proving it is next week's event. The row says so rather
|
||||||
|
than implying the run has happened.
|
||||||
|
|
||||||
| layer | demo-felhom | demo-hp |
|
## 4. R-190 — the grant repairs itself, and the repair is visible
|
||||||
|---|---|---|
|
|
||||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
|
||||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
|
||||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
|
||||||
|
|
||||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
**Mitigation.** On a missing grant the agent runs the existing root wrapper
|
||||||
|
`felhom-backup-target-apply grant <id>` and **re-reads once** — the pbsdr R-22 shape and its
|
||||||
|
restraint. **No new privileged surface:** `grant *` was already in the sudoers allowlist for any
|
||||||
|
storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants
|
||||||
|
both user and token. It existed, was permitted, and had only ever been called at storage **creation**
|
||||||
|
— the *built but never wired* shape in a verb rather than a seam. Bounded: one attempt per tier per
|
||||||
|
hour.
|
||||||
|
|
||||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
### The record — and how v0.124.0 got it wrong in production with a green test suite
|
||||||
|
|
||||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
§8.5 asked whether the hub's existing degraded↔ok edge sufficed **before building anything new**. It
|
||||||
|
does, as a *channel* — but only if the agent deliberately reports one degraded cycle, because a silent
|
||||||
|
repair produces no edge at all. So **nothing new was built**: no wire change, no hub change, no new
|
||||||
|
event type. The explanation rides `Feature`, because that is the field the hub interpolates into the
|
||||||
|
operator's e-mail (`monitor/host_capability.go` builds its message from names and features; `Reason`
|
||||||
|
does not travel).
|
||||||
|
|
||||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
**v0.124.0 implemented "one cycle" as one probe CALL, and that failed live:**
|
||||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
|
||||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
|
||||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
|
||||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
|
||||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
|
||||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
|
||||||
|
|
||||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
```
|
||||||
|
09:39:34 agent: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED … confirmed_by=re-read
|
||||||
|
09:39:34 agent: capabilities self-check ok=69 total=70 degraded=1
|
||||||
|
09:39:37 hub: host-report stored → pve:store-grant:felhom-backup = ok ← the record never arrived
|
||||||
|
```
|
||||||
|
|
||||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
`probeAll` is invoked **independently** by the self-check log and by the collector building a report.
|
||||||
incrementally) and the job then failed on the prune —
|
The repairing call was the log's; the report three seconds later found the grant present and sent
|
||||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
`ok`. The agent's journal had the record, the hub had nothing, and the operator would have learned
|
||||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
nothing — the exact silence R-190 exists for, re-created inside its own mitigation, **while every unit
|
||||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
test passed**. **v0.124.1** replaces it with a latch on **time** (20 min > the 900 s report interval).
|
||||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
|
||||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
|
||||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
|
||||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
|
||||||
namespaces.
|
|
||||||
|
|
||||||
### Two corrections to yesterday's record
|
### Live proof, twice, on demo-felhom
|
||||||
|
|
||||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
Grant deleted by hand (both rows — user and token), which reproduces R-190 and the R-185 symptom
|
||||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
(`{"data":[]}` through the token):
|
||||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
|
||||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
|
||||||
runbook and on the row.
|
|
||||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
|
||||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
|
||||||
since the R-86 work — and one **more** since, below.
|
|
||||||
|
|
||||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
```
|
||||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
10:00:29 agent: store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)
|
||||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
target=felhom-backup privilege=Datastore.AllocateSpace
|
||||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read
|
||||||
|
10:00:30 hub: host-report → pve:store-grant:felhom-backup = DEGRADED
|
||||||
|
feature: "…the agent's storage grant was MISSING and has been AUTOMATICALLY RESTORED…"
|
||||||
|
10:00:40 hub: Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)
|
||||||
|
10:00:40 hub: Operator email sent for demo-felhom/agent_capability_degraded
|
||||||
|
pveum acl list | grep -c felhom-backup → 2 (both rows restored)
|
||||||
|
```
|
||||||
|
|
||||||
## 7. The capability map
|
Scenario E is unchanged from v0.123.0 — a failed repair, or one that does not survive the re-read,
|
||||||
|
leaves the capability degraded with the failure in its reason.
|
||||||
|
|
||||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
## 5. Part 3 — one bounded pass at the mechanism, negatives named
|
||||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
|
||||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
|
||||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
|
||||||
|
|
||||||
## 8. Registers
|
**The lead is real as a CLASS and is documented in our own installer:** *"`pveum user token remove`
|
||||||
|
purges the token's ACL, so re-applying post-rotate is mandatory"* — a rotation would take every grant
|
||||||
|
and re-apply only the hardcoded `PVE_STORAGES` set, which is exactly the observed end state.
|
||||||
|
|
||||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
**It does not fit this box.** A rotation purges **all** the token's ACLs and mints a **new secret**;
|
||||||
correction in §6.
|
demo-felhom's token still authenticates with the same secret (`--selftest` OK), it kept its other
|
||||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced
|
||||||
the highest in use).
|
(no 2026-08-03 install log; host uptime 12 days at the time).
|
||||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
|
||||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
|
||||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
|
||||||
|
|
||||||
## 9. Observations — noticed, NOT acted on
|
**Ruled out, by name:** a host reinstall; any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00;
|
||||||
|
any cluster-log ACL entry; a token rotation **on this box**. **Not ruled out fleet-wide** — any
|
||||||
|
installer run still purges and re-grants only `PVE_STORAGES`, though installer 1.24.0's reuse-arm fix
|
||||||
|
now re-grants the backup target on that path.
|
||||||
|
|
||||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
**The mechanism remains open**, and the row says which half it reflects: mitigation shipped, cause
|
||||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
unexplained. No reproduction was attempted and the session was not extended for it.
|
||||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
|
||||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
## 6. Tests and red-proofs
|
||||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
|
||||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
Green gate both repos: `go build ./... && go vet ./... && go test ./...` rc=0; `agent_gates.py` and
|
||||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
`repo_gates.py` all OK. Test runs and commits always separate commands.
|
||||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
|
||||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
| # | Test | Mutation | Observed |
|
||||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
|---|---|---|---|
|
||||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
| G | `hostinstall_gates.py` offsite assertion | pin `keep_last: 2` back | **FAIL** — `arms a CLIENT-SIDE prune (keep_last=2)` |
|
||||||
the hub's database, not on the waiters.
|
| F | `TestGrantRepair_IsBounded` | drop the interval check | **FAIL** — `5 cycles produced 5 attempt(s)` |
|
||||||
|
| D(record) | `TestGrantRepair_ARepairedGrantIsReportedAsATransition` | report `ok` after a repair | **FAIL** — `got "ok" — the loss would be invisible` |
|
||||||
|
| D(latch) | `TestGrantRepair_TransitionSurvivesALaterProbe` | ignore the latch in the healthy path | **FAIL** — `got "ok" — the host-report would carry ok` |
|
||||||
|
| H | `TestMainWiresTheGrantRepair` | pass `nil` instead of the repairer | **FAIL** — `not passed the repairer` |
|
||||||
|
| — | `TestGrantRepair_CallsTheExistingWrapperVerb` | — | pass (argv is exactly `grant <id>`; anything else is a policy change) |
|
||||||
|
| — | `TestGrantRepair_FailureIsReturned`, `_NilIsSafe` | — | pass |
|
||||||
|
|
||||||
|
**Two red-proofs did not fail on the first attempt, and both are reported rather than quietly fixed:**
|
||||||
|
|
||||||
|
1. **The seam mutation (`grantRepairer` → `nil`) failed to COMPILE** (`declared and not used`), so the
|
||||||
|
test never ran. The guard holds — you cannot ship it — but it was not demonstrated. The mutation
|
||||||
|
that does fail it keeps the variable used (`_ = grantRepairer`) and passes `nil`.
|
||||||
|
2. **The latch red-proof passed**, because the test asserted the latch *helper* rather than the path
|
||||||
|
consuming it. Fixed by extracting `storeGrantHealthyVerdict` and pointing the test at it — after
|
||||||
|
which the mutation fails as it should. **This is the second hollow test caught in this file**; the
|
||||||
|
first asserted a `Status` it had constructed itself. Both are now production functions under test.
|
||||||
|
|
||||||
|
Timestamps in the repair tests carry jitter (`09:39:34`, `+13m07s`, `+27m51s`, …) rather than round
|
||||||
|
intervals, per §10.
|
||||||
|
|
||||||
|
## 7. Files, commits, tag
|
||||||
|
|
||||||
|
`felhom-agent`: `cmd/felhom-agent/main.go` (repairer, latch, `storeGrantHealthyVerdict`,
|
||||||
|
`storeGrantRepairedVerdict`, wiring), `cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`.
|
||||||
|
Commits `257c4d8` (v0.124.0) and the v0.124.1 fix. Released shas: **`5e417938…`** (0.124.0),
|
||||||
|
**`5c279bda…`** (0.124.1), both round-trip verified.
|
||||||
|
|
||||||
|
`felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`,
|
||||||
|
`manifests/webpage.yaml`, `documentation/backlog/OPEN-ITEMS.md`,
|
||||||
|
`documentation/architecture/07-backup-architecture.md`, `CONTEXT.md`, `STATUS.md`, `REPORT.md`.
|
||||||
|
**Installer tag: `installer-v1.25.0`.**
|
||||||
|
|
||||||
|
## 8. The served installer
|
||||||
|
|
||||||
|
```
|
||||||
|
$ curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 SCRIPT_VERSION
|
||||||
|
SCRIPT_VERSION="1.25.0"
|
||||||
|
$ … | grep -o '"keep_last":[0-9]*'
|
||||||
|
"keep_last":0
|
||||||
|
```
|
||||||
|
|
||||||
|
Both `--ref=` lines bumped to `installer-v1.25.0` (327, 372); ArgoCD Synced/Healthy; webpage rollout
|
||||||
|
complete. The served bytes carry the change, not just the version.
|
||||||
|
|
||||||
|
## 9. Deployed versions
|
||||||
|
|
||||||
|
Both demo hosts run **`felhom-agent 0.124.1`**, deployed from the **published** artifact (downloaded
|
||||||
|
from Gitea; deployed sha matches the published sha on both). `systemctl is-active` → active on both.
|
||||||
|
**NOT VOUCHED** — that stays the operator's act.
|
||||||
|
|
||||||
|
## 10. Registers
|
||||||
|
|
||||||
|
- **R-191 → CLOSED** (installer 1.25.0 + both boxes), carrying the Part 0 evidence *and* the
|
||||||
|
broken-instrument warning, because the latter is the more reusable lesson.
|
||||||
|
- **R-190 → MITIGATION SHIPPED, MECHANISM STILL OPEN** — stated in the status field itself so the row
|
||||||
|
cannot be misread as solved.
|
||||||
|
- **R-194 → filed** (grep established it free): PVE caches permissions — ~40 s in one run and
|
||||||
|
~16 minutes in another between deleting both ACL rows and the permission read reflecting it. It puts
|
||||||
|
a floor under detection speed, makes a single permission read a lagging indicator, and is a
|
||||||
|
candidate explanation for *when* R-190's refusal surfaced (not for what removed the grant).
|
||||||
|
- `07-backup-architecture.md`: the offsite retention row was **already correct** — the configuration
|
||||||
|
disagreed with it. Annotated, because a doc that states a contract does not enforce it; the gate does.
|
||||||
|
- `ROADMAP.md` holds none of these rows.
|
||||||
|
|
||||||
|
## 11. Teardown
|
||||||
|
|
||||||
|
The hand-removed grant was restored **by the agent**, which is the test, and confirmed present (2 ACL
|
||||||
|
rows). Nothing else was provisioned: no scratch storage, no fixture, no prune job, no probe tag.
|
||||||
|
Nothing was deleted from ep0 or from any tier.
|
||||||
|
|
||||||
|
## 12. Observations — noticed, NOT acted on
|
||||||
|
|
||||||
|
- **PVE's permission cache is the reason detection is not instant** (R-194). Notably, the storage
|
||||||
|
*content* listing reflected the loss immediately (`{"data":[]}`) while the *permission* read still
|
||||||
|
said present — two signals disagreeing, and today only one is read. Worth deciding whether the probe
|
||||||
|
should consult both.
|
||||||
|
- **`logCapabilities` runs `probeAll` independently of the report path.** The latch makes that safe
|
||||||
|
for this capability, but any future probe with one-shot semantics will hit the same trap. The
|
||||||
|
general fix would be one probe per report cycle, shared — deliberately not attempted here.
|
||||||
|
- **demo-hp's offsite tier is still `escrow_state: escrowed, snapshot_count: 0`** after this morning's
|
||||||
|
re-issue; whether the pre-rebuild 15 snapshots reattach is answered by its next offbox run (04:15),
|
||||||
|
and belongs to R-193 rather than to this session.
|
||||||
|
- **The `felhom-pbs` tier is armed on both boxes with `cadence=168h`**, so the R-191 fix cannot be
|
||||||
|
observed end-to-end until that cadence fires. Nothing was triggered manually to shortcut it.
|
||||||
|
|||||||
Reference in New Issue
Block a user