felhom.eu/REPORT.md

REPORT — RUNBOOK: the first host-tier restore-test, on both boxes

Date: 2026-08-03 → 2026-08-04 · Repos: felhom.eu docs + registers only. Nothing was built and no version was bumped. The only binary that moved is the already-published v0.123.0, onto the box that did not have it (P1). Baselines re-read and matched: felhom-agent 72161f6cf010 / v0.123.0; felhom.eu e3187c86d58d / hub v0.91.1, installer 1.24.0. Constants re-confirmed at source: defaultRestoreTestEvalInterval 6 h, defaultRestoreTestSettle 24 h.

Outcome: four scheduled runs, all passed, nothing triggered by hand.

---

1. Preconditions

#	demo-felhom	demo-hp
P1 agent	v0.123.0 ✓	0.120.0 — below the 0.121.0 floor, so it could not become due at all. Remediated with the published v0.123.0 (sha 74910135…, deployed sha identical)
P2 tiers	host felhom-backup + offsite felhom-pbs (weekly)	the same — the runbook expected demo-hp to have no offsite tier, and it has one, active with 2 snapshots
P3 grant	ok=70 total=70 degraded=0	ok=70 total=70 degraded=0 (once the probe existed)
P4 storage	/dev/sdb → /mnt/hdd_1, ext4, on the N100	/mnt/nvme-1tb on the t740 — different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe
P5 space	target 889 GB free; restore pool local-lvm 358 GB	target 925 GB free; restore pool data 53.9 G at 30.79 % (~37 GB free) against a 2.35 GB archive — adequate, and measured because that pool is the over-subscribed one
P6 candidate	…2026_08_02-04_42_14.tar.zst (08-02)	…2026_08_02-04_49_29.tar.zst (08-02) — both correctly the settled archive, not the day's
P7 other heavy work	daily backup ~04:44, outside the window	daily backup ~04:49, outside the window

2. The due verdicts before the run, quoted

demo-felhom  tier=felhom-backup due=true  archive="…2026_08_02-04_42_14.tar.zst"
               reason: newest settled archive … has not been proven; nothing proven on this tier yet
             tier=felhom-pbs    due=true  archive="…2026-07-28T04:49:43Z"
               reason: … has not been proven (last proven archive was a different one)

demo-hp      tier=felhom-backup due=true  archive="…2026_08_02-04_49_29.tar.zst"
             tier=felhom-pbs    due=true  archive="…2026-07-28T19:19:45Z"

Both boxes had BOTH tiers due, which made §4's ordering question live rather than theoretical.

3. The runs — all four SCHEDULED, none triggered

box	tier	due at	archive	result
demo-felhom	host	00:55:21	…2026_08_02-04_42_14.tar.zst	passed, 83.8 s, scratch torn down 00:56:45
demo-felhom	offsite	06:55:21	…2026-07-28T04:49:43Z	passed, 540.4 s, torn down 07:04:21
demo-hp	host	02:05:39	…2026_08_02-04_49_29.tar.zst	passed, 109.3 s, torn down 02:07:28
demo-hp	offsite	08:05:39	…2026-07-28T19:19:45Z	passed, 300.1 s, torn down 08:10:39

No box failed to fire, so Phase C was not entered and no --selftest was used as a proof. The only selftest invocations in this session were the read-only restore-test-due verdict prints in §2, which start nothing.

§4's question, answered live: each box took its host tier first — never-proven sorts ahead of proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the following evaluation six hours later. One heavy operation at a time, per box, with nobody sequencing it. That is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.

The asymmetry worth keeping: a host-tier restore is 83–109 s; an offsite one 300–540 s. The tier an ordinary recovery uses is also the cheapest to prove.

4. What the runs left behind

Persisted state — v3, naming the archive, the tier and what was verified:

demo-felhom  felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
                              proven_at 2026-08-03T22:56:45Z}
             felhom-pbs    → {archive …2026-07-28T04:49:43Z,        tier pbs,   proven_at 2026-08-04T05:04:21Z}
demo-hp      felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
             felhom-pbs    → {archive …2026-07-28T19:19:45Z,        tier pbs,   proven_at 2026-08-04T06:10:39Z}

The hub received the host-tier proofs — R-189's path carrying one for the first time. demo-felhom's latest report holds two entries, one per tier; the local one can only have come from the persisted state, because the in-memory store held only that morning's offsite run:

demo-felhom  tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
             tier=pbs   pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z          tested_at=2026-08-04T05:04:21Z
demo-hp      tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z

A subsequent evaluation runs nothing on a proven tier, quoted from demo-felhom now:

tier=felhom-pbs  due=false  proven="…2026-07-28T04:49:43Z"
  reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven

…while its host tier reads due=true again — on the 08-03 archive, which has now settled 24 h. That is not a defect: it is "proved daily, on its own archive", visible one day later.

5. Teardown — three layers, per box

layer	demo-felhom	demo-hp
the machine	pct list → 0 entries for 990000	0
the host	lvs → 0 volumes for 990000; local-lvm 1.95 % → 1.95 %	0; local-lvm 30.79 % → 40.86 % during the offsite run → 30.83 % after
the hub	the restore_tests[] entries are RETAINED DELIBERATELY — they are the proof the staleness check reads, so deleting them would delete the result	same

Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.

6. What the run surfaced — three findings, two of them corrections to my own record

R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning

A vzdump by felhom-agent@pve!agent with --storage felhom-backup completed OK at 04:44:50 on 2026-08-03; the first 403 … missing privilege Datastore.Allocate on that path is 09:24:56, and by ~14:50 pveum acl list held no row at all for it. Ruled out by measurement: a host reinstall (uptime 12 days), any pveum/ACL/user.cfg activity in syslog 04:00–10:00, any cluster-log ACL entry. Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before. A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and v0.123.0's probe detects the state but says nothing about the transition.

R-191 (new) — every weekly offsite backup reports FAILED although it worked

demo-felhom, 06:49–06:53 today: the upload succeeded (223 s, 629 MiB of 1.874 GiB, 67.2 % reused incrementally) and the job then failed on the prune — missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom → TASK ERROR: job errors, and the hub raised whole_guest_backup_failed. The token behaves exactly as R-89 designed (box tokens are write-only; ep0 prunes). What did not follow is the config: both boxes still arm the offsite tier with keep_last=2 prune_pbs_allowed=true, so every weekly run asks for a prune that must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary — an alarm whose text is true and whose trigger is not the thing you would act on. Not fixed here (§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two namespaces.

Two corrections to yesterday's record

1. The R-185 drift was NOT silent on the write path. demo-felhom's local-api backup jobs 403'd six times (09:24 → 17:34) on that storage and privilege, and the hub raised whole_guest_backup_failed at the first with edge-triggering suppressing the rest. My annotation said backups kept landing because writes go through a root path — wrong, and now corrected in the runbook and on the row.
2. My "no restore_test_* events at all" was an instrument error. The hub has no /events route; I grepped a 404 page. Read from the events table: five such events exist (2026-07-27/28), none since the R-86 work — and one more since, below.

One further event, correctly raised and worth stating: restore_test_stale for demo-felhom at 2026-08-03 22:33:42 UTC — 22 minutes before the host-tier run. It was true: at that moment the host tier had never been proven, and the hub said so on its own, without being asked. It has not re-fired; the signal is edge-triggered, so the return to healthy is silent by design.

7. The capability map

The unattended restore-proof row now reads PROVEN-LIVE for the host tier, unattended, on both demo boxes, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the teardown cited — and it states its scope explicitly: demo-felhom and demo-hp. The tester's box is untested and untouched, and one box proving something does not make it a fleet property.

8. Registers

• R-185 — its consequence is now demonstrated; the row already closed yesterday, and carries the correction in §6.
• R-190, R-191 — filed. grep established R-190 and R-191 were free before minting (R-189 was the highest in use).
• ROADMAP.md holds none of these rows, so nothing to collapse.
• STATUS.md rewritten for the operator and kept to one screen (85 lines); R-191 appears under "What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.

9. Observations — noticed, NOT acted on

• demo-hp's local-lvm thin pool reached 40.86 % during its offsite restore (from 30.79 %, returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes warn about, and the offsite archive is the larger of the two. A materially bigger guest would want the restore pointed at /mnt/nvme-1tb instead.
• demo-hp has an offsite tier, contrary to the runbook's §2 premise and to the note that it "has none". Nothing depends on that assumption now, but the operations notes still carry it.
• The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp) for the same guest role — a 2.7× difference worth understanding before either is used to size anything.
• Both waiters this session produced no output despite the runs completing; the evidence was gathered by direct query afterwards. A watcher that silently produces nothing is exactly the instrument class this project distrusts — the conclusions here rest on the boxes' own journals and the hub's database, not on the waiters.