felhom.eu/REPORT.md

# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes

**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.

**Outcome: four scheduled runs, all passed, nothing triggered by hand.**

---

## 1. Preconditions

| # | demo-felhom | demo-hp |
|---|---|---|
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |

## 2. The due verdicts before the run, quoted

```
demo-felhom  tier=felhom-backup due=true  archive="…2026_08_02-04_42_14.tar.zst"
               reason: newest settled archive … has not been proven; nothing proven on this tier yet
             tier=felhom-pbs    due=true  archive="…2026-07-28T04:49:43Z"
               reason: … has not been proven (last proven archive was a different one)

demo-hp      tier=felhom-backup due=true  archive="…2026_08_02-04_49_29.tar.zst"
             tier=felhom-pbs    due=true  archive="…2026-07-28T19:19:45Z"
```

**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.

## 3. The runs — all four SCHEDULED, none triggered

| box | tier | due at | archive | result |
|---|---|---|---|---|
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |

**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
which start nothing.

**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.

**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
tier an ordinary recovery uses is also the cheapest to prove.

## 4. What the runs left behind

**Persisted state — v3, naming the archive, the tier and what was verified:**

```json
demo-felhom  felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
                              proven_at 2026-08-03T22:56:45Z}
             felhom-pbs    → {archive …2026-07-28T04:49:43Z,        tier pbs,   proven_at 2026-08-04T05:04:21Z}
demo-hp      felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
             felhom-pbs    → {archive …2026-07-28T19:19:45Z,        tier pbs,   proven_at 2026-08-04T06:10:39Z}
```

**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
latest report holds **two** entries, one per tier; the `local` one can only have come from the
persisted state, because the in-memory store held only that morning's offsite run:

```
demo-felhom  tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
             tier=pbs   pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z          tested_at=2026-08-04T05:04:21Z
demo-hp      tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
```

**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:

```
tier=felhom-pbs  due=false  proven="…2026-07-28T04:49:43Z"
  reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
```

…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
That is not a defect: it is "proved daily, on its own archive", visible one day later.

## 5. Teardown — three layers, per box

| layer | demo-felhom | demo-hp |
|---|---|---|
| the machine | `pct list` → **0** entries for 990000 | **0** |
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |

Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.

## 6. What the run surfaced — three findings, two of them corrections to my own record

### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning

A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
and v0.123.0's probe detects the *state* but says nothing about the *transition*.

### R-191 (new) — every weekly offsite backup reports FAILED although it worked

demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
incrementally) and the job then failed on the prune —
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
namespaces.

### Two corrections to yesterday's record

1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
   **six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
   `whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
   said backups kept landing because writes go through a root path — wrong, and now corrected in the
   runbook and on the row.
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
   I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
   since the R-86 work — and one **more** since, below.

**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
host tier had never been proven, and the hub said so on its own, without being asked. It has not
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.

## 7. The capability map

The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
is untested and untouched, and one box proving something does not make it a fleet property.

## 8. Registers

- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
  correction in §6.
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
  the highest in use).
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
  "What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.

## 9. Observations — noticed, NOT acted on

- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
  returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
  warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
  the restore pointed at `/mnt/nvme-1tb` instead.
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
  none". Nothing depends on that assumption now, but the operations notes still carry it.
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
  role — a 2.7× difference worth understanding before either is used to size anything.
- **Both waiters this session produced no output** despite the runs completing; the evidence was
  gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
  instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
  the hub's database, not on the waiters.