hub v0.157.0: app_definition_refused (R-971), the vault time moves on a re-vault (R-954), the hub's own snapshot count (R-895 slice); uninstall knows felhom-thin-check
gates / gates (push) Successful in 6m58s

- hub: `app_definition_refused` in allowedEventTypes + operatorOnlyEvents (controller v0.328.0's catalog
  allow-list refusal); SaveHostRecoveryCredential moves set_at on a re-vault (found live: both rotated hosts
  kept their first vault's time); the daily key audit also lists the box's restic snapshots/ read-only over the
  login the hub already holds and logs count/newest/oldest (no alarm yet).
- installer (main only, rides the next tag): --uninstall removes /usr/local/sbin/felhom-thin-check (agent
  v0.159.0's bundle file); test_hostinstall.py knows it.
- runbook whole-guest-backup-orphan-thin.md: the human repair the agent's new orphan finding points to.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-17 16:27:51 +02:00
parent ab2fc21a9c
commit 8f97b8009b
12 changed files with 276 additions and 1 deletions
@@ -0,0 +1,42 @@
# RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012)
**When:** the operator mail „whole-guest backup failed" ends with „probable cause: thin pool `<vg>/<pool>` holds device
`<N>` that LVM does not know". The agent found it (agent ≥ v0.159.0); **it removed nothing.** The repair below is a
human step on purpose: it deletes a thin device, and only a person decides that.
**Why it happens:** the host's root filled up while vzdump took its storage snapshot; the kernel made the thin
snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next
snapshot asks for the same device number and fails („Creation of new snapshot <N> of device <M> failed"). First seen
on demo-hp 2026-10-14 (`audits/power-cut-fixes-2026-10-14/E/`). Agent v0.156.1's leftover cleanup keeps the root from
filling; this is the repair when it happened anyway.
## 1. Confirm (read-only)
On the PVE host, as root:
```bash
felhom-thin-check pve data # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok"
lvs -o lv_name,thin_id,pool_lv pve # LVM's view: every thin LV with its id
```
The orphan is a `pool <N>` line with no `lvm-before <N>` and no `lvm-after <N>`. If the helper refuses with
„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again.
**Stop here if the id belongs to an LV, or if a backup or restore is running on the host** (`pct list` shows a lock,
or `ps -ef | grep vzdump`). Write down the id and the LVM list in the evidence folder first (R-320).
## 2. Repair (deletes ONE thin device — the operator decides)
```bash
dmsetup message pve-data-tpool 0 "delete <N>"
felhom-thin-check pve data # now the pool's ids equal LVM's
```
(`pve-data-tpool` is `<vg>-<pool>-tpool`, with every `-` in a name doubled.) Then start one whole-guest backup of the
guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp
2026-10-14 12:02 UTC (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`).
## 3. Afterwards
- The register row for the box, with the id and the evidence path.
- Why the root filled: `df -h /`, `/var/lib/vz/dump` — the cleanup (R-1006) should have prevented it.