hub v0.157.0: app_definition_refused (R-971), the vault time moves on a re-vault (R-954), the hub's own snapshot count (R-895 slice); uninstall knows felhom-thin-check
gates / gates (push) Successful in 6m58s
gates / gates (push) Successful in 6m58s
- hub: `app_definition_refused` in allowedEventTypes + operatorOnlyEvents (controller v0.328.0's catalog allow-list refusal); SaveHostRecoveryCredential moves set_at on a re-vault (found live: both rotated hosts kept their first vault's time); the daily key audit also lists the box's restic snapshots/ read-only over the login the hub already holds and logs count/newest/oldest (no alarm yet). - installer (main only, rides the next tag): --uninstall removes /usr/local/sbin/felhom-thin-check (agent v0.159.0's bundle file); test_hostinstall.py knows it. - runbook whole-guest-backup-orphan-thin.md: the human repair the agent's new orphan finding points to. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012)
|
||||
|
||||
**When:** the operator mail „whole-guest backup failed" ends with „probable cause: thin pool `<vg>/<pool>` holds device
|
||||
`<N>` that LVM does not know". The agent found it (agent ≥ v0.159.0); **it removed nothing.** The repair below is a
|
||||
human step on purpose: it deletes a thin device, and only a person decides that.
|
||||
|
||||
**Why it happens:** the host's root filled up while vzdump took its storage snapshot; the kernel made the thin
|
||||
snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next
|
||||
snapshot asks for the same device number and fails („Creation of new snapshot <N> of device <M> failed"). First seen
|
||||
on demo-hp 2026-10-14 (`audits/power-cut-fixes-2026-10-14/E/`). Agent v0.156.1's leftover cleanup keeps the root from
|
||||
filling; this is the repair when it happened anyway.
|
||||
|
||||
## 1. Confirm (read-only)
|
||||
|
||||
On the PVE host, as root:
|
||||
|
||||
```bash
|
||||
felhom-thin-check pve data # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok"
|
||||
lvs -o lv_name,thin_id,pool_lv pve # LVM's view: every thin LV with its id
|
||||
```
|
||||
|
||||
The orphan is a `pool <N>` line with no `lvm-before <N>` and no `lvm-after <N>`. If the helper refuses with
|
||||
„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again.
|
||||
|
||||
**Stop here if the id belongs to an LV, or if a backup or restore is running on the host** (`pct list` shows a lock,
|
||||
or `ps -ef | grep vzdump`). Write down the id and the LVM list in the evidence folder first (R-320).
|
||||
|
||||
## 2. Repair (deletes ONE thin device — the operator decides)
|
||||
|
||||
```bash
|
||||
dmsetup message pve-data-tpool 0 "delete <N>"
|
||||
felhom-thin-check pve data # now the pool's ids equal LVM's
|
||||
```
|
||||
|
||||
(`pve-data-tpool` is `<vg>-<pool>-tpool`, with every `-` in a name doubled.) Then start one whole-guest backup of the
|
||||
guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp
|
||||
2026-10-14 12:02 UTC (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`).
|
||||
|
||||
## 3. Afterwards
|
||||
|
||||
- The register row for the box, with the id and the evidence path.
|
||||
- Why the root filled: `df -h /`, `/var/lib/vz/dump` — the cleanup (R-1006) should have prevented it.
|
||||
Reference in New Issue
Block a user