admin
8f97b8009b
- hub: `app_definition_refused` in allowedEventTypes + operatorOnlyEvents (controller v0.328.0's catalog allow-list refusal); SaveHostRecoveryCredential moves set_at on a re-vault (found live: both rotated hosts kept their first vault's time); the daily key audit also lists the box's restic snapshots/ read-only over the login the hub already holds and logs count/newest/oldest (no alarm yet). - installer (main only, rides the next tag): --uninstall removes /usr/local/sbin/felhom-thin-check (agent v0.159.0's bundle file); test_hostinstall.py knows it. - runbook whole-guest-backup-orphan-thin.md: the human repair the agent's new orphan finding points to. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2.3 KiB
RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012)
When: the operator mail „whole-guest backup failed" ends with „probable cause: thin pool <vg>/<pool> holds device
<N> that LVM does not know". The agent found it (agent ≥ v0.159.0); it removed nothing. The repair below is a
human step on purpose: it deletes a thin device, and only a person decides that.
Why it happens: the host's root filled up while vzdump took its storage snapshot; the kernel made the thin
snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next
snapshot asks for the same device number and fails („Creation of new snapshot of device failed"). First seen
on demo-hp 2026-10-14 (audits/power-cut-fixes-2026-10-14/E/). Agent v0.156.1's leftover cleanup keeps the root from
filling; this is the repair when it happened anyway.
1. Confirm (read-only)
On the PVE host, as root:
felhom-thin-check pve data # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok"
lvs -o lv_name,thin_id,pool_lv pve # LVM's view: every thin LV with its id
The orphan is a pool <N> line with no lvm-before <N> and no lvm-after <N>. If the helper refuses with
„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again.
Stop here if the id belongs to an LV, or if a backup or restore is running on the host (pct list shows a lock,
or ps -ef | grep vzdump). Write down the id and the LVM list in the evidence folder first (R-320).
2. Repair (deletes ONE thin device — the operator decides)
dmsetup message pve-data-tpool 0 "delete <N>"
felhom-thin-check pve data # now the pool's ids equal LVM's
(pve-data-tpool is <vg>-<pool>-tpool, with every - in a name doubled.) Then start one whole-guest backup of the
guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp
2026-10-14 12:02 UTC (audits/power-cut-fixes-2026-10-14/E/repair.txt, E/next-backup.txt).
3. Afterwards
- The register row for the box, with the id and the evidence path.
- Why the root filled:
df -h /,/var/lib/vz/dump— the cleanup (R-1006) should have prevented it.