admin
8f97b8009b
gates / gates (push) Successful in 6m58s
- hub: `app_definition_refused` in allowedEventTypes + operatorOnlyEvents (controller v0.328.0's catalog allow-list refusal); SaveHostRecoveryCredential moves set_at on a re-vault (found live: both rotated hosts kept their first vault's time); the daily key audit also lists the box's restic snapshots/ read-only over the login the hub already holds and logs count/newest/oldest (no alarm yet). - installer (main only, rides the next tag): --uninstall removes /usr/local/sbin/felhom-thin-check (agent v0.159.0's bundle file); test_hostinstall.py knows it. - runbook whole-guest-backup-orphan-thin.md: the human repair the agent's new orphan finding points to. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
43 lines
2.3 KiB
Markdown
43 lines
2.3 KiB
Markdown
# RUNBOOK — an orphan thin device stops a box's whole-guest backups (R-1012)
|
|
|
|
**When:** the operator mail „whole-guest backup failed" ends with „probable cause: thin pool `<vg>/<pool>` holds device
|
|
`<N>` that LVM does not know". The agent found it (agent ≥ v0.159.0); **it removed nothing.** The repair below is a
|
|
human step on purpose: it deletes a thin device, and only a person decides that.
|
|
|
|
**Why it happens:** the host's root filled up while vzdump took its storage snapshot; the kernel made the thin
|
|
snapshot, LVM could not write its own metadata, and the device stayed in the pool with no LV. From then on LVM's next
|
|
snapshot asks for the same device number and fails („Creation of new snapshot <N> of device <M> failed"). First seen
|
|
on demo-hp 2026-10-14 (`audits/power-cut-fixes-2026-10-14/E/`). Agent v0.156.1's leftover cleanup keeps the root from
|
|
filling; this is the repair when it happened anyway.
|
|
|
|
## 1. Confirm (read-only)
|
|
|
|
On the PVE host, as root:
|
|
|
|
```bash
|
|
felhom-thin-check pve data # the agent's own read-only helper; prints pool ids and LVM's, ends with "ok"
|
|
lvs -o lv_name,thin_id,pool_lv pve # LVM's view: every thin LV with its id
|
|
```
|
|
|
|
The orphan is a `pool <N>` line with no `lvm-before <N>` and no `lvm-after <N>`. If the helper refuses with
|
|
„a metadata snapshot … is already held", someone else is reading the pool; wait and run it again.
|
|
|
|
**Stop here if the id belongs to an LV, or if a backup or restore is running on the host** (`pct list` shows a lock,
|
|
or `ps -ef | grep vzdump`). Write down the id and the LVM list in the evidence folder first (R-320).
|
|
|
|
## 2. Repair (deletes ONE thin device — the operator decides)
|
|
|
|
```bash
|
|
dmsetup message pve-data-tpool 0 "delete <N>"
|
|
felhom-thin-check pve data # now the pool's ids equal LVM's
|
|
```
|
|
|
|
(`pve-data-tpool` is `<vg>-<pool>-tpool`, with every `-` in a name doubled.) Then start one whole-guest backup of the
|
|
guest from the controller („Mentés most") and read that it made its snapshot and finished. Done this way on demo-hp
|
|
2026-10-14 12:02 UTC (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`).
|
|
|
|
## 3. Afterwards
|
|
|
|
- The register row for the box, with the id and the evidence path.
|
|
- Why the root filled: `df -h /`, `/var/lib/vz/dump` — the cleanup (R-1006) should have prevented it.
|