Power-cut fixes E: agent 0.156.1 delivered to demo-hp, the pre-backup cleanup proven live; an orphan thin snapshot from C4 blocks demo-hp's backups (R-1012 filed); per-customer floors hold 0.309.0
gates / gates (push) Successful in 6m44s
gates / gates (push) Successful in 6m44s
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
Oct 14 13:42:33 demo-hp sudo[519542]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
Oct 14 13:42:40 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:40.495+02:00 level=WARN msg="vzdump-partial: removed a dead half-written backup" storage=local vmid=9201 name=vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat bytes=1161887744 why="written before this boot" helper="removed vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat 1161887744 B (written before this boot)"
|
||||
Oct 14 13:42:40 demo-hp sudo[526700]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
Oct 14 13:42:47 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:47.973+02:00 level=WARN msg="vzdump-partial: removed a dead half-written backup" storage=local vmid=9201 name=vzdump-lxc-9201-2026_10_14-10_42_25.tmp bytes=4096 why="written before this boot" helper="removed vzdump-lxc-9201-2026_10_14-10_42_25.tmp 571 B (written before this boot)"
|
||||
Oct 14 13:42:48 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:48.506+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=5648463064 need_bytes=8134320654 avail_bytes=14566670336
|
||||
Oct 14 13:42:50 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:50.583+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1791978152980804597 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:00082531:00104E7D:6ACF6AB8:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
|
||||
@@ -0,0 +1,7 @@
|
||||
/dev/mapper/pve-root 41424257024 25873854464 13413195776 66% /
|
||||
-rw-r--r-- 1 root root 654115664 Sep 13 22:16 felhom-golden-0.236.0.tar.zst
|
||||
-rw-r--r-- 1 root root 656970239 Aug 21 17:59 vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst
|
||||
-rw-r--r-- 1 root root 5648463064 Oct 14 09:35 vzdump-lxc-9201-2026_10_14-09_28_50.tar.zst
|
||||
-rw-r--r-- 1 root root 1161887744 Oct 14 10:44 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
drwxr-xr-x 3 root root 4096 Oct 14 10:42 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.309.0
|
||||
@@ -0,0 +1,14 @@
|
||||
2026-10-14 13:42:48 INFO: Starting Backup of VM 9201 (lxc)
|
||||
2026-10-14 13:42:48 INFO: status = running
|
||||
2026-10-14 13:42:48 INFO: CT Name: demo-hp
|
||||
2026-10-14 13:42:48 INFO: including mount point rootfs ('/') in backup
|
||||
2026-10-14 13:42:48 INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
2026-10-14 13:42:48 INFO: excluding bind mount point mp8 ('/mnt/felhom-drives') from backup (not a volume)
|
||||
2026-10-14 13:42:48 INFO: excluding bind mount point mp9 ('/etc/felhom-bootstrap') from backup (not a volume)
|
||||
2026-10-14 13:42:48 INFO: backup mode: snapshot
|
||||
2026-10-14 13:42:48 INFO: ionice priority: 7
|
||||
2026-10-14 13:42:48 INFO: suspend vm to make snapshot
|
||||
2026-10-14 13:42:48 INFO: create storage snapshot 'vzdump'
|
||||
2026-10-14 13:42:49 INFO: resume vm
|
||||
2026-10-14 13:42:49 INFO: guest is online again after 1 seconds
|
||||
2026-10-14 13:42:49 ERROR: Backup of VM 9201 failed - lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin pve/vm-9201-disk-1.
|
||||
@@ -0,0 +1,8 @@
|
||||
ALLOWED : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
ALLOWED : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.zst
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.log
|
||||
refused : felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat /etc/shadow
|
||||
refused : felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
refused : felhom-vzdump-partial-clean --help
|
||||
@@ -0,0 +1,6 @@
|
||||
Scanning data mappings
|
||||
DEV MAPPED CREATE_TIME SNAP_TIME
|
||||
1 1964MiB 0 139
|
||||
2 19GiB 0 138
|
||||
4 19GiB 138 138
|
||||
released
|
||||
@@ -115,3 +115,29 @@ its report (operator page, later the household), and run „thumbnails: all" in
|
||||
hook and CPU in proportion to the library; a missing original then needs a restore from the night backup. If nothing is
|
||||
done: broken previews stay until someone forces a regeneration by hand; a lost original stays unnoticed.
|
||||
|
||||
|
||||
## E — the live proof on demo-hp (agent 0.156.1 + bundle, delivered by signed jobs)
|
||||
|
||||
Vouched by the operator 2026-10-14 (read back from `/configuration`): agent 0.156.1 `83c7da27…`, golden 0.310.0 `2110d5e4…`,
|
||||
min_agent 0.155.0, global floor 0.310.0. demo-hp: `agent_update 0.156.1` (11:00 UTC) → `agent_config_update 0.156.1-step1`
|
||||
(11:15, „written=1", probe 70/71 — the helper not yet there) → `agent_config_update 0.156.1` (11:30, „BUNDLE WROTE
|
||||
/usr/local/sbin/felhom-vzdump-partial-clean (new)", probe **71/71**). The new agent's first start (11:00) already FOUND C3's
|
||||
partial and called it dead — and left it, „the root helper refused: sudo: a password is required" (the helper came with the
|
||||
bundle). Real sudo after the bundle: `E/sudo-live-demo-hp.txt` (the two shapes allowed, six attacks refused).
|
||||
|
||||
**Prediction (before the cut):** after the boot the agent's start-up sweep removes BOTH partials (C3's from 08:44 and this
|
||||
one), logged „removed a dead half-written backup … written before this boot"; the dead vzdump task is logged once as „worker
|
||||
process is gone" and the start-up stale-lock pass clears the lock itself (or finds Proxmox already did); the next backup runs
|
||||
and succeeds; host root returns below its use before this cut.
|
||||
|
||||
**Measured so far (11:42 UTC):**
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| floor 0.310.0 | not delivered: each demo customer (demo-felhom, demo-hp, Tester 1) has its OWN floor 0.309.0 (set 2026-10-12 16:07), which wins over the global one (hub log + `floor_raise_skipped` mail). demo-hp's guest stays on 0.309.0 |
|
||||
| backup pressed (11:42:33 UTC) | **the pre-backup cleanup worked live:** `removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat` (C3's 1.16 GB) and its `.tmp`, through the root helper (`sudo … felhom-vzdump-partial-clean local 9201 …`); each removal took ~7 s (the helper's `/proc` scan); then „space preflight passed" (`E/agent-cleanup-before-backup.txt`) |
|
||||
| the backup itself | **failed** in 1 s: `lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin`; kernel: „device-mapper: thin: Creation of new snapshot 4 of device 2 failed" (`E/failed-backup.log`) |
|
||||
| why | the thin pool holds an **orphan thin device 4** — a 19 GiB snapshot of 9201's data disk (device 2) created at pool time 138 — which LVM does not know (LVM: devices 1, 2; transaction 559 on both sides). It is what C4's full root disk left: the snapshot cleanup could not write LVM's metadata. Read with a metadata snapshot (`thin_ls --metadata-snap`, reserved and released; `E/thin-ls.txt`) |
|
||||
| consequence | **no whole-guest backup of 9201 on demo-hp can run until device 4 is deleted** (every snapshot create picks id 4). Caused by this session's C4 test |
|
||||
|
||||
The power-cut leg of E waits on the repair (one pool message, `dmsetup message pve-data-tpool 0 "delete 4"`) — an operator decision.
|
||||
|
||||
Reference in New Issue
Block a user