Power-cut fixes E: agent 0.156.1 delivered to demo-hp, the pre-backup cleanup proven live; an orphan thin snapshot from C4 blocks demo-hp's backups (R-1012 filed); per-customer floors hold 0.309.0
gates / gates (push) Successful in 6m44s
gates / gates (push) Successful in 6m44s
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
Oct 14 13:42:33 demo-hp sudo[519542]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
Oct 14 13:42:40 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:40.495+02:00 level=WARN msg="vzdump-partial: removed a dead half-written backup" storage=local vmid=9201 name=vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat bytes=1161887744 why="written before this boot" helper="removed vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat 1161887744 B (written before this boot)"
|
||||
Oct 14 13:42:40 demo-hp sudo[526700]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
Oct 14 13:42:47 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:47.973+02:00 level=WARN msg="vzdump-partial: removed a dead half-written backup" storage=local vmid=9201 name=vzdump-lxc-9201-2026_10_14-10_42_25.tmp bytes=4096 why="written before this boot" helper="removed vzdump-lxc-9201-2026_10_14-10_42_25.tmp 571 B (written before this boot)"
|
||||
Oct 14 13:42:48 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:48.506+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=5648463064 need_bytes=8134320654 avail_bytes=14566670336
|
||||
Oct 14 13:42:50 demo-hp felhom-agent[397764]: time=2026-10-14T13:42:50.583+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1791978152980804597 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:00082531:00104E7D:6ACF6AB8:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
|
||||
@@ -0,0 +1,7 @@
|
||||
/dev/mapper/pve-root 41424257024 25873854464 13413195776 66% /
|
||||
-rw-r--r-- 1 root root 654115664 Sep 13 22:16 felhom-golden-0.236.0.tar.zst
|
||||
-rw-r--r-- 1 root root 656970239 Aug 21 17:59 vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst
|
||||
-rw-r--r-- 1 root root 5648463064 Oct 14 09:35 vzdump-lxc-9201-2026_10_14-09_28_50.tar.zst
|
||||
-rw-r--r-- 1 root root 1161887744 Oct 14 10:44 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
drwxr-xr-x 3 root root 4096 Oct 14 10:42 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.309.0
|
||||
@@ -0,0 +1,14 @@
|
||||
2026-10-14 13:42:48 INFO: Starting Backup of VM 9201 (lxc)
|
||||
2026-10-14 13:42:48 INFO: status = running
|
||||
2026-10-14 13:42:48 INFO: CT Name: demo-hp
|
||||
2026-10-14 13:42:48 INFO: including mount point rootfs ('/') in backup
|
||||
2026-10-14 13:42:48 INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
2026-10-14 13:42:48 INFO: excluding bind mount point mp8 ('/mnt/felhom-drives') from backup (not a volume)
|
||||
2026-10-14 13:42:48 INFO: excluding bind mount point mp9 ('/etc/felhom-bootstrap') from backup (not a volume)
|
||||
2026-10-14 13:42:48 INFO: backup mode: snapshot
|
||||
2026-10-14 13:42:48 INFO: ionice priority: 7
|
||||
2026-10-14 13:42:48 INFO: suspend vm to make snapshot
|
||||
2026-10-14 13:42:48 INFO: create storage snapshot 'vzdump'
|
||||
2026-10-14 13:42:49 INFO: resume vm
|
||||
2026-10-14 13:42:49 INFO: guest is online again after 1 seconds
|
||||
2026-10-14 13:42:49 ERROR: Backup of VM 9201 failed - lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin pve/vm-9201-disk-1.
|
||||
@@ -0,0 +1,8 @@
|
||||
ALLOWED : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
ALLOWED : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tmp
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.zst
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.log
|
||||
refused : felhom-vzdump-partial-clean local 9201 ../../etc/vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
refused : felhom-vzdump-partial-clean local 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat /etc/shadow
|
||||
refused : felhom-vzdump-partial-clean local;id 9201 vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat
|
||||
refused : felhom-vzdump-partial-clean --help
|
||||
@@ -0,0 +1,6 @@
|
||||
Scanning data mappings
|
||||
DEV MAPPED CREATE_TIME SNAP_TIME
|
||||
1 1964MiB 0 139
|
||||
2 19GiB 0 138
|
||||
4 19GiB 138 138
|
||||
released
|
||||
@@ -115,3 +115,29 @@ its report (operator page, later the household), and run „thumbnails: all" in
|
||||
hook and CPU in proportion to the library; a missing original then needs a restore from the night backup. If nothing is
|
||||
done: broken previews stay until someone forces a regeneration by hand; a lost original stays unnoticed.
|
||||
|
||||
|
||||
## E — the live proof on demo-hp (agent 0.156.1 + bundle, delivered by signed jobs)
|
||||
|
||||
Vouched by the operator 2026-10-14 (read back from `/configuration`): agent 0.156.1 `83c7da27…`, golden 0.310.0 `2110d5e4…`,
|
||||
min_agent 0.155.0, global floor 0.310.0. demo-hp: `agent_update 0.156.1` (11:00 UTC) → `agent_config_update 0.156.1-step1`
|
||||
(11:15, „written=1", probe 70/71 — the helper not yet there) → `agent_config_update 0.156.1` (11:30, „BUNDLE WROTE
|
||||
/usr/local/sbin/felhom-vzdump-partial-clean (new)", probe **71/71**). The new agent's first start (11:00) already FOUND C3's
|
||||
partial and called it dead — and left it, „the root helper refused: sudo: a password is required" (the helper came with the
|
||||
bundle). Real sudo after the bundle: `E/sudo-live-demo-hp.txt` (the two shapes allowed, six attacks refused).
|
||||
|
||||
**Prediction (before the cut):** after the boot the agent's start-up sweep removes BOTH partials (C3's from 08:44 and this
|
||||
one), logged „removed a dead half-written backup … written before this boot"; the dead vzdump task is logged once as „worker
|
||||
process is gone" and the start-up stale-lock pass clears the lock itself (or finds Proxmox already did); the next backup runs
|
||||
and succeeds; host root returns below its use before this cut.
|
||||
|
||||
**Measured so far (11:42 UTC):**
|
||||
|
||||
| Step | Result |
|
||||
|---|---|
|
||||
| floor 0.310.0 | not delivered: each demo customer (demo-felhom, demo-hp, Tester 1) has its OWN floor 0.309.0 (set 2026-10-12 16:07), which wins over the global one (hub log + `floor_raise_skipped` mail). demo-hp's guest stays on 0.309.0 |
|
||||
| backup pressed (11:42:33 UTC) | **the pre-backup cleanup worked live:** `removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat` (C3's 1.16 GB) and its `.tmp`, through the root helper (`sudo … felhom-vzdump-partial-clean local 9201 …`); each removal took ~7 s (the helper's `/proc` scan); then „space preflight passed" (`E/agent-cleanup-before-backup.txt`) |
|
||||
| the backup itself | **failed** in 1 s: `lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin`; kernel: „device-mapper: thin: Creation of new snapshot 4 of device 2 failed" (`E/failed-backup.log`) |
|
||||
| why | the thin pool holds an **orphan thin device 4** — a 19 GiB snapshot of 9201's data disk (device 2) created at pool time 138 — which LVM does not know (LVM: devices 1, 2; transaction 559 on both sides). It is what C4's full root disk left: the snapshot cleanup could not write LVM's metadata. Read with a metadata snapshot (`thin_ls --metadata-snap`, reserved and released; `E/thin-ls.txt`) |
|
||||
| consequence | **no whole-guest backup of 9201 on demo-hp can run until device 4 is deleted** (every snapshot create picks id 4). Caused by this session's C4 test |
|
||||
|
||||
The power-cut leg of E waits on the repair (one pool message, `dmsetup message pve-data-tpool 0 "delete 4"`) — an operator decision.
|
||||
|
||||
@@ -149,10 +149,11 @@ stopping line that lies.
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
|
||||
## Backup & restore — 29 rows (P2 4, P3 15, P4 10)
|
||||
## Backup & restore — 30 rows (P2 5, P3 15, P4 10)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1012** | Backup & restore | P2 | **A full host root can leave an orphan thin snapshot in the LVM pool that LVM no longer knows; from then on every whole-guest backup of that guest fails at the snapshot step („Creation of new snapshot 4 of device 2 failed"), and nothing in the product finds or names the cause.** Made on demo-hp 2026-10-14 by this session's C4 test (the root filled while vzdump snapshotted; LVM could not write its metadata): `thin_ls --metadata-snap` shows device 4 (19 GiB, a snapshot of 9201's data disk), LVM only 1 and 2 (`audits/power-cut-fixes-2026-10-14/E/thin-ls.txt`). R-1006's cleanup keeps the root from filling; this row is the repair and the detection. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | operator: the repair on demo-hp | Operator: yes/no to `dmsetup message pve-data-tpool 0 "delete 4"` on demo-hp; then CC: a check that compares the pool's thin ids with LVM's and names an orphan in the backup failure | CC |
|
||||
| **R-1006** | Backup & restore | P2 | **A power cut during the night's whole-guest backup leaves the half-written archive on the host, and from then on every local whole-guest backup of that box is skipped „not enough space" — nothing removes it.** Measured 2026-10-14 on demo-hp (`audits/night-power-2026-10-13/` R3): the cut at 02:52 left `vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat` (12.2 GB) + `.tmp`; host root at 92 %; agent retry 03:08 „skipped: not enough space: local has 2.9 GiB free; the last archive … was 4.6 GiB". The operator got `whole_guest_backup_failed` (true). Second part: the space preflight sizes the new archive from the LAST one (7.2 GB planned) while the guest had grown (12.4 GB written when cut) — a backup that outgrows the plan can fill the host root. Third part, the same rule the other way: after one big archive (14.4 GB at 05:17, while bench data sat on the guest disk) the next preflight would need ~21 GB of 5.5 GB free — the box skips every later backup though the guest is small again; cleared by hand at 07:28 UTC (`night-power-2026-10-13/README.md`). | **WAITING-ON-OPERATOR** — 2026-10-14: built and released in agent v0.156.1 (0.156.0 superseded); `audits/power-cut-fixes-2026-10-14/` (spike C1–C4, 19 red-proofs); part 2 (the preflight sizing) settled by C4 — vzdump cleans its own ENOSPC partial, `spaceFits` stays | operator: vouch agent 0.156.1 + bundle | CC: after the vouch — deliver (agent → step bundle → bundle) to demo-hp; one real cut mid-backup; prove the partial is gone, the next backup succeeds; then Tester 1 and the N100 | CC |
|
||||
| **R-978** | Backup & restore | P3 | **Starting the escrow ceremony again can make the customer's written-down recovery code useless: the new package replaces the old one before the new code is confirmed.** It can happen to an ordinary customer who starts and walks away. Fix: keep the previous package until the new code is confirmed. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix in the agent with a test | CC |
|
||||
| **R-973** | Backup & restore | P3 | **A compromised controller can make the whole-guest backup run again and again, and each run prunes older restore points: the local tier directly, and the off-site tier through the store's own nightly keep-last-2.** So `07` §8 row 10 (ransomware) is weaker than it reads. The second reader confirmed the local half; the off-site half is inferred. Fix: a cadence or daily cap on the local-API backup route, and off-site snapshots that a nightly prune cannot remove. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Add the cap with a test; decide the off-site retention shape | CC |
|
||||
|
||||
Reference in New Issue
Block a user