felhom.eu/documentation/audits/power-cut-fixes-2026-10-14

Power-cut fixes 2026-10-14 — spike (Part C) and live proof (Part E)

Times UTC. Box: demo-hp (Tier 0). No key, token or password in this folder.

C1 — what the night's cut left (from audits/night-power-2026-10-13/R3/)

Item	Value
left on disk	/var/lib/vz/dump/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat (file, root:root 0644, 12.4 GB at the cut, 12.2 GB after the boot — the unwritten tail was lost) and vzdump-lxc-9201-2026_10_14-04_42_18.tmp/ (directory, root:root 0755). No .log (vzdump writes it at the end)
name pattern	vzdump-lxc-<vmid>-YYYY_MM_DD-HH_MM_SS + .tar.dat / .tmp; the timestamp is the host's local time
the task	UPID:demo-hp:000A8C2E:000DFB52:6ACEEC0A:vzdump:9201:felhom-agent@pve!agent: — started by the agent's API token
after the boot	02:52:56 the agent: „a vzdump backup is genuinely in-flight — leaving the lock" — so the dead task WAS in tasks?source=active 1 min after the boot. How long it stays: not recorded → C3
who may delete	the files are root's in a root-owned directory; the agent runs as felhom-agent → it cannot delete them itself

C2 — predictions (written before the run)

Question	Prediction
left on disk after kill -9 of the vzdump worker	the same pair (.tar.dat + .tmp); the tar/zstd children may keep writing for a moment, then die on the broken pipe
StorageContent (backup) lists it	no — Proxmox lists vzdump-*.tar.zst/.vma.zst etc.; .tar.dat is a temporary name
the active-task list keeps the task	no — the PVE task daemon sees the worker die and ends the task with an error (unlike a power cut, where nobody ends it)
lock: backup on the guest	stays (vzdump's cleanup did not run)
the next backup	the space preflight sizes from the last COMPLETE archive (5.6 GB) and passes while space is free; the partial still occupies the disk

C2 — a partial without a power cut (C2/; 9201, because 9202 has no agent link — its „back up now" cannot run)

Backup pressed through the product 08:07:30 (POST /api/guest-backup/trigger, 200); at 1.5 GB written the vzdump worker (task UPID:…:vzdump:9201:…) was killed -9 at 08:09:44.

Question	Prediction	Measured
left on disk	.tar.dat + .tmp	yes — vzdump-lxc-9201-2026_10_14-10_08_05.tar.dat + .tmp/, root 0644 / 0755, no .log
the writers	die on a broken pipe	no — tar and zstd (children of a bash -c, none named „vzdump") kept writing for 4 min 40 s, to 5.6 GB (a whole archive's size), then exited. The file was never renamed. A „no vzdump process" check alone would call a file dead while it is being written
StorageContent lists it	no	no — only the complete …09_28_50.tar.zst is listed
active-task list keeps the task	no	no — gone within seconds (the PVE daemon ended it: „backup job failed" 08:09:46)
lock	backup stays	the agent's 10-min janitor cleared it at 08:10:18 while the orphan tar still read the snapshot; its snapshot delete failed („contains a filesystem in use") → lock: snapshot-delete; the next janitor pass (08:20:18) cleared that lock and removed the snapshot
the next backup	passes the preflight	none came: a failed manual press is not retried (the night loop retries; this one did not within 30 min)
clean-up	—	the 5.6 GB partial removed by hand at 08:41 (host root 78 % → 63 %)

C3 — one real power cut mid-write (9201's backup, pressed through the product)

Predictions (written before the cut):

Question	Prediction
the dead vzdump task in tasks?source=active after the boot	yes, and it stays there (nobody ends a task whose node died) — for the whole 15 min
BackupRunning (the agent's startup stale-lock line)	true → „genuinely in-flight — leaving the lock"; Proxmox's start-all removes the lock anyway
the guest restart request (POST /api/server/reboot, the dashboard's)	refused „backup running" while the dead task is listed
the household power-off request	refused — „too soon" first (host up < 10 min), and on demo-hp „other guests" anyway (9202 and Tester 1 run); it cannot reach the backup check here
files	.tar.dat + .tmp stay, nothing removes them

Measured (C3/): backup pressed 08:42:25; at 1.5 GB the cut: 08:44:08.089 off → 08:44:18.165 on (plug countdown). Boot btime 1791967481; the dead task's start 1791967345 (136 s before the boot).

Question	Prediction	Measured
dead task in the active list	stays all 15 min	listed RUNNING at 08:44:57; gone at 08:45:30 — Proxmox's own start-up clears it within ~30–60 s of the boot (at 10:45:00 local pvesh/pve-guests: „removed left over backup lock from '9201'!")
BackupRunning	true	true at the agent's start (08:44:59: „genuinely in-flight — leaving the lock") — the agent starts inside that short window
guest restart request (POST /api/server/reboot, 08:47:09)	refused „backup running"	202 — it restarted 9201. The restart route never asks BackupRunning; the guestpower.go caller is the guest-power WATCHDOG (it starts stopped guests), not the restart
power-off request (08:48:41)	refused	404 — demo-hp's 9201 runs controller 0.309.0, which has no power-off route; the guard can be exercised only after the floor raise (Part E)
files	stay	stay all 15 min: …10_42_25.tar.dat 1.16 GB (1.5 GB at the cut — the unwritten tail lost) + .tmp/
lock	Proxmox removes it	removed by Proxmox at boot

What this changes (STOP-rule check): the night's mechanism is the same — the partial is the damage, Proxmox does not list it as a backup. R-1008's window is short (≈ 1 min after a boot): the startup stale-lock pass, the first guest-power watchdog sweep and the controller supervisor in that minute see a dead task as running. D1 stays (one helper, cheap, correct); its live effect is small. D2 is the fix that matters.

C4 — a backup that runs out of space (prediction before the run)

On scratch guest 9202, vzdump 9202 --storage local --mode snapshot --compress zstd run as root on the host (9202 has no agent link; the question is vzdump's own behaviour), with the host root filled by a test file to ~600 MB free. Prediction: vzdump fails with „No space left on device" and removes its own .tar.dat/.tmp (its error path runs, unlike a power cut); the lock clears; nothing is left.

Measured (C4/, host root filled to 600 MB free by C4-fill-test.bin, removed after):

Run	Result
vzdump 9202 (dir storage → vzdump's rsync path, suspend mode)	„No space left on device" in the first sync; vzdump removed its .tmp; only vzdump-lxc-9202-…11_01_36.log (2 KB) left; space back
vzdump 9201 (LVM-thin → the snapshot + tar … | zstd > .tar.dat path, the night's path)	zstd: error 70 … No space left on device; vzdump removed its .tar.dat; only the .log left. But the full disk broke LVM: „Failed to write metadata … Backup of volume group pve metadata failed"; the vzdump snapshot was half removed (disk-0's gone, disk-1's left) → lock: snapshot-delete
the agent's janitor after it	11:14:59 local: cleared the lock, then its snapshot delete failed („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") — it would fail the same way every 10 min, the guest's snapshot list stuck. Fixed by hand 09:19 UTC: pct unlock 9201; pct delsnapshot 9201 vzdump --force (removed disk-1's snapshot LV, cleared the entry)

D3 decided by C4: vzdump cleans its own partial on ENOSPC, so the preflight's sizing (R-1006 part 2) costs only wasted time, never a leftover; spaceFits stays as it is. The real danger C4 shows is different: a host root that fills up breaks LVM's own metadata writes — the night's 12 GB partial took demo-hp to 92 %. Removing a dead partial (D2) is what keeps the root disk from filling. New finding: the janitor's snapshot delete has no --force path for a half-removed snapshot (row).

F — can Immich repair the power cut's damage by itself? (R-1007, spike only)

Prediction (before the run): Immich's thumbnail job with „missing" does NOT redo a zero-byte thumbnail (the database says it exists); the „all" (force) run does; nothing in Immich restores a deleted original, but a storage/integrity report lists it.

Measured (F/immich-repair.txt, F/immich-delete-then-missing.txt; Immich v3.2.4 on 9202, 30 photos, deployed and removed through the product):

Act	Result
10 assets' thumbnail + preview truncated to 0 bytes; 1 original deleted (the night's damage, R2)	the API serves 0-byte thumbnails/previews; the deleted original: 404
integrity jobs integrity-missing-files, -untracked-files, -checksum-mismatch (POST /api/jobs)	the deleted original is found: GET /api/admin/integrity/report?type=missing_file lists it with its asset id. The zero-byte thumbnails are not reported (the checks cover originals)
thumbnails „missing" (PUT /api/jobs/thumbnailGeneration {force:false})	does not fix a zero-byte thumbnail — the database says it exists
zero-byte files deleted first, then „missing"	still not fixed (Immich decides from its database, not the disk)
thumbnails „all" ({force:true})	fixes all 10 — every thumbnail regenerated (30 photos: 16 s; a family library of 50,000: hours of CPU)
the deleted original	nothing in Immich restores it — only a backup

Prediction: right on „missing" and „all"; right that nothing restores an original; the integrity report exists in this version (/api/admin/integrity/…) and finds the missing original.

Proposal for R-1007 (an operator decision): after an unclean boot, run Immich's „integrity-missing-files" job and show its report (operator page, later the household), and run „thumbnails: all" in the night window. Cost: one catalog-level hook and CPU in proportion to the library; a missing original then needs a restore from the night backup. If nothing is done: broken previews stay until someone forces a regeneration by hand; a lost original stays unnoticed.

B2 — can Immich rebuild single photos? (R-1007, after the operator ruling „fix only the broken photos", spike only)

Immich v3.2.4 on scratch guest 9202, deployed and removed through the product; 30 bench photos (B2/).

What the running server answers (asked, not remembered): POST /api/assets/jobs with a bad body → 400 listing the names refresh-faces, refresh-metadata, regenerate-thumbnail, transcode-video; body {"assetIds": [<uuid>…], "name": "regenerate-thumbnail"}, permission AssetUpdate (an admin or the owner's key). In the server code (dist/services/asset.service.js run()): one AssetGenerateThumbnails job per id given — nothing else is queued.

Path layout: <upload>/thumbs/<user id>/<a[0:2]>/<a[2:4]>/<asset id>_thumbnail.webp and …_preview.jpeg (+ one .immich marker file). Mapping an empty file to its asset: the first 36 characters of the file name ARE the asset id. A box finds the broken photos from the disk alone: find <upload>/thumbs -type f -size 0, cut the names to 36 characters, unique.

Step	Result
damage	assets 1–5: thumbnail → 0 B; assets 6–10: preview → 0 B; the API served the 0-byte files
found from the disk	10 zero-byte files → 10 asset ids (the mapping above)
POST /api/assets/jobs regenerate-thumbnail for those 10	204; no zero-byte file left after 1.55 s; queue idle after 16.9 s (5 s polling); CPU 5.3 s (all Immich containers, cgroup usage_usec)
the 10 assets	all 20 files rewritten (the undamaged sibling too), none empty; the API serves real thumbnails and previews
the other 20 assets	0 of 40 files rewritten (mtimes before/after)
positive control: thumbnails „all" (force=true)	rewrote 40 of 40 of those same files (the 41st counted file is the .immich marker); CPU 16.5 s for 30 photos
teardown	Immich removed through the product; no folder, container, volume or image left; 9202's standing apps untouched

Answer: yes — v3.2.4 rebuilds single photos. Cost ≈ 0.5 s CPU per photo here (a family library with 80 broken photos, like the night's R2: well under a minute) against „all" in proportion to the whole library. A lost original is still found only by integrity-missing-files and restored only from a backup.

E — the live proof on demo-hp (agent 0.156.1 + bundle, delivered by signed jobs)

Vouched by the operator 2026-10-14 (read back from /configuration): agent 0.156.1 83c7da27…, golden 0.310.0 2110d5e4…, min_agent 0.155.0, global floor 0.310.0. demo-hp: agent_update 0.156.1 (11:00 UTC) → agent_config_update 0.156.1-step1 (11:15, „written=1", probe 70/71 — the helper not yet there) → agent_config_update 0.156.1 (11:30, „BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new)", probe 71/71). The new agent's first start (11:00) already FOUND C3's partial and called it dead — and left it, „the root helper refused: sudo: a password is required" (the helper came with the bundle). Real sudo after the bundle: E/sudo-live-demo-hp.txt (the two shapes allowed, six attacks refused).

Prediction (before the cut): after the boot the agent's start-up sweep removes BOTH partials (C3's from 08:44 and this one), logged „removed a dead half-written backup … written before this boot"; the dead vzdump task is logged once as „worker process is gone" and the start-up stale-lock pass clears the lock itself (or finds Proxmox already did); the next backup runs and succeeds; host root returns below its use before this cut.

Measured so far (11:42 UTC):

Step	Result
floor 0.310.0	not delivered: each demo customer (demo-felhom, demo-hp, Tester 1) has its OWN floor 0.309.0 (set 2026-10-12 16:07), which wins over the global one (hub log + floor_raise_skipped mail). demo-hp's guest stays on 0.309.0
backup pressed (11:42:33 UTC)	the pre-backup cleanup worked live: removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat (C3's 1.16 GB) and its .tmp, through the root helper (sudo … felhom-vzdump-partial-clean local 9201 …); each removal took ~7 s (the helper's /proc scan); then „space preflight passed" (E/agent-cleanup-before-backup.txt)
the backup itself	failed in 1 s: lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin; kernel: „device-mapper: thin: Creation of new snapshot 4 of device 2 failed" (E/failed-backup.log)
why	the thin pool holds an orphan thin device 4 — a 19 GiB snapshot of 9201's data disk (device 2) created at pool time 138 — which LVM does not know (LVM: devices 1, 2; transaction 559 on both sides). It is what C4's full root disk left: the snapshot cleanup could not write LVM's metadata. Read with a metadata snapshot (thin_ls --metadata-snap, reserved and released; E/thin-ls.txt)
consequence	no whole-guest backup of 9201 on demo-hp can run until device 4 is deleted (every snapshot create picks id 4). Caused by this session's C4 test

Repair (operator „B2 - A", 12:02 UTC): dmsetup message pve-data-tpool 0 "delete 4" → thin_ls shows devices 1 and 2 only, pool transaction 559 unchanged, LVM agrees (E/repair.txt). Identity, not size: device 4 was the one id LVM does not know, a snapshot (CREATE_TIME 138) of device 2.

The cut (E/): fences clear 12:04:48; backup pressed through the product 12:04:48 (POST /api/guest-backup/trigger, 200); the apps were stopped for the snapshot and started again at 12:05:30 (snapshot create worked — the repair held); at 1.53 GB written, plugcut.py --plug hp --off-seconds 10: 12:06:52.325 off → 12:07:02.409 on (plug countdown; watchdog not needed). Root use at the cut 26,249,101,312 B; before the press 24,720,375,808 B. SSH back 12:07:45.

Prediction	Measured
dead task logged once as „worker process is gone"	yes — 12:07:44 „backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running" (the agent started 12:07:42, inside the 30–60 s window in which Proxmox still lists the dead task — the R-1008 case, now judged right)
the start-up stale-lock pass clears the lock, or finds Proxmox did	both, in the same second: the agent decided „clearing a stale backup lock" and ran pct unlock 9201; Proxmox's start-all had just removed it („removed left over backup lock from '9201'!", 12:07:44) → the agent logged ERROR „pct unlock failed … no lock found". Harmless (the guest started 12:07:47), but an ERROR for an outcome that is the goal — R-1013
the start-up sweep removes the partial	yes — 12:07:48 „removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-14_05_21.tar.dat 964,239,360 B (written before this boot)" and 12:07:52 its .tmp — through the root helper. 1.53 GB were written before the cut, 0.96 GB survived (the unwritten tail lost, as in C1)
guards do not refuse because of the dead task	the start-up sweep IS a BackupRunning caller: it went ahead only because the dead task was judged not running. The household power-off request at 12:09:53 (uptime 147 s) was refused too_soon_after_boot only (E/requests.txt). The guest restart is not guarded by BackupRunning at all (C3)
the next backup succeeds	yes — pressed 12:10:06, „found old vzdump snapshot (force removal)" (vzdump removed the cut's LVM snapshot itself), archive vzdump-lxc-9201-2026_10_14-14_10_50.tar.zst 5,650,197,172 B at 12:17:08; the older archive pruned (keep-last 1) (E/next-backup.txt)
host root back below its use before this cut	yes — 24,689,217,536 B after the boot sweep; 24,691,134,464 B after the new backup (before the press: 24,720,375,808 B)
no orphan thin device this time	yes — pool transaction 567, LVM lists only vm-9201-disk-0/1

E/demo-hp-journal-boot.txt is the whole boot journal (scanned: no key, token or password). R-1006 and R-1008 are proven live on demo-hp.

Delivery to the other two boxes (E/delivery-tester1-n100.txt): signed jobs agent_update 0.156.1 (12:21:24 UTC) → agent_config_update 0.156.1-step1 (12:35:37) → agent_config_update 0.156.1 (12:50:14). Tester 1: agent 0.156.1 committed 12:24, bundle done 12:53 (probe 71/71); N100: agent 12:34 (it reports every 15 min), bundle done 13:04 (probe 71/71). On both, real sudo -l -U felhom-agent allows the cleanup shape and refuses a finished archive (.tar.zst). The operator cleared the per-customer floors (demo-hp and demo-felhom 12:00, Tester 1 12:44 UTC): every guest runs controller 0.310.0. Both plugs ON: the HP relay back on by its countdown (12:07:02, E/plugcut.log), the N100's never cut; both hosts answer.