Power-cut fixes 2026-10-14 — spike (Part C) and live proof (Part E)
Times UTC. Box: demo-hp (Tier 0). No key, token or password in this folder.
C1 — what the night's cut left (from audits/night-power-2026-10-13/R3/)
| Item | Value |
|---|---|
| left on disk | /var/lib/vz/dump/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat (file, root:root 0644, 12.4 GB at the cut, 12.2 GB after the boot — the unwritten tail was lost) and vzdump-lxc-9201-2026_10_14-04_42_18.tmp/ (directory, root:root 0755). No .log (vzdump writes it at the end) |
| name pattern | vzdump-lxc-<vmid>-YYYY_MM_DD-HH_MM_SS + .tar.dat / .tmp; the timestamp is the host's local time |
| the task | UPID:demo-hp:000A8C2E:000DFB52:6ACEEC0A:vzdump:9201:felhom-agent@pve!agent: — started by the agent's API token |
| after the boot | 02:52:56 the agent: „a vzdump backup is genuinely in-flight — leaving the lock" — so the dead task WAS in tasks?source=active 1 min after the boot. How long it stays: not recorded → C3 |
| who may delete | the files are root's in a root-owned directory; the agent runs as felhom-agent → it cannot delete them itself |
C2 — predictions (written before the run)
| Question | Prediction |
|---|---|
left on disk after kill -9 of the vzdump worker |
the same pair (.tar.dat + .tmp); the tar/zstd children may keep writing for a moment, then die on the broken pipe |
StorageContent (backup) lists it |
no — Proxmox lists vzdump-*.tar.zst/.vma.zst etc.; .tar.dat is a temporary name |
| the active-task list keeps the task | no — the PVE task daemon sees the worker die and ends the task with an error (unlike a power cut, where nobody ends it) |
lock: backup on the guest |
stays (vzdump's cleanup did not run) |
| the next backup | the space preflight sizes from the last COMPLETE archive (5.6 GB) and passes while space is free; the partial still occupies the disk |
C2 — a partial without a power cut (C2/; 9201, because 9202 has no agent link — its „back up now" cannot run)
Backup pressed through the product 08:07:30 (POST /api/guest-backup/trigger, 200); at 1.5 GB written the vzdump worker
(task UPID:…:vzdump:9201:…) was killed -9 at 08:09:44.
| Question | Prediction | Measured |
|---|---|---|
| left on disk | .tar.dat + .tmp |
yes — vzdump-lxc-9201-2026_10_14-10_08_05.tar.dat + .tmp/, root 0644 / 0755, no .log |
| the writers | die on a broken pipe | no — tar and zstd (children of a bash -c, none named „vzdump") kept writing for 4 min 40 s, to 5.6 GB (a whole archive's size), then exited. The file was never renamed. A „no vzdump process" check alone would call a file dead while it is being written |
StorageContent lists it |
no | no — only the complete …09_28_50.tar.zst is listed |
| active-task list keeps the task | no | no — gone within seconds (the PVE daemon ended it: „backup job failed" 08:09:46) |
| lock | backup stays |
the agent's 10-min janitor cleared it at 08:10:18 while the orphan tar still read the snapshot; its snapshot delete failed („contains a filesystem in use") → lock: snapshot-delete; the next janitor pass (08:20:18) cleared that lock and removed the snapshot |
| the next backup | passes the preflight | none came: a failed manual press is not retried (the night loop retries; this one did not within 30 min) |
| clean-up | — | the 5.6 GB partial removed by hand at 08:41 (host root 78 % → 63 %) |
C3 — one real power cut mid-write (9201's backup, pressed through the product)
Predictions (written before the cut):
| Question | Prediction |
|---|---|
the dead vzdump task in tasks?source=active after the boot |
yes, and it stays there (nobody ends a task whose node died) — for the whole 15 min |
BackupRunning (the agent's startup stale-lock line) |
true → „genuinely in-flight — leaving the lock"; Proxmox's start-all removes the lock anyway |
the guest restart request (POST /api/server/reboot, the dashboard's) |
refused „backup running" while the dead task is listed |
| the household power-off request | refused — „too soon" first (host up < 10 min), and on demo-hp „other guests" anyway (9202 and Tester 1 run); it cannot reach the backup check here |
| files | .tar.dat + .tmp stay, nothing removes them |
Measured (C3/): backup pressed 08:42:25; at 1.5 GB the cut: 08:44:08.089 off → 08:44:18.165 on (plug countdown).
Boot btime 1791967481; the dead task's start 1791967345 (136 s before the boot).
| Question | Prediction | Measured |
|---|---|---|
| dead task in the active list | stays all 15 min | listed RUNNING at 08:44:57; gone at 08:45:30 — Proxmox's own start-up clears it within ~30–60 s of the boot (at 10:45:00 local pvesh/pve-guests: „removed left over backup lock from '9201'!") |
BackupRunning |
true | true at the agent's start (08:44:59: „genuinely in-flight — leaving the lock") — the agent starts inside that short window |
guest restart request (POST /api/server/reboot, 08:47:09) |
refused „backup running" | 202 — it restarted 9201. The restart route never asks BackupRunning; the guestpower.go caller is the guest-power WATCHDOG (it starts stopped guests), not the restart |
| power-off request (08:48:41) | refused | 404 — demo-hp's 9201 runs controller 0.309.0, which has no power-off route; the guard can be exercised only after the floor raise (Part E) |
| files | stay | stay all 15 min: …10_42_25.tar.dat 1.16 GB (1.5 GB at the cut — the unwritten tail lost) + .tmp/ |
| lock | Proxmox removes it | removed by Proxmox at boot |
What this changes (STOP-rule check): the night's mechanism is the same — the partial is the damage, Proxmox does not list it as a backup. R-1008's window is short (≈ 1 min after a boot): the startup stale-lock pass, the first guest-power watchdog sweep and the controller supervisor in that minute see a dead task as running. D1 stays (one helper, cheap, correct); its live effect is small. D2 is the fix that matters.
C4 — a backup that runs out of space (prediction before the run)
On scratch guest 9202, vzdump 9202 --storage local --mode snapshot --compress zstd run as root on the host (9202 has no
agent link; the question is vzdump's own behaviour), with the host root filled by a test file to ~600 MB free.
Prediction: vzdump fails with „No space left on device" and removes its own .tar.dat/.tmp (its error path runs,
unlike a power cut); the lock clears; nothing is left.
Measured (C4/, host root filled to 600 MB free by C4-fill-test.bin, removed after):
| Run | Result |
|---|---|
vzdump 9202 (dir storage → vzdump's rsync path, suspend mode) |
„No space left on device" in the first sync; vzdump removed its .tmp; only vzdump-lxc-9202-…11_01_36.log (2 KB) left; space back |
vzdump 9201 (LVM-thin → the snapshot + tar … | zstd > .tar.dat path, the night's path) |
zstd: error 70 … No space left on device; vzdump removed its .tar.dat; only the .log left. But the full disk broke LVM: „Failed to write metadata … Backup of volume group pve metadata failed"; the vzdump snapshot was half removed (disk-0's gone, disk-1's left) → lock: snapshot-delete |
| the agent's janitor after it | 11:14:59 local: cleared the lock, then its snapshot delete failed („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") — it would fail the same way every 10 min, the guest's snapshot list stuck. Fixed by hand 09:19 UTC: pct unlock 9201; pct delsnapshot 9201 vzdump --force (removed disk-1's snapshot LV, cleared the entry) |
D3 decided by C4: vzdump cleans its own partial on ENOSPC, so the preflight's sizing (R-1006 part 2) costs only wasted
time, never a leftover; spaceFits stays as it is. The real danger C4 shows is different: a host root that fills up
breaks LVM's own metadata writes — the night's 12 GB partial took demo-hp to 92 %. Removing a dead partial (D2) is what
keeps the root disk from filling. New finding: the janitor's snapshot delete has no --force path for a half-removed
snapshot (row).
F — can Immich repair the power cut's damage by itself? (R-1007, spike only)
Prediction (before the run): Immich's thumbnail job with „missing" does NOT redo a zero-byte thumbnail (the database says it exists); the „all" (force) run does; nothing in Immich restores a deleted original, but a storage/integrity report lists it.
Measured (F/immich-repair.txt, F/immich-delete-then-missing.txt; Immich v3.2.4 on 9202, 30 photos, deployed and
removed through the product):
| Act | Result |
|---|---|
| 10 assets' thumbnail + preview truncated to 0 bytes; 1 original deleted (the night's damage, R2) | the API serves 0-byte thumbnails/previews; the deleted original: 404 |
integrity jobs integrity-missing-files, -untracked-files, -checksum-mismatch (POST /api/jobs) |
the deleted original is found: GET /api/admin/integrity/report?type=missing_file lists it with its asset id. The zero-byte thumbnails are not reported (the checks cover originals) |
thumbnails „missing" (PUT /api/jobs/thumbnailGeneration {force:false}) |
does not fix a zero-byte thumbnail — the database says it exists |
| zero-byte files deleted first, then „missing" | still not fixed (Immich decides from its database, not the disk) |
thumbnails „all" ({force:true}) |
fixes all 10 — every thumbnail regenerated (30 photos: 16 s; a family library of 50,000: hours of CPU) |
| the deleted original | nothing in Immich restores it — only a backup |
Prediction: right on „missing" and „all"; right that nothing restores an original; the integrity report exists in this
version (/api/admin/integrity/…) and finds the missing original.
Proposal for R-1007 (an operator decision): after an unclean boot, run Immich's „integrity-missing-files" job and show its report (operator page, later the household), and run „thumbnails: all" in the night window. Cost: one catalog-level hook and CPU in proportion to the library; a missing original then needs a restore from the night backup. If nothing is done: broken previews stay until someone forces a regeneration by hand; a lost original stays unnoticed.