felhom.eu/documentation/audits/power-cut-fixes-2026-10-14/README.md

# Power-cut fixes 2026-10-14 — spike (Part C) and live proof (Part E)

Times UTC. Box: demo-hp (Tier 0). No key, token or password in this folder.

## C1 — what the night's cut left (from `audits/night-power-2026-10-13/R3/`)

| Item | Value |
|---|---|
| left on disk | `/var/lib/vz/dump/vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat` (file, `root:root 0644`, 12.4 GB at the cut, 12.2 GB after the boot — the unwritten tail was lost) and `vzdump-lxc-9201-2026_10_14-04_42_18.tmp/` (directory, `root:root 0755`). **No `.log`** (vzdump writes it at the end) |
| name pattern | `vzdump-lxc-<vmid>-YYYY_MM_DD-HH_MM_SS` + `.tar.dat` / `.tmp`; the timestamp is the host's local time |
| the task | `UPID:demo-hp:000A8C2E:000DFB52:6ACEEC0A:vzdump:9201:felhom-agent@pve!agent:` — started by the agent's API token |
| after the boot | 02:52:56 the agent: „a vzdump backup is genuinely in-flight — leaving the lock" — so the dead task WAS in `tasks?source=active` 1 min after the boot. How long it stays: not recorded → C3 |
| who may delete | the files are root's in a root-owned directory; the agent runs as `felhom-agent` → it **cannot** delete them itself |

## C2 — predictions (written before the run)

| Question | Prediction |
|---|---|
| left on disk after `kill -9` of the vzdump worker | the same pair (`.tar.dat` + `.tmp`); the tar/zstd children may keep writing for a moment, then die on the broken pipe |
| `StorageContent` (backup) lists it | **no** — Proxmox lists `vzdump-*.tar.zst`/`.vma.zst` etc.; `.tar.dat` is a temporary name |
| the active-task list keeps the task | **no** — the PVE task daemon sees the worker die and ends the task with an error (unlike a power cut, where nobody ends it) |
| `lock: backup` on the guest | stays (vzdump's cleanup did not run) |
| the next backup | the space preflight sizes from the last COMPLETE archive (5.6 GB) and passes while space is free; the partial still occupies the disk |

## C2 — a partial without a power cut (`C2/`; 9201, because 9202 has no agent link — its „back up now" cannot run)

Backup pressed through the product 08:07:30 (`POST /api/guest-backup/trigger`, 200); at 1.5 GB written the vzdump worker
(`task UPID:…:vzdump:9201:…`) was killed `-9` at 08:09:44.

| Question | Prediction | Measured |
|---|---|---|
| left on disk | `.tar.dat` + `.tmp` | **yes** — `vzdump-lxc-9201-2026_10_14-10_08_05.tar.dat` + `.tmp/`, root 0644 / 0755, no `.log` |
| the writers | die on a broken pipe | **no** — `tar` and `zstd` (children of a `bash -c`, none named „vzdump") kept writing for **4 min 40 s**, to 5.6 GB (a whole archive's size), then exited. The file was never renamed. **A „no vzdump process" check alone would call a file dead while it is being written** |
| `StorageContent` lists it | no | **no** — only the complete `…09_28_50.tar.zst` is listed |
| active-task list keeps the task | no | **no** — gone within seconds (the PVE daemon ended it: „backup job failed" 08:09:46) |
| lock | `backup` stays | the agent's 10-min janitor cleared it at 08:10:18 **while the orphan `tar` still read the snapshot**; its snapshot delete failed („contains a filesystem in use") → `lock: snapshot-delete`; the next janitor pass (08:20:18) cleared that lock and removed the snapshot |
| the next backup | passes the preflight | none came: a failed manual press is not retried (the night loop retries; this one did not within 30 min) |
| clean-up | — | the 5.6 GB partial removed by hand at 08:41 (host root 78 % → 63 %) |

## C3 — one real power cut mid-write (9201's backup, pressed through the product)

**Predictions (written before the cut):**

| Question | Prediction |
|---|---|
| the dead vzdump task in `tasks?source=active` after the boot | **yes**, and it stays there (nobody ends a task whose node died) — for the whole 15 min |
| `BackupRunning` (the agent's startup stale-lock line) | true → „genuinely in-flight — leaving the lock"; Proxmox's start-all removes the lock anyway |
| the guest restart request (`POST /api/server/reboot`, the dashboard's) | refused „backup running" while the dead task is listed |
| the household power-off request | refused — „too soon" first (host up < 10 min), and on demo-hp „other guests" anyway (9202 and Tester 1 run); it cannot reach the backup check here |
| files | `.tar.dat` + `.tmp` stay, nothing removes them |


**Measured** (`C3/`): backup pressed 08:42:25; at 1.5 GB the cut: 08:44:08.089 off → 08:44:18.165 on (plug countdown).
Boot `btime` 1791967481; the dead task's start 1791967345 (136 s before the boot).

| Question | Prediction | Measured |
|---|---|---|
| dead task in the active list | stays all 15 min | **listed RUNNING at 08:44:57; gone at 08:45:30** — Proxmox's own start-up clears it within ~30–60 s of the boot (at 10:45:00 local `pvesh`/`pve-guests`: „removed left over backup lock from '9201'!") |
| `BackupRunning` | true | **true at the agent's start** (08:44:59: „genuinely in-flight — leaving the lock") — the agent starts inside that short window |
| guest restart request (`POST /api/server/reboot`, 08:47:09) | refused „backup running" | **202 — it restarted 9201.** The restart route never asks `BackupRunning`; the `guestpower.go` caller is the guest-power WATCHDOG (it starts stopped guests), not the restart |
| power-off request (08:48:41) | refused | **404** — demo-hp's 9201 runs controller 0.309.0, which has no power-off route; the guard can be exercised only after the floor raise (Part E) |
| files | stay | **stay all 15 min**: `…10_42_25.tar.dat` 1.16 GB (1.5 GB at the cut — the unwritten tail lost) + `.tmp/` |
| lock | Proxmox removes it | removed by Proxmox at boot |

**What this changes (STOP-rule check):** the night's mechanism is the same — the partial is the damage, Proxmox does not
list it as a backup. R-1008's window is **short** (≈ 1 min after a boot): the startup stale-lock pass, the first guest-power
watchdog sweep and the controller supervisor in that minute see a dead task as running. D1 stays (one helper, cheap,
correct); its live effect is small. D2 is the fix that matters.

## C4 — a backup that runs out of space (prediction before the run)

On scratch guest 9202, `vzdump 9202 --storage local --mode snapshot --compress zstd` run as root on the host (9202 has no
agent link; the question is vzdump's own behaviour), with the host root filled by a test file to ~600 MB free.
**Prediction:** vzdump fails with „No space left on device" and **removes its own `.tar.dat`/`.tmp`** (its error path runs,
unlike a power cut); the lock clears; nothing is left.

**Measured** (`C4/`, host root filled to 600 MB free by `C4-fill-test.bin`, removed after):

| Run | Result |
|---|---|
| `vzdump 9202` (dir storage → vzdump's rsync path, `suspend` mode) | „No space left on device" in the first sync; **vzdump removed its `.tmp`**; only `vzdump-lxc-9202-…11_01_36.log` (2 KB) left; space back |
| `vzdump 9201` (LVM-thin → the snapshot + `tar … \| zstd > .tar.dat` path, the night's path) | `zstd: error 70 … No space left on device`; **vzdump removed its `.tar.dat`**; only the `.log` left. **But the full disk broke LVM:** „Failed to write metadata … Backup of volume group pve metadata failed"; the `vzdump` snapshot was half removed (disk-0's gone, disk-1's left) → `lock: snapshot-delete` |
| the agent's janitor after it | 11:14:59 local: cleared the lock, then its snapshot delete **failed** („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") — it would fail the same way every 10 min, the guest's snapshot list stuck. Fixed by hand 09:19 UTC: `pct unlock 9201; pct delsnapshot 9201 vzdump --force` (removed disk-1's snapshot LV, cleared the entry) |

**D3 decided by C4:** vzdump cleans its own partial on ENOSPC, so the preflight's sizing (R-1006 part 2) costs only wasted
time, never a leftover; `spaceFits` stays as it is. The real danger C4 shows is different: **a host root that fills up
breaks LVM's own metadata writes** — the night's 12 GB partial took demo-hp to 92 %. Removing a dead partial (D2) is what
keeps the root disk from filling. New finding: the janitor's snapshot delete has no `--force` path for a half-removed
snapshot (row).


## F — can Immich repair the power cut's damage by itself? (R-1007, spike only)

**Prediction (before the run):** Immich's thumbnail job with „missing" does NOT redo a zero-byte thumbnail (the database
says it exists); the „all" (force) run does; nothing in Immich restores a deleted original, but a storage/integrity
report lists it.

**Measured** (`F/immich-repair.txt`, `F/immich-delete-then-missing.txt`; Immich v3.2.4 on 9202, 30 photos, deployed and
removed through the product):

| Act | Result |
|---|---|
| 10 assets' thumbnail + preview truncated to 0 bytes; 1 original deleted (the night's damage, R2) | the API serves 0-byte thumbnails/previews; the deleted original: 404 |
| integrity jobs `integrity-missing-files`, `-untracked-files`, `-checksum-mismatch` (`POST /api/jobs`) | **the deleted original is found**: `GET /api/admin/integrity/report?type=missing_file` lists it with its asset id. The zero-byte thumbnails are not reported (the checks cover originals) |
| thumbnails „missing" (`PUT /api/jobs/thumbnailGeneration {force:false}`) | **does not fix** a zero-byte thumbnail — the database says it exists |
| zero-byte files deleted first, then „missing" | **still not fixed** (Immich decides from its database, not the disk) |
| thumbnails „all" (`{force:true}`) | **fixes all 10** — every thumbnail regenerated (30 photos: 16 s; a family library of 50,000: hours of CPU) |
| the deleted original | nothing in Immich restores it — only a backup |

**Prediction:** right on „missing" and „all"; right that nothing restores an original; the integrity report exists in this
version (`/api/admin/integrity/…`) and finds the missing original.

**Proposal for R-1007 (an operator decision):** after an unclean boot, run Immich's „integrity-missing-files" job and show
its report (operator page, later the household), and run „thumbnails: all" in the night window. Cost: one catalog-level
hook and CPU in proportion to the library; a missing original then needs a restore from the night backup. If nothing is
done: broken previews stay until someone forces a regeneration by hand; a lost original stays unnoticed.


## B2 — can Immich rebuild single photos? (R-1007, after the operator ruling „fix only the broken photos", spike only)

Immich v3.2.4 on scratch guest 9202, deployed and removed through the product; 30 bench photos (`B2/`).

**What the running server answers** (asked, not remembered): `POST /api/assets/jobs` with a bad body → 400 listing the
names `refresh-faces`, `refresh-metadata`, `regenerate-thumbnail`, `transcode-video`; body `{"assetIds": [<uuid>…],
"name": "regenerate-thumbnail"}`, permission `AssetUpdate` (an admin or the owner's key). In the server code
(`dist/services/asset.service.js` `run()`): one `AssetGenerateThumbnails` job per id given — nothing else is queued.

**Path layout:** `<upload>/thumbs/<user id>/<a[0:2]>/<a[2:4]>/<asset id>_thumbnail.webp` and `…_preview.jpeg` (+ one
`.immich` marker file). **Mapping an empty file to its asset:** the first 36 characters of the file name ARE the asset
id. A box finds the broken photos from the disk alone: `find <upload>/thumbs -type f -size 0`, cut the names to 36
characters, unique.

| Step | Result |
|---|---|
| damage | assets 1–5: thumbnail → 0 B; assets 6–10: preview → 0 B; the API served the 0-byte files |
| found from the disk | 10 zero-byte files → 10 asset ids (the mapping above) |
| `POST /api/assets/jobs regenerate-thumbnail` for those 10 | **204**; no zero-byte file left after **1.55 s**; queue idle after 16.9 s (5 s polling); **CPU 5.3 s** (all Immich containers, cgroup `usage_usec`) |
| the 10 assets | all 20 files rewritten (the undamaged sibling too), none empty; the API serves real thumbnails and previews |
| the other 20 assets | **0 of 40 files rewritten** (mtimes before/after) |
| positive control: thumbnails „all" (`force=true`) | rewrote **40 of 40** of those same files (the 41st counted file is the `.immich` marker); CPU 16.5 s for 30 photos |
| teardown | Immich removed through the product; no folder, container, volume or image left; 9202's standing apps untouched |

**Answer:** yes — v3.2.4 rebuilds single photos. Cost ≈ 0.5 s CPU per photo here (a family library with 80 broken photos,
like the night's R2: well under a minute) against „all" in proportion to the whole library. A lost original is still
found only by `integrity-missing-files` and restored only from a backup.

## E — the live proof on demo-hp (agent 0.156.1 + bundle, delivered by signed jobs)

Vouched by the operator 2026-10-14 (read back from `/configuration`): agent 0.156.1 `83c7da27…`, golden 0.310.0 `2110d5e4…`,
min_agent 0.155.0, global floor 0.310.0. demo-hp: `agent_update 0.156.1` (11:00 UTC) → `agent_config_update 0.156.1-step1`
(11:15, „written=1", probe 70/71 — the helper not yet there) → `agent_config_update 0.156.1` (11:30, „BUNDLE WROTE
/usr/local/sbin/felhom-vzdump-partial-clean (new)", probe **71/71**). The new agent's first start (11:00) already FOUND C3's
partial and called it dead — and left it, „the root helper refused: sudo: a password is required" (the helper came with the
bundle). Real sudo after the bundle: `E/sudo-live-demo-hp.txt` (the two shapes allowed, six attacks refused).

**Prediction (before the cut):** after the boot the agent's start-up sweep removes BOTH partials (C3's from 08:44 and this
one), logged „removed a dead half-written backup … written before this boot"; the dead vzdump task is logged once as „worker
process is gone" and the start-up stale-lock pass clears the lock itself (or finds Proxmox already did); the next backup runs
and succeeds; host root returns below its use before this cut.

**Measured so far (11:42 UTC):**

| Step | Result |
|---|---|
| floor 0.310.0 | not delivered: each demo customer (demo-felhom, demo-hp, Tester 1) has its OWN floor 0.309.0 (set 2026-10-12 16:07), which wins over the global one (hub log + `floor_raise_skipped` mail). demo-hp's guest stays on 0.309.0 |
| backup pressed (11:42:33 UTC) | **the pre-backup cleanup worked live:** `removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-10_42_25.tar.dat` (C3's 1.16 GB) and its `.tmp`, through the root helper (`sudo … felhom-vzdump-partial-clean local 9201 …`); each removal took ~7 s (the helper's `/proc` scan); then „space preflight passed" (`E/agent-cleanup-before-backup.txt`) |
| the backup itself | **failed** in 1 s: `lvcreate snapshot 'pve/snap_vm-9201-disk-1_vzdump' error: Failed to suspend thin snapshot origin`; kernel: „device-mapper: thin: Creation of new snapshot 4 of device 2 failed" (`E/failed-backup.log`) |
| why | the thin pool holds an **orphan thin device 4** — a 19 GiB snapshot of 9201's data disk (device 2) created at pool time 138 — which LVM does not know (LVM: devices 1, 2; transaction 559 on both sides). It is what C4's full root disk left: the snapshot cleanup could not write LVM's metadata. Read with a metadata snapshot (`thin_ls --metadata-snap`, reserved and released; `E/thin-ls.txt`) |
| consequence | **no whole-guest backup of 9201 on demo-hp can run until device 4 is deleted** (every snapshot create picks id 4). Caused by this session's C4 test |

**Repair (operator „B2 - A", 12:02 UTC):** `dmsetup message pve-data-tpool 0 "delete 4"` → thin_ls shows devices 1 and 2
only, pool transaction 559 unchanged, LVM agrees (`E/repair.txt`). Identity, not size: device 4 was the one id LVM does not
know, a snapshot (CREATE_TIME 138) of device 2.

**The cut (`E/`):** fences clear 12:04:48; backup pressed through the product 12:04:48 (`POST /api/guest-backup/trigger`,
200); the apps were stopped for the snapshot and started again at 12:05:30 (snapshot create worked — the repair held);
at 1.53 GB written, `plugcut.py --plug hp --off-seconds 10`: **12:06:52.325 off → 12:07:02.409 on** (plug countdown;
watchdog not needed). Root use at the cut 26,249,101,312 B; before the press 24,720,375,808 B. SSH back 12:07:45.

| Prediction | Measured |
|---|---|
| dead task logged once as „worker process is gone" | **yes** — 12:07:44 „backup-in-flight: a vzdump task is listed as active but its worker process is gone — it is dead, not running" (the agent started 12:07:42, inside the 30–60 s window in which Proxmox still lists the dead task — the R-1008 case, now judged right) |
| the start-up stale-lock pass clears the lock, or finds Proxmox did | **both, in the same second:** the agent decided „clearing a stale backup lock" and ran `pct unlock 9201`; Proxmox's start-all had just removed it („removed left over backup lock from '9201'!", 12:07:44) → the agent logged **ERROR „pct unlock failed … no lock found"**. Harmless (the guest started 12:07:47), but an ERROR for an outcome that is the goal — R-1013 |
| the start-up sweep removes the partial | **yes** — 12:07:48 „removed a dead half-written backup … vzdump-lxc-9201-2026_10_14-14_05_21.tar.dat 964,239,360 B (written before this boot)" and 12:07:52 its `.tmp` — through the root helper. 1.53 GB were written before the cut, 0.96 GB survived (the unwritten tail lost, as in C1) |
| guards do not refuse because of the dead task | the start-up sweep IS a `BackupRunning` caller: it went ahead only because the dead task was judged not running. The household power-off request at 12:09:53 (uptime 147 s) was refused **`too_soon_after_boot`** only (`E/requests.txt`). The guest restart is not guarded by `BackupRunning` at all (C3) |
| the next backup succeeds | **yes** — pressed 12:10:06, „found old vzdump snapshot (force removal)" (vzdump removed the cut's LVM snapshot itself), archive `vzdump-lxc-9201-2026_10_14-14_10_50.tar.zst` 5,650,197,172 B at 12:17:08; the older archive pruned (keep-last 1) (`E/next-backup.txt`) |
| host root back below its use before this cut | **yes** — 24,689,217,536 B after the boot sweep; 24,691,134,464 B after the new backup (before the press: 24,720,375,808 B) |
| no orphan thin device this time | **yes** — pool transaction 567, LVM lists only vm-9201-disk-0/1 |

`E/demo-hp-journal-boot.txt` is the whole boot journal (scanned: no key, token or password). R-1006 and R-1008 are proven
live on demo-hp.

**Delivery to the other two boxes (`E/delivery-tester1-n100.txt`):** signed jobs `agent_update 0.156.1` (12:21:24 UTC) →
`agent_config_update 0.156.1-step1` (12:35:37) → `agent_config_update 0.156.1` (12:50:14). Tester 1: agent 0.156.1 committed
12:24, bundle done 12:53 (probe 71/71); N100: agent 12:34 (it reports every 15 min), bundle done 13:04 (probe 71/71). On
both, real `sudo -l -U felhom-agent` allows the cleanup shape and refuses a finished archive (`.tar.zst`). The operator
cleared the per-customer floors (demo-hp and demo-felhom 12:00, Tester 1 12:44 UTC): every guest runs controller 0.310.0.
Both plugs ON: the HP relay back on by its countdown (12:07:02, `E/plugcut.log`), the N100's never cut; both hosts answer.