Power-cut fixes E done: a real cut mid-backup on demo-hp, the box cleared up by itself; agent 0.156.1 on all three boxes; R-1006 + R-1008 closed, R-1013 filed

demo-hp 12:06:52 UTC, 1.53 GB written: at boot the dead task was judged dead, the partial and its .tmp
removed by the root helper, the next backup finished (5.65 GB), host root back below its pre-cut use.
Orphan thin device 4 repaired first (operator B2 - A); R-1012 P2 -> P3, detection stays open.
Delivered by signed jobs to Tester 1 and the N100 (probe 71/71, real sudo checked). Controller 0.310.0
everywhere after the operator cleared the per-customer floors. Register 134 -> 134 (opened 4, closed 4).
Report: REPORT-power-cut-fixes-2026-10-14.md. Both plugs ON.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb

documentation/backlog/CLOSED-ITEMS.md

@@ -26,6 +26,13 @@
 
 ---
 
+## 2026-10-14 — a box clears up after a power cut mid-backup
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-1006** | **A power cut during the night's whole-guest backup leaves the half-written archive on the host, and from then on every local whole-guest backup of that box is skipped „not enough space" — nothing removes it** (P2) | CLOSED 2026-10-14 — agent v0.156.1 + config bundle (new root helper `felhom-vzdump-partial-clean`, sudoers `FELHOM_VZPARTIAL`): at start-up and before every backup the agent removes a dead `.tar.dat`/`.vma.dat`/`.tmp` of that guest on a dir storage; the helper re-checks name, storage, any vzdump on the host, open files, and death (before this boot, or quiet 120 s). Proven live on demo-hp with a real power cut mid-backup (12:06:52 UTC): the 0.96 GB partial removed at boot, the next backup finished (5.65 GB), host root back below its pre-cut use. Delivered by signed jobs to demo-hp, Tester 1 and the N100 (probe 71/71 each, real sudo checked). Part 2 (preflight sizing) settled by C4: vzdump removes its own ENOSPC partial; `spaceFits` stays | felhom-agent `v0.156.1` (`7ae3081`); `audits/power-cut-fixes-2026-10-14/` C1–C4, E |
+| **R-1008** | **After a power cut the agent's stale-lock check calls the dead backup „genuinely in-flight" and leaves `lock: backup` on the guest** (P4) | CLOSED 2026-10-14 — agent v0.156.1: a vzdump task is in flight only while its worker process lives (Proxmox's own pid + pstart rule; a zombie or a reused pid is dead; unreadable `/proc` fails safe). Proven live on demo-hp 12:07:44 UTC: „a vzdump task is listed as active but its worker process is gone — it is dead, not running". 0.156.0's wall-clock rule was fail-open under a clock step (security review) — superseded, never vouched. The ERROR the unlock then logged when Proxmox had cleared the lock first is R-1013 | felhom-agent `v0.156.1`; `audits/power-cut-fixes-2026-10-14/E/after-boot.txt` |
+
 ## 2026-10-14 — the FAQ says who sets the BIOS, and how to turn the machine off
 
 | Row | What | Closed | Evidence |

documentation/backlog/OPEN-ITEMS.md

@@ -149,12 +149,11 @@ stopping line that lies.
 | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
 | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
 
-## Backup & restore — 30 rows (P2 5, P3 15, P4 10)
+## Backup & restore — 29 rows (P2 3, P3 16, P4 10)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
-| **R-1012** | Backup & restore | P2 | **A full host root can leave an orphan thin snapshot in the LVM pool that LVM no longer knows; from then on every whole-guest backup of that guest fails at the snapshot step („Creation of new snapshot 4 of device 2 failed"), and nothing in the product finds or names the cause.** Made on demo-hp 2026-10-14 by this session's C4 test (the root filled while vzdump snapshotted; LVM could not write its metadata): `thin_ls --metadata-snap` shows device 4 (19 GiB, a snapshot of 9201's data disk), LVM only 1 and 2 (`audits/power-cut-fixes-2026-10-14/E/thin-ls.txt`). R-1006's cleanup keeps the root from filling; this row is the repair and the detection. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | operator: the repair on demo-hp | Operator: yes/no to `dmsetup message pve-data-tpool 0 "delete 4"` on demo-hp; then CC: a check that compares the pool's thin ids with LVM's and names an orphan in the backup failure | CC |
-| **R-1006** | Backup & restore | P2 | **A power cut during the night's whole-guest backup leaves the half-written archive on the host, and from then on every local whole-guest backup of that box is skipped „not enough space" — nothing removes it.** Measured 2026-10-14 on demo-hp (`audits/night-power-2026-10-13/` R3): the cut at 02:52 left `vzdump-lxc-9201-2026_10_14-04_42_18.tar.dat` (12.2 GB) + `.tmp`; host root at 92 %; agent retry 03:08 „skipped: not enough space: local has 2.9 GiB free; the last archive … was 4.6 GiB". The operator got `whole_guest_backup_failed` (true). Second part: the space preflight sizes the new archive from the LAST one (7.2 GB planned) while the guest had grown (12.4 GB written when cut) — a backup that outgrows the plan can fill the host root. Third part, the same rule the other way: after one big archive (14.4 GB at 05:17, while bench data sat on the guest disk) the next preflight would need ~21 GB of 5.5 GB free — the box skips every later backup though the guest is small again; cleared by hand at 07:28 UTC (`night-power-2026-10-13/README.md`). | **WAITING-ON-OPERATOR** — 2026-10-14: built and released in agent v0.156.1 (0.156.0 superseded); `audits/power-cut-fixes-2026-10-14/` (spike C1–C4, 19 red-proofs); part 2 (the preflight sizing) settled by C4 — vzdump cleans its own ENOSPC partial, `spaceFits` stays | operator: vouch agent 0.156.1 + bundle | CC: after the vouch — deliver (agent → step bundle → bundle) to demo-hp; one real cut mid-backup; prove the partial is gone, the next backup succeeds; then Tester 1 and the N100 | CC |
+| **R-1012** | Backup & restore | P3 | **A full host root can leave an orphan thin snapshot in the LVM pool that LVM no longer knows; from then on every whole-guest backup of that guest fails at the snapshot step („Creation of new snapshot 4 of device 2 failed"), and nothing in the product finds or names the cause.** Made on demo-hp 2026-10-14 by this session's C4 test (the root filled while vzdump snapshotted; LVM could not write its metadata): `thin_ls --metadata-snap` shows device 4 (19 GiB, a snapshot of 9201's data disk), LVM only 1 and 2 (`audits/power-cut-fixes-2026-10-14/E/thin-ls.txt`). R-1006's cleanup keeps the root from filling; this row is the repair and the detection. | **OPEN** — filed 2026-10-14; **repaired on demo-hp 2026-10-14 12:02 UTC** (operator „B2 - A"): `dmsetup message pve-data-tpool 0 "delete 4"`, then thin_ls and LVM agree on devices 1 and 2, and the next backup (12:10, after a real power cut) made its snapshot and finished (`audits/power-cut-fixes-2026-10-14/E/repair.txt`, `E/next-backup.txt`). What is left is the detection. 2026-10-14: P2 → P3 — the cause (a full host root) is what agent v0.156.1's leftover cleanup now prevents | — | CC: a check that compares the pool's thin ids with LVM's and names an orphan in the backup failure | CC |
 | **R-978** | Backup & restore | P3 | **Starting the escrow ceremony again can make the customer's written-down recovery code useless: the new package replaces the old one before the new code is confirmed.** It can happen to an ordinary customer who starts and walks away. Fix: keep the previous package until the new code is confirmed. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix in the agent with a test | CC |
 | **R-973** | Backup & restore | P3 | **A compromised controller can make the whole-guest backup run again and again, and each run prunes older restore points: the local tier directly, and the off-site tier through the store's own nightly keep-last-2.** So `07` §8 row 10 (ransomware) is weaker than it reads. The second reader confirmed the local half; the off-site half is inferred. Fix: a cadence or daily cap on the local-API backup route, and off-site snapshots that a nightly prune cannot remove. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Add the cap with a test; decide the off-site retention shape | CC |
 | **R-926** | Backup & restore | P3 | **After a household removes an `after_install` app keeping its backups and then restores it, nobody has checked whether the app page still shows a password that works.** A `type: password` deploy field is NOT in `stacks.PortableSecretEnvVars` (that list is `type: secret` only, `deploy.go:1199`), so it does not travel in the backup unit — the same shape as R-765, where Radicale's restored login was replaced by a value no page showed and every phone lost its calendar. MEASURED on 9202 2026-10-10 for grocy: remove keeping the backups, restore, and **the household's own password still signs in and the seeded data reads back** — so the DATABASE half is right. What was NOT settled is the PAGE half: `app.yaml` holds the value encrypted at rest, so reading it proves nothing (a 76-character string that of course did not sign in), and `POST /apps/grocy/initial-credentials/reveal` answered 404 for this app. Seven apps are in the same class: bookstack, calibre-web, claper, dawarich, grocy, mealie, wger. | **OPEN — the question is unanswered, not the answer bad** | — | Ask the controller what the app page renders for a generated `type: password` after a restore (one endpoint read, one app), and if it differs from the one that works, decide: carry the value in the unit, or re-run `after_install` after a restore | CC |
@@ -226,7 +225,7 @@ stopping line that lies.
 | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
 | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | — | — | CC |
 | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
-| **R-1008** | Box system & updates | P4 | **After a power cut the agent's stale-lock check calls the dead backup „genuinely in-flight" and leaves `lock: backup` on the guest.** Harmless today only because Proxmox's own start-all removes it in the same second („removed left over backup lock from '9201'!", demo-hp 2026-10-14 02:52:56, `audits/night-power-2026-10-13/R3/lock-lines.txt`). Nothing can be in flight on a fresh boot. | **WAITING-ON-OPERATOR** — 2026-10-14: built in agent v0.156.1 — in flight = the worker process lives (Proxmox's pid + pstart rule), not a wall-clock start (0.156.0's btime rule was fail-open under a clock step — security review); measured: the dead task stays listed only ~30–60 s after a boot (C3), so the live harm is that first minute | operator: vouch agent 0.156.1 | CC: the live proof with R-1006 (the agent's start-up line after the cut) | CC |
+| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
 | **R-1010** | Box system & updates | P3 | **The agent's stale-lock janitor cannot remove a half-removed `vzdump` snapshot: Proxmox's delete fails on the first missing volume („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") and the janitor retries the same call every 10 min forever, leaving the other volume's snapshot and the snapshot entry.** Seen 2026-10-14 after a backup ran out of space on a full host root (LVM could not write its metadata; `audits/power-cut-fixes-2026-10-14/README.md` C4). Fixed by hand with `pct delsnapshot 9201 vzdump --force`. | **READY** — filed 2026-10-14 | — | CC: on that error, retry once with `--force` (a sudoers line or the API's `force=1`), with a test that replays the error text; next agent release | CC |