felhom.eu/REPORT-night-power-2026-10-13.md

# REPORT — night 2026-10-13 → 14: real power cuts on the HP, the estimator's numbers, a night of wall power

Evidence: `documentation/audits/night-power-2026-10-13/README.md` (rounds, mails, power) and
`documentation/audits/sizing-check-2026-10-14/CHECK.md` (the estimator). Times UTC (CEST = UTC + 2).
Architecture read: `14-host-power-off.md`, `05-hub-architecture.md` (staleness, crash, recovery mails),
`07-backup-architecture.md` (the night schedule), `03-host-agent.md` (crash guard).

## Decisions I took (each reversible; each is written where it happened)

- The countdown spike ran on the **HP's** plug inside R1, not on the N100's (the N100 was running).
- Rounds ran **earlier** than the table where the fences allowed (R1, R4, R6, R5 before the 00:30 db-dump); R3 used the
  night's **own** whole-guest backup instead of a product-started one; R2 waited for the night backups to end.
- The 8 GB test ran on **scratch guest 9202**, not 9201; restarted with `pct reboot` (its restart button has no agent link).
- The family set on 4 GB ran **twice** (the first lacked Nextcloud: its kept data refused the install).
- The two over-size backup archives were **removed by hand** after their round was recorded, and a fresh backup was started
  through the product, so demo-hp's next night backup is not blocked.

## 1. Does the HP start by itself after a power cut?

**Yes — every time, 8 cuts in 6 rounds** (R1, R4 ×2, R6, R5 ×2, R3, R2), with the BIOS set to „After Power Loss =
Previous State" by the operator the evening before. Before that setting it did not start (2026-10-13 20:05). The N100's
setting is still unknown. The FAQ promises this setting for every installation; a self-install never gets it (R-1003, P1).

## 2. The rounds

| Round | Cut | Back alone? | To dashboard / all healthy | What broke | Told (operator / household) | True? |
|---|---|---|---|---|---|---|
| R1 + spike | idle, 10 s; the tool killed after the off | yes; the plug's countdown switched on | 2 min 43 s / 3 min 42 s | nothing; each guest volume waits ~43 s in ext4 MMP (R-1004) | crash mail / none | yes |
| R4 | flicker 2 s off, 3 s on, 2 s off | yes | 3 min 0 s / 4 min 12 s | nothing | crash recorded, mail suppressed by cooldown / none | yes (the page has it) |
| R6 | 50 min off | yes | 3 min 5 s / 4 min 15 s | nothing | `node_stale`, `host_stale` at 45 min, then `*_recovered` pairs, crash mail / none | yes; the recovery mail says „stale for 8m" for a 51-min outage (R-1005) |
| R5 | 10 s off, again 40 s after power-on | yes | 2 min 38 s / 3 min 51 s | nothing | as R4 | yes |
| R3 | 10 s off during the night's whole-guest backup | yes; Proxmox removed the stale backup lock | 2 min 46 s / 3 min 57 s | **the 12.2 GB half-backup stayed; every later backup skipped „not enough space"** until removed by hand (R-1006); the agent called the dead backup „in flight" (R-1008) | crash mail, `whole_guest_backup_failed`, later `…_recovered` / none | yes |
| R2 | 10 s off at mid-import of 200 photos | yes | 2 min 37 s / 4 min 16 s | **Immich: 81 of 101 new photos with an empty thumbnail, 1 with no original — shown as done, never repaired** (R-1007) | as R4 / none | the crash yes; Immich's own view no |

Planted files (20, written hours before): intact after every round. Databases repaired themselves every time.

## 3. The N100

It ran all night (the operator had pressed its button at 21:48, so the prompt's „off all night" did not apply). Normal
night: database dumps, Tier 2, Tier 3, whole-guest backup, OS steps — all ran; **no alarm, no mail**. 66 Wh in 7.8 h
(8.5 W average). There is no morning catch-up to do.

## 4. Power

| | Value |
|---|---|
| HP at rest (9201 with its 11 apps + Tester 1's VM + scratch 9202) | 24–27 W (median 24.1 W) |
| HP, three heavy apps at once (R1) | 71 W average, 81.6 W max |
| HP boots after a cut | 72–81 W peak, 2.7–3.4 Wh each, 4 min to all healthy |
| HP backups | Tier 3 restic 43.8 W avg (2.6 Wh); whole-guest vzdump 39–44 W avg, 7–7.5 Wh |
| HP, the test night 22:12–06:00 | 245 Wh (a bench night, not a household night) |
| N100, the night 22:12–06:00 | **66 Wh**, 8.5 W average, 20.6 W max (its night backup: 19.8 W max) |
| The plug's countdown | yes — data point 9, up to 86,400 s; it switched on by itself in every cut, also with the tool killed |

## 5. The estimator check (`audits/sizing-check-2026-10-14/CHECK.md`)

- **Do the numbers add up when apps run together?** Yes, closely. The „Family starter set" at rest: page 2,791 MB,
  HP 3,007 MB (+8 %). Its real peak with three heavy apps busy at once: **4,900 MB against the page's 5,045 + 470** — the
  page's „all at peak at once" is about right, not pessimistic (my prediction of 60–80 % was wrong).
- **The three answers on an 8 GB box** (guest set to the installer's 4,096 MiB, on the nearly empty scratch guest):

| Selection | Page says | Result |
|---|---|---|
| „Just files and passwords" | fits well | **true** — 0 OOM kills, 19 MB swap, upload 28 s |
| Immich alone | fits, but tight | **true** — works, but 14 OOM kills in its ML container and swap full; import 295 s |
| „Family starter set" | does not fit | **true** — real harm: 84 of 200 photo uploads failed, transcode 15× slower, Paperless did nothing in 30 min, swap full, load 151 |

- **More than 30 % off:** Nextcloud peak −45 %, Vaultwarden peak −60 %, Home Assistant peak +32 %, Calibre-Web peak −41 %,
  Grimmory typical +30 %, qBittorrent typical −89 % (idle). 14 of 16 weak figures re-measured at rest agree within ±30 %.
- **Decision for the operator (R-1009):** (A, recommended) a small figure update — Immich, Jellyfin, Home Assistant up;
  Nextcloud, Vaultwarden down; Grimmory typical up. Cost: one data edit and the page's data file rebuilt; no verdict
  changes (the site rule: family 5,515 → 5,659 MB). (B) also one sentence for Immich on 8 GB. **If you do nothing:** the
  page keeps today's figures; its three answers were all true, so no household is misled about fit.

## 6. The countdown fail-safe

**Yes.** The plug has data point 9 `countdown_1` (0–86,400 s). Sent in the same command as the off, it switched the relay
on by itself in every cut, also with the tool killed (R1) and for 50 minutes (R6). The detached watchdog never had to act.

## 7. Claims in the prompt that turned out wrong

| Claim | What is true |
|---|---|
| „The N100 stays off all night (you switched it off with the button)" | it ran: the operator pressed its button at 21:48. Part N became a normal night; there is no morning catch-up (no `MORNING-n100.md`) |
| „Prove the countdown on the N100's plug — the N100 is off, a safe bench" | not possible (it ran); the spike ran on the HP plug as part of R1, with the watchdog armed |
| „HP `standby_w` is not measured" | it was, the evening before: 0–3.2 W (stored in the private plug file) |
| R3: „`lock: backup` can stop the guest from starting" | Proxmox 9 removes a left-over backup lock at start-all; the guest started. The real damage was the half-written archive (R-1006) |
| „all at peak at once: pessimistic on purpose" (`sizing.js`) | the real simultaneous peak was 97 % of the sum |
| Baselines (felhom.eu `95d293b9`, controller `e24430cc`, agent `10ca63aa`) | main had moved: `50bd3f16`, `acbbf4b`, `e7d4ecd` (the power-off session's last commits) |
| „Paperless: the 20 scans" (through its API) | the box's own Paperless has no password the bench knows; the scans went through its consume folder |
| „Add an HP row and a one-night row to `sizing.json`" and „no website change" | they conflict: `sizing.json`'s checksum is in the website's data file. The rows went into `SIZING.md` §4 only |
| „8 GB test on guest 9201" | ran on scratch guest 9202 (9201's own 11 apps would fill 2.6 of the 4 GB); its restart button answered 503 (no agent link by design), so `pct reboot` |

## 8. Records

| Item | Value |
|---|---|
| Commits (felhom.eu) | `13504b90` (plugcut + round table), `8ac7e149` (rounds R1–R6 + R1 set), and the closing commit |
| Register | **127 before, 134 after; opened 7 (R-1003 … R-1009), closed 0** |
| STATUS.md | **1,358 lines → 50**; the dated sections moved byte for byte to `documentation/backlog/STATUS-HISTORY.md` (diff proven) |
| Plugs at the end | HP relay **on** (22.5 W), N100 relay **on** (6.6 W, running) |
| HP guest memory at the end | 9201 `25898` (never changed), 9202 `25898` (4096 from 03:17 to 05:57 for Part R3) |
| Capability map | G18 corrected (kernel lane proven 2026-10-08/10 → PROVEN-LIVE); F5 note: operator recovery leg seen live |
| Instruction files | none needed a fix (`nodes.md`: both `ssh demo-hp` and `ssh hp` work) |
| System poster | no fact changed |
| Teardown | machine: bench apps removed through the product, bench folders, planted files, Paperless bench scans, `/root/sizing` removed; host: the two over-size archives removed by hand (README „Teardown"), crash guard still tripped (re-arms 2026-10-15 03:10); hub: nothing set |
| Fixed without a row | the bench's own scripts only (`plugcut.py` is new; no product code) |