Bench: plugcut.py — a real power cut on the HP with three ways back on; the night's round table written before round 1

2026-10-14 00:24:18 +02:00
parent 50bd3f168b
commit 13504b90de
6 changed files with 359 additions and 0 deletions
@@ -0,0 +1,49 @@
+# Night 2026-10-13 → 14 — real power cuts on demo-hp, the estimator check, a night of wall power
+
+Times are **UTC** (the boxes' schedules are in CEST = UTC+2; both are given where it matters). Tier 0 boxes only
+(`runbooks/target-selection.md`): demo-hp and Tester 1 (VM 341 on it) are cut; the N100 is read only; DooPlex runs
+the plug tools and reads the hub. Tool: `scripts/bench/plugcut.py` (new). No key, device id or token in this folder.
+
+## Facts read before round 1 (22:10–22:40)
+
+| Item | Value |
+|---|---|
+| N100 | **ON** — the operator pressed its power button at 21:48:43 (the prompt assumed it stays off). Read only tonight |
+| HP host | AMD Ryzen Embedded V1756B, 29,994 MiB RAM, swap 8 GiB; guests 9201 (`memory: 25898`), 9202 (25898), VM 341 Tester 1 (8192); 9401 stopped |
+| HP guest 9201 | 19 containers; 11 apps deployed (Home Assistant and Paperless among them; `paperless-ngx` and `romm` have no running container) |
+| HP crash guard | `armed: false`, `kernel.panic 0` — tripped by today's 20:13 cut, re-arms 24 h later |
+| Plug countdown | both plugs have data point 9 `countdown_1` (0–86400 s) and DP 38 `relay_status` = `memory` |
+| SSH alias | both `ssh demo-hp` and `ssh hp` reach 192.168.0.104 from DooPlex; `nodes.md` is right |
+
+**The HP's night schedule** (controller log 20:15:57 + agent journal):
+
+| CEST | UTC | Job | Talks off-box? |
+|---|---|---|---|
+| 02:30 | 00:30 | db-dump (Tier 1) | no |
+| 03:30 | 01:30 | tier2-backup, fill-watch; ep0 prunes households (DooPlex timeline) | ep0 prune — fenced |
+| 04:15 | 02:15 | offbox-backup (Tier 3, restic → Storage Box) | **yes — fenced** |
+| ~04:33–04:44 | ~02:33–02:44 | whole-guest vzdump, local (last 3 nights: 04:33, 04:36, 04:38 start, ~5.5 min) | no |
+| ~04:45 | ~02:45 | OS step (`osupdate` guest + host, ring 0) | **fenced** |
+| 05:10 / 05:30 / 06:00 | 03:10 / 03:30 / 04:00 | offsite abandon-sweep / proof / integrity | Storage Box — checked live before a cut |
+| weekly | — | PBS whole-guest → ep0: HP last 2026-10-08 20:34, Tester 1 last 2026-10-11 20:02 (cadence 168 h) — **not due tonight** | — |
+
+Fences before every cut (§9 rule 3): no `vzdump`, `proxmox-backup-client`, `restic`, `felhom-os-apply`, `apt`/`dpkg`
+process on the HP host, guest 9201 or Tester 1; no scheduled start within 10 min; ≥ 10 min after the last full recovery.
+
+## The round table (written 22:45, before round 1)
+
+| Round | Planned UTC | The cut | Prediction |
+|---|---|---|---|
+| A + R1 | 23:00 | idle, 10 s, the countdown spike on the HP plug (the N100 is on: its plug is not a safe bench) | relay back on by the plug countdown; HP starts (BIOS „Previous State", set 20:10 today) in ≤ 40 s to ping; guests in 2–3 min; crash guard `last_boot_unclean: true`; hub `host_crash_restart` mail for demo-hp and Tester 1; no stale/down (well under 45 min) |
+| R4 | 23:25 | flicker: off 2 s, on 3 s, off 2 s, on | as R1. Risk: the BIOS may treat the second off as a stop during POST and stay off |
+| R6 | 23:50 | 50 min off (`--long`) | `host_stale` 30–45 min after the last report, for demo-hp AND Tester 1 (and `node_stale` from the controllers); at 50 min no `host_down` yet (90 min); on return: paired `*_recovered` mails + `host_crash_restart`; the 00:30 db-dump is missed while off and caught up after the start |
+| R5 | 01:00 | 10 s off, 40 s after a power-on | the box is in POST/kernel start; it starts again; nothing is damaged (no guest was running) |
+| — | 01:20–03:05 | **Part R1/R2 measurement** (no cut): the family set, rest, actions together, rest, across the 01:30 Tier 2, 02:15 Tier 3 and 02:33 vzdump | — |
+| R2 | 03:15 | 10 s off at mid-import of a second 200-photo Immich upload | Immich DB recovers (postgres WAL replay); the half-import is not shown as done; the controller marks no app done that is not |
+| R3 | 03:45 | 10 s off during a whole-guest backup started through the product | a `lock: backup` may stay in `pct config 9201` and stop the guest starting (the July finding); predict: the guest **does not** start by itself |
+| — | 04:15–05:45 | Part R3: the 8 GB box test | — |
+
+A round may move later; none is dropped without a line here saying why.
+
+## Rounds (filled as they run)
+
@@ -0,0 +1,20 @@
+14189638d73b22ca70e52ed9c9268ee42e905db1b56968cb81ca2cdd9b8452b8  plant-01.bin
+78c98745537aa24664ae74361fd3f20309f2934162b58b041e517836bcc417d6  plant-02.bin
+badddd7dad4214cd6207cb29cfde3edc4e4baab3f63ea23f0c7520a8026992f9  plant-03.bin
+08ee138f7a6446b8ffee5aed1198921ac3bb016d28d23fd92b4fa4f3c9631269  plant-04.bin
+ede81bd0d1a89edd367f39bd71345a67178ce49d661b41cf558283da8c6d342f  plant-05.bin
+83d6d2fbd0c8db497216aeddda66c0383d0b9f0f5d527e971f18e4f8f38a96f9  plant-06.bin
+dbaf2e9ce71cbad9c03208175e9ea582cfe8cdc72da4600301b3dc353a829a7f  plant-07.bin
+24066a276fb71410c1880be8e6d200c949efa55fd7699a70cf4cf0829c66f545  plant-08.bin
+42612af9c55c9a33c131b1761c4cde0a97e41cc48f78c118f5249ae5cf87ec46  plant-09.bin
+0aabce9d5e76b23626b5b04c3d35731568d8cc965e87006d4458e505d14c83dc  plant-10.bin
+ded0a457507c935a86d14abcb603b81747139fe107657f7a21ab8f2293747dd6  plant-11.bin
+5ac05274d9f4cd0daebc62a02df0fc76045fb722a309a702a6a7874ec7a34ec5  plant-12.bin
+6593d15aa3e580970660fc406cbfb7f4116499ba6d0f0495d6dab88c73238ce8  plant-13.bin
+b1810d59dc548600c23f49e8ec56fc121f6d7ba67a4acaea4db1380f85cd70c8  plant-14.bin
+8197f84a48f001fc1b056e96ca9bd7302d7bee232b449903486db7406e30203d  plant-15.bin
+32436352885d939d451d72cecf358558d2696cf25f5685d28686ce87029121ed  plant-16.bin
+b2556174851fff1c3c21cb8c28a14887d3459e33f96f894aa85882a402da83ab  plant-17.bin
+dd0ceedb353679e6d61876c4898477f29362e3d1fb3707ff67e78921ab03aac4  plant-18.bin
+6c842b657602840f3e866971007feb00a16378838bebb37f04d4dcf826df6247  plant-19.bin
+c088602f92943929018670b76607a77f621ab1c1997c022577a371834a4d4d45  plant-20.bin