Power-off live proof complete (R-998, R-999 closed): the hub held 2 h 56 min, the power button brought the N100 back, the hold ended; HP plug + BIOS; R-1002 filed; F11 PROVEN-LIVE
gates / gates (push) Successful in 6m45s
gates / gates (push) Successful in 6m45s
This commit is contained in:
@@ -0,0 +1,80 @@
|
||||
# REPORT — the „turn the machine off" button: spike on the N100, then built and proven (2026-10-13)
|
||||
|
||||
Architecture: `documentation/architecture/14-host-power-off.md` (new), `03-host-agent.md` §6, `13-threat-model.md`,
|
||||
`05` staleness. Evidence: `documentation/audits/poweroff-2026-10-13/README.md` (Part A, Part B, HP). Times UTC.
|
||||
|
||||
## 1. Watts and the restart
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| N100 at rest | 8.5 W mean, 7.2 W median |
|
||||
| N100 at work (Immich, 200 photos) | 20.7 W mean, 24.0 W busiest minute, 25.2 W max |
|
||||
| N100 off | 0.0 W (below the plug's 0.1 W step), 2 h 56 min |
|
||||
| N100 plug off/on | did **not** start (BIOS not read — no monitor) |
|
||||
| N100 power button | Part A: dashboard 200 in 42 s. Part B: dashboard 200 within 50 s |
|
||||
| HP t740 off, plug on | 0–3.2 W |
|
||||
| HP after a power cut | did not start (BIOS „After Power Loss" Off) → operator set „Previous State" → starts by itself, controller healthy 2 min 38 s after power returned |
|
||||
|
||||
## 2. The hub during the hand test (Part A, before the build)
|
||||
|
||||
| Item | Result |
|
||||
|---|---|
|
||||
| 18 min after power-off | hosts page ONLINE, „Last Report 18 min ago"; no event, no mail |
|
||||
| Code reading | without the build: `host_stale` at 45 min, `host_down` at 90 min, and the 05:00 `expected_backup_missed` pair — as Tester 2 gets every day |
|
||||
|
||||
## 3. The button
|
||||
|
||||
| Who | What they see |
|
||||
|---|---|
|
||||
| Household | Beállítások → Rendszer → „A gép kikapcsolása". They type KIKAPCSOLÁS (EN: TURN OFF). Text: to start it again, press the machine's power button. The plug trick is not mentioned. Refusals in plain words: too soon after start, already done this boot, backup, update, drive work, another machine running |
|
||||
| Old restart button | now „A Felhom újraindítása" / „Restart Felhom" (R-999); reboot is POST only |
|
||||
| Hub | event `host_deliberate_off` at once; host page „Power: Kikapcsolva a háztartás által … riasztás szünetel eddig: +14 days"; badge KIKAPCSOLVA once the report is stale (45 min — R-1002); no stale/down/backup/offsite/restore-test/OS alarm for 14 days; OS ring-0 stall is still raised on purpose |
|
||||
| Hub, machine back | `host_deliberate_off_ended` (info) with the first report; staleness back to ok; no `*_recovered`, no crash mail |
|
||||
| Live proof | press 18:52:34 → 202 → 0.0 W 18:53:07 → 2 h 56 min quiet (mailbox checked; positive control: the HP crash mails arrived) → button 21:48:43 → clean boot → hub cleared 21:49:22 |
|
||||
|
||||
## 4. Versions and what waits on the operator
|
||||
|
||||
| Part | Version | Where |
|
||||
|---|---|---|
|
||||
| Agent | v0.155.0 + config bundle | all three boxes by signed job; sudo checked live on each |
|
||||
| Hub | v0.149.0 | live (ArgoCD, hub Deployment only) |
|
||||
| Controller | v0.310.0, MinAgent 0.155.0 | N100 guest 9201 only, by hand |
|
||||
|
||||
| Waiting on the operator | If nothing is done |
|
||||
|---|---|
|
||||
| Golden bake (weekly) | the waiver covers it until 2026-10-20 |
|
||||
| Vouch agent 0.155.0 + bundle | a fresh install gets 0.154.0, without the power-off lines |
|
||||
| Floor 0.310.0 with `min_agent` 0.155.0 | demo-hp and Tester 1 keep 0.309.0: no button there |
|
||||
| N100 BIOS „After Power Loss" | unknown; a deliberately-off N100 stays off after a power cut (looks like „last state") |
|
||||
| HP BIOS: „Bypass F1 on configuration changes" → Enable | a changed BIOS can stop the HP at an F1 prompt after a cut |
|
||||
|
||||
## 5. Claims in the spec that were wrong
|
||||
|
||||
| Claim | What is true |
|
||||
|---|---|
|
||||
| „guestpower.go guards a reboot during a vzdump" | only the watchdog path does; the new route takes the heavy-operation gate itself |
|
||||
| „the plug can be switched; the spike runs without the operator" | the N100 does not start on plug power; the operator pressed the button twice |
|
||||
| A0 „never cut a running machine" as written | needed a measured standby value first (`standby_w`) and a 20 s reading window; the first live `off` refused on 16 s |
|
||||
|
||||
Decisions taken in the session (doc 14): refuse while another guest or VM runs; `os_ring0_stalled` not held; the
|
||||
unaccented KIKAPCSOLAS accepted; the golden waiver renewed to 2026-10-20 to let the hub deploy pass its gate.
|
||||
|
||||
## 6. Commits, register, STATUS
|
||||
|
||||
| Repo | Commits |
|
||||
|---|---|
|
||||
| felhom.eu | `6a1f715a`, `b4dcf4b3`, `750b55e9`, `50e47e29`, `a6acb0f7`, `35e546a9`, `e8c8fc30`, `81e64c6f`, `0544ae0d`, `95d293b9`, and this one |
|
||||
| felhom-agent | `3403c87`, `f38b03c`, `c6d9435`, `bdf25f0`, `68a2130`, `10ca63a`, `e7d4ecd` |
|
||||
| felhom-controller | `201f936`, `e24430c`, `acbbf4b` |
|
||||
|
||||
Register: **125 before, 127 after.** Opened R-998, R-999, R-1000, R-1001, R-1002 (5). Closed R-998, R-999 (2).
|
||||
R-997 was closed by another session in between (−1).
|
||||
|
||||
Fixed without a row: an escrow-ceremony data race (agent); `plugswitch` read window (bench tool); my wrong
|
||||
`os_ring0_stalled` sentence in the hub CHANGELOG and doc 14.
|
||||
|
||||
Teardown: machine — the N100 and the HP run as before, Immich removed in Part A; host — no change left on either host
|
||||
apart from the released agent and bundle; hub — the hold was set and cleared by the product. The bench meters are
|
||||
stopped. Home Assistant on demo-hp stays (Task 1, on purpose).
|
||||
|
||||
System poster: no fact in `felhom-system-poster.facts.md` changed.
|
||||
@@ -2,7 +2,16 @@
|
||||
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
|
||||
|
||||
**Updated 2026-10-12 (evening): demo-hp, demo-felhom and Tester 1 run agent 0.154.0 and controller 0.309.0. The open-items list is at 126 (2026-10-13; the machine estimator was listed and its row R-997 closed). Reports: `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Tuesday 2026-10-13 (night): a household can turn the machine off
|
||||
|
||||
- **Done:** the dashboard has „A gép kikapcsolása" (Beállítások → Rendszer). The household types KIKAPCSOLÁS; the machine shuts down cleanly. The hub shows KIKAPCSOLVA and sends no alarm for 14 days. Tested on the N100: off for 3 hours, no mail; the power button brought it back, and the hub cleared the state by itself (R-998, R-999 closed).
|
||||
- **Waiting on you — vouch** agent 0.155.0 and its bundle. If you do nothing: a fresh install gets 0.154.0, which cannot turn the machine off.
|
||||
- **Waiting on you — raise the floor** to controller 0.310.0 with `min_agent` 0.155.0. If you do nothing: only the N100 has the button.
|
||||
- **Waiting on you — golden bake** at the weekly slot. If you do nothing: the waiver ends 2026-10-20 and pushes stop at the gate.
|
||||
- **Waiting on you — N100 BIOS:** read „After Power Loss" when a monitor is at hand. If you do nothing: a switched-off N100 stays off after a power cut (that is fine); a running one we only infer comes back.
|
||||
- **Waiting on you — HP BIOS:** set „Bypass F1 Prompt on Configuration Changes" to Enable. If you do nothing: after a BIOS change the HP can stop at an F1 prompt instead of starting.
|
||||
|
||||
## Tuesday 2026-10-13 (night): the machine estimator is public
|
||||
|
||||
|
||||
@@ -217,7 +217,7 @@
|
||||
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
||||
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
|
||||
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
|
||||
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PARTIAL** | built and tested; live proof on the N100 pending | `architecture/14-host-power-off.md`; `audits/poweroff-2026-10-13/`; R-998 |
|
||||
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
|
||||
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
|
||||
|
||||
## G. Fleet & operator (hub)
|
||||
|
||||
@@ -189,10 +189,10 @@ the machine's power button brings it back, and the hub clears the state.
|
||||
**Components:** agent v0.155.0 (`POST /host/poweroff`), hub v0.149.0 (`hosts.deliberate_off_at`, the hold), controller
|
||||
v0.310.0 (the card, `POST /api/server/poweroff`).
|
||||
|
||||
**Status:** PARTIAL — built and tested (each guard red-proofed); live proof on the N100 pending.
|
||||
**Status:** PROVEN-LIVE on the N100, 2026-10-13 — button pressed 18:52:34Z, 0.0 W from 18:53:07, hub held 2 h 56 min with no alarm and no mail, power button 21:48:43, clean boot, hub cleared the state 21:49:22. Not yet on the fleet: the floor still delivers controller 0.309.0 (golden bake, vouch and floor wait on the operator).
|
||||
|
||||
**Evidence:** design `14-host-power-off.md`; spike `audits/poweroff-2026-10-13/`.
|
||||
**Evidence:** design `14-host-power-off.md`; spike and live proof `audits/poweroff-2026-10-13/README.md` (Part A, Part B).
|
||||
|
||||
**Gap / roadmap:** the website FAQ line (R-1000).
|
||||
**Gap / roadmap:** the website FAQ line (R-1000); the hosts-page badge reads ONLINE for the first 45 min (R-1002).
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
Tue Oct 13 21:50:00 UTC 2026
|
||||
23:50:00 up 1 min, 1 user, load average: 1.12, 0.43, 0.15
|
||||
felhom-agent 0.155.0
|
||||
"last_boot_at": "2026-10-13T21:48:55Z",
|
||||
"last_boot_unclean": false,
|
||||
"tripped": false,
|
||||
"unclean_boots": [
|
||||
"unclean_boots_24h": 1,
|
||||
"unclean_boots_in_window": 0,
|
||||
VMID Status Lock Name
|
||||
9201 running demo-felhom
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.310.0
|
||||
felhom-controller Up 45 seconds (healthy)
|
||||
opengist Up 46 seconds (healthy)
|
||||
cloudflared Up 46 seconds (healthy)
|
||||
filebrowser Up 46 seconds (healthy)
|
||||
traefik Up 46 seconds
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
Hosts — Felhom Hub Felhom Hub Dashboard Customers Apps Hosts System Offsite Configuration Hosts Host Customer Agent Proxmox / kernel Status Guests CPU Mem Disk Cloudflared Worst Storage Tester-2-be8404 Tester 2 0.142.0 9.2.2 7.0.2-6-pve DOWN 1/1 4% 13% 6% running (connected) 6% local demo-felhom-8363b5 Demo Ügyfél 0.155.0 9.2.21 7.0.14-23-pve KIKAPCSOLVA Kikapcsolva a háztartás által, 2026-10-13 18:52 UTC — riasztás szünetel eddig: 2026-10-27 18:52 UTC 1/2 0% 14% 33% running (connected) 33% local demo-hp-bb76ea Demo HP 0.155.0 9.2.21 7.0.14-23-pve ONLINE 0/2 0% 6% 58% unknown (no running customer guest) 58% local tester-1-d70be4 Tester 1 0.155.0 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 18% 21% unknown (no running customer guest) 21% local Felhom Hub 0.149.0
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
press 2026-10-13T18:52:34Z
|
||||
POST /api/server/poweroff -> 202 {'data': {'powering_off': True}, 'ok': True}
|
||||
no_ping 2026-10-13T18:53:02Z
|
||||
@@ -0,0 +1,5 @@
|
||||
ALLOWED : /usr/bin/systemctl poweroff
|
||||
ALLOWED : /usr/sbin/pct shutdown 9201 --timeout 120
|
||||
refused : /usr/bin/systemctl poweroff --force
|
||||
refused : /usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1
|
||||
refused : /usr/bin/systemctl reboot
|
||||
@@ -0,0 +1,8 @@
|
||||
ALLOWED : /usr/bin/systemctl poweroff
|
||||
ALLOWED : /usr/sbin/pct shutdown 9201 --timeout 120
|
||||
refused : /usr/bin/systemctl poweroff --force
|
||||
refused : /usr/bin/systemctl poweroff -ff
|
||||
refused : /usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1
|
||||
refused : /usr/sbin/pct shutdown 9201
|
||||
refused : /usr/bin/systemctl reboot
|
||||
refused : /usr/bin/systemctl halt
|
||||
@@ -0,0 +1,6 @@
|
||||
felhom-agent 0.155.0
|
||||
ALLOWED : /usr/bin/systemctl poweroff
|
||||
ALLOWED : /usr/sbin/pct shutdown 9201 --timeout 120
|
||||
refused : /usr/bin/systemctl poweroff --force
|
||||
refused : /usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1
|
||||
refused : /usr/bin/systemctl reboot
|
||||
@@ -0,0 +1,8 @@
|
||||
20:04:49
|
||||
22:04:49 up 39 min, 1 user, load average: 0.91, 0.71, 0.74
|
||||
|
||||
341 night1004-tester1 running 8192 200.00 1827
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
{'last_boot_at': '2026-10-13T19:25:35Z', 'last_boot_unclean': False, 'unclean_boots_24h': 0, 'tripped': False}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,10 @@
|
||||
2026-10-13T21:00:39.092Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:00:39.193Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:13:49.897Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:13:50.014Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:22:04.745Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:27:46.025Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:35:53.483Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:41:46.819Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:45:11.332Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:45:11.413Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
@@ -0,0 +1,3 @@
|
||||
20:05:06 CUT: plug off (operator-ordered outage test)
|
||||
20:05:21 RESTORE: plug on
|
||||
20:05:21 switch on = True
|
||||
@@ -0,0 +1 @@
|
||||
{'last_boot_at': '2026-10-13T20:10:53Z', 'last_boot_unclean': True, 'unclean_boots_24h': 1, 'unclean_boots_in_window': 1, 'tripped': False}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,4 @@
|
||||
2026-10-13T20:46:31.792Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T20:46:31.922Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:19:55.520Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
2026-10-13T21:27:46.028Z plugmeter: read failed (RuntimeError: tuya error 914 Check device key or version)
|
||||
@@ -0,0 +1,11 @@
|
||||
20:13:16 CUT: plug off
|
||||
20:13:31 RESTORE: plug on
|
||||
20:13:31 switch on = True
|
||||
20:14:09 HP answers ping (+38s after power back)
|
||||
20:16:09 controller healthy
|
||||
341 night1004-tester1 running 8192
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
containers: 22 running, 0 unhealthy
|
||||
{'last_boot_at': '2026-10-13T20:13:54Z', 'last_boot_unclean': True, 'unclean_boots_in_window': 2, 'tripped': True}
|
||||
@@ -0,0 +1,5 @@
|
||||
19:19:14 qm shutdown 341 rc=0 took 113s -> status: stopped
|
||||
19:20:54 pct shutdown 9202 rc=0 took 99s -> status: stopped
|
||||
19:22:39 pct shutdown 9201 rc=0 took 104s -> status: stopped
|
||||
19:22:45 host poweroff sent
|
||||
19:23:10 no ping from demo-hp
|
||||
@@ -32,3 +32,39 @@ the hub page captures is replaced by `<household e-mail>`.
|
||||
(hub `host_crash_restart`). Consistent with the BIOS „after power loss = last state": a machine that was off stays off.
|
||||
Not read in the BIOS (no monitor) — an inference. That is the behaviour the household text promises („press the power
|
||||
button"), so the spike passes for the button; the plug cannot restart a deliberately-off N100 on the bench.
|
||||
|
||||
# Part B — the button, proven live on the N100, 2026-10-13
|
||||
|
||||
Released: agent v0.155.0 + config bundle (signed jobs), hub v0.149.0 (ArgoCD), controller v0.310.0 (hand-deployed on
|
||||
guest 9201 only — the floor is the operator's). Method: endpoint-level, the exact `POST /api/server/poweroff` the
|
||||
dashboard card calls (no browser on DooPlex). Hub read through the operator UI; mail read in the operator mailbox.
|
||||
|
||||
| Step | What happened | File |
|
||||
|---|---|---|
|
||||
| sudo on the box | real `sudo -l` checks on the N100: the 2 new lines allowed (`pct shutdown <id> --timeout <n>`, `systemctl poweroff`); 6 attack lines refused (`--force`, `-ff`, `--forceStop 1`, no timeout, `reboot`, `halt`) | `B-sudo-live-n100.txt` |
|
||||
| sudo on the fleet | the same checks on demo-hp and Tester 1 (agent 0.155.0, bundle installed): the 2 lines allowed; `--force`, `--forceStop 1`, `reboot` refused | `B-sudo-live-demo-hp.txt`, `B-sudo-live-tester-1.txt` |
|
||||
| refusals | no confirm word / a wrong word → 400; `GET /api/server/reboot` → 405 | session log |
|
||||
| press | 18:52:34Z → **202** `powering_off: true`; no ping 18:53:02 (28 s); **0.0 W from 18:53:07** | `B-press.log`, `B-power.csv` |
|
||||
| hub at once | event `host_deliberate_off` (info) 18:52; host page „Power: Kikapcsolva a háztartás által, 18:52 UTC — riasztás szünetel eddig: 2026-10-27 18:52 UTC" from 18:53 | `B-hub-watch.log` |
|
||||
| hub at 33 min | controller staleness `ok → deliberately_off (household power-off, no event)` 19:25 | hub log |
|
||||
| hub at 46 min | hosts badge **KIKAPCSOLVA** from 19:38 (before that: ONLINE plus the Power line — R-1002) | `B-hub-watch.log` |
|
||||
| hub at 92–95 min | no `host_stale`, `host_down`, `node_stale`, `node_down` event; no notification in the customer's last 10 | `B-hub-*-at-9xmin.txt` |
|
||||
| mailbox | no demo-felhom mail after 13:58 for the whole 2 h 56 min. Positive control: the HP power cut's `host_crash_restart` / `host_crash_guard_tripped` mails arrived 20:29 | operator mailbox |
|
||||
| off | **0.0 W** for 2 h 56 min, 2346 readings, max 0.0 | `B-power.csv` |
|
||||
| power button | operator pressed it: first watts 21:48:43 → boot 21:48:55 → **dashboard 200 by 21:49:33** (first check, ≤ 50 s) | `B-power.csv`, `B-after.txt` |
|
||||
| hub clears | 21:49:22 `deliberate power-off ended (was 18:52:28Z)` → event `host_deliberate_off_ended` (info); 21:50 staleness `deliberately_off → ok`; badge ONLINE | hub log, `B-hub-watch.log` |
|
||||
| after | crash guard `last_boot_unclean: false` (no crash mail, correct); guest 9201 running; controller 0.310.0; the same 5 containers healthy; no `*_recovered` mail | `B-after.txt` |
|
||||
| next report | the first report after boot (21:49:22) was sent before the guest started: „Guests 0/2", cloudflared „unknown"; the next one (22:04:27, 15 min later) read 1/2 and „running (connected)" | hub log |
|
||||
|
||||
# HP t740 (demo-hp) on the second plug — operator-ordered, 2026-10-13
|
||||
|
||||
| Step | What happened | File |
|
||||
|---|---|---|
|
||||
| clean shutdown by hand | VM 341 113 s, guest 9202 99 s, guest 9201 104 s, then `poweroff`; no ping 19:23:10 | `HP-poweroff.log` |
|
||||
| off, plug on | **0.0–3.2 W** (flickers; NIC/BIOS standby) → `standby_w` 3.2 in the private plug file | `HP-cut-power.csv` |
|
||||
| cut 1 (20:05:06, 15 s) | the HP did **not** start: BIOS „After Power Loss" was Off. Operator set „Previous State" | `HP-cut.log` |
|
||||
| cut 2 (20:13:16, 15 s) | started by itself: ping +38 s, controller healthy 20:16:09, all 3 guests running, 22 containers, 0 unhealthy | `HP-cut2*.{log,txt}` |
|
||||
| crash guard | 2 unclean boots in 60 min → tripped, `kernel.panic=0`, re-arms in 24 h — the guard working, not a defect; mails at 20:29 | `HP-cut2.log` |
|
||||
| idle watts | running, 3 guests: mean 25.2 W, median 23.0 W, max 45.5 W | `HP-cut-power.csv` |
|
||||
| Tester 1 (VM 341 on the HP) | the cut killed it too: `host_crash_restart` + `host_crash_guard_tripped` for `tester-1-d70be4`, mails 20:29 | operator mailbox |
|
||||
| meter note | from 20:13 two `plugmeter --plug hp` readers ran at once (the first was not stopped), so `HP-cut-power.csv` and `HP-cut2-power.csv` overlap and hold near-duplicate rows; the figures above are from one file | — |
|
||||
|
||||
@@ -26,6 +26,15 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-13 (night) — the household can turn the machine off
|
||||
|
||||
Built and proven live on the N100 (`documentation/audits/poweroff-2026-10-13/`, architecture `14-host-power-off.md`). Fleet delivery waits on the operator: golden bake, vouch agent 0.155.0 + bundle, floor 0.310.0 with min_agent 0.155.0.
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-998** | **Nothing can turn a Felhom machine off: a household that moves house or goes away can only pull the plug** (P3) | CLOSED 2026-10-13 — agent v0.155.0 `POST /host/poweroff` (felhom-agent `3403c87`..`68a2130`), hub v0.149.0 14-day hold (`50e47e29`, `a6acb0f7`, deployed `35e546a9`), controller v0.310.0 „A gép kikapcsolása" (felhom-controller `201f936`). Live on the N100: press 18:52:34Z → 202 → 0.0 W from 18:53:07; hub `host_deliberate_off`, KIKAPCSOLVA, no alarm and no mail for 2 h 56 min; power button 21:48:43 → clean boot (crash guard `last_boot_unclean: false`), dashboard 200 by 21:49:33, hub `host_deliberate_off_ended` 21:49:22, staleness back to ok 21:50. **Reasoning kept:** the agent refuses while another guest or VM runs (it would cut someone else's machine); `os_ring0_stalled` is NOT held — a box off for 14 days still needs its OS updates. | `audits/poweroff-2026-10-13/B-*`, full text `git show 750b55e9:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-999** | **„Kiszolgáló újraindítása" says it restarts „the whole server", but it restarts only the Felhom part (the guest)** (P3) | CLOSED 2026-10-13 — controller v0.310.0: „A Felhom újraindítása" / „Restart Felhom", and the reboot route is POST only (GET → 405) | felhom-controller `201f936`; full text `git show 750b55e9:documentation/backlog/OPEN-ITEMS.md` |
|
||||
|
||||
## 2026-10-13 (night) — the machine estimator is listed
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -209,12 +209,10 @@ stopping line that lies.
|
||||
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
|
||||
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
|
||||
|
||||
## Box system & updates — 10 rows (P2 1, P3 8, P4 1)
|
||||
## Box system & updates — 8 rows (P2 1, P3 6, P4 1)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-999** | Box system & updates | P3 | **„Kiszolgáló újraindítása" says it restarts „the whole server", but it restarts only the Felhom part (the guest)** (`settings_system.html`, controller hu/en locale). Found 2026-10-13 reading `HandleServerReboot` → agent `pct reboot`. | **OPEN** — filed 2026-10-13, being built | — | CC: honest strings in the power-off release | CC |
|
||||
| **R-998** | Box system & updates | P3 | **Nothing can turn a Felhom machine off: a household that moves house or goes away can only pull the plug** (spike 2026-10-13 on the N100, `audits/poweroff-2026-10-13/`). Building: agent `POST /host/poweroff`, hub „deliberately off" state (14 days without alarms), controller button „A gép kikapcsolása". | **OPEN** — filed 2026-10-13, being built | — | CC: build, release, live proof on the N100 | CC |
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
| **R-982** | Box system & updates | P3 | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: `audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt`. | OPEN — design-proposal | — | Decide; then set it in the installer and the config bundle | CC |
|
||||
| **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
|
||||
@@ -236,7 +234,7 @@ stopping line that lies.
|
||||
| **R-724** | Monitoring & notifications | P3 | **[P3-LOW] The status pages disagree with each other in small ways a household notices.** MEASURED 2026-09-29 on a fresh box: Beállítások reads „Mentés ütemezés 02:30 / 03:00" while Biztonsági mentés says 02:30 / 03:30 / 04:15 / 04:30–08:30; Beállítások shows a raw `2026-09-29T19:22:30Z`; its „Helyi cím (LAN)" and „Átjáró" read „nem állapítható meg" on the page the household is asked to read out for remote help; the backups overview still says „Következő mentés — 0 órája" (the age of the LAST run, under the word "next" — noted 2026-09-14, never filed); the dashboard shows the backup at 19:33 where the backup pages say 21:33 (R-500, still reproducing). **FIXED 2026-09-30 (controller v0.283.0):** Beállítások shows the real schedule and a local time; the dashboard's last backup is local time (R-500); „az előző N órája készült" replaces „0 órája". **NARROWED — remaining:** „Helyi cím (LAN)" / „Átjáró" read through the file-sharing container, so a box without it says „nem állapítható meg" — not a text fix (the read needs another path). | **NARROWED — the LAN/gateway read only; owner: CC** **2026-10-06 night: confirmed not a controller-only fix** — the only host-network container is samba (`infra/samba.go:123`) and the agent's local API has no guest-network route; the fix is a new agent route (e.g. the PVE lxc interfaces read) + a controller reader — a two-repo design. | — | — | CC |
|
||||
| **R-285** | Monitoring & notifications | P4 | **A planned, supervised reinstall pages the operator as if the machine had died — there is no notion of expected downtime anywhere.** During the 2026-08-09 rehearsal the hub sent, all `status: sent` to the operator channel: `host_stale` 08:58 UTC, `node_stale` 09:00, **`host_down` 09:28 (error)**, **`node_down` 09:30 (error)**, `host_leaf_changed` 09:31, `host_recovered` 09:31, `node_recovered` 09:34, `offsite_delivery_stuck` 09:34 — eight operator mails for work that was deliberate, attended and announced. **This is the OPPOSITE gap from the one R-281 filed:** the alarms are not missing, they are indiscriminate. `host_stale` at 30 min and `host_down` at 60 min (`monitor/host_staleness.go:22-23`, `downAfter = 2 * threshold`) cannot distinguish a wiped-on-purpose box from a dead one, and `host_leaf_changed` firing on a reinstall is correct-but-expected. **Note the interaction with the mute used on 2026-08-09 evening:** blocking a customer silences everything, so today the only two settings are *page me for planned work* and *tell me nothing at all*. **What is owed is a middle:** a maintenance window, or an operator-set expected-downtime flag, that suppresses staleness and leaf-change while leaving genuine faults audible | **READY (M) — NEW 2026-08-09** | — | The evidence is the operator's mailbox plus `events`/`notification_log` for 2026-08-09 | CC |
|
||||
|
||||
## Hub & operator — 10 rows (P3 6, P4 4)
|
||||
## Hub & operator — 11 rows (P3 6, P4 5)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -250,6 +248,7 @@ stopping line that lies.
|
||||
| **R-451** | Hub & operator | P4 | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **DEFERRED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the fleet sweep (report field, hub denormalisation, fleet page) is ruled and not built — deferred until the fleet grows (`09` §6.3)) — **RULED — build deferred until the fleet grows; owner: CC** | — | — | CC |
|
||||
| **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
||||
| **R-844** | Hub & operator | P4 | **The household's OS-update line exists only on the hub's customer timeline.** 2026-10-04: the box itself has no event surface for agent results (the controller UI shows no timeline), so `os_update_applied` is a hub customer event (info: recorded, never mailed). Its stored text is the hub's English sentence; the hu/en bundle text (`mail.event.os_update_applied`) is used only if it is ever mailed. Fix direction: a controller-side line (the controller already polls the agent's local API) when the box gets a household timeline. `audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A DESIGN** — a household timeline on the box does not exist yet. | — | — | CC |
|
||||
| **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
|
||||
|
||||
## Business & legal — 13 rows (P2 4, P3 2, P4 7)
|
||||
|
||||
|
||||
@@ -194,6 +194,11 @@ with `tinytuya scan` if it stops answering). Read it from DooPlex with
|
||||
installed on purpose**: it reads the plug through Tuya's cloud and keeps its history. Keys live only in DooPlex
|
||||
`~/.config/felhom-bench/`. `audits/plugmeter-2026-10-13/`.
|
||||
|
||||
**demo-hp is on a second metered plug** (LAN `192.168.0.107`; `--plug hp`). Off with the plug on it draws 0–3.2 W.
|
||||
**After a power cut:** demo-hp starts by itself (BIOS „After Power Loss = Previous State", set 2026-10-13); the N100 does
|
||||
NOT start after a plug off/on when it was switched off first — BIOS not read yet (no monitor). `plugswitch.py off` refuses
|
||||
a running machine. `audits/poweroff-2026-10-13/`.
|
||||
|
||||
## Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION
|
||||
|
||||
> **Read this before any product-shape audit.** `demo-hp` is **customer-shaped** — it is a normal
|
||||
|
||||
Reference in New Issue
Block a user