From ecfc09c3efa829883d21c6979685c3022bc0a87e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 22 Oct 2026 13:17:17 +0200 Subject: [PATCH] day 2026-10-22: R-1199 reopened and closed (hub 0.165.0 + controller 0.339.0; demo-felhom's list restored), R-1216 closed (D28 = A), R-1217 filed; Tester 2 hand-over list; STATUS, CONTEXT, D28 to history; rows 178 -> 178 --- CONTEXT.md | 13 ++++ REPORT-day-2026-10-22.md | 70 +++++++++++++++++++ STATUS.md | 10 +-- .../day-2026-10-22/01-prefs-before-deploy.txt | 13 ++++ .../day-2026-10-22/02-restore-demo-felhom.txt | 1 + .../day-2026-10-22/03-prefs-after-restore.txt | 7 ++ .../day-2026-10-22/04-release-0339-boxes.txt | 11 +++ .../audits/day-2026-10-22/05-https-only.txt | 33 +++++++++ .../day-2026-10-22/06-prefs-after-release.txt | 18 +++++ documentation/backlog/CLOSED-ITEMS.md | 8 ++- documentation/backlog/OPEN-ITEMS.md | 6 +- documentation/backlog/STATUS-HISTORY.md | 4 ++ documentation/pilot/TESTER2-handover.md | 61 ++++++++++++++++ 13 files changed, 244 insertions(+), 11 deletions(-) create mode 100644 REPORT-day-2026-10-22.md create mode 100644 documentation/audits/day-2026-10-22/01-prefs-before-deploy.txt create mode 100644 documentation/audits/day-2026-10-22/02-restore-demo-felhom.txt create mode 100644 documentation/audits/day-2026-10-22/03-prefs-after-restore.txt create mode 100644 documentation/audits/day-2026-10-22/04-release-0339-boxes.txt create mode 100644 documentation/audits/day-2026-10-22/05-https-only.txt create mode 100644 documentation/audits/day-2026-10-22/06-prefs-after-release.txt create mode 100644 documentation/pilot/TESTER2-handover.md diff --git a/CONTEXT.md b/CONTEXT.md index c4aa470d..1d3372fb 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,19 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-22 day — operator rulings (11:42) and what was built.** Rulings: **D28 = A** (loosen the host-swap check); +> agent 0.162.0 **vouched** (read back: `artifact_agent_version` 0.162.0 at 07:57 UTC); Tester 2's laptop was installed +> with **our installer**; Tester 2 is off, no date. Built: **R-1199 reopened** — controller 0.338.0's start-up default +> push overwrote demo-felhom's own saved mail list (13 → 17 events, 07:54:55 UTC); hub **0.165.0** marks a list the +> household saved (`household_saved`, backfilled once: Tester-2, Tester4 false; the other four true) and never lets a +> default overwrite it; it answers a box's self-scoped read; controller **0.339.0** reads the hub's list when it has +> none, adopts a household's, pushes its default once only when no household saved one. demo-felhom's list restored +> from the hub's 00:00 UTC snapshot through the hub's own save route; all six rows equal the snapshot after. Found and +> fixed with it: an empty saved list read back as the 17 defaults (`omitempty`; Tester 1 and demo-hp hold that shape), +> and the add-only seeds ran on a new box's second start. **R-1216 (D28 = A):** short host swap = none, or below 90 % +> of min(guests' swap, 8192 MB) — the installer's maximum (`Install.pm compute_swapsize`); one case table in hub and +> controller, `scripts/host_swap_rule_gate.py`. Tester 2 hand-over: `documentation/pilot/TESTER2-handover.md`. + > **2026-10-22 morning — the operator's five yeses, carried out.** Controller **0.338.0** released (image > `sha256:e8a529f3…`, MinAgent 0.158.1): Tester 1, demo-hp, demo-felhom through per-customer floors, checks green > (https-only 28/28, 0 unhealthy, 0 errors), then the global floor 0.338.0 (read back with the WAL). Agent **0.162.0** diff --git a/REPORT-day-2026-10-22.md b/REPORT-day-2026-10-22.md new file mode 100644 index 00000000..a07b214d --- /dev/null +++ b/REPORT-day-2026-10-22.md @@ -0,0 +1,70 @@ +# REPORT — the day of 2026-10-22: the family's mail list, the swap check (D28 = A), the Tester 2 hand-over + +**Operator rulings (11:42):** D28 = A; agent 0.162.0 vouched (read back: hub `artifact_agent_version` 0.162.0, 07:57 +UTC); Tester 2's laptop used our installer; Tester 2 off, no date. Design read: `08-alarm-ladder.md`, +`05-hub-architecture.md`, `03-host-agent.md`. Baselines: controller `08fa440d`, agent `2d5840ba`, felhom.eu `a9283741`. + +## Four lines + +1. **Mail list fix live** on the hub (0.165.0) and on Tester 1, demo-hp and demo-felhom (controller 0.339.0, global + floor). A box with no list of its own now adopts the family's list from the hub; a default never overwrites it. +2. **One family list restored: demo-felhom** (13 events, from the hub's 00:00 UTC snapshot; this morning's release had + made it 17). After the release demo-felhom adopted it; all six hub rows equal the snapshot. +3. **Swap warning rule:** none, or below 90 % of min(the guests' swap, 8 GB). The two morning alarms cleared at the next + report (demo-hp 11:04, demo-felhom 11:09 UTC), no mail. No agent change. +4. **Hand-over list:** `documentation/pilot/TESTER2-handover.md`, linked from STATUS. + +## Versions, digests, CI + +| | version | digest | CI (commit → job) | +|---|---|---|---| +| hub | 0.165.0, DEPLOYED (operator yes; only `Deployment/hub` synced, manifest `25dae2f5`) | `sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537` | `5ae5984c` → 2207 success; `25dae2f5` → 2208 success | +| controller | 0.339.0-rc.1 (9202) | `sha256:7d4c88528b50adeb891e409cf50343f44fb168a84e890d42864b8124f3051fdd` | `add310c` → 2206 success | +| controller | 0.339.0, released, global floor (MinAgent 0.158.1) | `sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8` | `fc5a20f` → 2209 success | + +Agent: no change. Evidence: `documentation/audits/day-2026-10-22/` (rows before, the restore, rows after, the release +on the three boxes, https-only). + +## What was built + +- **Hub:** `customer_notifications.household_saved` — set only by the household's own save (`source: household`), + never cleared; any other push over a marked list → `{"status":"kept"}` (an older controller's push with neither + address nor clear counts as not the household's — that is 0.338.0's default push); `GET /api/v1/preferences` + (customer-scoped); one-time backfill (written after creation → a box wrote it): Tester-2 false, Tester4 false, + demo-felhom / demo-hp / tester-1 / `_resend-rotation-test` true. `host_swap_short` by the new rule. +- **Controller:** start-up — a saved list as before (R-922); none saved → read the hub, adopt a household's, else push + the default once and save it; unreadable hub → nothing. The swap sentence by the same rule. +- **Contract:** one case table in both repos, byte-identical, each run by a Go test; `scripts/host_swap_rule_gate.py` + with 4 decoys (`test_host_swap_rule_gate.py`). +- **Red-proofs seen failing:** hub A / A-legacy / C (refusal removed), the swap table and the clear (morning rule), the + backfill (marking off); controller A / B / unreadable hub (0.338.0's rule), the empty-list test (getter restored), + the swap table (morning rule). + +## Rows + +Before 178, after 178. Opened 2: R-1199 (reopened), R-1217 (demo-hp's hub list has 3 app events its page does not +show — P4). Closed 2: R-1199, R-1216. + +## Observations + +- An empty saved list read back as the 17 defaults (Tester 1, demo-hp) — **NOT-A-FINDING: fixed in this session** + (size rule; controller CHANGELOG „Fixed without a row"). +- The add-only seeds re-added a just-switched-off mail on a new box's second start — **NOT-A-FINDING: fixed in this + session** (same entry). +- demo-hp: hub mails 3 app events the page does not show — **FILED: R-1217**. +- The cleared swap alarms sent no mail (`host_swap_ok` is info) — **NOT-A-FINDING: the ladder sends no operator mail + for an info clear; the event row is the record.** +- A docstring test refused my first push (`9c.` is not a gate-list number) — **NOT-A-FINDING: my numbering; fixed + before the push.** +- The safety check blocked one `sh -c` with a temp-file removal; rewritten with no file to remove — **NOT-A-FINDING: + nothing was run.** +- 9202 has no hub link, so the rc there proved start-up only — **NOT-A-FINDING: the mail path was proven on the three + test boxes through the floor.** + +## Teardown + +- **Machine:** nothing created. 9202 now runs 0.339.0-rc.1 (the scratch box; the floor moves it at its next report if + it reports — it has no hub link, so set it by hand next time it is needed). +- **Host:** the three test boxes on controller 0.339.0 — intended. +- **Hub:** one row written (demo-felhom's restored list, through `POST /api/v1/preferences`); global floor 0.339.0; + no per-customer floor left. diff --git a/STATUS.md b/STATUS.md index 8c053d8a..9c2a38e6 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,8 +1,8 @@ # STATUS — what works, what's broken, what's next -**Ready for the first real tester (Tester 2)? Nearly — one click left:** vouch agent 0.162.0 (Configs → Day-0 artifacts), so new boxes install it. Tester 2 is off; nothing was sent to it. +**Ready for Tester 2? Yes — when the laptop is on: the operator sends agent 0.162.0 first (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-22 (morning, your yeses).** Released: controller 0.338.0 (global floor), agent 0.162.0 (on the 3 test boxes), hub 0.164.0, installer 1.30.0 and the website download button. The new host-swap check warns on boxes bigger than 8 GB; that is too strict (decide D28). +**Updated 2026-10-22 (day).** A rebuilt box no longer overwrites the family's saved mail list (hub 0.165.0, controller 0.339.0); demo-felhom's own list, replaced this morning, is back. The swap warning now fires only when a machine has clearly too little swap (your D28 = A). ## What works - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4). @@ -30,7 +30,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D28 | R-1216 | The host-swap check warns when host swap is below the guests' whole swap. A 16 GB box with the installer's 8 GB swap trips it, but the soak used 2.7 GB. Loosen it to „below what the installer gives"? | **Loosen it (A)** (no mail for a healthy box; 938 MB still warns) | Keep (B): you get a mail per big box, and the install page tells such a household to call us | demo-hp and demo-felhom stay flagged; one mail each, already sent | — | | D27 | R-1207 | A host with no swap gets nothing from the swap fix, and the agent never adds swap. Add host swap automatically (BYO too), or refuse an 8 GB host without swap at install? | **Report only for now** (our installer gives 7.75 GB, and Tester 2's laptop used our installer; the alarm and the install-page sentence now tell you) — decide after the first BYO box | Add it: the agent changes a disk its owner set up; refuse: a BYO household cannot start | a no-swap host can still freeze; you get the alarm | — | | D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | @@ -53,10 +52,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. **Today, 2026-10-22.** -1. Vouch agent 0.162.0 (Configs → Day-0 artifacts; sha256 `5fd7a23b…`, bundle `ac772967…`). If not: new boxes install 0.161.0, without the swap fix. -2. Discard the two test appliances 47 and 48 on the Hosts page (the installer proof). If not: they stay as „unclaimed". -3. Decide D28 above. If not: big boxes keep the warning. -4. When Tester 2 is back on: say so, and I send it agent 0.162.0 (the swap fix). +1. When Tester 2's laptop is on: follow the hand-over list, step 1 first (tell CC; CC sends agent 0.162.0). If not: the box stays on old software without the swap fix. **The rest** - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. diff --git a/documentation/audits/day-2026-10-22/01-prefs-before-deploy.txt b/documentation/audits/day-2026-10-22/01-prefs-before-deploy.txt new file mode 100644 index 00000000..2a7d8f5b --- /dev/null +++ b/documentation/audits/day-2026-10-22/01-prefs-before-deploy.txt @@ -0,0 +1,13 @@ +# hub customer_notifications BEFORE hub 0.165.0 (live, with WAL) and in the 2026-10-22T00:00Z snapshot; no address, only whether one is set +live Tester-2 addr 2026-10-04 16:30:00 2026-10-04 16:30:00 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical'] +live Tester4 addr 2026-10-18 18:18:34 2026-10-18 18:18:34 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical'] +live _resend-rotation-test noaddr 2026-06-29 06:02:43 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone'] +live demo-felhom addr 2026-02-16 19:22:40 2026-10-22 07:54:55 6 17 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'backup_integrity_failed', 'crossdrive_failed', 'db_dump_failed', 'disk_critical', 'disk_warning', 'expected_backup_missed', 'expected_dbdump_missed', 'health_critical', 'health_recovered', 'node_down', 'offbox_enlarge_blocked', 'storage_disconnected', 'storage_reconnected'] +live demo-hp addr 2026-08-23 09:54:03 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone'] +live tester-1 addr 2026-10-09 06:09:57 2026-10-10 07:53:24 6 0 [] +snapshot 00:00Z Tester-2 addr 2026-10-04 16:30:00 2026-10-04 16:30:00 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical'] +snapshot 00:00Z Tester4 addr 2026-10-18 18:18:34 2026-10-18 18:18:34 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical'] +snapshot 00:00Z _resend-rotation-test noaddr 2026-06-29 06:02:43 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone'] +snapshot 00:00Z demo-felhom addr 2026-02-16 19:22:40 2026-09-24 09:39:35 6 13 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'db_dump_failed', 'disk_critical', 'disk_warning', 'expected_backup_missed', 'expected_dbdump_missed', 'health_critical', 'node_down', 'offbox_enlarge_blocked', 'storage_disconnected'] +snapshot 00:00Z demo-hp addr 2026-08-23 09:54:03 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone'] +snapshot 00:00Z tester-1 addr 2026-10-09 06:09:57 2026-10-10 07:53:24 6 0 [] diff --git a/documentation/audits/day-2026-10-22/02-restore-demo-felhom.txt b/documentation/audits/day-2026-10-22/02-restore-demo-felhom.txt new file mode 100644 index 00000000..62275268 --- /dev/null +++ b/documentation/audits/day-2026-10-22/02-restore-demo-felhom.txt @@ -0,0 +1 @@ +{"status":"ok"} HTTP 200 diff --git a/documentation/audits/day-2026-10-22/03-prefs-after-restore.txt b/documentation/audits/day-2026-10-22/03-prefs-after-restore.txt new file mode 100644 index 00000000..7697036c --- /dev/null +++ b/documentation/audits/day-2026-10-22/03-prefs-after-restore.txt @@ -0,0 +1,7 @@ +# after hub 0.165.0 + the demo-felhom restore (live, with WAL); compared with the 00:00Z snapshot +Tester-2 2026-10-04 16:30:00 cooldown 6 household_saved 0 9 events | events = snapshot: True | address = snapshot: True +Tester4 2026-10-18 18:18:34 cooldown 6 household_saved 0 9 events | events = snapshot: True | address = snapshot: True +_resend-rotation-test 2026-09-24 09:39:35 cooldown 6 household_saved 1 3 events | events = snapshot: True | address = snapshot: True +demo-felhom 2026-10-22 10:56:44 cooldown 6 household_saved 1 13 events | events = snapshot: True | address = snapshot: True +demo-hp 2026-09-24 09:39:35 cooldown 6 household_saved 1 3 events | events = snapshot: True | address = snapshot: True +tester-1 2026-10-10 07:53:24 cooldown 6 household_saved 1 0 events | events = snapshot: True | address = snapshot: True diff --git a/documentation/audits/day-2026-10-22/04-release-0339-boxes.txt b/documentation/audits/day-2026-10-22/04-release-0339-boxes.txt new file mode 100644 index 00000000..5a7c5f5d --- /dev/null +++ b/documentation/audits/day-2026-10-22/04-release-0339-boxes.txt @@ -0,0 +1,11 @@ +### root@192.168.0.154 +containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 35 seconds (healthy) +errors=0 +### demo-hp +containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 36 seconds (healthy) +errors=0 +### felhom-pve +containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 39 seconds (healthy) +errors=0 +2026/10/22 11:12:30 [INFO] notification prefs: adopted the household's saved list from the hub (13 events, address set=true) — nothing pushed (R-1199) +[('min_controller_version', '0.339.0', '2026-10-22 11:14:21'), ('min_controller_version_declared_min_agent', '0.339.0=0.158.1', '2026-10-22 11:14:21')] diff --git a/documentation/audits/day-2026-10-22/05-https-only.txt b/documentation/audits/day-2026-10-22/05-https-only.txt new file mode 100644 index 00000000..8682d689 --- /dev/null +++ b/documentation/audits/day-2026-10-22/05-https-only.txt @@ -0,0 +1,33 @@ +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 14 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https diff --git a/documentation/audits/day-2026-10-22/06-prefs-after-release.txt b/documentation/audits/day-2026-10-22/06-prefs-after-release.txt new file mode 100644 index 00000000..648fc9b9 --- /dev/null +++ b/documentation/audits/day-2026-10-22/06-prefs-after-release.txt @@ -0,0 +1,18 @@ +# after controller 0.339.0 on the three test boxes (live, with WAL), against the 00:00Z snapshot +Tester-2 updated 2026-10-04 16:30:00 household_saved 0 9 events | events = snapshot: True | address = snapshot: True +Tester4 updated 2026-10-18 18:18:34 household_saved 0 9 events | events = snapshot: True | address = snapshot: True +_resend-rotation-test updated 2026-09-24 09:39:35 household_saved 1 3 events | events = snapshot: True | address = snapshot: True +demo-felhom updated 2026-10-22 10:56:44 household_saved 1 13 events | events = snapshot: True | address = snapshot: True +demo-hp updated 2026-09-24 09:39:35 household_saved 1 3 events | events = snapshot: True | address = snapshot: True +tester-1 updated 2026-10-10 07:53:24 household_saved 1 0 events | events = snapshot: True | address = snapshot: True +# swap events today +('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short') +('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short') +('2026-10-22 11:04:27', 'demo-hp', 'host_swap_ok') +('2026-10-22 11:09:41', 'demo-felhom', 'host_swap_ok') +# operator mails for host_swap today +('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short', 'sent', 'operator') +('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short', 'skipped', 'customer') +('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short', 'sent', 'operator') +('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short', 'skipped', 'customer') +demo-felhom box file: {'email': 'set', 'enabled_events': 13, 'cooldown_hours': 6} diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 300a1deb..144ed778 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,13 @@ --- +## 2026-10-22 day — the mail list and the swap rule (full text: R-1216 `git show 25dae2f5:documentation/backlog/OPEN-ITEMS.md`; R-1199's original row `git show 115f15c8:documentation/backlog/OPEN-ITEMS.md` — its reopened text of 2026-10-22 was never committed, and its substance is the Closed cell below) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1199** | **REOPENED 2026-10-22: the night's push (controller 0.338.0) overwrites, on a rebuilt box, the mail list the household saved on its old box — and repeats at every start, because the push creates no local save.** (P2) | CLOSED 2026-10-22 day — REOPENED the same morning (0.338.0's start-up default overwrote demo-felhom's own 13-event list at 07:54:55 UTC) and fixed: hub **0.165.0** (`household_saved`, set only by the household's own save, backfilled once; a default never overwrites a marked list; self-scoped `GET /api/v1/preferences`) + controller **0.339.0** (a box with no list reads the hub's: adopts a household's, pushes its default once only when none; every push names its source). demo-felhom's list restored from the hub's 00:00 UTC snapshot; at 0.339.0's start demo-felhom ADOPTED it (13 events, nothing pushed), Tester 1 and demo-hp sent nothing; all six hub rows equal the snapshot after the release. The night's rule (*decided by CC — operator may reverse*) stands, narrowed: the default goes only where no household ever chose. | `audits/day-2026-10-22/01..06` | +| **R-1216** | **The host-swap check is too strict for a guest larger than the installer's 8 GB host swap: on demo-hp and demo-felhom the agent warned and the hub mailed the operator `host_swap_short` within minutes of agent 0.162.0, and the controller's install page will tell such a household „nincs elég cserehely… Szólj a Felhomnak” when memory is tight.** (P3) | CLOSED 2026-10-22 day — D28 = A (operator 11:42): short = no host swap, or below 90 % of min(the guests' swap, 8192 MB), the installer's maximum (`Install.pm compute_swapsize`). Hub 0.165.0 (`host_swap_short`) and controller 0.339.0 (the install-page sentence) by one case table, `scripts/host_swap_rule_gate.py` (4 decoys). No agent change. The morning's two alarms cleared at the next report: `host_swap_ok` demo-hp 11:04:27, demo-felhom 11:09:41 UTC (no mail — info). | `audits/day-2026-10-22/06-prefs-after-release.txt` | + ## 2026-10-22 morning — the releases (full text of the rows: `git show 115f15c8:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | @@ -36,7 +43,6 @@ | **R-1178** | **The second drive's „Teljes visszaállítás" can change files, then refuse the database half, leave the app STOPPED, and tell the household „the app and its data are untouched".** (P2) | CLOSED 2026-10-22 — shipped in controller **v0.338.0** (`UnitRestoreOptions.CheckOnly`: a refusal moves nothing); test-proven. | controller CHANGELOG v0.338.0 | | **R-1180** | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** (P2) | CLOSED 2026-10-22 — shipped in controller **v0.338.0** (the night's mirror skips an app while its restore runs); test-proven. | controller CHANGELOG v0.338.0 | | **R-1195** | **A power cut in the middle of a full off-site restore can leave the app running on the BACKUP's data while the page says only „the restore was interrupted, start it again" — the household's newer data is gone and nothing says so.** (P2) | CLOSED 2026-10-22 — shipped in controller **v0.338.0**; proven on 9202 the night before (a controller killed mid-restore → put back at the next start, the record says so in Hungarian). | `audits/night-2026-10-21/P3-*` | -| **R-1199** | **The household's notification page shows disk-warning, drive-disconnected and backup-failed mails switched ON, but on the rehearsal day only the OPERATOR got them — the household's mailbox received one alarm (a failed off-site run) out of nine.** (P2) | CLOSED 2026-10-22 — the hub mailed the household by a list saved 2026-10-18, before the box existed; the box never pushed its own because it pushes only a saved choice. Fixed in controller `ba22f9b` (a box with no saved choice pushes the default list its page shows, no address), shipped in **v0.338.0**. *Decided by CC unattended 2026-10-21 — operator may reverse* (the box's page is the source, `08`). `TestR1199_*` red-proved. | `REPORT-night-2026-10-21.md` §6, `audits/night-2026-10-21/P5-checks/` | ## 2026-10-21 night — the CI time limit of the decoy suite (full text of the row: `git show f576d7f7:documentation/backlog/OPEN-ITEMS.md`) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7f8cd523..bff5e599 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -236,7 +236,7 @@ stopping line that lies. | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC | | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f ` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC | -## Box system & updates — 14 rows (P2 2, P3 8, P4 4) +## Box system & updates — 13 rows (P2 2, P3 7, P4 4) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -247,7 +247,6 @@ stopping line that lies. | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC | | **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR **NO-SWAP TEST 2026-10-22 04:08–04:38 UTC (`audits/night-2026-10-21/P2-noswap/`):** the soak box, `swapoff -a` on the host, the same burst → the freeze CAME BACK within 20 s: `/api/health` dead for 289 s, then 172 s and 68 s (24 of 125 probes failed), memory „full" 36 % host / 47 % guest, the VM reading ~40 GB per 25 s from disk; Immich lost 207 of 306 photos and its stream broke; the box recovered only when the Immich stream stopped (04:23). After `swapon -a`: 0 failed probes, „full" ≤ 1.3 %. The agent logged its one WARN at the swapoff (host 938 MB < guests 4096 MB). **Operator answer 2026-10-22: Tester 2's laptop was installed with OUR installer** — so it has host swap (the installer gives ≈ RAM up to 8 GB) and the swap fix helps it once its agent is ≥ 0.162.0. The decision for BYO hosts stays open. | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator | | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | OPEN | — | Measure the tier2 job alone on the 8 GB box (per-step pressure); decide whether 20 % over 10 s is the right line, or limit the copy (ionice / cgroup IO weight) | CC | -| **R-1216** | Box system & updates | P3 | **The host-swap check is too strict for a guest larger than the installer's 8 GB host swap: on demo-hp and demo-felhom the agent warned and the hub mailed the operator `host_swap_short` within minutes of agent 0.162.0, and the controller's install page will tell such a household „nincs elég cserehely… Szólj a Felhomnak” when memory is tight.** Measured 2026-10-22 (`audits/morning-2026-10-22/release/04-agent-delivery.txt`, hub events 07:57:07 UTC): demo-hp host swap 8191 MB vs guests' swap 25898 MB; demo-felhom 8191 vs 12288; Tester 1 (8 GB machine) 7871 vs 4096 — fine. The check compares the host's swap with the guests' WHOLE swap allowance (swap = RAM since R-1192); Proxmox's installer gives host swap = RAM capped at 8 GB, so every machine above ~10 GB RAM with a big guest trips it. The soak's burst used at most 2.7 GB of host swap (8 GB machine, 4 GB guest). Three places compare the same two numbers: agent (WARN), hub (`host_swap_short`, operator-only), controller (install-page sentence). | OPEN | the operator's pick: (A) „short" = host swap below min(guests' swap, 8 GB) − 10 %, i.e. below what the installer itself gives — agent, hub and controller patch releases; or (B) keep the strict check and accept the operator mail for big boxes, the page sentence only when memory is tight | Operator picks A or B; CC builds A in the three repos with a test each (a 16 GB host with 8 GB swap and a 12 GB guest is NOT short; 938 MB is) | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | @@ -256,7 +255,7 @@ stopping line that lies. | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -## Monitoring & notifications — 14 rows (P2 1, P3 10, P4 3) +## Monitoring & notifications — 15 rows (P2 1, P3 10, P4 4) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -273,6 +272,7 @@ stopping line that lies. | **R-1049** | Monitoring & notifications | P3 | **A pulled drive mails the operator „unexpectedly disconnected” (error), and nothing ever mails that it came back.** MEASURED 2026-10-14 night (`audits/media-drive-events-2026-10-14/K8-hub-mails.txt`, the hub DB with its WAL from the running pod, and the mailbox as the second channel): two pulls → two `storage_disconnected` mails; the hub RECEIVED both `storage_reconnected` and both `health_recovered` (severity info) and logged no notification for any of them — they are not in `recoveredPairedDownTypes` (`hub/internal/notify/dispatcher.go`), so the severity gate drops them. The R-97a shape: told it broke, never told it healed — the operator cannot tell a returned drive from one still gone without opening the hub. **BUILT 2026-10-15 — hub v0.151.0 (`12552aca`): both pairs in `recoveredPairedDownTypes`, cooldowns ended by the next loss (`TestR1049_DriveBackMailsBothEdges`, red-proved). Not deployed: the operator's „yes” first (decision 162); the image build waits for it too.** | **WATCHING 2026-10-20** — the fix is live: hub 0.151.0 and every later hub carry it (hub 0.162.0 runs today, `audits/status-view-2026-10-20/hub-reads.txt`), so it no longer waits on the operator. Left: read one real „came back” mail. **WAITING-ON-OPERATOR** — hub v0.151.0 code pushed, not built, not deployed | a drive pulled and put back on a test box | On the next pulled-and-returned drive (any test box): read the „came back” mail in the operator mailbox, then close | CC | | **R-1170** | Monitoring & notifications | P3 | **A box whose tunnel container disappears raises no alarm, even when its tunnel is configured.** The hub alarms on `not_running` only (`checkTunnel`). A guest with no cloudflared container read `unknown` since Docker 29 (lower-case „no such object", R-1070) and reads `absent` with the next agent release — neither alarms. On a box whose customer has a tunnel token, a missing container means the box is unreachable from outside. Found 2026-10-19 while fixing R-1070. | OPEN — filed 2026-10-19 | `hub/internal/api/handler.go` `checkTunnel` | Hub: treat `absent` as down when the customer's config carries a tunnel token (two reports in a row, as today); a test | CC | | **R-1208** | Monitoring & notifications | P3 | **The operator gets `node_stale` / `node_down` („No report received for 1h30m") for a customer whose box the operator has just DELETED.** 2026-10-21: Tester4's box was torn down 16:27 UTC and its host deleted by the operator 16:49 UTC (escrow acknowledged); `node_down` was mailed to the operator at 17:55 UTC. The customer-level staleness (`hub/internal/monitor/staleness.go` `Check`) counts from the last controller report and does not know the host was deleted on purpose. Not fixed tonight: the brief allowed one hub image and 0.164.0 was already built. | OPEN | — | Skip (state „deleted", no event) a customer with no live host whose newest host deletion is after its last report; a new box's first report → ok with no event (as `StateDisabled`); test both | CC | +| **R-1217** | Monitoring & notifications | P4 | **On demo-hp the hub mails three app events (`app_update_undone`, `app_update_held`, `app_stopped_unhealthy`) while the box's page shows none ticked.** The household saved an empty list; the hub's one-time add-only seed of 2026-09-24 added the three types to that empty list, while the box's own seeds skip a list stored as missing (the old `omitempty` shape: `{"cooldown_hours": 6}`). Read 2026-10-22: hub row 3 events, box file no `enabled_events`. Tester 1 (also empty) is not affected (its row was saved after the seed). Test boxes only; a household saving its page once makes the two agree. | OPEN | — | Either let the box adopt the hub's list when its own saved list is the missing-events shape, or leave it (test boxes only) — decide at the next notification change | CC | | **R-1211** | Monitoring & notifications | P3 | **The household's `backup_failed` mail says only „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." — no app named, no next step, and the formal „Kérjük".** Rehearsal 2026-10-20 (the one alarm the household got, `audits/rehearsal-2026-10-20/B9-words/words.md` W25); the box's own page now says which app and that the box retries the next night (controller `5776d54`, W25), but the mail is the hub's `customerMessages` entry. Found by the night helper 2026-10-21. | OPEN | — | Hub: build the household mail from the event's app names and the classifier's class, in the te-form, with „a doboz a következő éjjel újra próbálkozik"; mail golden | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 5240de3a..ffe83006 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -7,6 +7,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S ## Moved from STATUS on 2026-10-22 +**Updated 2026-10-22 (morning, your yeses).** Released: controller 0.338.0 (global floor), agent 0.162.0 (on the 3 test boxes), hub 0.164.0, installer 1.30.0 and the website download button. The new host-swap check warns on boxes bigger than 8 GB; that is too strict (decide D28). + +**Decided 2026-10-22 11:42 — D28 = A** (the operator): | D28 | R-1216 | The host-swap check warns when host swap is below the guests' whole swap. A 16 GB box with the installer's 8 GB swap trips it, but the soak used 2.7 GB. Loosen it to „below what the installer gives"? | **Loosen it (A)** (no mail for a healthy box; 938 MB still warns) | Keep (B): you get a mail per big box, and the install page tells such a household to call us | demo-hp and demo-felhom stay flagged; one mail each, already sent | — | + **Updated 2026-10-22 (morning, after the night run).** A failed or cut restore now puts the data back and starts the app (proven on the scratch box); clearer first-hour texts; a host short of swap is reported. Built and tested, nothing released. **Updated 2026-10-21 (evening).** A rebuilt box now gets the household's off-site backups back (hub 0.163.0, proven live: 3 031 files byte-identical), and shared folders, imported apps and away drives keep their copies and locks (controller 0.337.0, the floor). The 8 GB freeze is explained — the guest's swap was capped at 512 MB — and the agent fix is ready but not released. Report: `REPORT-fixes-2026-10-21.md`. diff --git a/documentation/pilot/TESTER2-handover.md b/documentation/pilot/TESTER2-handover.md new file mode 100644 index 00000000..70b9c93f --- /dev/null +++ b/documentation/pilot/TESTER2-handover.md @@ -0,0 +1,61 @@ +# Tester 2 — the hand-over list (written 2026-10-22) + +Tester 2's laptop (`Tester-2-be8404`, customer `Tester-2`) has been off since 2026-10-04. It was installed with **our +installer** (operator answer, 2026-10-22), so it has host swap and the swap fix will work on it. When it reports +again it runs old software: agent 0.142.0, controller 0.292.0. This page says what happens then, in order. + +## 1. The operator, when the laptop is on — in this order + +| # | What | Where | Done when | +|---|---|---|---| +| 1 | **Tell CC „Tester 2 is on".** CC signs the agent update to the vouched **0.162.0** (one signed `agent_update` job, the same as for the three test boxes on 2026-10-22). | this chat | CC reports `update committed` within about 15 minutes | +| 2 | **Read the host swap.** No new `host_swap_short` alarm for Tester 2 within 15 minutes of the update means the swap is fine. Our installer gives about the machine's RAM, at most 8 GB. CC also reads the two numbers from the host report. | Hub → Hosts → `Tester-2-be8404`, and the operator mailbox | no alarm; CC confirms the numbers | +| 3 | **The bootstrap step (R-862)** — once, by hand, because this box is older than the config bundle. Connect your WireGuard operator tunnel; `ssh -p 8822 felhom-op@10.77.0.5`; Hub → Hosts → `Tester-2-be8404` → Console access → **Reveal**; `su -` with it; then the three commands in `runbooks/config-bundle.md` („A box from before agent v0.143.0"). | your tunnel | the script ends with `DONE. This box can now take signed config bundles.` Then tell CC „bootstrap done" — CC sends the bundles and reads them back | +| 4 | **The emergency root password (R-1139)** — in the same root shell: `bash felhom-host-install.sh --rotate-recovery-only` (`runbooks/break-glass.md`). It sets a new random console password and stores it in the hub. | the same root shell | it prints `root@pam password set + vaulted to the hub`; tell CC — CC proves the new one logs in and the old one is refused | +| 5 | **One off-site check (R-862).** You believed on 2026-10-07 that Tester 2 never set up the off-site backup. Look: Hub → Customers → `Tester-2` → Off-site. If it is not set up, decide with the tester whether to set it up now. | the hub | you tell CC what you saw | +| 6 | **Tell CC „Tester 2 hand-over done".** | this chat | CC runs section 3 | + +The controller needs nothing from you: once the agent is 0.162.0, the hub's floor moves the controller to the current +release at the next report (the floor waits for an agent ≥ 0.158.1). + +**What happens if you do nothing:** the box keeps agent 0.142.0 and controller 0.292.0 (the floor holds it there), +without the swap fix — a big upload can freeze it for minutes — and with the old emergency password. + +## 2. For the tester (Hungarian, one page — send it as it is) + +> **Szia! A Felhom dobozod újra bekapcsolható.** +> +> **Mi változott október 4. óta?** +> 1. A doboz nagy feltöltéskor (például sok fénykép egyszerre) nem akad meg többé percekre. +> 2. Ha egy visszaállítás félbeszakad (hiba vagy áramszünet miatt), az adataid visszakerülnek oda, ahol előtte voltak. +> 3. Áramszünet után a doboz magától rendbe teszi magát, és az alkalmazások újraindulnak. +> 4. Az értesítéseid listáját a doboz megőrzi; a levelek és a képernyők szövegei érthetőbbek lettek. +> 5. A doboz minden címe csak biztonságos (https) kapcsolaton érhető el. +> +> **Mit próbálj ki az első órában?** +> - Nyisd meg a doboz kezdőlapját, és nézd meg, hogy minden alkalmazásod fut-e. +> - Tölts fel néhány fényképet vagy dokumentumot abba az alkalmazásba, amit a legtöbbet használsz. +> - Nézd meg a Beállítások → Értesítések oldalt: azok a levelek vannak-e bekapcsolva, amelyeket kérni szeretnél. +> - Nyomd meg a „Mentés most" gombot a Mentések oldalon, és várd meg, amíg kész lesz. +> +> **Ha valami nem működik:** írd le röviden, mit csináltál, mit vártál, és mi történt helyette. Ha tudsz, küldj +> képernyőképet is. Küldd el a Felhom üzemeltetőjének azon a címen, ahol eddig is levelezettetek. +> +> **Amit ne csinálj:** +> - Visszaállítás közben ne húzd ki a dobozt a konnektorból, és ne kapcsold ki. +> - A helyreállítási kódodat tartsd papíron, a dobozon kívül. A dobozon vagy a laptopon ne tárold. + +## 3. What CC checks after the hand-over + +1. **A read-only box walk of Tester 2's box:** `scripts/https_only_check.py` with Tester 2 in it (R-1104); the + containers, unhealthy and restarting counts, and the controller's error lines (the 2026-10-22 release shape); the + agent's capability self-check (`ok=N total=N`) and the config bundle read back (R-862). +2. **The next night chain:** the database dump, the second-drive copy (if the tester has added a drive), the off-site + copy (if set up), the integrity check — each ran, read from the box's log the next morning. +3. **The hub alarms:** for 24 hours after it is back, every alarm Tester 2 raises, and whether the household's mail + list is the one the household saved (the hub keeps it; the box adopts it — R-1199). +4. **Tester 2's mail list:** its hub row was written at the claim and never saved by the household + (`household_saved` = false), so the box will push its default list once and keep it. If the tester had chosen a + different list on 2026-10-04, ask them to save it again on the box. + +Rows: R-862, R-1139, R-1104, R-1207, R-1199.