close-out 2026-10-29 afternoon release: R-1343/R-1342/R-1340/R-1341 closed (controller 0.348.0, proven live); R-1349 (guest reboot, decide D65), R-1350 (app_start_failed after a clear) filed; R-1344 numbers so far; CONTEXT (D64), STATUS, report, evidence
gates / gates (push) Successful in 1m47s
gates / gates (push) Successful in 1m47s
This commit is contained in:
admin committed
1 parent
e4d91043e4
commit
dd3e4137ee
16 files changed
+237
-9
No files matched your search
+24
@@ -16,6 +16,30 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — RELEASED: hub v0.176.1, agent v0.169.0 (VOUCHED), controller v0.348.0 (global
|
||||
> floor). Operator ruling D64 = B (`09` §3 decision 283, `04` §6.1):** „the hub's „Clear hold" works for every hold reason
|
||||
> with no operator signature; the operator accepted that a compromised hub could clear a data-protecting hold. R-379
|
||||
> („only a person clears a hold") now reads „a person — on the box, or the operator on the hub"." **Built:** (1) „Restart
|
||||
> controller" per guest on the hub's host page, done by the AGENT (its own closed list: `restart_controller <vmid>` on
|
||||
> the host-report reply, the result on the next host report; the hub pokes the agent over the tunnel). The supervisor's
|
||||
> guards; the brake 3 restarts / 15 min counts operator restarts, then 30 min refused (`opBrakeUntil`). (2) `clear_hold`
|
||||
> as the controller's fifth verb; the running controller clears and saves at once through `<data>/ctl/hold.sock`, and the
|
||||
> box command goes through it too (R-1343). The hub's button per held-app row has a confirm step and, for a
|
||||
> data-protecting hold, the restore warning. (3) R-1344: a controller that runs but does not answer → `controller_unresponsive`
|
||||
> (operator-only) — ALARM ONLY until the night's health log (to 2026-10-30 07:00 UTC) is read; so far 631 probes, max
|
||||
> 163 ms, no failed probe. **Proven live on Tester 1 on the release builds:** restart 14:49 CET, „done" on the hub 16 s
|
||||
> after the press; crash-loop hold cleared 14:57:48 CET, gone on the box 3 s later and still gone after the next settings
|
||||
> save (15:04:23). Earlier on rc builds: a restart with the controller stopped by hand, the fourth press refused; a
|
||||
> restore_mixed hold cleared with the warning; R-1340/R-1341/R-1342 live. **Versions:** hub 0.176.0 (`sha256:53372198…`,
|
||||
> `3d0c907b`) then 0.176.1 (`sha256:c105cb9a…`, from `2ccf180b`, manifest `c1f3a163`): 0.176.0 gated the restart button
|
||||
> on a controller version the agent never reports, and named a token press „basic auth" — both found live, fixed red
|
||||
> first. Agent 0.169.0 (`fc066de6…`, bundle `3ca33f9c…`, from `637c92c`), vouched 14:06:09 CET, `pinned.day0_agent`
|
||||
> 0.169.0 (`db25eb0`). Controller 0.348.0 (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515; rc.2 on 9202): per-box floors
|
||||
> 14:41 CET, walk green on all three + https-only, global floor 14:48 CET (MinAgent 0.158.1). Rows: closed R-1343,
|
||||
> R-1342, R-1340, R-1341; opened R-1344…R-1350 (R-1350: an `app_start_failed` 30 s after a clear — the hub's cooldown
|
||||
> hid it). Register 226 → 229. Tester4 (a demo-hp test VM, off for 2 days) moves to the floor when it starts. Report:
|
||||
> `REPORT-release-2026-10-29b.md`.
|
||||
|
||||
> **2026-10-29 (attended release run) — RELEASED: hub v0.175.0, agent v0.168.0 (VOUCHED), controller v0.347.0 (global
|
||||
> floor), catalog `313eaf6`. Operator rulings D62 + D63 (`09` §3 decisions 281/282, `07` §6.3).** D62: an unused restore
|
||||
> copy stays 30 days (was the night's 14, decided by CC — superseded); the kept-copies list shows each restore copy (date,
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# REPORT — release run 2026-10-29 (afternoon, attended): restart a box's controller and clear a held app from the hub
|
||||
|
||||
## Five lines
|
||||
|
||||
1. **Restart works with the controller stopped: yes.** Stopped by hand on Tester 1, one press restarted it 5 s after
|
||||
the stop. On the release builds one press read „done" on the hub 16 s after the press; the fourth press in 15
|
||||
minutes was refused.
|
||||
2. **Stuck check: alarm only, for now.** So far 631 probes on Tester 1 and demo-hp: median 57–61 ms, max 163 ms, no failed
|
||||
probe, no 5-s timeout. The night chain is not in it yet; the log runs to 2026-10-30 07:00 UTC, then CC decides (R-1344).
|
||||
3. **Clear stays after a save: yes.** On the release builds: cleared 3 s after the press, still gone after the next
|
||||
settings save 6.5 minutes later; the hub's list emptied.
|
||||
4. **Released:** hub 0.176.1, agent 0.169.0 (vouched), controller 0.348.0 (the global floor since 14:48 CET).
|
||||
5. **What you do:** nothing is blocked. One new decide row (D65: a „Reboot guest" button — CC says not now). Tester4 is
|
||||
off since 10-27; it updates when you start it.
|
||||
|
||||
## Readings taken
|
||||
|
||||
1. **The restart is the agent's, not the controller's** — a stopped or stuck controller cannot restart itself. It uses
|
||||
`felhom-controller-bootstrap.service`, the same unit as the supervisor (R-523), not the host's bootstrap unit.
|
||||
2. **The brake counts operator restarts with the supervisor's**, and a 4th restart in 15 minutes opens a 30-minute
|
||||
refusal for presses (the supervisor's own pause is separate).
|
||||
3. **Part 2 stopped at the alarm.** A day run cannot hold the night chain (the whole-guest vzdump), so the „no false fire"
|
||||
condition is not shown yet. Building the restart now would introduce a mechanism nobody has measured.
|
||||
4. **The clear goes through the running controller's socket**, never a file edit under a running controller — the
|
||||
file edit was R-1343's cause.
|
||||
|
||||
## Versions, digests, CI by commit
|
||||
|
||||
| Repo | Commit | CI job | What |
|
||||
|---|---|---|---|
|
||||
| felhom.eu (hub) | `3d0c907b` | 2505 green | the two buttons, the agent channel, `controller_unresponsive`, D64 in `04`/`09` — **hub 0.176.0** (`sha256:53372198…`) |
|
||||
| felhom.eu | `682015db` | 2506 green | manifest → 0.176.0 (Deployment/hub only) |
|
||||
| felhom.eu (hub) | `d0dbc6f1` | 2510 green | restart button for every guest (0.176.0 gated it on a version the agent does not report); R-1345 |
|
||||
| felhom.eu (hub) | `2ccf180b` | 2511 green | a token press named by its token — **hub 0.176.1** (`sha256:c105cb9a…`) |
|
||||
| felhom.eu | `c1f3a163` | 2512 green | manifest → 0.176.1 |
|
||||
| felhom.eu | `e4d91043` | 2516 | `03`/`05`/`08` [DESIGN], R-1346…R-1348, evidence |
|
||||
| agent | `637c92c` | 2504, 2507 green | **0.169.0 built from here** (`fc066de6…`, bundle `3ca33f9c…`, tag `v0.169.0`) |
|
||||
| agent | `db25eb0` | 2513 green | CHANGELOG; `pinned.day0_agent` = 0.169.0 |
|
||||
| controller | `5b3b541`, `6077ac0`, `99fac06` | 2508, 2509, 2514 green | clear_hold, the socket, R-1340/R-1341, the fallback, the second R-1340 producer |
|
||||
| controller | `2a0a17a` | 2515 green | v0.348.0 heading — **0.348.0 built from here** (`sha256:672fc1c1…`); rc.2 `1d96f60c…` on 9202 |
|
||||
|
||||
The close-out commits' CI is read at the end of the session.
|
||||
|
||||
## Rows
|
||||
|
||||
- **Closed (4):** R-1343, R-1342, R-1340, R-1341 — each released in 0.348.0 and proven live.
|
||||
- **Opened (7):** R-1344 (stuck controller: alarm; restart after the measurement), R-1345 (token scopes cannot be edited),
|
||||
R-1346 (hub pages slow; the vouch's 504), R-1347 (update-held flag after a restore lift), R-1348 (restore_mixed set at
|
||||
start-up sends no mail), R-1349 (guest reboot from the hub — needs your word), R-1350 (`app_start_failed` 30 s after
|
||||
a clear).
|
||||
- **Register:** 226 at the start of the brief → 229.
|
||||
|
||||
## Observations
|
||||
|
||||
- **F2 (an `app_start_failed` after a clear) is real, filed as R-1350.** The 12:44 clear showed none in 18 minutes; the
|
||||
14:57 clear did, 30 s later — the hub's cooldown swallowed the mail. The text is true; the timing is noise.
|
||||
- **A hub history line from rc.1** reads „held after an off-site restore failed its database replay" for a local restore.
|
||||
That is the stored result of a 13:17 UTC clear, written before `99fac06` fixed the second producer: NOT-A-FINDING
|
||||
(history, not live text).
|
||||
- **Hub speed (R-1346):** the hub itself answers its pages in 0.01–0.7 s and the registry in ~0.1 s, measured in the
|
||||
cluster; the 504 comes from the path in front of the hub.
|
||||
- **Tester 1 tidy:** the test app Gokapi removed through the product (no container, no app file, no volume); 9
|
||||
PrivateBin scratch volumes from today's backup presses removed by name (277 → 268; the night's 6 belong to R-1267);
|
||||
the empty `restore-undo` folder removed. Radicale is not installed and holds nothing.
|
||||
|
||||
## Teardown
|
||||
|
||||
Machine: Tester 1 as above; 9202 runs rc.2 (the scratch box is not linked to the hub, so no floor moves it; it stays there until the next test).
|
||||
Host: nothing provisioned. Hub: no test customer or host made; the per-box floors cleared after the 30-minute watch.
|
||||
|
||||
## Docs
|
||||
|
||||
`04` §6.1 [DESIGN] D64 + the agent's list; `09` §3 decision 283; `03` [DESIGN] operator restart + R-1344; `05` [DESIGN]
|
||||
the two buttons; `08` table rows `controller_unresponsive`, `restore_copy_deleted`. The system poster: no fact changed
|
||||
(no machine, path, backup tier or time).
|
||||
|
||||
Evidence: `documentation/audits/release-2026-10-29b/`.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-29 (attended release).** On your yes: hub 0.175.0, agent 0.168.0 (your vouch), controller 0.347.0 as the floor for every box, and the catalog (Plex night updates; Wanderer's memory figure). D62 is in (restore copies stay 30 days; „Törlés most" deletes one) and D63 held: both untested branches were proven live, nothing reverted. Register 226 → 226 (4 opened, 4 closed); the 10:39 Tester 1 error mail was my test's false alarm.
|
||||
**Updated 2026-10-29 (afternoon release).** On your yes: hub 0.176.1, agent 0.169.0 (your vouch) and controller 0.348.0 as the floor for every box. From the hub's host page you can now restart a box's controller (it works when the controller is stopped; 3 presses in 15 minutes, then 30 minutes refused) and clear any held app (D64 = B) — both proven on Tester 1 on the release builds. Register 226 → 229 (7 opened, 4 closed); a stuck-but-running controller only alarms you until CC reads tonight's measurement.
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -32,6 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D65 | R-1349 | Add „Reboot guest" to the hub's host page, beside „Restart controller"? | **Not now** (restart controller covers a stuck controller; a reboot stops every app of the household for a minute and needs its own guards) | yes: a brief with its guards, about one session to build and prove | nothing changes; a stuck Docker or network still needs a person at the host | — |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
@@ -48,7 +49,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
|
||||
|
||||
**2026-10-29 (release run).** 1. Clear test app Radicale's hold on Tester 1, in ONE command (R-1343): `ssh root@192.168.0.154 "pct exec 9201 -- bash -c 'docker exec felhom-controller felhom-controller --clear-restore-hold radicale && systemctl restart felhom-controller-bootstrap.service'"`. If not: Radicale stays held there. 2. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||
**2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
== felhom.eu
|
||||
2489 gates 2026-10-29T07:57:48Z bf69bb91 completed success
|
||||
2490 gates 2026-10-29T08:27:25Z 41f8f670 completed success
|
||||
2491 gates 2026-10-29T08:38:04Z d6352cd8 completed success
|
||||
2494 gates 2026-10-29T09:15:55Z 650aa0cf completed success
|
||||
2502 gates 2026-10-29T10:20:11Z 7c69c726 completed success
|
||||
2503 gates 2026-10-29T10:57:45Z 70862fa7 completed success
|
||||
2505 gates 2026-10-29T11:34:43Z 3d0c907b completed success
|
||||
2506 gates 2026-10-29T11:46:20Z 682015db completed success
|
||||
2510 gates 2026-10-29T12:00:23Z d0dbc6f1 completed success
|
||||
2511 gates 2026-10-29T12:06:43Z 2ccf180b completed success
|
||||
2512 gates 2026-10-29T12:42:36Z c1f3a163 completed success
|
||||
2516 gates 2026-10-29T13:48:46Z e4d91043 in_progress None
|
||||
== felhom-agent
|
||||
2429 gates 2026-10-27T14:49:44Z 00719934 completed success
|
||||
2439 gates 2026-10-27T19:45:32Z 3055bc54 completed success
|
||||
2473 gates 2026-10-28T21:51:14Z 5ddd124c completed success
|
||||
2475 gates 2026-10-28T22:22:46Z 6f933d6e completed success
|
||||
2481 gates 2026-10-29T01:51:06Z 91bd96c7 completed success
|
||||
2492 gates 2026-10-29T08:51:36Z 91bd96c7 completed success
|
||||
2495 gates 2026-10-29T09:16:05Z f4e63a55 completed success
|
||||
2497 gates 2026-10-29T09:34:04Z 6b0b0931 completed success
|
||||
2499 gates 2026-10-29T09:48:43Z 63e63ee8 completed success
|
||||
2504 gates 2026-10-29T11:31:13Z 637c92c7 completed success
|
||||
2507 gates 2026-10-29T11:48:05Z 637c92c7 completed success
|
||||
2513 gates 2026-10-29T13:09:11Z db25eb0e completed success
|
||||
== felhom-controller
|
||||
2476 gates 2026-10-28T22:23:00Z eed23e75 completed success
|
||||
2477 gates 2026-10-28T22:56:49Z 20c58770 completed success
|
||||
2485 gates 2026-10-29T04:14:22Z a815b37f completed success
|
||||
2488 gates 2026-10-29T07:51:16Z e3fce09d completed success
|
||||
2493 gates 2026-10-29T08:55:57Z fd75c7c7 completed success
|
||||
2498 gates 2026-10-29T09:48:37Z 6c72e3ff completed success
|
||||
2500 gates 2026-10-29T10:16:26Z 652a690e completed success
|
||||
2501 gates 2026-10-29T10:16:49Z c7bb776e completed success
|
||||
2508 gates 2026-10-29T11:50:00Z 5b3b5416 completed success
|
||||
2509 gates 2026-10-29T11:59:55Z 6077ac00 completed success
|
||||
2514 gates 2026-10-29T13:35:22Z 99fac06a completed success
|
||||
2515 gates 2026-10-29T13:35:45Z 2a0a17a7 completed success
|
||||
@@ -0,0 +1,8 @@
|
||||
2026/10/29 14:56:22 [INFO] Event from tester-1: app_stopped_unhealthy (warning) — A(z) gokapi alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). A Felhom ügyfélszolgálatát értesítettük.
|
||||
2026/10/29 14:57:21 [INFO] Event from tester-1: app_held (info) — App gokapi is held: stopped by the box for a crash loop (2026-10-29T13:56:21Z) — the household's Start gives it one more try
|
||||
2026/10/29 14:57:21 [DEBUG] [held-apps] tester-1/gokapi held (unhealthy_stop since 2026-10-29T13:56:21Z)
|
||||
2026/10/29 14:57:48 [INFO] operator action #10 clear_hold gokapi requested for tester-1 (host tester-1-d70be4) by access token "cc-hosts" from 10.42.0.1 — the box acts on its next report
|
||||
2026/10/29 14:58:08 [INFO] operator action #10 clear_hold gokapi for tester-1 (pressed by access token "cc-hosts" from 10.42.0.1): done — the hold on gokapi is cleared (it was: stopped by the box for a crash loop (2026-10-29T13:56:21Z) — the household's Start gives it one more try). The app is NOT started by this — the hou
|
||||
2026/10/29 14:58:21 [INFO] Event from tester-1: app_hold_cleared (info) — App gokapi is no longer held
|
||||
2026/10/29 14:58:21 [INFO] [held-apps] tester-1/gokapi no longer held (a row existed: true)
|
||||
2026/10/29 14:58:21 [INFO] Operator email suppressed for tester-1/app_start_failed — cooldown (key=tester-1:app_start_failed:gokapi)
|
||||
@@ -0,0 +1 @@
|
||||
13:57:36 Clear the hold of gokapi (unhealthy_stop, since 2026-10-29 13:56)? The box's controller clears it at once; the household can then start the app.
|
||||
@@ -0,0 +1,12 @@
|
||||
press at 2026-10-29T13:49:15Z
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /hosts/tester-1-d70be4#operator-actions
|
||||
2026/10/29 14:49:15 [INFO] host operator action #9 restart_controller 9201 requested for host tester-1-d70be4 by access token "cc-hosts" from 10.42.0.1 — the agent acts on its next report (poked now)
|
||||
2026/10/29 14:49:30 [INFO] host operator action #9 restart_controller 9201 on tester-1-d70be4 (pressed by access token "cc-hosts" from 10.42.0.1): done — restarted; the controller answered after 9 s (Docker's health check: healthy)
|
||||
result seen at 13:49:31
|
||||
Oct 29 14:49:19 felhom sudo[886737]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips
|
||||
Oct 29 14:49:19 felhom felhom-agent[697615]: time=2026-10-29T14:49:19.230+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=9 unit=felhom-controller-bootstrap.service
|
||||
Oct 29 14:49:19 felhom sudo[886742]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- systemctl restart felhom-controller-bootstrap.service
|
||||
Oct 29 14:49:28 felhom felhom-agent[697615]: time=2026-10-29T14:49:28.047+01:00 level=WARN msg="audit: operator action" id=9 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller answered aft
|
||||
Oct 29 14:49:30 felhom sudo[888018]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.348.0 Up 12 seconds (healthy)
|
||||
@@ -0,0 +1,4 @@
|
||||
9
|
||||
268
|
||||
restore-undo-removed
|
||||
privatebin Up 51 minutes (healthy)
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
== tester-1
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
== demo-hp
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
== demo-felhom
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
2026/10/29 15:21:48 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/10/29 15:21:48 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
|
||||
2026/10/29 15:21:48 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/10/29 15:21:48 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
|
||||
2026/10/29 15:21:48 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
|
||||
2026/10/29 15:21:48 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
|
||||
v0.348.0 ●
|
||||
v0.348.0 ●
|
||||
0.348.0
|
||||
v0.348.0
|
||||
0.348.0
|
||||
v0.348.0
|
||||
0.348.0
|
||||
v0.348.0
|
||||
+1
@@ -0,0 +1 @@
|
||||
2026-10-29T13:48:23Z
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
Managed updates — global floor
|
||||
Saving takes effect immediately — boxes below the floor update on their next
|
||||
report, no customer action. Blank = no global floor. This setting is independent of the Day-0
|
||||
Effective floor:
|
||||
v0.348.0
|
||||
Save global floor…
|
||||
DB override: v0.348.0
|
||||
function confirmGlobalFloor() {
|
||||
2026/10/29 14:48:24 [INFO] Global controller-version floor set to "0.348.0" (declared MinAgent "0.158.1")
|
||||
2026/10/29 14:48:24 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
|
||||
@@ -0,0 +1,2 @@
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
@@ -0,0 +1,15 @@
|
||||
13:49:00 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 1
|
||||
13:52:01 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
13:55:01 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
13:58:02 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 2
|
||||
14:01:02 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 2
|
||||
14:04:02 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
14:07:03 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
14:10:03 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
14:13:04 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
14:16:04 versions: 0.345.1 0.348.0 0.348.0 0.348.0 hub-log error/held/refused lines (3m): 0
|
||||
2026/10/29 14:48:44 [WARN] access token "cc-releases-2" refused: session only — never by a token — GET /customers → 403
|
||||
2026/10/29 14:57:21 [INFO] Event from tester-1: app_held (info) — App gokapi is held: stopped by the box for a crash loop (2026-10-29T13:56:21Z) — the household's Start gives it one more try
|
||||
2026/10/29 14:57:21 [DEBUG] [held-apps] tester-1/gokapi held (unhealthy_stop since 2026-10-29T13:56:21Z)
|
||||
2026/10/29 14:58:21 [INFO] Event from tester-1: app_hold_cleared (info) — App gokapi is no longer held
|
||||
2026/10/29 14:58:21 [INFO] [held-apps] tester-1/gokapi no longer held (a row existed: true)
|
||||
@@ -26,6 +26,15 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-29 (afternoon release) — R-1343, R-1342, R-1340 and R-1341 closed (full text of the rows: `git show e4d91043:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1343** | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). The running controller clears the hold through its own socket and saves at once; the hub's „Clear hold" (hub 0.176.1, D64 = B) goes through it. Proven live on Tester 1: a crash-loop hold cleared from the hub at 12:44 UTC on rc.1 was still gone after the next settings save (13:00:45), and again on the release builds (see the evidence). | `audits/release-2026-10-29b/``p3-clear-hold/p3-*.log`, `final/` |
|
||||
| **R-1342** | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). Proven live on rc.1 (Tester 1): during the stable-start watch the list says „checking", the button is off and a delete gets 409; removing the app ends the watch with no alarm. | `audits/release-2026-10-29b/``p3-clear-hold/r1342-*.log` |
|
||||
| **R-1340** | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). Both producers (the hold mail and `RestoreHold.Why()`) build the text from the restore kind, the cause, the failed step and the replay; test red first. Proven live: a local restore's mixed hold mail names the local restore, the failed put-back stop and the kept copy, and no database. | `audits/release-2026-10-29b/``p3-clear-hold/r1340-*.log` |
|
||||
| **R-1341** | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** (P4) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). The OOM line is logged once per new kill (test red first); live on Tester 1: one WARN line over 4 scans and one `app_oom` event. | `audits/release-2026-10-29b/``p3-clear-hold/r1341-*.log` |
|
||||
|
||||
## 2026-10-29 (release run) — R-1238, R-1338, R-1256 and R-1255 closed (full text of the rows: `git show 650aa0cf:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -169,7 +169,7 @@ stopping line that lies.
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | OPEN | — | the restore's lift calls ForgetUpdateHeld too, red-first test (restore after an update hold → the watcher watches again) | CC |
|
||||
|
||||
## Backup & restore — 48 rows (P2 4, P3 26, P4 18)
|
||||
## Backup & restore — 45 rows (P2 4, P3 23, P4 18)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -217,9 +217,6 @@ stopping line that lies.
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
|
||||
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
|
||||
| **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
|
||||
| **R-1342** | Backup & restore | P3 | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** Found live on Tester 1, 2026-10-29: Gokapi restored 10:08 CET, its copy deleted by hand 10:10 (D62's own live proof), the app removed 10:11; at 10:39 CET the operator got a `backup_run_failures` error mail — both halves of its sentence false. (1) is a gap in D62 (released in controller 0.347.0): the operator's rule is „never while an operation runs on the app"; (2) is older (the R-1270 watch, since 2026-10-26). **Fixed on main** (controller `652a690`, not released — one release per session): the watch marks its copy (`Manager.watching`, under `m.mu`), the list reads it „checking" with one line and no delete date, the delete refuses it; the watch ends quietly when the provider says the app was removed (`StackRemovalReporter`, implemented by the production adapter — a compile-time assertion). Tests `TestR1342_*`, each seen failing by a mutation. Evidence: `audits/d63-2026-10-29/` (hub log of the mail, `run-p4b-d62.log`). | OPEN — fixed on main, not released; rides the next controller release | — | release it (next controller release) | CC |
|
||||
| **R-1343** | Backup & restore | P3 | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** Measured 2026-10-29 on Tester 1: the operator ran `docker exec felhom-controller felhom-controller --clear-restore-hold radicale` at 11:42 CET („restore hold CLEARED … Settings saved"), then `systemctl restart felhom-bootstrap.service` on the Proxmox HOST (the hint's unit name does not exist there; the host's own bootstrap unit skipped itself — no harm); the guest controller was restarted at 11:53 and loaded `settings.json` WITH the hold again (its mtime: the old controller's save after a report). The CLI's own text knew half of this („the running controller keeps refusing until it reloads", measured on demo-hp 2026-08-22). Fix options: the CLI refuses while a controller runs and says „stop it first"; or the clear goes through the running controller (an operator-only local call) so memory and file agree; and the hint names „in the guest (pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service), at once". | OPEN | — | pick the fix, red-first test that a save after the clear does not bring the hold back | CC |
|
||||
| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | OPEN | — | send the operator mail from the start-up put-back failure (the R-1340 sentence), red-first test | CC |
|
||||
|
||||
## Storage & devices — 14 rows (P3 4, P4 10)
|
||||
@@ -321,11 +318,11 @@ stopping line that lies.
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
|
||||
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
|
||||
| **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1350** | Monitoring & notifications | P4 | **When the operator clears a crash-loop hold from the hub, the box sends `app_start_failed` („Telepített alkalmazás nem fut") for that app 30 s later — about an app the operator just chose to leave stopped until the household's Start.** Seen live 2026-10-29 on Tester 1 (controller 0.348.0): clear at 13:57:51 UTC, `app_hold_cleared` and `app_start_failed` both at 13:58:21; the hub's cooldown swallowed the mail only because the crash loop had sent one 3 minutes before. The text is true (the app is installed and not running); the timing is noise. The 12:44 clear (rc.1) showed none in 18 minutes. | OPEN | `audits/release-2026-10-29b/final/clear-hub.txt` | decide whether a box-stopped app whose hold the operator cleared counts as stopped on purpose until the next Start; test red first | CC |
|
||||
|
||||
## Hub & operator — 14 rows (P3 4, P4 10)
|
||||
## Hub & operator — 15 rows (P3 4, P4 11)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -343,6 +340,7 @@ stopping line that lies.
|
||||
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
|
||||
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
|
||||
| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. | OPEN | — | (1) a slow-request line in the hub (method, path, status, duration ≥ 2 s); (2) time the public path from outside the LAN; (3) then fix where the time goes | CC |
|
||||
| **R-1349** | Hub & operator | P4 | **The hub can restart a box's controller (agent, 2026-10-29) but cannot restart the whole guest — a guest whose Docker or network is stuck still needs a person at the Proxmox host.** Operator brief 2026-10-29: „a row for a guest reboot from the hub, if it should come later". Not built: the agent's closed list holds `restart_controller` only (`04` §6.1). A `reboot_guest` verb would stop every app of the household for a minute, so it needs the operator's word on the guards (no backup or restore running, the same brake, a typed confirm) before any code. | OPEN — needs a decision | — | operator: yes/no to a `reboot_guest` verb; if yes, a brief with its guards | operator |
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (afternoon release)
|
||||
|
||||
**Updated 2026-10-29 (attended release).** On your yes: hub 0.175.0, agent 0.168.0 (your vouch), controller 0.347.0 as the floor for every box, and the catalog (Plex night updates; Wanderer's memory figure). D62 is in (restore copies stay 30 days; „Törlés most" deletes one) and D63 held: both untested branches were proven live, nothing reverted. Register 226 → 226 (4 opened, 4 closed); the 10:39 Tester 1 error mail was my test's false alarm.
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (release run)
|
||||
|
||||
**Updated 2026-10-29 (night, unattended).** Nothing released; CC decided (you may reverse) that an unused restore copy is deleted after 14 days. On main, each test seen failing first: four restore fixes, two escrow fixes, a memory kill Docker misses now alarms (proven on Tester 1), and Plex's update test works on both test machines; two promises walked, both pass, register 225 → 226 (1 opened, 0 closed).
|
||||
|
||||
Reference in new issue
Block a user