close-out 2026-10-29 afternoon release: R-1343/R-1342/R-1340/R-1341 closed (controller 0.348.0, proven live); R-1349 (guest reboot, decide D65), R-1350 (app_start_failed after a clear) filed; R-1344 numbers so far; CONTEXT (D64), STATUS, report, evidence
gates / gates (push) Successful in 1m47s
gates / gates (push) Successful in 1m47s
This commit is contained in:
1 parent
e4d91043e4
commit
dd3e4137ee
16 files changed
+237
-9
No files matched your search
@@ -26,6 +26,15 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-29 (afternoon release) — R-1343, R-1342, R-1340 and R-1341 closed (full text of the rows: `git show e4d91043:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1343** | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). The running controller clears the hold through its own socket and saves at once; the hub's „Clear hold" (hub 0.176.1, D64 = B) goes through it. Proven live on Tester 1: a crash-loop hold cleared from the hub at 12:44 UTC on rc.1 was still gone after the next settings save (13:00:45), and again on the release builds (see the evidence). | `audits/release-2026-10-29b/``p3-clear-hold/p3-*.log`, `final/` |
|
||||
| **R-1342** | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). Proven live on rc.1 (Tester 1): during the stable-start watch the list says „checking", the button is off and a delete gets 409; removing the app ends the watch with no alarm. | `audits/release-2026-10-29b/``p3-clear-hold/r1342-*.log` |
|
||||
| **R-1340** | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** (P3) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). Both producers (the hold mail and `RestoreHold.Why()`) build the text from the restore kind, the cause, the failed step and the replay; test red first. Proven live: a local restore's mixed hold mail names the local restore, the failed put-back stop and the kept copy, and no database. | `audits/release-2026-10-29b/``p3-clear-hold/r1340-*.log` |
|
||||
| **R-1341** | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** (P4) | CLOSED 2026-10-29 — released in controller **0.348.0** (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515), the global floor since 14:48 CET (MinAgent 0.158.1). The OOM line is logged once per new kill (test red first); live on Tester 1: one WARN line over 4 scans and one `app_oom` event. | `audits/release-2026-10-29b/``p3-clear-hold/r1341-*.log` |
|
||||
|
||||
## 2026-10-29 (release run) — R-1238, R-1338, R-1256 and R-1255 closed (full text of the rows: `git show 650aa0cf:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -169,7 +169,7 @@ stopping line that lies.
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | OPEN | — | the restore's lift calls ForgetUpdateHeld too, red-first test (restore after an update hold → the watcher watches again) | CC |
|
||||
|
||||
## Backup & restore — 48 rows (P2 4, P3 26, P4 18)
|
||||
## Backup & restore — 45 rows (P2 4, P3 23, P4 18)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -217,9 +217,6 @@ stopping line that lies.
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
|
||||
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
|
||||
| **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
|
||||
| **R-1342** | Backup & restore | P3 | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** Found live on Tester 1, 2026-10-29: Gokapi restored 10:08 CET, its copy deleted by hand 10:10 (D62's own live proof), the app removed 10:11; at 10:39 CET the operator got a `backup_run_failures` error mail — both halves of its sentence false. (1) is a gap in D62 (released in controller 0.347.0): the operator's rule is „never while an operation runs on the app"; (2) is older (the R-1270 watch, since 2026-10-26). **Fixed on main** (controller `652a690`, not released — one release per session): the watch marks its copy (`Manager.watching`, under `m.mu`), the list reads it „checking" with one line and no delete date, the delete refuses it; the watch ends quietly when the provider says the app was removed (`StackRemovalReporter`, implemented by the production adapter — a compile-time assertion). Tests `TestR1342_*`, each seen failing by a mutation. Evidence: `audits/d63-2026-10-29/` (hub log of the mail, `run-p4b-d62.log`). | OPEN — fixed on main, not released; rides the next controller release | — | release it (next controller release) | CC |
|
||||
| **R-1343** | Backup & restore | P3 | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** Measured 2026-10-29 on Tester 1: the operator ran `docker exec felhom-controller felhom-controller --clear-restore-hold radicale` at 11:42 CET („restore hold CLEARED … Settings saved"), then `systemctl restart felhom-bootstrap.service` on the Proxmox HOST (the hint's unit name does not exist there; the host's own bootstrap unit skipped itself — no harm); the guest controller was restarted at 11:53 and loaded `settings.json` WITH the hold again (its mtime: the old controller's save after a report). The CLI's own text knew half of this („the running controller keeps refusing until it reloads", measured on demo-hp 2026-08-22). Fix options: the CLI refuses while a controller runs and says „stop it first"; or the clear goes through the running controller (an operator-only local call) so memory and file agree; and the hint names „in the guest (pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service), at once". | OPEN | — | pick the fix, red-first test that a save after the clear does not bring the hold back | CC |
|
||||
| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | OPEN | — | send the operator mail from the start-up put-back failure (the R-1340 sentence), red-first test | CC |
|
||||
|
||||
## Storage & devices — 14 rows (P3 4, P4 10)
|
||||
@@ -321,11 +318,11 @@ stopping line that lies.
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
|
||||
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
|
||||
| **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1350** | Monitoring & notifications | P4 | **When the operator clears a crash-loop hold from the hub, the box sends `app_start_failed` („Telepített alkalmazás nem fut") for that app 30 s later — about an app the operator just chose to leave stopped until the household's Start.** Seen live 2026-10-29 on Tester 1 (controller 0.348.0): clear at 13:57:51 UTC, `app_hold_cleared` and `app_start_failed` both at 13:58:21; the hub's cooldown swallowed the mail only because the crash loop had sent one 3 minutes before. The text is true (the app is installed and not running); the timing is noise. The 12:44 clear (rc.1) showed none in 18 minutes. | OPEN | `audits/release-2026-10-29b/final/clear-hub.txt` | decide whether a box-stopped app whose hold the operator cleared counts as stopped on purpose until the next Start; test red first | CC |
|
||||
|
||||
## Hub & operator — 14 rows (P3 4, P4 10)
|
||||
## Hub & operator — 15 rows (P3 4, P4 11)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -343,6 +340,7 @@ stopping line that lies.
|
||||
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
|
||||
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
|
||||
| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. | OPEN | — | (1) a slow-request line in the hub (method, path, status, duration ≥ 2 s); (2) time the public path from outside the LAN; (3) then fix where the time goes | CC |
|
||||
| **R-1349** | Hub & operator | P4 | **The hub can restart a box's controller (agent, 2026-10-29) but cannot restart the whole guest — a guest whose Docker or network is stuck still needs a person at the Proxmox host.** Operator brief 2026-10-29: „a row for a guest reboot from the hub, if it should come later". Not built: the agent's closed list holds `restart_controller` only (`04` §6.1). A `reboot_guest` verb would stop every app of the household for a minute, so it needs the operator's word on the guards (no backup or restore running, the same brake, a typed confirm) before any code. | OPEN — needs a decision | — | operator: yes/no to a `reboot_guest` verb; if yes, a brief with its guards | operator |
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (afternoon release)
|
||||
|
||||
**Updated 2026-10-29 (attended release).** On your yes: hub 0.175.0, agent 0.168.0 (your vouch), controller 0.347.0 as the floor for every box, and the catalog (Plex night updates; Wanderer's memory figure). D62 is in (restore copies stay 30 days; „Törlés most" deletes one) and D63 held: both untested branches were proven live, nothing reverted. Register 226 → 226 (4 opened, 4 closed); the 10:39 Tester 1 error mail was my test's false alarm.
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (release run)
|
||||
|
||||
**Updated 2026-10-29 (night, unattended).** Nothing released; CC decided (you may reverse) that an unused restore copy is deleted after 14 days. On main, each test seen failing first: four restore fixes, two escrow fixes, a memory kill Docker misses now alarms (proven on Tester 1), and Plex's update test works on both test machines; two promises walked, both pass, register 225 → 226 (1 opened, 0 closed).
|
||||
|
||||
Reference in new issue
Block a user