close-out 2026-10-29 evening release (D65 = A): 04/09/03/05/08 [DESIGN]; R-1349, R-1350 closed (proven live); R-1351 filed; R-1346 measured (the save's own DB step); R-1344 planned events; CONTEXT, STATUS, report, evidence
gates / gates (push) Successful in 1m51s
gates / gates (push) Successful in 1m51s
This commit is contained in:
admin committed
1 parent
8cc555be6c
commit
e7d9c46370
61 files changed
+1838
-11
No files matched your search
@@ -26,6 +26,13 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-29 (evening release) — R-1349 and R-1350 closed (full text of the rows: `git show 8cc555be:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1349** | **The hub can restart a box's controller (agent, 2026-10-29) but cannot restart the whole guest — a guest whose Docker or network is stuck still needs a person at the Proxmox host.** (P4) | CLOSED 2026-10-29 — D65 = A (operator 15:50): `reboot_guest` in the agent's closed list, released in agent **0.170.0** (`da8d12a0…`, vouched 18:57 CET) with hub **0.177.0** (the button) and controller **0.349.0** (the busy file, the household line; global floor 19:24 CET). Proven live on Tester 1: refused during an off-site run (the controller's busy file), done in 67 s / 79 s, a second press inside the hour refused; on the release builds one press 20:05 CET: 'rebooted; the guest was up after 66 s, the controller answered after 78 s', every app back, the dashboard line 'A Felhom ügyfélszolgálata 20:05-kor újraindította a dobozt.' | `audits/release-2026-10-29c/``p1-reboot/`, `final/` |
|
||||
| **R-1350** | **When the operator clears a crash-loop hold from the hub, the box sends `app_start_failed` („Telepített alkalmazás nem fut") for that app 30 s later — about an app the operator just chose to leave stopped until the household's Start.** (P4) | CLOSED 2026-10-29 — released in controller **0.349.0** (`sha256:c5341883…`, from `94049a7`), the global floor since 19:24 CET. The operator's clear records the stop as wanted (desired state 'stopped') before the hold is lifted; test red first. Proven live on Tester 1 (rc.1): a crash-loop hold cleared from the hub, no `app_start_failed` in 6.5 minutes (before: 30 s), the dead-app scan meanwhile '0 currently down'. | `audits/release-2026-10-29c/``p3-r1350/` |
|
||||
|
||||
## 2026-10-29 (afternoon release) — R-1343, R-1342, R-1340 and R-1341 closed (full text of the rows: `git show e4d91043:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -272,7 +272,7 @@ stopping line that lies.
|
||||
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
|
||||
| **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC |
|
||||
|
||||
## Box system & updates — 14 rows (P3 9, P4 5)
|
||||
## Box system & updates — 15 rows (P3 9, P4 6)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -290,9 +290,10 @@ stopping line that lies.
|
||||
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
|
||||
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
|
||||
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
|
||||
| **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/<id>_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `<id>_filebrowser`), FileBrowser „Up (healthy)", files.<domain> 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | serialise the two FileBrowser start paths at boot (one waits for the other, or the bring-up skips a stack another path is recreating); test red first | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 25 rows (P2 2, P3 15, P4 8)
|
||||
## Monitoring & notifications — 24 rows (P2 2, P3 15, P4 7)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -319,10 +320,9 @@ stopping line that lies.
|
||||
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
|
||||
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1350** | Monitoring & notifications | P4 | **When the operator clears a crash-loop hold from the hub, the box sends `app_start_failed` („Telepített alkalmazás nem fut") for that app 30 s later — about an app the operator just chose to leave stopped until the household's Start.** Seen live 2026-10-29 on Tester 1 (controller 0.348.0): clear at 13:57:51 UTC, `app_hold_cleared` and `app_start_failed` both at 13:58:21; the hub's cooldown swallowed the mail only because the crash loop had sent one 3 minutes before. The text is true (the app is installed and not running); the timing is noise. The 12:44 clear (rc.1) showed none in 18 minutes. | OPEN | `audits/release-2026-10-29b/final/clear-hub.txt` | decide whether a box-stopped app whose hold the operator cleared counts as stopped on purpose until the next Start; test red first | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. **Planned events in the log window (2026-10-29 evening, leave them out of the reading):** Tester 1 — 18:00 UTC rc swap, 18:05 and 19:05 UTC operator reboots (D65 proofs), 18:21 UTC floor update; demo-hp — 18:21 UTC floor update; 9202 is not logged. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
|
||||
## Hub & operator — 15 rows (P3 4, P4 11)
|
||||
## Hub & operator — 14 rows (P3 4, P4 10)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -339,8 +339,7 @@ stopping line that lies.
|
||||
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
|
||||
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
|
||||
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
|
||||
| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. | OPEN | — | (1) a slow-request line in the hub (method, path, status, duration ≥ 2 s); (2) time the public path from outside the LAN; (3) then fix where the time goes | CC |
|
||||
| **R-1349** | Hub & operator | P4 | **The hub can restart a box's controller (agent, 2026-10-29) but cannot restart the whole guest — a guest whose Docker or network is stuck still needs a person at the Proxmox host.** Operator brief 2026-10-29: „a row for a guest reboot from the hub, if it should come later". Not built: the agent's closed list holds `restart_controller` only (`04` §6.1). A `reboot_guest` verb would stop every app of the household for a minute, so it needs the operator's word on the guards (no backup or restore running, the same brake, a typed confirm) before any code. | OPEN — needs a decision | — | operator: yes/no to a `reboot_guest` verb; if yes, a brief with its guards | operator |
|
||||
| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. **2026-10-29 evening (hub 0.177.0) — WHERE THE TIME GOES: the save itself, in the hub's own database step.** The hub's access log holds seven vouch saves (POST `/configuration/artifacts`) since 2026-10-25 of 33–170 s; the two 170-s saves (13:03:19Z, 13:04:59Z) overlapped the hub's 13:00Z database snapshot slot. ingress-nginx's default 60-s read timeout turns any save over 60 s into the browser's 504 while the hub finishes it; the hub's own `WriteTimeout` is 60 s too. Not the road: the page loads in 6 ms in the cluster and 65–255 ms through the public name; not the registry: each of the save's calls (tag, 2 package probes, 2 sha lists, bundle sha) 0.08–0.25 s from DooPlex and 0.01–0.03 s from inside the hub pod. 0.177.0's per-leg line on the operator's 0.170.0 vouch (18:57:48 CET): registry probes 65 ms, sha lookups 27 ms, bundle sha 43 ms, **fleet check + save 38 117 ms**, total 38 255 ms. That step is `NewestReportedControllerVersion` (a `DISTINCT` scan of `reports`, 21 563 rows, no index on the column — 0.08 s on a tmpfs copy of the snapshot, so not the scan alone) and `SetArtifactManifest` (7 separate `hub_settings` writes, each its own transaction, each waiting for the write lock under `busy_timeout` 5 s). The slow-request line also named `/apps` 25–30 s, `/customers/tester-1` 7–9 s and `/system` 5.8 s in the last days (access log). Evidence: `audits/release-2026-10-29c/r1346/`. | OPEN — measured | — | (1) time `NewestReportedControllerVersion` and each `setSetting` separately inside the save (one more line); (2) if the writes wait: one transaction for the manifest, and find who holds the write lock that long; (3) the same for `/apps` and the customer page | CC |
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (evening release)
|
||||
|
||||
**Updated 2026-10-29 (afternoon release).** On your yes: hub 0.176.1, agent 0.169.0 (your vouch) and controller 0.348.0 as the floor for every box. From the hub's host page you can now restart a box's controller (it works when the controller is stopped; 3 presses in 15 minutes, then 30 minutes refused) and clear any held app (D64 = B) — both proven on Tester 1 on the release builds. Register 226 → 229 (7 opened, 4 closed); a stuck-but-running controller only alarms you until CC reads tonight's measurement.
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (afternoon release)
|
||||
|
||||
**Updated 2026-10-29 (attended release).** On your yes: hub 0.175.0, agent 0.168.0 (your vouch), controller 0.347.0 as the floor for every box, and the catalog (Plex night updates; Wanderer's memory figure). D62 is in (restore copies stay 30 days; „Törlés most" deletes one) and D63 held: both untested branches were proven live, nothing reverted. Register 226 → 226 (4 opened, 4 closed); the 10:39 Tester 1 error mail was my test's false alarm.
|
||||
|
||||
Reference in new issue
Block a user