close-out 2026-10-30: R-1344 and R-1346 closed, R-1228 switched (D35) and narrowed; R-1352, R-1353 filed; 01/03/05 [DESIGN], total-loss + publish-train runbooks, poster facts; instruction files: the site publishes to Cloudflare Pages
gates / gates (push) Successful in 1m47s
gates / gates (push) Successful in 1m47s
This commit is contained in:
admin committed
1 parent
e0e55bf852
commit
3cda7e8d6c
30 files changed
+561
-17
No files matched your search
@@ -26,6 +26,13 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-30 — R-1344 and R-1346 closed (full text of the rows: `git show e0e55bf8:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1344** | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** (P3) | CLOSED 2026-10-30 — measured, then decided: 4 733 probes on Tester 1 and demo-hp (2026-10-29 11:12 → 10-30 07:00 UTC) through the night chain: no failed probe, slowest 303 ms against the 5 s timeout. Agent 0.171.0 (`bd18347`, vouched 10:47:36 CET) restarts at 10 unhealthy sweeps (alarm at 6), refused by a fresh or unreadable busy file and the brake; hub 0.178.0 mails `controller_unresponsive_restart`. Live on Tester 1: frozen 09:24:50 UTC → alarm 09:29:21 → restart 09:31:22 → healthy 09:31:24. Order of the two mails: R-1352 | `audits/release-2026-10-30/p1-r1344/` |
|
||||
| **R-1346** | **Hub pages load slowly, and the vouch save ended in a 504 while it went through.** (P3) | CLOSED 2026-10-30 — the cause per statement (`store/r1346_speed.go`): whole-table reads on a network disk with a 256 MiB pod for a 570 MB database. Four indexes + one hint (pages read 35 623 → 90, 38 570 → 118, 30 097 → 1 013, 15 404 → 1 437), the manifest in one transaction, the slow-statement line, memory 1 GiB (operator yes). Hub 0.178.0 deployed; the operator's vouch save: 38 255 ms → **132 ms**; `/apps` 25–30 s → 0.45 s warm, `/system` 5.8 s → 0.45 s. What still exceeds 1 s: R-1353 | `audits/release-2026-10-30/p2-r1346/` |
|
||||
|
||||
## 2026-10-29 (evening release) — R-1349 and R-1350 closed (full text of the rows: `git show 8cc555be:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -211,7 +211,7 @@ stopping line that lies.
|
||||
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
|
||||
| **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
|
||||
| **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: STARTED** (operator brief 2026-10-30 Part 3; the dated check is removed — the work is under way). OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
|
||||
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
@@ -293,7 +293,7 @@ stopping line that lies.
|
||||
| **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/<id>_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `<id>_filebrowser`), FileBrowser „Up (healthy)", files.<domain> 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | serialise the two FileBrowser start paths at boot (one waits for the other, or the bring-up skips a stack another path is recreating); test red first | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 24 rows (P2 2, P3 15, P4 7)
|
||||
## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -320,9 +320,9 @@ stopping line that lies.
|
||||
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
|
||||
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. **Planned events in the log window (2026-10-29 evening, leave them out of the reading):** Tester 1 — 18:00 UTC rc swap, 18:05 and 19:05 UTC operator reboots (D65 proofs), 18:21 UTC floor update; demo-hp — 18:21 UTC floor update; 9202 is not logged. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
|
||||
| **R-1352** | Monitoring & notifications | P4 | **When a controller stops answering AND the agent restarts it between two host reports, the operator gets the restart mail first and the „does not answer — the agent restarts it in 2 minutes" alarm second, in the same second.** Seen live 2026-10-30 (the R-1344 proof on Tester 1: alarm 09:29:21, restart 09:31:22, both reached the hub with the 09:42:56 report; `hub/internal/monitor/controller_supervisor.go` `observe()` emits the restart before the alarm). The text of each mail is true; only the order reads backwards. | OPEN | — | emit the alarm before the restart in `observe()`, or skip the alarm when the restart for the same episode arrives with it (one line + a test) | CC |
|
||||
|
||||
## Hub & operator — 14 rows (P3 4, P4 10)
|
||||
## Hub & operator — 14 rows (P3 3, P4 11)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -339,7 +339,7 @@ stopping line that lies.
|
||||
| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
|
||||
| **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
|
||||
| **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
|
||||
| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. **2026-10-29 evening (hub 0.177.0) — WHERE THE TIME GOES: the save itself, in the hub's own database step.** The hub's access log holds seven vouch saves (POST `/configuration/artifacts`) since 2026-10-25 of 33–170 s; the two 170-s saves (13:03:19Z, 13:04:59Z) overlapped the hub's 13:00Z database snapshot slot. ingress-nginx's default 60-s read timeout turns any save over 60 s into the browser's 504 while the hub finishes it; the hub's own `WriteTimeout` is 60 s too. Not the road: the page loads in 6 ms in the cluster and 65–255 ms through the public name; not the registry: each of the save's calls (tag, 2 package probes, 2 sha lists, bundle sha) 0.08–0.25 s from DooPlex and 0.01–0.03 s from inside the hub pod. 0.177.0's per-leg line on the operator's 0.170.0 vouch (18:57:48 CET): registry probes 65 ms, sha lookups 27 ms, bundle sha 43 ms, **fleet check + save 38 117 ms**, total 38 255 ms. That step is `NewestReportedControllerVersion` (a `DISTINCT` scan of `reports`, 21 563 rows, no index on the column — 0.08 s on a tmpfs copy of the snapshot, so not the scan alone) and `SetArtifactManifest` (7 separate `hub_settings` writes, each its own transaction, each waiting for the write lock under `busy_timeout` 5 s). The slow-request line also named `/apps` 25–30 s, `/customers/tester-1` 7–9 s and `/system` 5.8 s in the last days (access log). Evidence: `audits/release-2026-10-29c/r1346/`. | OPEN — measured | — | (1) time `NewestReportedControllerVersion` and each `setSetting` separately inside the save (one more line); (2) if the writes wait: one transaction for the manifest, and find who holds the write lock that long; (3) the same for `/apps` and the customer page | CC |
|
||||
| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC |
|
||||
|
||||
## Business & legal — 14 rows (P2 4, P3 2, P4 8)
|
||||
|
||||
@@ -409,6 +409,7 @@ stopping line that lies.
|
||||
| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
|
||||
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
|
||||
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
|
||||
| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
|
||||
| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC |
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-30
|
||||
|
||||
**Updated 2026-10-29 (evening release).** On your yes: hub 0.177.0, agent 0.170.0 (your vouch) and controller 0.349.0 for every box; „Reboot guest" on the hub works and refuses during backups, updates and twice in an hour, and no alarm follows your hold clear any more — proven on Tester 1. The slow vouch is the hub's own database step (38 of 38.3 s); the stuck-controller decision waits for tomorrow's night log.
|
||||
|
||||
## Moved from STATUS on 2026-10-29 (evening release)
|
||||
|
||||
**Updated 2026-10-29 (afternoon release).** On your yes: hub 0.176.1, agent 0.169.0 (your vouch) and controller 0.348.0 as the floor for every box. From the hub's host page you can now restart a box's controller (it works when the controller is stopped; 3 presses in 15 minutes, then 30 minutes refused) and clear any held app (D64 = B) — both proven on Tester 1 on the release builds. Register 226 → 229 (7 opened, 4 closed); a stuck-but-running controller only alarms you until CC reads tonight's measurement.
|
||||
|
||||
Reference in new issue
Block a user