6.0 KiB
REPORT — release run 2026-10-30 (day, attended): the stuck-controller decision, the slow hub, the site on Cloudflare
Read first: 03-host-agent.md (the supervisor), 05-hub-architecture.md (the store, the snapshot, the pages),
01-topology-and-trust.md, runbooks/total-loss-of-dooplex.md, runbooks/iso-release-gate.md,
runbooks/publish-train-rules.md, audits/dooplex-survival-2026-10-22/PLAN.md (G8); rows R-1344, R-1346, R-1228, R-1326.
Five lines
- The stuck-controller decision: RESTART. The night's log (4 733 probes, Tester 1 and demo-hp, 2026-10-29 11:12 → 10-30 07:00 UTC, through both vzdumps, the DB dumps, tier-2, the off-site push, a restore test and two updates) had no failed probe and no 5-s timeout; the slowest was 303 ms. So agent 0.171.0 restarts a controller that runs but does not answer at 10 unhealthy sweeps (alarm at 6) — proven on Tester 1: frozen 09:24:50 → restarted 09:31:22 UTC.
- The hub's slowness: five statements read whole tables from the network disk, with 256 MiB for a 570 MB database.
Four indexes, one hint, one transaction and 1 GiB fixed it: your vouch save 38 255 ms → 132 ms;
/apps25–30 s → 0.45 s (3.4 s cold);/system5.8 s → 0.45 s. - The site on Cloudflare: switched at 10:02:35 UTC on your yes. preview.felhom.eu and then felhom.eu/www: 30 of 30
pages, the installer byte-identical to
installer-v1.36.1, a preflight from a fresh ISO install passed with both. - A new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35). It no longer needs DooPlex to fetch the installer or show the site.
- What you do: discard the unclaimed appliance (uuid
d3529d2d…, my deleted proof VM) on the Hosts page. Nothing else is blocked; on 2026-11-06 I ask whether to retire DooPlex's old website copy.
Versions, digests, CI by commit
| Repo | Commit | CI job | What |
|---|---|---|---|
| agent | bd18347 |
2532 green | R-1344 restart — 0.171.0 built from here (sha256 c888ca7b…202a5, bundle 95f977f6…3d2b, tag v0.171.0) |
| agent | 84726c9 |
(close-out) | v0.171.0 heading; pinned.day0_agent = 0.171.0 (vouched 10:47:36 CET) |
| felhom.eu | be538975 |
(no own job; covered by 2533) | hub: controller_unresponsive_restart, the alarm text follows the agent |
| felhom.eu | a13c69b0 |
(covered by 2533) | hub 0.178.0: indexes, hint, one-transaction manifest, slow-statement line; evidence |
| felhom.eu | f1ab634c |
2533 green | R-1228 dated check removed — hub 0.178.0 built from here (sha256:9c15c510…d925f) |
| felhom.eu | cea8a907 |
2535 green | manifest: hub 0.178.0, memory 1 GiB (Deployment/hub only, Synced/Healthy) |
| felhom.eu | bd74a613, 53107f1b, e0e55bf8 |
read at the close-out | api.felhom.eu ingress; the publish script + the form endpoint; the ingress narrowed to /api/contact + 20 req/min |
Cloudflare: Pages project felhom-site (2 uploads), custom domains preview.felhom.eu, felhom.eu, www.felhom.eu;
DNS: preview and api added, felhom.eu and www switched (undo ids in p3-cloudflare/switch-undo.txt).
Rows
Before 228, after 228. Closed (2): R-1344, R-1346. Opened (2): R-1352 (the alarm mail arrives after the restart mail when both fall between two reports, P4), R-1353 (statements still over 1 s, P4). Narrowed: R-1228 (switched; DooPlex copy until 2026-11-06, DUE-CHECKS).
Observations
- FILED: R-1352 — live, the hub mailed
controller_unresponsive_restartandcontroller_unresponsivein the same second, restart first. - FILED: R-1353 — the new line named a background deadline check reading days of full host-report JSON (up to
9.2 s cold), the start-up prune of
events/app_log_issues(1–1.8 s), and the customer page's newest-report read. - NOT-A-FINDING: the first probe run slowed the live hub (one host report took 25 s) — my own probe's random reads on the shared Longhorn volume; stopped at once, and the measurement itself is the finding (R-1346, fixed).
- NOT-A-FINDING: the hub was down ~2 min at the deploy — the one-time index build (103 s) inside the 180-s startup probe; R-1326's start-up is otherwise unchanged (the index check is four tiny queries).
- NOT-A-FINDING: ArgoCD shows
Ingress/contact-mailerOutOfSync — the live object matches git field by field; not synced (outside this run's change). - NOT-A-FINDING: demo-hp's local whole-guest backup skipped for space (09:41 CET) — R-548.
- NOT-A-FINDING: a 70-s 522 on felhom.eu at the switch — the time Cloudflare took to activate the custom domains; one 522 on the first full pass, then 30 of 30 twice.
- NOT-A-FINDING: the full install (after the bind) was not run — the bind is a session-only operator act and the install would create Tester4's tunnel and off-site account; the preflight (both addresses) and the byte identity carry the proof.
Teardown
Machine: VM 3461 destroyed on demo-hp, no disk left on nvme-scratch; the DB copies (tmpfs on DooPlex and
/data/r1346-probe in the hub pod) deleted, the access-log copy shredded, the probe binary removed from the pod.
Host: nothing else provisioned. Hub: one unclaimed appliance entry (uuid d3529d2d…) — the operator's to discard.
Docs
03 [DESIGN] R-1344 restart; 05 [DESIGN] R-1346 + the restart event; 01 [DESIGN] the site on Cloudflare (row +
paragraph); runbooks/total-loss-of-dooplex.md (what a new install still needs); runbooks/publish-train-rules.md §6;
the system poster's facts (website on Cloudflare, mailer on DooPlex) — the drawing needs a refresh (STATUS line).
Instruction files corrected (stale facts, D35): CLAUDE.md „To publish / To roll back" (+ the upload script);
.claude/rules/website.md „Deploy" (git-sync is the way back, the script publishes); skills/felhom-build-deploy
„Website" (same). unproven.py --summary: not walked 31 of 55, unchanged.
Evidence: documentation/audits/release-2026-10-30/ (p1-r1344/, p2-r1346/, p3-cloudflare/).