felhom.eu/REPORT-release-2026-10-30.md

# REPORT — release run 2026-10-30 (day, attended): the stuck-controller decision, the slow hub, the site on Cloudflare

Read first: `03-host-agent.md` (the supervisor), `05-hub-architecture.md` (the store, the snapshot, the pages),
`01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`, `runbooks/iso-release-gate.md`,
`runbooks/publish-train-rules.md`, `audits/dooplex-survival-2026-10-22/PLAN.md` (G8); rows R-1344, R-1346, R-1228, R-1326.

## Five lines

1. **The stuck-controller decision: RESTART.** The night's log (4 733 probes, Tester 1 and demo-hp, 2026-10-29 11:12 →
   10-30 07:00 UTC, through both vzdumps, the DB dumps, tier-2, the off-site push, a restore test and two updates)
   had no failed probe and no 5-s timeout; the slowest was 303 ms. So agent 0.171.0 restarts a controller that runs but
   does not answer at 10 unhealthy sweeps (alarm at 6) — proven on Tester 1: frozen 09:24:50 → restarted 09:31:22 UTC.
2. **The hub's slowness:** five statements read whole tables from the network disk, with 256 MiB for a 570 MB database.
   Four indexes, one hint, one transaction and 1 GiB fixed it: your vouch save 38 255 ms → **132 ms**; `/apps` 25–30 s →
   0.45 s (3.4 s cold); `/system` 5.8 s → 0.45 s.
3. **The site on Cloudflare: switched** at 10:02:35 UTC on your yes. preview.felhom.eu and then felhom.eu/www: 30 of 30
   pages, the installer byte-identical to `installer-v1.36.1`, a preflight from a fresh ISO install passed with both.
4. **A new install still needs DooPlex for** the hub (claim, config, first report) and Gitea (the agent package, the
   golden) — by design (D35). It no longer needs DooPlex to fetch the installer or show the site.
5. **What you do:** discard the unclaimed appliance (uuid `d3529d2d…`, my deleted proof VM) on the Hosts page. Nothing
   else is blocked; on 2026-11-06 I ask whether to retire DooPlex's old website copy.

## Versions, digests, CI by commit

| Repo | Commit | CI job | What |
|---|---|---|---|
| agent | `bd18347` | 2532 green | R-1344 restart — **0.171.0 built from here** (sha256 `c888ca7b…202a5`, bundle `95f977f6…3d2b`, tag `v0.171.0`) |
| agent | `84726c9` | (close-out) | v0.171.0 heading; `pinned.day0_agent` = 0.171.0 (vouched 10:47:36 CET) |
| felhom.eu | `be538975` | (no own job; covered by 2533) | hub: `controller_unresponsive_restart`, the alarm text follows the agent |
| felhom.eu | `a13c69b0` | (covered by 2533) | hub 0.178.0: indexes, hint, one-transaction manifest, slow-statement line; evidence |
| felhom.eu | `f1ab634c` | 2533 green | R-1228 dated check removed — **hub 0.178.0 built from here** (`sha256:9c15c510…d925f`) |
| felhom.eu | `cea8a907` | 2535 green | manifest: hub 0.178.0, memory 1 GiB (Deployment/hub only, Synced/Healthy) |
| felhom.eu | `bd74a613`, `53107f1b`, `e0e55bf8` | read at the close-out | `api.felhom.eu` ingress; the publish script + the form endpoint; the ingress narrowed to `/api/contact` + 20 req/min |

Cloudflare: Pages project `felhom-site` (2 uploads), custom domains `preview.felhom.eu`, `felhom.eu`, `www.felhom.eu`;
DNS: `preview` and `api` added, `felhom.eu` and `www` switched (undo ids in `p3-cloudflare/switch-undo.txt`).

## Rows

Before 228, after 228. **Closed (2):** R-1344, R-1346. **Opened (2):** R-1352 (the alarm mail arrives after the
restart mail when both fall between two reports, P4), R-1353 (statements still over 1 s, P4). **Narrowed:** R-1228
(switched; DooPlex copy until 2026-11-06, DUE-CHECKS).

## Observations

- **FILED: R-1352** — live, the hub mailed `controller_unresponsive_restart` and `controller_unresponsive` in the same
  second, restart first.
- **FILED: R-1353** — the new line named a background deadline check reading days of full host-report JSON (up to
  9.2 s cold), the start-up prune of `events`/`app_log_issues` (1–1.8 s), and the customer page's newest-report read.
- **NOT-A-FINDING: the first probe run slowed the live hub** (one host report took 25 s) — my own probe's random reads
  on the shared Longhorn volume; stopped at once, and the measurement itself is the finding (R-1346, fixed).
- **NOT-A-FINDING: the hub was down ~2 min at the deploy** — the one-time index build (103 s) inside the 180-s startup
  probe; R-1326's start-up is otherwise unchanged (the index check is four tiny queries).
- **NOT-A-FINDING: ArgoCD shows `Ingress/contact-mailer` OutOfSync** — the live object matches git field by field; not
  synced (outside this run's change).
- **NOT-A-FINDING: demo-hp's local whole-guest backup skipped for space** (09:41 CET) — R-548.
- **NOT-A-FINDING: a 70-s 522 on felhom.eu at the switch** — the time Cloudflare took to activate the custom domains;
  one 522 on the first full pass, then 30 of 30 twice.
- **NOT-A-FINDING: the full install (after the bind) was not run** — the bind is a session-only operator act and the
  install would create Tester4's tunnel and off-site account; the preflight (both addresses) and the byte identity
  carry the proof.

## Teardown

Machine: VM 3461 destroyed on demo-hp, no disk left on `nvme-scratch`; the DB copies (tmpfs on DooPlex and
`/data/r1346-probe` in the hub pod) deleted, the access-log copy shredded, the probe binary removed from the pod.
Host: nothing else provisioned. Hub: one unclaimed appliance entry (uuid `d3529d2d…`) — the operator's to discard.

## Docs

`03` [DESIGN] R-1344 restart; `05` [DESIGN] R-1346 + the restart event; `01` [DESIGN] the site on Cloudflare (row +
paragraph); `runbooks/total-loss-of-dooplex.md` (what a new install still needs); `runbooks/publish-train-rules.md` §6;
the system poster's facts (website on Cloudflare, mailer on DooPlex) — the drawing needs a refresh (STATUS line).
Instruction files corrected (stale facts, D35): `CLAUDE.md` „To publish / To roll back" (+ the upload script);
`.claude/rules/website.md` „Deploy" (git-sync is the way back, the script publishes); `skills/felhom-build-deploy`
„Website" (same). `unproven.py --summary`: not walked 31 of 55, unchanged.

Evidence: `documentation/audits/release-2026-10-30/` (`p1-r1344/`, `p2-r1346/`, `p3-cloudflare/`).