close-out 2026-10-30: R-1344 and R-1346 closed, R-1228 switched (D35) and narrowed; R-1352, R-1353 filed; 01/03/05 [DESIGN], total-loss + publish-train runbooks, poster facts; instruction files: the site publishes to Cloudflare Pages
gates / gates (push) Successful in 1m47s

This commit is contained in:
admin committed 2026-10-30 11:13:17 +01:00
1 parent e0e55bf852
commit 3cda7e8d6c
30 files changed
+561 -17

No files matched your search

@@ -129,6 +129,7 @@ credentials.
| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) |
| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
| new box ↔ public site | the installer script (`felhom.eu/scripts/felhom-host-install.sh`) + the website | **[DESIGN] 2026-10-30 (D35 / R-1228): Cloudflare Pages** (project `felhom-site`), uploaded by `scripts/publish-site-cloudflare.sh` from `main` + the installer tag; the contact form posts to `api.felhom.eu` (DooPlex's mailer). Before: DooPlex nginx + git-sync, kept 7 days as the way back | a changed installer runs as root on every new box — the script refuses an upload whose installer is not the tag's byte for byte |
**Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is
a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the
@@ -288,6 +289,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
work.** The one check that is off today is the route's own: `*.<domain>` → `https://traefik` with `noTLSVerify`
(R-510; traefik shows its self-signed default certificate on a leg that never leaves the guest) — a recorded
exception, not a pattern to copy.
- **[DESIGN] 2026-10-30 — the public site and the installer are on Cloudflare Pages (D35 / R-1228).** `felhom.eu`
and `www` point to the Pages project `felhom-site` (uploaded by `scripts/publish-site-cloudflare.sh`); the contact
form posts to `api.felhom.eu` (DooPlex's mailer). The paragraph below describes DooPlex's copy, kept running 7 days
from the switch as the way back (`audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5).
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
@@ -214,6 +214,20 @@ by verb**:
`unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a
parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both
test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long.
- **`[DESIGN]` 2026-10-30 — R-1344 decided from the measurement: the alarm becomes a RESTART (agent `bd18347`,
released v0.171.0; hub 0.178.0 `controller_unresponsive_restart`).** The measurement (`audits/release-2026-10-30/
p1-r1344/`): 4 733 probes on Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, through demo-hp's PBS
vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test and two controller updates —
no failed probe, no 5-s timeout, slowest 303 ms (demo-hp, during the PBS vzdump). So: the alarm stays at 6 unhealthy
sweeps; at **10** (2 minutes later, ~6.5 minutes after the first failed probe) the agent restarts the bootstrap unit
— unless the controller's busy file is fresh (a backup, a restore, an update, an export) or unreadable, or the
supervisor's brake refuses (3 restarts in 15 minutes → the crash-loop pause). A swap, a parked or locked guest and a
vzdump keep resetting the count. The restart counts in both brakes (fast and R-539's slow counter); after it the count
starts from zero. The stanza carries `unresponsive_restart_after_s` (the hub's alarm text says when the restart comes,
and says „does not restart" for an older agent) and `unresponsive_restart_at` (the hub mails
`controller_unresponsive_restart` once — not also `controller_restarted_by_agent`). Proven live on Tester 1
2026-10-30: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the
restart 09:31:22 → healthy 09:31:24.
- **`[DESIGN]` 2026-10-29 — the operator's „Reboot guest" (D65 = A, agent `ca15a31`, released v0.170.0; `04` §6.1,
`09` §3 decision 284).** The agent's list gains `reboot_guest <vmid>`: the existing guest-reboot path (`pct reboot`
of THAT guest — never the host), for a stuck Docker or a stuck network inside the box. Guards, each refusing with the
@@ -833,3 +833,20 @@ stores an AGENT row `reboot_guest <vmid>` and pokes the box like the restart. **
itself when it comes back. The agent's result closes the row with one `operator_action` event. **R-1346:** the access-log
middleware also writes `[WARN] slow request: <method> <path> <status> <ms>` for a request over 5 s (path only;
`/api/v1/wait` exempt), and the artifact vouch save logs each leg's time.
**[DESIGN] 2026-10-30 — the hub's database speed (R-1346, hub 0.178.0).** Measured on a copy of the live database
inside the hub pod (`audits/release-2026-10-30/p2-r1346/`): the pod had 256 MiB for a 570 MB database, so nearly every
page read went to Longhorn, and five statements read whole tables. `controller_version` and `agent_version` sit after
`report_json` in their rows, so reading them walks each row's JSON. Four indexes (`idx_reports_controller_version`,
`idx_host_reports_agent_version`, `idx_reports_received`, `idx_host_reports_received`) and one hint (`INDEXED BY
idx_app_telemetry_prune` on the apps page) cut the pages read 35 623 → 90 (the vouch's fleet check), 38 570 → 118 (the
OS alarm's agent versions), 30 097 → 1 013 (the 04:30 prune, inside its write lock) and 15 404 → 1 437 (the apps page);
the SQL lives in `store/r1346_speed.go` and each plan is pinned by `TestR1346_HeavyStatementsUseTheirIndex`. The
vouch's manifest is one transaction. Every statement over 1 s is logged as `[WARN] slow statement: <ms> (over 1s):
<SQL>` (`store/timeddriver.go`; the SQL only, never a value). The memory limit is 1 GiB (operator yes 2026-10-30). The
6-hourly snapshot (`VACUUM INTO`) takes no write lock — 9 600 writes ran during one on a copy, the slowest 1.5 ms; its
cost is 3 minutes of disk load. The vouch save answers in the request: its database step is now milliseconds, so no
background save is needed. **R-1344 restart:** the hub mails `controller_unresponsive_restart` (warning, operator-only)
when the stanza's `unresponsive_restart_at` moves, and not also `controller_restarted_by_agent` for that act; the
`controller_unresponsive` text says when the agent restarts (`unresponsive_restart_after_s`), or that an older agent
does not.
@@ -106,7 +106,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
- **The public website** felhom.eu (served from DooPlex — Cloudflare DNS only, the traffic does not pass through it; MEASURED 2026-10-09: the public A record is the home address, not a Cloudflare one, and no `cf-ray` header comes back) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
- **The public website** felhom.eu, with the installer script (**served by Cloudflare Pages since 2026-10-30**, D35 / R-1228 — MEASURED: `server: cloudflare` and a `cf-ray` header; before, it was served from DooPlex with Cloudflare DNS only) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
## Zone 3 — the operator's home
@@ -116,7 +116,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
- **Vaultwarden** — the operator's password manager.
- **Prometheus + Alertmanager** — the operator's alarms.
- **The signing keys** — they sign every agent update and config bundle a box accepts.
- **The website** (felhom.eu) and the contact form's mailer.
- **The contact form's mailer** (`api.felhom.eu`) — and, until 2026-11-06, the old copy of the website (the way back after the 2026-10-30 move to Cloudflare).
- **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine").
- **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies).