close-out 2026-10-30: R-1344 and R-1346 closed, R-1228 switched (D35) and narrowed; R-1352, R-1353 filed; 01/03/05 [DESIGN], total-loss + publish-train runbooks, poster facts; instruction files: the site publishes to Cloudflare Pages
gates / gates (push) Successful in 1m47s
gates / gates (push) Successful in 1m47s
This commit is contained in:
1 parent
e0e55bf852
commit
3cda7e8d6c
30 files changed
+561
-17
No files matched your search
@@ -129,6 +129,7 @@ credentials.
|
||||
| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) |
|
||||
| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
|
||||
| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
|
||||
| new box ↔ public site | the installer script (`felhom.eu/scripts/felhom-host-install.sh`) + the website | **[DESIGN] 2026-10-30 (D35 / R-1228): Cloudflare Pages** (project `felhom-site`), uploaded by `scripts/publish-site-cloudflare.sh` from `main` + the installer tag; the contact form posts to `api.felhom.eu` (DooPlex's mailer). Before: DooPlex nginx + git-sync, kept 7 days as the way back | a changed installer runs as root on every new box — the script refuses an upload whose installer is not the tag's byte for byte |
|
||||
|
||||
**Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is
|
||||
a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the
|
||||
@@ -288,6 +289,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
||||
work.** The one check that is off today is the route's own: `*.<domain>` → `https://traefik` with `noTLSVerify`
|
||||
(R-510; traefik shows its self-signed default certificate on a leg that never leaves the guest) — a recorded
|
||||
exception, not a pattern to copy.
|
||||
- **[DESIGN] 2026-10-30 — the public site and the installer are on Cloudflare Pages (D35 / R-1228).** `felhom.eu`
|
||||
and `www` point to the Pages project `felhom-site` (uploaded by `scripts/publish-site-cloudflare.sh`); the contact
|
||||
form posts to `api.felhom.eu` (DooPlex's mailer). The paragraph below describes DooPlex's copy, kept running 7 days
|
||||
from the switch as the way back (`audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5).
|
||||
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
|
||||
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
|
||||
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
|
||||
|
||||
@@ -214,6 +214,20 @@ by verb**:
|
||||
`unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a
|
||||
parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both
|
||||
test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long.
|
||||
- **`[DESIGN]` 2026-10-30 — R-1344 decided from the measurement: the alarm becomes a RESTART (agent `bd18347`,
|
||||
released v0.171.0; hub 0.178.0 `controller_unresponsive_restart`).** The measurement (`audits/release-2026-10-30/
|
||||
p1-r1344/`): 4 733 probes on Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, through demo-hp's PBS
|
||||
vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test and two controller updates —
|
||||
no failed probe, no 5-s timeout, slowest 303 ms (demo-hp, during the PBS vzdump). So: the alarm stays at 6 unhealthy
|
||||
sweeps; at **10** (2 minutes later, ~6.5 minutes after the first failed probe) the agent restarts the bootstrap unit
|
||||
— unless the controller's busy file is fresh (a backup, a restore, an update, an export) or unreadable, or the
|
||||
supervisor's brake refuses (3 restarts in 15 minutes → the crash-loop pause). A swap, a parked or locked guest and a
|
||||
vzdump keep resetting the count. The restart counts in both brakes (fast and R-539's slow counter); after it the count
|
||||
starts from zero. The stanza carries `unresponsive_restart_after_s` (the hub's alarm text says when the restart comes,
|
||||
and says „does not restart" for an older agent) and `unresponsive_restart_at` (the hub mails
|
||||
`controller_unresponsive_restart` once — not also `controller_restarted_by_agent`). Proven live on Tester 1
|
||||
2026-10-30: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the
|
||||
restart 09:31:22 → healthy 09:31:24.
|
||||
- **`[DESIGN]` 2026-10-29 — the operator's „Reboot guest" (D65 = A, agent `ca15a31`, released v0.170.0; `04` §6.1,
|
||||
`09` §3 decision 284).** The agent's list gains `reboot_guest <vmid>`: the existing guest-reboot path (`pct reboot`
|
||||
of THAT guest — never the host), for a stuck Docker or a stuck network inside the box. Guards, each refusing with the
|
||||
|
||||
@@ -833,3 +833,20 @@ stores an AGENT row `reboot_guest <vmid>` and pokes the box like the restart. **
|
||||
itself when it comes back. The agent's result closes the row with one `operator_action` event. **R-1346:** the access-log
|
||||
middleware also writes `[WARN] slow request: <method> <path> <status> <ms>` for a request over 5 s (path only;
|
||||
`/api/v1/wait` exempt), and the artifact vouch save logs each leg's time.
|
||||
|
||||
**[DESIGN] 2026-10-30 — the hub's database speed (R-1346, hub 0.178.0).** Measured on a copy of the live database
|
||||
inside the hub pod (`audits/release-2026-10-30/p2-r1346/`): the pod had 256 MiB for a 570 MB database, so nearly every
|
||||
page read went to Longhorn, and five statements read whole tables. `controller_version` and `agent_version` sit after
|
||||
`report_json` in their rows, so reading them walks each row's JSON. Four indexes (`idx_reports_controller_version`,
|
||||
`idx_host_reports_agent_version`, `idx_reports_received`, `idx_host_reports_received`) and one hint (`INDEXED BY
|
||||
idx_app_telemetry_prune` on the apps page) cut the pages read 35 623 → 90 (the vouch's fleet check), 38 570 → 118 (the
|
||||
OS alarm's agent versions), 30 097 → 1 013 (the 04:30 prune, inside its write lock) and 15 404 → 1 437 (the apps page);
|
||||
the SQL lives in `store/r1346_speed.go` and each plan is pinned by `TestR1346_HeavyStatementsUseTheirIndex`. The
|
||||
vouch's manifest is one transaction. Every statement over 1 s is logged as `[WARN] slow statement: <ms> (over 1s):
|
||||
<SQL>` (`store/timeddriver.go`; the SQL only, never a value). The memory limit is 1 GiB (operator yes 2026-10-30). The
|
||||
6-hourly snapshot (`VACUUM INTO`) takes no write lock — 9 600 writes ran during one on a copy, the slowest 1.5 ms; its
|
||||
cost is 3 minutes of disk load. The vouch save answers in the request: its database step is now milliseconds, so no
|
||||
background save is needed. **R-1344 restart:** the hub mails `controller_unresponsive_restart` (warning, operator-only)
|
||||
when the stanza's `unresponsive_restart_at` moves, and not also `controller_restarted_by_agent` for that act; the
|
||||
`controller_unresponsive` text says when the agent restarts (`unresponsive_restart_after_s`), or that an older agent
|
||||
does not.
|
||||
@@ -106,7 +106,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
||||
- **The public website** felhom.eu (served from DooPlex — Cloudflare DNS only, the traffic does not pass through it; MEASURED 2026-10-09: the public A record is the home address, not a Cloudflare one, and no `cf-ray` header comes back) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
|
||||
- **The public website** felhom.eu, with the installer script (**served by Cloudflare Pages since 2026-10-30**, D35 / R-1228 — MEASURED: `server: cloudflare` and a `cf-ray` header; before, it was served from DooPlex with Cloudflare DNS only) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
|
||||
|
||||
## Zone 3 — the operator's home
|
||||
|
||||
@@ -116,7 +116,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
- **Vaultwarden** — the operator's password manager.
|
||||
- **Prometheus + Alertmanager** — the operator's alarms.
|
||||
- **The signing keys** — they sign every agent update and config bundle a box accepts.
|
||||
- **The website** (felhom.eu) and the contact form's mailer.
|
||||
- **The contact form's mailer** (`api.felhom.eu`) — and, until 2026-11-06, the old copy of the website (the way back after the 2026-10-30 move to Cloudflare).
|
||||
- **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine").
|
||||
- **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies).
|
||||
|
||||
|
||||
Reference in new issue
Block a user