diff --git a/.claude/rules/website.md b/.claude/rules/website.md index c34cee8d..a53ee5a3 100644 --- a/.claude/rules/website.md +++ b/.claude/rules/website.md @@ -6,8 +6,10 @@ paths: ["website/**"] ## Deploy -The website **auto-deploys via git-sync from `main`** — just push, live in 1–2 min. No ArgoCD sync, -no image build. Emergency edits: https://files.felhom.eu. +**Since 2026-10-30 (D35 / R-1228) the public site is served by Cloudflare Pages** (project `felhom-site`): a push to +`main` reaches the public site only after `sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run` +first; it uploads `website/` from `main` and `/scripts/` from the installer tag). DooPlex's git-sync copy keeps tracking +`main` as the way back (`documentation/audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5). No image build. **Do NOT pin the website to the installer tag.** The sparse-checkout used to cover `/website/` and `/scripts/` in one sync; pinning that would turn every copy edit into a release. The installer's diff --git a/CLAUDE.md b/CLAUDE.md index ee57e547..d083ab62 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,8 +62,9 @@ box**. runs two git-syncs: the website from `main`, and `/scripts/` from the tag `installer-v`. - **To publish:** cut `installer-v`, bump the `--ref` in `webpage.yaml` (**both** - the sidecar and the init container), commit, sync. -- **To roll back:** move the tag back and wait ~30 s. No ArgoCD sync, no deploy — that is the + the sidecar and the init container), commit, sync — **and** (since 2026-10-30, D35: felhom.eu is served by Cloudflare + Pages) run `sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh`, which uploads the tag the manifest names. +- **To roll back:** move the tag back, then run the same script (~1 min; it re-reads the tag). That is the emergency lever; fix forward afterwards. - **Do NOT pin the website to the tag**, and the URL never carries a ref — `felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit. diff --git a/CONTEXT.md b/CONTEXT.md index 03ebbae2..378f0838 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,21 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-30 (day, attended) — RELEASED: hub v0.178.0 (1 GiB), agent v0.171.0 (VOUCHED 10:47:36 CET); the site on +> Cloudflare (D35, switched 10:02:35 UTC).** (1) **R-1344 decided from the measurement** (4 733 probes through the night +> chain, slowest 303 ms, no failure): the agent restarts a running-but-unanswering controller at 10 unhealthy sweeps +> (alarm at 6) unless the busy file or the brake refuses (`controllersupervisor.go restartUnresponsive`, +> `supervisorBrakeAllows`, `recordSupervisorRestart`); the hub mails `controller_unresponsive_restart` +> (`monitor/controller_supervisor.go`). Proven on Tester 1 (SIGSTOP → restart in 6.5 min). (2) **R-1346 fixed**: five +> whole-table reads on Longhorn with a 256 MiB pod (`store/r1346_speed.go`: four indexes, one INDEXED BY hint, plans +> pinned); manifest save in one transaction; `store/timeddriver.go` logs any statement over 1 s (SQL only); memory 1 +> GiB. The vouch save 38 255 ms → 132 ms. First start built the indexes in 103 s (startup probe allows 180 s; once). +> (3) **D35**: `scripts/publish-site-cloudflare.sh` uploads `website/` (main) + `scripts/` (the manifest's installer +> tag) to Pages project `felhom-site` (account `9e432dea…`, token `CF_PAGES_TOKEN` in `sec`); `felhom.eu`/`www` are +> proxied CNAMEs to `felhom-site.pages.dev`; the contact form posts to `api.felhom.eu` (ingress `contact-mailer-api`, +> exact `/api/contact`, limit-rpm 20). DooPlex's webpage copy is the way back until 2026-11-06 (DUE-CHECKS R-1228). +> Rows: 228 → 228 (closed R-1344, R-1346; opened R-1352, R-1353). Report: `REPORT-release-2026-10-30.md`. + > **2026-10-29 (evening, attended) — RELEASED: hub v0.177.0, agent v0.170.0 (VOUCHED 18:57:48 CET), controller v0.349.0 > (global floor 19:24 CET, MinAgent 0.158.1). Operator ruling D65 = A (15:50; `09` §3 decision 284, `04` §6.1):** > `reboot_guest ` joins the AGENT's closed list, no signature (a reboot destroys nothing, `03`). **Built:** (1) the diff --git a/REPORT-release-2026-10-30.md b/REPORT-release-2026-10-30.md new file mode 100644 index 00000000..4ee7b2ac --- /dev/null +++ b/REPORT-release-2026-10-30.md @@ -0,0 +1,78 @@ +# REPORT — release run 2026-10-30 (day, attended): the stuck-controller decision, the slow hub, the site on Cloudflare + +Read first: `03-host-agent.md` (the supervisor), `05-hub-architecture.md` (the store, the snapshot, the pages), +`01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`, `runbooks/iso-release-gate.md`, +`runbooks/publish-train-rules.md`, `audits/dooplex-survival-2026-10-22/PLAN.md` (G8); rows R-1344, R-1346, R-1228, R-1326. + +## Five lines + +1. **The stuck-controller decision: RESTART.** The night's log (4 733 probes, Tester 1 and demo-hp, 2026-10-29 11:12 → + 10-30 07:00 UTC, through both vzdumps, the DB dumps, tier-2, the off-site push, a restore test and two updates) + had no failed probe and no 5-s timeout; the slowest was 303 ms. So agent 0.171.0 restarts a controller that runs but + does not answer at 10 unhealthy sweeps (alarm at 6) — proven on Tester 1: frozen 09:24:50 → restarted 09:31:22 UTC. +2. **The hub's slowness:** five statements read whole tables from the network disk, with 256 MiB for a 570 MB database. + Four indexes, one hint, one transaction and 1 GiB fixed it: your vouch save 38 255 ms → **132 ms**; `/apps` 25–30 s → + 0.45 s (3.4 s cold); `/system` 5.8 s → 0.45 s. +3. **The site on Cloudflare: switched** at 10:02:35 UTC on your yes. preview.felhom.eu and then felhom.eu/www: 30 of 30 + pages, the installer byte-identical to `installer-v1.36.1`, a preflight from a fresh ISO install passed with both. +4. **A new install still needs DooPlex for** the hub (claim, config, first report) and Gitea (the agent package, the + golden) — by design (D35). It no longer needs DooPlex to fetch the installer or show the site. +5. **What you do:** discard the unclaimed appliance (uuid `d3529d2d…`, my deleted proof VM) on the Hosts page. Nothing + else is blocked; on 2026-11-06 I ask whether to retire DooPlex's old website copy. + +## Versions, digests, CI by commit + +| Repo | Commit | CI job | What | +|---|---|---|---| +| agent | `bd18347` | 2532 green | R-1344 restart — **0.171.0 built from here** (sha256 `c888ca7b…202a5`, bundle `95f977f6…3d2b`, tag `v0.171.0`) | +| agent | `84726c9` | (close-out) | v0.171.0 heading; `pinned.day0_agent` = 0.171.0 (vouched 10:47:36 CET) | +| felhom.eu | `be538975` | (no own job; covered by 2533) | hub: `controller_unresponsive_restart`, the alarm text follows the agent | +| felhom.eu | `a13c69b0` | (covered by 2533) | hub 0.178.0: indexes, hint, one-transaction manifest, slow-statement line; evidence | +| felhom.eu | `f1ab634c` | 2533 green | R-1228 dated check removed — **hub 0.178.0 built from here** (`sha256:9c15c510…d925f`) | +| felhom.eu | `cea8a907` | 2535 green | manifest: hub 0.178.0, memory 1 GiB (Deployment/hub only, Synced/Healthy) | +| felhom.eu | `bd74a613`, `53107f1b`, `e0e55bf8` | read at the close-out | `api.felhom.eu` ingress; the publish script + the form endpoint; the ingress narrowed to `/api/contact` + 20 req/min | + +Cloudflare: Pages project `felhom-site` (2 uploads), custom domains `preview.felhom.eu`, `felhom.eu`, `www.felhom.eu`; +DNS: `preview` and `api` added, `felhom.eu` and `www` switched (undo ids in `p3-cloudflare/switch-undo.txt`). + +## Rows + +Before 228, after 228. **Closed (2):** R-1344, R-1346. **Opened (2):** R-1352 (the alarm mail arrives after the +restart mail when both fall between two reports, P4), R-1353 (statements still over 1 s, P4). **Narrowed:** R-1228 +(switched; DooPlex copy until 2026-11-06, DUE-CHECKS). + +## Observations + +- **FILED: R-1352** — live, the hub mailed `controller_unresponsive_restart` and `controller_unresponsive` in the same + second, restart first. +- **FILED: R-1353** — the new line named a background deadline check reading days of full host-report JSON (up to + 9.2 s cold), the start-up prune of `events`/`app_log_issues` (1–1.8 s), and the customer page's newest-report read. +- **NOT-A-FINDING: the first probe run slowed the live hub** (one host report took 25 s) — my own probe's random reads + on the shared Longhorn volume; stopped at once, and the measurement itself is the finding (R-1346, fixed). +- **NOT-A-FINDING: the hub was down ~2 min at the deploy** — the one-time index build (103 s) inside the 180-s startup + probe; R-1326's start-up is otherwise unchanged (the index check is four tiny queries). +- **NOT-A-FINDING: ArgoCD shows `Ingress/contact-mailer` OutOfSync** — the live object matches git field by field; not + synced (outside this run's change). +- **NOT-A-FINDING: demo-hp's local whole-guest backup skipped for space** (09:41 CET) — R-548. +- **NOT-A-FINDING: a 70-s 522 on felhom.eu at the switch** — the time Cloudflare took to activate the custom domains; + one 522 on the first full pass, then 30 of 30 twice. +- **NOT-A-FINDING: the full install (after the bind) was not run** — the bind is a session-only operator act and the + install would create Tester4's tunnel and off-site account; the preflight (both addresses) and the byte identity + carry the proof. + +## Teardown + +Machine: VM 3461 destroyed on demo-hp, no disk left on `nvme-scratch`; the DB copies (tmpfs on DooPlex and +`/data/r1346-probe` in the hub pod) deleted, the access-log copy shredded, the probe binary removed from the pod. +Host: nothing else provisioned. Hub: one unclaimed appliance entry (uuid `d3529d2d…`) — the operator's to discard. + +## Docs + +`03` [DESIGN] R-1344 restart; `05` [DESIGN] R-1346 + the restart event; `01` [DESIGN] the site on Cloudflare (row + +paragraph); `runbooks/total-loss-of-dooplex.md` (what a new install still needs); `runbooks/publish-train-rules.md` §6; +the system poster's facts (website on Cloudflare, mailer on DooPlex) — the drawing needs a refresh (STATUS line). +Instruction files corrected (stale facts, D35): `CLAUDE.md` „To publish / To roll back" (+ the upload script); +`.claude/rules/website.md` „Deploy" (git-sync is the way back, the script publishes); `skills/felhom-build-deploy` +„Website" (same). `unproven.py --summary`: not walked 31 of 55, unchanged. + +Evidence: `documentation/audits/release-2026-10-30/` (`p1-r1344/`, `p2-r1346/`, `p3-cloudflare/`). diff --git a/STATUS.md b/STATUS.md index b4220a54..3efc1ab4 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-29 (evening release).** On your yes: hub 0.177.0, agent 0.170.0 (your vouch) and controller 0.349.0 for every box; „Reboot guest" on the hub works and refuses during backups, updates and twice in an hour, and no alarm follows your hold clear any more — proven on Tester 1. The slow vouch is the hub's own database step (38 of 38.3 s); the stuck-controller decision waits for tomorrow's night log. +**Updated 2026-10-30.** On your yes, felhom.eu and the installer are served by Cloudflare (DooPlex's copy stays until 11-06 as the way back), and the contact form posts to api.felhom.eu. Hub 0.178.0 (1 GiB) saves your vouch in 0.13 s instead of 38 s and opens the apps page in under a second. Agent 0.171.0 (your vouch) restarts a controller that runs but does not answer — proven on Tester 1. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -24,7 +24,7 @@ ## What we work on - Apps still waiting for the HTTP2Shell fix: 6 — ActualBudget, cal.com, Calibre-Web, Paperless, RomM and SparkyFitness have no fixed release yet. -- Dated checks, run by CC. 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). +- Dated checks, run by CC. 2026-11-06: ask you to retire DooPlex's old copy of the website (R-1228). ## Waiting on you — decide @@ -48,13 +48,13 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14) 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12) -**2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes. +**2026-10-30.** On the hub's Hosts page: Discard the unclaimed appliance registered 10:44:59 (uuid d3529d2d…, CC's deleted proof VM). If not: a dead entry alarms. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes. **2026-10-23 do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms. 3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. **The rest** -- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. +- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the website now on Cloudflare, only its mail form on DooPlex (D35). If not: the drawing is out of date. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). (since 10-09) - Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). (since 10-14) - Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784). diff --git a/documentation/architecture/01-topology-and-trust.md b/documentation/architecture/01-topology-and-trust.md index 3294b019..1ff0b725 100644 --- a/documentation/architecture/01-topology-and-trust.md +++ b/documentation/architecture/01-topology-and-trust.md @@ -129,6 +129,7 @@ credentials. | box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) | | guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — | | hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF | +| new box ↔ public site | the installer script (`felhom.eu/scripts/felhom-host-install.sh`) + the website | **[DESIGN] 2026-10-30 (D35 / R-1228): Cloudflare Pages** (project `felhom-site`), uploaded by `scripts/publish-site-cloudflare.sh` from `main` + the installer tag; the contact form posts to `api.felhom.eu` (DooPlex's mailer). Before: DooPlex nginx + git-sync, kept 7 days as the way back | a changed installer runs as root on every new box — the script refuses an upload whose installer is not the tag's byte for byte | **Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the @@ -288,6 +289,10 @@ app. Controller down → the gated app answers an error, never the app. Measured work.** The one check that is off today is the route's own: `*.` → `https://traefik` with `noTLSVerify` (R-510; traefik shows its self-signed default certificate on a leg that never leaves the guest) — a recorded exception, not a pattern to copy. +- **[DESIGN] 2026-10-30 — the public site and the installer are on Cloudflare Pages (D35 / R-1228).** `felhom.eu` + and `www` point to the Pages project `felhom-site` (uploaded by `scripts/publish-site-cloudflare.sh`); the contact + form posts to `api.felhom.eu` (DooPlex's mailer). The paragraph below describes DooPlex's copy, kept running 7 days + from the switch as the way back (`audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5). - **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls `website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read `http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 25d7783d..8a193977 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -214,6 +214,20 @@ by verb**: `unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long. +- **`[DESIGN]` 2026-10-30 — R-1344 decided from the measurement: the alarm becomes a RESTART (agent `bd18347`, + released v0.171.0; hub 0.178.0 `controller_unresponsive_restart`).** The measurement (`audits/release-2026-10-30/ + p1-r1344/`): 4 733 probes on Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, through demo-hp's PBS + vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test and two controller updates — + no failed probe, no 5-s timeout, slowest 303 ms (demo-hp, during the PBS vzdump). So: the alarm stays at 6 unhealthy + sweeps; at **10** (2 minutes later, ~6.5 minutes after the first failed probe) the agent restarts the bootstrap unit + — unless the controller's busy file is fresh (a backup, a restore, an update, an export) or unreadable, or the + supervisor's brake refuses (3 restarts in 15 minutes → the crash-loop pause). A swap, a parked or locked guest and a + vzdump keep resetting the count. The restart counts in both brakes (fast and R-539's slow counter); after it the count + starts from zero. The stanza carries `unresponsive_restart_after_s` (the hub's alarm text says when the restart comes, + and says „does not restart" for an older agent) and `unresponsive_restart_at` (the hub mails + `controller_unresponsive_restart` once — not also `controller_restarted_by_agent`). Proven live on Tester 1 + 2026-10-30: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the + restart 09:31:22 → healthy 09:31:24. - **`[DESIGN]` 2026-10-29 — the operator's „Reboot guest" (D65 = A, agent `ca15a31`, released v0.170.0; `04` §6.1, `09` §3 decision 284).** The agent's list gains `reboot_guest `: the existing guest-reboot path (`pct reboot` of THAT guest — never the host), for a stuck Docker or a stuck network inside the box. Guards, each refusing with the diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 36fec2b9..b0660c63 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -833,3 +833,20 @@ stores an AGENT row `reboot_guest ` and pokes the box like the restart. ** itself when it comes back. The agent's result closes the row with one `operator_action` event. **R-1346:** the access-log middleware also writes `[WARN] slow request: ` for a request over 5 s (path only; `/api/v1/wait` exempt), and the artifact vouch save logs each leg's time. + +**[DESIGN] 2026-10-30 — the hub's database speed (R-1346, hub 0.178.0).** Measured on a copy of the live database +inside the hub pod (`audits/release-2026-10-30/p2-r1346/`): the pod had 256 MiB for a 570 MB database, so nearly every +page read went to Longhorn, and five statements read whole tables. `controller_version` and `agent_version` sit after +`report_json` in their rows, so reading them walks each row's JSON. Four indexes (`idx_reports_controller_version`, +`idx_host_reports_agent_version`, `idx_reports_received`, `idx_host_reports_received`) and one hint (`INDEXED BY +idx_app_telemetry_prune` on the apps page) cut the pages read 35 623 → 90 (the vouch's fleet check), 38 570 → 118 (the +OS alarm's agent versions), 30 097 → 1 013 (the 04:30 prune, inside its write lock) and 15 404 → 1 437 (the apps page); +the SQL lives in `store/r1346_speed.go` and each plan is pinned by `TestR1346_HeavyStatementsUseTheirIndex`. The +vouch's manifest is one transaction. Every statement over 1 s is logged as `[WARN] slow statement: (over 1s): +` (`store/timeddriver.go`; the SQL only, never a value). The memory limit is 1 GiB (operator yes 2026-10-30). The +6-hourly snapshot (`VACUUM INTO`) takes no write lock — 9 600 writes ran during one on a copy, the slowest 1.5 ms; its +cost is 3 minutes of disk load. The vouch save answers in the request: its database step is now milliseconds, so no +background save is needed. **R-1344 restart:** the hub mails `controller_unresponsive_restart` (warning, operator-only) +when the stanza's `unresponsive_restart_at` moves, and not also `controller_restarted_by_agent` for that act; the +`controller_unresponsive` text says when the agent restarts (`unresponsive_restart_after_s`), or that an older agent +does not. diff --git a/documentation/architecture/felhom-system-poster.facts.md b/documentation/architecture/felhom-system-poster.facts.md index fe23da9a..2b8f2d75 100644 --- a/documentation/architecture/felhom-system-poster.facts.md +++ b/documentation/architecture/felhom-system-poster.facts.md @@ -106,7 +106,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1– Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data." - **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys. - **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households. -- **The public website** felhom.eu (served from DooPlex — Cloudflare DNS only, the traffic does not pass through it; MEASURED 2026-10-09: the public A record is the home address, not a Cloudflare one, and no `cf-ray` header comes back) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted. +- **The public website** felhom.eu, with the installer script (**served by Cloudflare Pages since 2026-10-30**, D35 / R-1228 — MEASURED: `server: cloudflare` and a `cf-ray` header; before, it was served from DooPlex with Cloudflare DNS only) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted. ## Zone 3 — the operator's home @@ -116,7 +116,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1– - **Vaultwarden** — the operator's password manager. - **Prometheus + Alertmanager** — the operator's alarms. - **The signing keys** — they sign every agent update and config bundle a box accepts. -- **The website** (felhom.eu) and the contact form's mailer. +- **The contact form's mailer** (`api.felhom.eu`) — and, until 2026-11-06, the old copy of the website (the way back after the 2026-10-30 move to Cloudflare). - **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine"). - **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies). diff --git a/documentation/audits/release-2026-10-30/p1-r1344/live/agent-journal.txt b/documentation/audits/release-2026-10-30/p1-r1344/live/agent-journal.txt new file mode 100644 index 00000000..770f0ed0 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p1-r1344/live/agent-journal.txt @@ -0,0 +1,5 @@ +2026-10-30T10:22:21+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:22:21.806+01:00 level=INFO msg="controller-supervisor: alive" sweeps_since_boot=20 guests_evaluated=1 controllers_not_running=0 +2026-10-30T10:29:21+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:29:21.932+01:00 level=ERROR msg="controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so" vmid=9201 unhealthy_sweeps=6 interval=30s restart_after_sweeps=10 +2026-10-30T10:31:22+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:31:22.888+01:00 level=WARN msg="controller-supervisor: the controller does not answer — restarting the bootstrap unit" vmid=9201 unhealthy_sweeps=10 since=2026-10-30T09:29:21Z unit=felhom-controller-bootstrap.service +2026-10-30T10:31:24+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:31:24.361+01:00 level=WARN msg="controller-supervisor: RESTARTED the controller" vmid=9201 reason="the controller ran but did not answer its health check for 10 sweeps (since 2026-10-30T09:29:21Z)" restarts_24h=1 +2026-10-30T10:31:51+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:31:51.841+01:00 level=INFO msg="controller-supervisor: the controller answers its health check again" vmid=9201 diff --git a/documentation/audits/release-2026-10-30/p1-r1344/live/r1344-live.txt b/documentation/audits/release-2026-10-30/p1-r1344/live/r1344-live.txt new file mode 100644 index 00000000..ce0f7c9d --- /dev/null +++ b/documentation/audits/release-2026-10-30/p1-r1344/live/r1344-live.txt @@ -0,0 +1,18 @@ +09:25:02 running healthy streak=0 started=2026-10-29T19:07:00.843131965Z +09:25:33 running healthy streak=1 started=2026-10-29T19:07:00.843131965Z +09:26:05 running healthy streak=2 started=2026-10-29T19:07:00.843131965Z +09:26:36 running unhealthy streak=3 started=2026-10-29T19:07:00.843131965Z +09:27:08 running unhealthy streak=3 started=2026-10-29T19:07:00.843131965Z +09:27:39 running unhealthy streak=4 started=2026-10-29T19:07:00.843131965Z +09:28:11 running unhealthy streak=5 started=2026-10-29T19:07:00.843131965Z +09:28:42 running unhealthy streak=6 started=2026-10-29T19:07:00.843131965Z +09:29:13 running unhealthy streak=7 started=2026-10-29T19:07:00.843131965Z +09:29:45 running unhealthy streak=8 started=2026-10-29T19:07:00.843131965Z +09:30:16 running unhealthy streak=9 started=2026-10-29T19:07:00.843131965Z +09:30:48 running unhealthy streak=10 started=2026-10-29T19:07:00.843131965Z +09:31:19 running unhealthy streak=11 started=2026-10-29T19:07:00.843131965Z +09:31:50 running healthy streak=0 started=2026-10-30T09:31:24.212057217Z +DONE +2026-10-30T10:29:21+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:29:21.932+01:00 level=ERROR msg="controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so" vmid=9201 unhealthy_sweeps=6 interval=30s restart_after_sweeps=10 +2026-10-30T10:31:22+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:31:22.888+01:00 level=WARN msg="controller-supervisor: the controller does not answer — restarting the bootstrap unit" vmid=9201 unhealthy_sweeps=10 since=2026-10-30T09:29:21Z unit=felhom-controller-bootstrap.service +2026-10-30T10:31:24+01:00 felhom felhom-agent[2609044]: time=2026-10-30T10:31:24.361+01:00 level=WARN msg="controller-supervisor: RESTARTED the controller" vmid=9201 reason="the controller ran but did not answer its health check for 10 sweeps (since 2026-10-30T09:29:21Z)" restarts_24h=1 diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/PLAN.md b/documentation/audits/release-2026-10-30/p3-cloudflare/PLAN.md new file mode 100644 index 00000000..0ef939ed --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/PLAN.md @@ -0,0 +1,83 @@ +# D35 / R-1228 — the website and the installer on Cloudflare (plan, 2026-10-30) + +Operator ruling D35 = YES (2026-10-23): the static website and the installer are served from Cloudflare like the ISO; +the hub and Gitea stay on DooPlex (R-1228's option B). This plan was written before anything on Cloudflare changed. + +## 1. What is served today (read from the source, 2026-10-30) + +| Address | Served by | What a box, the ISO or a person fetches | +|---|---|---| +| `felhom.eu`, `www.felhom.eu` (CNAME → the home line, not proxied) | DooPlex: nginx + git-sync of `website/` at `main` (`manifests/webpage.yaml`) | the 19 Hungarian and 11 English pages, `sitemap.xml`, `robots.txt`, `assets/` | +| `felhom.eu/scripts/…` | DooPlex: a second git-sync of `scripts/` at the tag `installer-v1.36.1` | **`/scripts/felhom-host-install.sh`** — the only `/scripts/` file anything fetches: the ISO's first boot (`scripts/iso/felhom-bootstrap.sh:168`, `FELHOM_INSTALL_URL` default) and the hub's day-0 command (`customer_unified.html`). sha256 `e18cdd17…aa1f1` = the tag's | +| `felhom.eu/api/contact` | DooPlex: `contact-mailer` (`manifests/contact-mailer.yaml`, Resend) | the contact form's POST (`kapcsolat.html`, `en/contact.html`) — **the one dynamic part** | +| `stats.felhom.eu` | DooPlex: Umami | the analytics script every page loads (a separate name — unchanged) | +| `iso.felhom.eu` | Cloudflare R2 (proxied CNAME) | the installer ISO — already off DooPlex | +| `hub.felhom.eu`, `gitea.dooplex.hu` | DooPlex | the box's claim and reports; the agent download — **stay on DooPlex (D35)** | + +Inside the installer every other address is the hub, Gitea (the agent package) and `download.proxmox.com` — nothing +else from `felhom.eu`. + +## 2. Where each goes — Cloudflare Pages, one project `felhom-site` + +**Pick: Cloudflare Pages, not R2.** Fewest moving parts: Pages serves clean URLs (`/kapcsolat` → `kapcsolat.html`), +the site's `404.html` with status 404, directory indexes (`/en/`) and compression with no configuration — exactly what +nginx's `try_files $uri $uri.html $uri/` does today. R2 behind a custom domain serves objects by exact key only: no +clean URLs, no index, no 404 page, so it would need a Worker in front — one more piece to own. + +- `/` ← `website/` from `origin/main`; `/scripts/` ← `scripts/` from the installer tag named in `manifests/webpage.yaml` + (the same two refs, the same files as DooPlex — 487 files, 25 MB). +- **The contact form cannot be static.** It stays on DooPlex (`contact-mailer`, which already answers CORS for + `https://felhom.eu`). After the switch `felhom.eu/api/contact` would reach Cloudflare, not DooPlex, so the form + needs its own name on DooPlex — e.g. `api.felhom.eu` → the same ingress, and the form's `formEndpoint` pointing there. + That is a DooPlex ingress change and a DNS name: **it waits for the operator's word** (the switch STOP). Without it the + form shows an error on Cloudflare; the page's `mailto:info@felhom.eu` link still works. +- `stats.felhom.eu`, `files.felhom.eu`, `hub.felhom.eu`, `ntfy.felhom.eu`: unchanged. + +## 3. How a release reaches it + +`scripts/publish-site-cloudflare.sh` (run under `sec run CF_PAGES_TOKEN --`): stages both trees from git, REFUSES when +the staged installer is not the tag's byte for byte, when a must-have page is missing, or when the manifest names more +than one tag; `--dry-run` stages and prints without uploading. The upload is wrangler in a pinned node container; the +token reaches it as an environment variable name only. + +- **A website edit:** push to `main` (DooPlex picks it up as before) AND run the script (Cloudflare). After the switch + only the script matters. +- **An installer release:** cut `installer-v`, bump BOTH `--ref`s in `manifests/webpage.yaml` (as today), then run + the script — it reads the tag from the manifest, so the two copies cannot differ. + +## 4. What stays on DooPlex + +The hub (`hub.felhom.eu`), Gitea and the registry (`gitea.dooplex.hu` — the agent package a new install downloads), +the contact mailer, Umami, FileBrowser, ntfy. **A new install still needs DooPlex for:** the hub's claim and its first +report (`hub.felhom.eu`), and the agent package from Gitea. It no longer needs DooPlex to fetch the installer. + +## 5. The switch (only after the operator's yes) and the way back + +Switch: in the Pages project add the custom domains `felhom.eu` and `www.felhom.eu`; Cloudflare replaces the two +CNAMEs to the home line with proxied records to `felhom-site.pages.dev`. Way back (7 days, DooPlex's webpage keeps +running): set the two records back to `CNAME dooplex.hopto.org`, not proxied (the values in `dns-before.json`), and +remove the two custom domains from the project. DooPlex's Let's Encrypt certificate for felhom.eu is valid until its +next renewal; a renewal after the switch fails (HTTP-01 reaches Cloudflare), which is the reason the way back is 7 days +and not open-ended. + +## 6. Result (2026-10-30, after the plan) + +- **Test address:** Pages project `felhom-site` + `preview.felhom.eu` (proxied CNAME). `sitecheck.sh`: 30 of 30 (every + page by its clean URL, the sitemap, robots, an asset, a missing page = 404, the installer's sha256 = the tag's). + Control: the same check against DooPlex's felhom.eu before the switch, 30 of 30. +- **Install proof:** throwaway VM 3461 on demo-hp (scratch NVMe, never the thin pool) from the public ISO 1.30.0 + (sha256 = `iso.felhom.eu`'s). Booted with the cable unplugged, `FELHOM_INSTALL_URL` pointed at the test address, then + plugged in: the first boot reached the hub and registered as an unclaimed appliance (pairing mode — the installer is + fetched after a bind, which is a session-only operator act). The installer fetched from the test address (sha256 = + the tag's) passed `--preflight-only` (customer Tester4, read-only `GET /config`). After the switch the same preflight + with the installer fetched from `felhom.eu` passed again. The full install was not run: it needs the operator's bind + and would create Tester4's tunnel and off-site account. VM 3461 destroyed; the hub's unclaimed entry (uuid + `d3529d2d…`) is the operator's to discard. +- **The contact form:** `api.felhom.eu` (DNS to the home line, not proxied; ingress `contact-mailer-api`, only + `/api/contact`, 20 requests a minute per client; certificate issued). Preflight from `https://felhom.eu` 204; one test + message 200, and it arrived at info@felhom.eu (read through the mailbox). +- **Switch:** 10:02:35 UTC (operator yes): `felhom.eu` and `www` → `felhom-site.pages.dev`, proxied; the project's two + custom domains active at 10:03:47 (about 70 s of 522 in between). After: 30 of 30 on both names (one 522 on the first + pass, then 30 of 30 twice), `server: cloudflare`, the installer's sha256 = `installer-v1.36.1`. +- **Undo (until 2026-11-06):** `switch-undo.txt` holds the two record ids; set both back to `CNAME dooplex.hopto.org`, + `proxied: false`, and remove the two custom domains from `felhom-site`. DooPlex's webpage still serves both names. diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/cf-deploy-2.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/cf-deploy-2.txt new file mode 100644 index 00000000..2fb7ac24 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/cf-deploy-2.txt @@ -0,0 +1,20 @@ +project : felhom-site (account 9e432dea3b6befc7c87a41npm warn deprecated sourcemap-316049f162) +website : origin/main e0e55bf8 +installer : installer-v1.36.1 — scripts/felhom-host-install.sh sha256 e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 +files : 487 (25M) +Uploading... (484/487) +Uploading... (485/487) +Uploadingcodec@1.4.8: Please use @jridgewell/sourcemap-codec instead +npm warn deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject. +npm ... (486/487) +Uploading... (487/487) +✨ Success! Uploaded 3 files (484 already uploaded) (1.36 sec) + +🌎 Deploying... +✨ Deployment complete! Takenotice +npm notice New major version of npm available! 10.8.2 -> 12.2.1 +npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.2.1 +npm notice To update run: npm install -g npm@12.2.1 +npm notice + a peek over at https://a833afb5.felhom-site.pages.dev +UPLOADED — check it from outside: the pages answer, and the installer's sha256 is e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/dns-before.json b/documentation/audits/release-2026-10-30/p3-cloudflare/dns-before.json new file mode 100644 index 00000000..df511e8c --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/dns-before.json @@ -0,0 +1,135 @@ +[ + { + "type": "A", + "name": "ep0.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "" + }, + { + "type": "AAAA", + "name": "ep0.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "" + }, + { + "type": "CNAME", + "name": "felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "CNAME", + "name": "files.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "CNAME", + "name": "hub.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "CNAME", + "name": "iso.felhom.eu", + "proxied": true, + "ttl": 1, + "content": "public.r2.dev" + }, + { + "type": "CNAME", + "name": "ntfy.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "CNAME", + "name": "stats.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "CNAME", + "name": "www.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "dooplex.hopto.org" + }, + { + "type": "MX", + "name": "felhom.eu", + "proxied": false, + "ttl": 1, + "content": "route3.mx.cloudflare.net" + }, + { + "type": "MX", + "name": "felhom.eu", + "proxied": false, + "ttl": 1, + "content": "route2.mx.cloudflare.net" + }, + { + "type": "MX", + "name": "felhom.eu", + "proxied": false, + "ttl": 1, + "content": "route1.mx.cloudflare.net" + }, + { + "type": "MX", + "name": "send.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "feedback-smtp.eu-west-1.amazonses.com" + }, + { + "type": "TXT", + "name": "cf2024-1._domainkey.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "\"v=DKIM1; h=sha256; k=rsa; p=MIIBIjANBgk\u2026" + }, + { + "type": "TXT", + "name": "_dmarc.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "\"v=DMARC1; p=quarantine; rua=mailto:a18d\u2026" + }, + { + "type": "TXT", + "name": "felhom.eu", + "proxied": false, + "ttl": 3600, + "content": "\"google-site-verification=Iiq-C0udb1fHvE\u2026" + }, + { + "type": "TXT", + "name": "felhom.eu", + "proxied": false, + "ttl": 1, + "content": "\"v=spf1 include:_spf.mx.cloudflare.net ~\u2026" + }, + { + "type": "TXT", + "name": "resend._domainkey.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "\"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBg\u2026" + }, + { + "type": "TXT", + "name": "send.felhom.eu", + "proxied": false, + "ttl": 1, + "content": "\"v=spf1 include:amazonses.com ~all\"\u2026" + } +] \ No newline at end of file diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-felhom-eu.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-felhom-eu.txt new file mode 100644 index 00000000..f1e0deb3 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-felhom-eu.txt @@ -0,0 +1,5 @@ +[INFO] felhom-host-install v1.36.1 — mode=appliance customer=Tester4 vmid=9201 +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[OK] PRE-FLIGHT PASS (mode=appliance) — no state written, no install step executed +ls: cannot access '/root/.pp': No such file or directory diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-preview.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-preview.txt new file mode 100644 index 00000000..045187d5 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/preflight-preview.txt @@ -0,0 +1,24 @@ +[INFO] auto-sized guest RAM: 4096 MiB (host 7872 MiB; clamp(host-4096, min 4096, max host-2048), ceiling host-1024) +[INFO] auto-sized guest cores: 3 (host 4 cores; host-1, min 2) + +[INFO] felhom-host-install v1.36.1 — mode=appliance customer=Tester4 vmid=9201 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: r-proof (auto) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~19 GiB +[WARN] local-lvm free ~19 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately +[INFO] free RAM: ~6276 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storage 'felhom-pbs' not present yet — expected: the PBS-DR tier creates it; the grant is pre-positioned deliberately +[INFO] dnsmasq: not present before Felhom — would be recorded at install; uninstall will remove it again if we install it +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[INFO] golden: none local — will fetch + verify from Gitea in step 7/8 +[OK] pre-flight passed + +[OK] PRE-FLIGHT PASS (mode=appliance) — no state written, no install step executed +ls: cannot access '/root/.pp': No such file or directory diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-after-switch.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-after-switch.txt new file mode 100644 index 00000000..2047506f --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-after-switch.txt @@ -0,0 +1,7 @@ +BAD /gepigeny -> 522 (want 200) +installer sha256 e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 = installer-v1.36.1 +https://felhom.eu: 29 ok, 1 bad +installer sha256 e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 = installer-v1.36.1 +https://www.felhom.eu: 30 ok, 0 bad +https://felhom.eu: 30 ok, 0 bad +https://felhom.eu: 30 ok, 0 bad diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-preview.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-preview.txt new file mode 100644 index 00000000..35d8241b --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck-preview.txt @@ -0,0 +1,2 @@ +installer sha256 e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 = installer-v1.36.1 +https://preview.felhom.eu: 30 ok, 0 bad diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck.sh b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck.sh new file mode 100755 index 00000000..02dbf059 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/sitecheck.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# sitecheck.sh — every page of website/ (clean URL), the sitemap, an asset, a missing page (must be 404), +# and the installer's sha256 against installer-v1.36.1. Prints one line per check and a summary. +B="$1"; R=/mnt/5_hdd/felhom.eu/git/felhom.eu; ok=0; bad=0 +chk() { code=$(curl -s -o /dev/null -w '%{http_code}' "$B$1"); if [ "$code" = "$2" ]; then ok=$((ok+1)); else bad=$((bad+1)); echo "BAD $1 -> $code (want $2)"; fi; } +for f in $(cd $R/website && find . -name '*.html' | sed 's|^\./||' | sort); do + case "$f" in 404.html) continue;; index.html) p=/;; */index.html) p=/${f%index.html};; *) p=/${f%.html};; esac + chk "$p" 200 +done +chk /sitemap.xml 200; chk /robots.txt 200; chk /assets/site.css 200; chk /no-such-page-r1228 404 +want=$(git -C $R show installer-v1.36.1:scripts/felhom-host-install.sh | sha256sum | cut -d' ' -f1) +got=$(curl -fsS "$B/scripts/felhom-host-install.sh" | sha256sum | cut -d' ' -f1) +[ "$want" = "$got" ] && { ok=$((ok+1)); echo "installer sha256 $got = installer-v1.36.1"; } || { bad=$((bad+1)); echo "BAD installer sha256 $got (want $want)"; } +echo "$B: $ok ok, $bad bad" diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/switch-undo.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/switch-undo.txt new file mode 100644 index 00000000..76eeac28 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/switch-undo.txt @@ -0,0 +1,2 @@ +felhom.eu eb47a1a1767c004c45d66d8578416080 dooplex.hopto.org False 1 +www.felhom.eu a2915d01210d784bbef00ec5ab174f95 dooplex.hopto.org False 1 diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/upload-1.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/upload-1.txt new file mode 100644 index 00000000..4cfba1f6 --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/upload-1.txt @@ -0,0 +1,38 @@ +project : felhom-site (account 9e432dea3b6befc7c87a41Unable to find image 'node:20.17.0-alpine' locally +20.17.0-alpine: Pulling from library/node +737da86e5e86: Pulling fs layer +43c4264eed91: Pulling fs layer +842cd80f9366: Pulling fs layer +c36270121b0c: Pulling fs layer +737da86e5e86: Download complete +842cd80f9366: Download complete +43c4264eed91: Download complete +43c4264eed91: Pull complete +7f15c96a8a88: Download complete +c36270121b0c: Download complete +46839ec416f9: Download complete +c36270121b0c: Pull complete +737da86e5e86: Pull complete +842cd80f9366: Pull complete +Digest: sha256:2d07db07a2df6830718ae2a47db6fedce6745f5bcd174c398f2acdda90a11c03 +Status: Downloaded newer image for node:20.17.0-alpine +npm warn deprecated rollup-plu316049f162) +website : origin/main cea8a907 +installer : installer-v1.36.1 — scripts/felhom-host-install.sh sha256 e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 +files : 487 (25M) +Uploading... (0/487) +Uploading..gin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject. +npm warn deprecated sourcemap-codec@1.4.8: Please use @jridgewell/sourcemap-codec instead +npm . (162/487) +Uploading... (324/487) +Uploading... (487/487) +✨ Success! Uploaded 487 files (6.09 sec) + +🌎 Deploying... +✨ Deployment complete! Takenotice +npm notice New major version of npm available! 10.8.2 -> 12.2.1 +npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.2.1 +npm notice To update run: npm install -g npm@12.2.1 +npm notice + a peek over at https://06026223.felhom-site.pages.dev +UPLOADED — check it from outside: the pages answer, and the installer's sha256 is e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/boot2.png b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/boot2.png new file mode 100644 index 00000000..f5489920 Binary files /dev/null and b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/boot2.png differ diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/bootstrap-journal.txt b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/bootstrap-journal.txt new file mode 100644 index 00000000..a77f834d --- /dev/null +++ b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/bootstrap-journal.txt @@ -0,0 +1,36 @@ +2026-10-30T00:41:21-09:00 r-proof systemd[1]: Starting felhom-bootstrap.service - Felhom host bootstrap (fetch + run felhom-host-install.sh unattended, retry until success)... +2026-10-30T00:41:21-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: console: pvebanner.service masked (it would rewrite /etc/issue with the Proxmox admin URL on every boot) +2026-10-30T00:41:21-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: console: /etc/issue is the Felhom text (no admin URL) +2026-10-30T00:41:31-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: hub unreachable (https://hub.felhom.eu) — diagnosing (R-59) +2026-10-30T00:41:33-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: NIC status: +2026-10-30T00:41:33-09:00 r-proof felhom-bootstrap.sh[1356]: nic0 bc:24:11:f5:ca:95 kábel: nincs ? +2026-10-30T00:41:33-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network sweep: trying vmbr0 -> nic0 (bounded DHCP 20s + hub probe) +2026-10-30T00:41:54-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: sweep found no NIC that reaches the hub — console screen + retry every 60s +2026-10-30T00:41:54-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: console font -> Lat2-Terminus16 (Latin-2, ő/ű capable) +2026-10-30T00:43:06-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: NIC status: +2026-10-30T00:43:06-09:00 r-proof felhom-bootstrap.sh[1356]: nic0 bc:24:11:f5:ca:95 kábel: nincs ? +2026-10-30T00:43:06-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network sweep: trying vmbr0 -> nic0 (bounded DHCP 20s + hub probe) +2026-10-30T00:43:28-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: sweep found no NIC that reaches the hub — console screen + retry every 60s +2026-10-30T00:44:40-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network gate: NIC status: +2026-10-30T00:44:40-09:00 r-proof felhom-bootstrap.sh[1356]: nic0 bc:24:11:f5:ca:95 kábel: nincs ? +2026-10-30T00:44:40-09:00 r-proof felhom-bootstrap.sh[1356]: felhom-bootstrap: network sweep: trying vmbr0 -> nic0 (bounded DHCP 20s + hub probe) +2026-10-30T00:44:51-09:00 r-proof systemd[1]: felhom-bootstrap.service: Main process exited, code=killed, status=15/TERM +2026-10-30T00:44:51-09:00 r-proof systemd[1]: felhom-bootstrap.service: Failed with result 'signal'. +2026-10-30T00:44:51-09:00 r-proof systemd[1]: Stopped felhom-bootstrap.service - Felhom host bootstrap (fetch + run felhom-host-install.sh unattended, retry until success). +2026-10-30T00:44:51-09:00 r-proof systemd[1]: felhom-bootstrap.service: Consumed 2.739s CPU time, 31.1M memory peak. +2026-10-30T00:44:51-09:00 r-proof systemd[1]: Starting felhom-bootstrap.service - Felhom host bootstrap (fetch + run felhom-host-install.sh unattended, retry until success)... +2026-10-30T00:44:52-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: console: pvebanner.service masked (it would rewrite /etc/issue with the Proxmox admin URL on every boot) +2026-10-30T00:44:52-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: console: /etc/issue is the Felhom text (no admin URL) +2026-10-30T00:44:52-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: network gate: hub unreachable (https://hub.felhom.eu) — diagnosing (R-59) +2026-10-30T00:44:54-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: network gate: NIC status: +2026-10-30T00:44:54-09:00 r-proof felhom-bootstrap.sh[2309]: nic0 bc:24:11:f5:ca:95 kábel: nincs ? +2026-10-30T00:44:54-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: network sweep: trying vmbr0 -> nic0 (bounded DHCP 20s + hub probe) +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: network self-heal: vmbr0 -> nic0 (bc:24:11:f5:ca:95), hub reachable +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: PAIRING mode (generic ISO, no baked customer/passphrase) — hub=https://hub.felhom.eu +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: registering unclaimed appliance at the hub +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: registered — appliance token stored (0600); waiting for the operator or a customer self-bind +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: console font -> Lat2-Terminus16 (Latin-2, ő/ű capable) +2026-10-30T00:44:59-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error) +2026-10-30T00:55:01-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: still waiting to be bound (10m elapsed; polling continues) +2026-10-30T01:05:03-09:00 r-proof felhom-bootstrap.sh[2309]: felhom-bootstrap: still waiting to be bound (20m elapsed; polling continues) +FELHOM_INSTALL_URL=https://preview.felhom.eu/scripts/felhom-host-install.sh diff --git a/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/con2.png b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/con2.png new file mode 100644 index 00000000..5afb2f1e Binary files /dev/null and b/documentation/audits/release-2026-10-30/p3-cloudflare/vm-proof/con2.png differ diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 3c9d5f3a..83b9383d 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,13 @@ --- +## 2026-10-30 — R-1344 and R-1346 closed (full text of the rows: `git show e0e55bf8:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1344** | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** (P3) | CLOSED 2026-10-30 — measured, then decided: 4 733 probes on Tester 1 and demo-hp (2026-10-29 11:12 → 10-30 07:00 UTC) through the night chain: no failed probe, slowest 303 ms against the 5 s timeout. Agent 0.171.0 (`bd18347`, vouched 10:47:36 CET) restarts at 10 unhealthy sweeps (alarm at 6), refused by a fresh or unreadable busy file and the brake; hub 0.178.0 mails `controller_unresponsive_restart`. Live on Tester 1: frozen 09:24:50 UTC → alarm 09:29:21 → restart 09:31:22 → healthy 09:31:24. Order of the two mails: R-1352 | `audits/release-2026-10-30/p1-r1344/` | +| **R-1346** | **Hub pages load slowly, and the vouch save ended in a 504 while it went through.** (P3) | CLOSED 2026-10-30 — the cause per statement (`store/r1346_speed.go`): whole-table reads on a network disk with a 256 MiB pod for a 570 MB database. Four indexes + one hint (pages read 35 623 → 90, 38 570 → 118, 30 097 → 1 013, 15 404 → 1 437), the manifest in one transaction, the slow-statement line, memory 1 GiB (operator yes). Hub 0.178.0 deployed; the operator's vouch save: 38 255 ms → **132 ms**; `/apps` 25–30 s → 0.45 s warm, `/system` 5.8 s → 0.45 s. What still exceeds 1 s: R-1353 | `audits/release-2026-10-30/p2-r1346/` | + ## 2026-10-29 (evening release) — R-1349 and R-1350 closed (full text of the rows: `git show 8cc555be:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index bc6dedc3..f4c09e1b 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -211,7 +211,7 @@ stopping line that lies. | **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC | | **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC | | **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC | -| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: STARTED** (operator brief 2026-10-30 Part 3; the dated check is removed — the work is under way). OPEN | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) | +| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) | | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator | | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC | | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | @@ -293,7 +293,7 @@ stopping line that lies. | **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `_filebrowser`), FileBrowser „Up (healthy)", files. 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | serialise the two FileBrowser start paths at boot (one waits for the other, or the bring-up skips a stack another path is recreating); test red first | CC | -## Monitoring & notifications — 24 rows (P2 2, P3 15, P4 7) +## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -320,9 +320,9 @@ stopping line that lies. | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC | | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | -| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). **Read so far (2026-10-29 11:12–13:50 UTC, 631 probes):** Tester 1 314 probes, median 57 ms, max 163 ms; demo-hp 317 probes, median 61 ms, max 79 ms; no failed probe, no 5-s timeout. One record read „unhealthy": Tester 1 at 12:01 UTC, the controller stopped by hand for the restart proof (its probes passed). Four operator restarts and one controller update in the window never read „unhealthy". The night chain is not in it yet — so the decision waits for the night: alarm only until then. **Planned events in the log window (2026-10-29 evening, leave them out of the reading):** Tester 1 — 18:00 UTC rc swap, 18:05 and 19:05 UTC operator reboots (D65 proofs), 18:21 UTC floor update; demo-hp — 18:21 UTC floor update; 9202 is not logged. | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC | +| **R-1352** | Monitoring & notifications | P4 | **When a controller stops answering AND the agent restarts it between two host reports, the operator gets the restart mail first and the „does not answer — the agent restarts it in 2 minutes" alarm second, in the same second.** Seen live 2026-10-30 (the R-1344 proof on Tester 1: alarm 09:29:21, restart 09:31:22, both reached the hub with the 09:42:56 report; `hub/internal/monitor/controller_supervisor.go` `observe()` emits the restart before the alarm). The text of each mail is true; only the order reads backwards. | OPEN | — | emit the alarm before the restart in `observe()`, or skip the alarm when the restart for the same episode arrives with it (one line + a test) | CC | -## Hub & operator — 14 rows (P3 4, P4 10) +## Hub & operator — 14 rows (P3 3, P4 11) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -339,7 +339,7 @@ stopping line that lies. | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC | -| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. **2026-10-29 evening (hub 0.177.0) — WHERE THE TIME GOES: the save itself, in the hub's own database step.** The hub's access log holds seven vouch saves (POST `/configuration/artifacts`) since 2026-10-25 of 33–170 s; the two 170-s saves (13:03:19Z, 13:04:59Z) overlapped the hub's 13:00Z database snapshot slot. ingress-nginx's default 60-s read timeout turns any save over 60 s into the browser's 504 while the hub finishes it; the hub's own `WriteTimeout` is 60 s too. Not the road: the page loads in 6 ms in the cluster and 65–255 ms through the public name; not the registry: each of the save's calls (tag, 2 package probes, 2 sha lists, bundle sha) 0.08–0.25 s from DooPlex and 0.01–0.03 s from inside the hub pod. 0.177.0's per-leg line on the operator's 0.170.0 vouch (18:57:48 CET): registry probes 65 ms, sha lookups 27 ms, bundle sha 43 ms, **fleet check + save 38 117 ms**, total 38 255 ms. That step is `NewestReportedControllerVersion` (a `DISTINCT` scan of `reports`, 21 563 rows, no index on the column — 0.08 s on a tmpfs copy of the snapshot, so not the scan alone) and `SetArtifactManifest` (7 separate `hub_settings` writes, each its own transaction, each waiting for the write lock under `busy_timeout` 5 s). The slow-request line also named `/apps` 25–30 s, `/customers/tester-1` 7–9 s and `/system` 5.8 s in the last days (access log). Evidence: `audits/release-2026-10-29c/r1346/`. | OPEN — measured | — | (1) time `NewestReportedControllerVersion` and each `setSetting` separately inside the save (one more line); (2) if the writes wait: one transaction for the manifest, and find who holds the write lock that long; (3) the same for `/apps` and the customer page | CC | +| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) @@ -409,6 +409,7 @@ stopping line that lies. | R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row | | R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row | | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row | +| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | | **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 1d4cf965..3478cd76 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-10-30 + +**Updated 2026-10-29 (evening release).** On your yes: hub 0.177.0, agent 0.170.0 (your vouch) and controller 0.349.0 for every box; „Reboot guest" on the hub works and refuses during backups, updates and twice in an hour, and no alarm follows your hold clear any more — proven on Tester 1. The slow vouch is the hub's own database step (38 of 38.3 s); the stuck-controller decision waits for tomorrow's night log. + ## Moved from STATUS on 2026-10-29 (evening release) **Updated 2026-10-29 (afternoon release).** On your yes: hub 0.176.1, agent 0.169.0 (your vouch) and controller 0.348.0 as the floor for every box. From the hub's host page you can now restart a box's controller (it works when the controller is stopped; 3 presses in 15 minutes, then 30 minutes refused) and clear any held app (D64 = B) — both proven on Tester 1 on the release builds. Register 226 → 229 (7 opened, 4 closed); a stuck-but-running controller only alarms you until CC reads tonight's measurement. diff --git a/documentation/runbooks/publish-train-rules.md b/documentation/runbooks/publish-train-rules.md index 2fc7dac0..9db5cd94 100644 --- a/documentation/runbooks/publish-train-rules.md +++ b/documentation/runbooks/publish-train-rules.md @@ -110,3 +110,13 @@ one stops an ISO from carrying a golden the floor has already outrun. > default rotted twice. Treat these as a sanity check on what you read, never as a substitute for > reading it. (Floor raised to 0.216.0 by the operator on 2026-08-18 in a separate save after the > vouch, per rule 2 — see R-343, which records that the raise moved `demo-felhom` nine seconds later.) + + +## 6. The website and the installer reach the public through Cloudflare (D35 / R-1228, 2026-10-30) + +`felhom.eu` (site + `/scripts/felhom-host-install.sh`) is Cloudflare Pages since 2026-10-30. A push to `main` and a +new `installer-v…` tag change only DooPlex's copy (the way back); the public copy changes with +`sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run` first). The script reads the installer tag +from `manifests/webpage.yaml`, so bump both `--ref`s there FIRST, then upload — and it refuses an installer that is not +the tag's byte for byte. Check after: `felhom.eu/scripts/felhom-host-install.sh`'s sha256 equals +`git show installer-v:scripts/felhom-host-install.sh | sha256sum`. diff --git a/documentation/runbooks/total-loss-of-dooplex.md b/documentation/runbooks/total-loss-of-dooplex.md index cae00f64..25d6bcc8 100644 --- a/documentation/runbooks/total-loss-of-dooplex.md +++ b/documentation/runbooks/total-loss-of-dooplex.md @@ -122,7 +122,7 @@ sha256 matched, every image re-verified against its digest, and controller 0.340 | the hub DB | S2 (or Vaultwarden) | sheet, or via S3 | | sign a box update | S6 | **sheet — today exists on DooPlex only** | | hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild | -| Gitea `main` → `felhom.eu` website, ISO host | Cloudflare (S11), the build tree (lost) | S11 on the sheet; the build tree is rebuilt from code | +| `felhom.eu` website + the installer script, ISO host | **Cloudflare** (S11): since 2026-10-30 (D35 / R-1228) the site and `/scripts/felhom-host-install.sh` are served by Cloudflare Pages, the ISO by R2 — **both keep answering with DooPlex gone**; a CHANGE to them needs the repo (Gitea copy) and `scripts/publish-site-cloudflare.sh`. **A new install still needs DooPlex for two things:** the hub (`hub.felhom.eu`: the claim, the box's config, its first report) and Gitea (`gitea.dooplex.hu`: the agent package and the golden). The contact form (`api.felhom.eu`) stops; the page's `mailto:` link works | S11 on the sheet; the build tree is rebuilt from code | | Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope | **Result:** with a filled sheet (S1, S2, S4, S5, S6, S7, S8, S11 + the ep0 fingerprint) and the master password in diff --git a/skills/felhom-build-deploy/SKILL.md b/skills/felhom-build-deploy/SKILL.md index fa757a17..e9a96876 100644 --- a/skills/felhom-build-deploy/SKILL.md +++ b/skills/felhom-build-deploy/SKILL.md @@ -226,7 +226,8 @@ Green gate before any hub commit: `go build ./... && go vet ./... && go test ./. ## Website (felhom.eu/website) -Push to `main` = deployed (git-sync sidecar, live in ~1–2 min). **Run `python3 scripts/site_gates.py` +Since 2026-10-30 (D35) the public site is Cloudflare Pages: push to `main`, then `sec run CF_PAGES_TOKEN -- +scripts/publish-site-cloudflare.sh` (`--dry-run` first; DooPlex's git-sync copy is only the way back). **Run `python3 scripts/site_gates.py` first, after ANY website change** (BOM, emoji, nav parity, cache-bust `?v=N` — bump it when css/svg change). New pages must be added to the script's `PAGES` list. Emergency edits: https://files.felhom.eu.