R-1228: retire DooPlex's copy of the website (operator yes); the installer pin moves to INSTALLER-TAG
gates / gates (push) Successful in 3m25s
gates / gates (push) Successful in 3m25s
- INSTALLER-TAG (one line) is the only installer pin; publish-site-cloudflare.sh reads it. - hostinstall gate 6c: pin file holds one tag, no manifest pins one, publish script takes the site from origin/main and scripts/ from the tag. Red seen on five mutants. - manifests: webpage.yaml keeps only FileBrowser; contact-mailer.yaml loses the old felhom.eu/api Ingress (an orphan: glued to the next doc by a `80---` typo). - R-110 fence text, manifests/website rules, build skill, runbooks, poster facts updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
7c6d3236df
commit
c28ce7ce2f
16 files changed
+107
-418
No files matched your search
@@ -116,7 +116,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
- **Vaultwarden** — the operator's password manager.
|
||||
- **Prometheus + Alertmanager** — the operator's alarms.
|
||||
- **The signing keys** — they sign every agent update and config bundle a box accepts.
|
||||
- **The contact form's mailer** (`api.felhom.eu`) — and, until 2026-11-06, the old copy of the website (the way back after the 2026-10-30 move to Cloudflare).
|
||||
- **The contact form's mailer** (`api.felhom.eu`). The old copy of the website (the way back after the 2026-10-30 move to Cloudflare) was retired on 2026-11-07.
|
||||
- **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine").
|
||||
- **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies).
|
||||
|
||||
|
||||
@@ -210,7 +210,7 @@ stopping line that lies.
|
||||
| **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
|
||||
| **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
|
||||
| **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED **2026-11-07: asked** (demo-check session, in chat); the due date moved to 2026-11-14 to wait for the answer. | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED **2026-11-07: asked, and the operator said RETIRE IT (in chat, 23:0x CET).** The copy has no visitors (`felhom.eu` resolves to Cloudflare, also from DooPlex). It is not just a Deployment to delete: `manifests/webpage.yaml` is also where the installer TAG is pinned — `publish-site-cloudflare.sh` reads it and `hostinstall_gates.py` gate 6 checks it, and the R-110 fence in `CLAUDE.md` + `.claude/rules/manifests.md` describes it. So the retirement moves the tag pin to its own home first (script + gate 6 + its decoys + the fence text), then removes the webpage Deployment, Service, Ingress, its two ConfigMaps and TLS Secret by a deliberate ArgoCD sync (FileBrowser in the same file stays). Not started at night during an agent rollout. | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
|
||||
| **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
|
||||
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
@@ -406,6 +406,5 @@ stopping line that lies.
|
||||
| R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
|
||||
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
|
||||
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
|
||||
| R-1228 | 2026-11-14 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
|
||||
@@ -233,8 +233,9 @@ directions: it makes an operator think a pushed fix is live when it is not, and
|
||||
is live when it is not. (Measured 2026-08-12: this URL served `1.25.0` while `main` held `1.27.0`,
|
||||
three and a half hours after the push.)
|
||||
|
||||
`manifests/webpage.yaml` runs **two** git-syncs against **different refs**: the website tracks `main`,
|
||||
and `/scripts/` is checked out from the tag **`installer-v<SCRIPT_VERSION>`**. So this URL serves the
|
||||
felhom.eu is Cloudflare Pages (D35). `scripts/publish-site-cloudflare.sh` uploads the website from `main` and
|
||||
`/scripts/` from the tag **`installer-v<SCRIPT_VERSION>`** named in `INSTALLER-TAG` (one line; until 2026-11-07 the
|
||||
pin was two `--ref` lines in `manifests/webpage.yaml`, DooPlex's retired copy — R-1228). So this URL serves the
|
||||
**tagged** installer, not `main`.
|
||||
|
||||
**What actually publishes it — three acts, and the middle one is two lines, not one:**
|
||||
@@ -242,15 +243,13 @@ and `/scripts/` is checked out from the tag **`installer-v<SCRIPT_VERSION>`**. S
|
||||
1. Bump `SCRIPT_VERSION` in `scripts/felhom-host-install.sh` (the single version source) and push to
|
||||
`main`. *Nothing is published yet.*
|
||||
2. Cut and push the tag `installer-v<new SCRIPT_VERSION>`.
|
||||
3. Move **BOTH** `--ref=installer-v…` pins in `manifests/webpage.yaml` — the git-sync **sidecar** and
|
||||
the **init container** (today lines 327 and 372) — commit, and sync ArgoCD. **Moving one pin is the
|
||||
trap**: the running pod keeps serving until it restarts, and then a fresh pod seeded by the stale
|
||||
init container serves the OLD script with no error anywhere.
|
||||
3. Put the new tag in `INSTALLER-TAG`, commit, push, then upload:
|
||||
`sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run` first).
|
||||
|
||||
**To roll back:** move the tag back and wait ~30 s. No ArgoCD sync, no deploy — that is the emergency
|
||||
lever; fix forward afterwards. **Do not pin the website to the tag**, or every copy edit becomes a
|
||||
release. `hostinstall_gates.py` gate 6 fails if the manifest stops naming an `installer-v…` tag or if
|
||||
the website stops tracking `main`.
|
||||
**To roll back:** move the tag back, then run the same script (~1 min; it re-reads the tag). That is the
|
||||
emergency lever; fix forward afterwards. **Do not pin the website to the tag**, or every copy edit becomes a
|
||||
release. `hostinstall_gates.py` gate 6 fails if `INSTALLER-TAG` does not name exactly one tag, if a manifest
|
||||
pins a second one, or if the publish script stops taking the site from `main` and `/scripts/` from the tag.
|
||||
|
||||
**How to verify from OUTSIDE that the published version actually changed** — a push, a green sync and a
|
||||
correct-looking manifest are each consistent with nothing having been published, so ask the public URL
|
||||
|
||||
@@ -114,9 +114,9 @@ one stops an ISO from carrying a golden the floor has already outrun.
|
||||
|
||||
## 6. The website and the installer reach the public through Cloudflare (D35 / R-1228, 2026-10-30)
|
||||
|
||||
`felhom.eu` (site + `/scripts/felhom-host-install.sh`) is Cloudflare Pages since 2026-10-30. A push to `main` and a
|
||||
new `installer-v…` tag change only DooPlex's copy (the way back); the public copy changes with
|
||||
`sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run` first). The script reads the installer tag
|
||||
from `manifests/webpage.yaml`, so bump both `--ref`s there FIRST, then upload — and it refuses an installer that is not
|
||||
`felhom.eu` (site + `/scripts/felhom-host-install.sh`) is Cloudflare Pages since 2026-10-30; DooPlex's copy (the way
|
||||
back) was retired on 2026-11-07. A push to `main` or a new `installer-v…` tag changes nothing public; the public copy
|
||||
changes with `sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run` first). The script reads the
|
||||
installer tag from `INSTALLER-TAG`, so put the new tag there FIRST, then upload — and it refuses an installer that is not
|
||||
the tag's byte for byte. Check after: `felhom.eu/scripts/felhom-host-install.sh`'s sha256 equals
|
||||
`git show installer-v<X>:scripts/felhom-host-install.sh | sha256sum`.
|
||||
@@ -280,7 +280,7 @@ attempted; reachability is the point.
|
||||
|
||||
The operator ruled that `felhom.eu` stays anonymously readable for the moment. Making it private
|
||||
breaks two live paths that clone it with **no credentials**: the website git-sync in
|
||||
`manifests/webpage.yaml`, and the installer fetched from the `installer-v…` tag by every new box
|
||||
`manifests/webpage.yaml` (retired 2026-11-07, R-1228 — no longer a reason), and the installer fetched from the `installer-v…` tag by every new box
|
||||
(R-110). Both would have to be given credentials first. The standing consequence of that ruling:
|
||||
**no secret may ever enter this repo again** — which `manifest_bearer_gate.py` now enforces with no
|
||||
exemption. If the decision is ever reversed, the 32 `<!-- source: … -->` comments served on
|
||||
|
||||
Reference in new issue
Block a user