From e2c35007b5c46039356a244cca31a6c7a3619145 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 19 Oct 2026 11:51:54 +0200 Subject: [PATCH] Website git-sync in-cluster: R-1163 closed with evidence, R-1164 opened (P4); report --- REPORT-website-gitsync-2026-10-19.md | 81 +++++++++++++++++++ .../gitsync-part1-incluster.txt | 2 + .../gitsync-part1-public.txt | 2 + .../gitsync-pre-pods.txt | 2 + .../gitsync-scenarioA.txt | 26 ++++++ .../gitsync-scenarioBC.txt | 16 ++++ documentation/backlog/CLOSED-ITEMS.md | 6 ++ documentation/backlog/OPEN-ITEMS.md | 3 +- 8 files changed, 137 insertions(+), 1 deletion(-) create mode 100644 REPORT-website-gitsync-2026-10-19.md create mode 100644 documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-incluster.txt create mode 100644 documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-public.txt create mode 100644 documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-pre-pods.txt create mode 100644 documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioA.txt create mode 100644 documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioBC.txt diff --git a/REPORT-website-gitsync-2026-10-19.md b/REPORT-website-gitsync-2026-10-19.md new file mode 100644 index 00000000..a01a7212 --- /dev/null +++ b/REPORT-website-gitsync-2026-10-19.md @@ -0,0 +1,81 @@ +# REPORT — the website keeps syncing during an internet outage (2026-10-19) + +## In plain words + +1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait + and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer + script both answer normally. +2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the + address must go back to the public one — register row R-1164. + +## Part 1 — anonymous in-cluster fetch (before any commit) + +Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials: +`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0` + +| ref | in-cluster | public (anonymous) | +|---|---|---| +| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal | +| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` | + +Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`. + +## Part 2 — the change + +Commit **`fd75003e`** (no Co-Authored-By, per the brief): +- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`; + the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and + `nginx-config-rev` unchanged. +- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing + ref checks unchanged. + +Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0 +(`ok: manifest: all four git-syncs read the same repo (…)`). + +## Part 4 — deploy and proof + +**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e` +(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod +terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new +ReplicaSet existed; re-run after the sync it waited for the old replica and finished.) + +**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and +`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init +containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh` +**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`. + +**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after), +`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`: +- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**. +- **C, without:** `too many failures, aborting`, restarts **1**. +- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to + time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s. + +**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo +values differ`, rc 1; restored byte-identical → rc 0. + +## Part 3 — docs changed + +- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/` + (main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the + DooPlex cluster. +- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name + one repo (gate 6), and the sidecars carry `--max-failures=-1`. +- `scripts/CHANGELOG.md`: one entry. + +## Register + +Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub & +operator: move the website to Hetzner; then the git-syncs must use the public address again). + +## Observations + +- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) — + NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it. +- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was + OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says. + +## Teardown + +Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0). +No hub, controller, agent, customer box, demo box or ep0 touched. diff --git a/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-incluster.txt b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-incluster.txt new file mode 100644 index 00000000..06d8f431 --- /dev/null +++ b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-incluster.txt @@ -0,0 +1,2 @@ +5d3cb5c132c2467e303442b15a86c37078b0136e refs/heads/main +add58eec9b2d8f0ab405862ddc827b64c86f1b56 refs/tags/installer-v1.34.0 diff --git a/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-public.txt b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-public.txt new file mode 100644 index 00000000..06d8f431 --- /dev/null +++ b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-part1-public.txt @@ -0,0 +1,2 @@ +5d3cb5c132c2467e303442b15a86c37078b0136e refs/heads/main +add58eec9b2d8f0ab405862ddc827b64c86f1b56 refs/tags/installer-v1.34.0 diff --git a/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-pre-pods.txt b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-pre-pods.txt new file mode 100644 index 00000000..c2631cd2 --- /dev/null +++ b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-pre-pods.txt @@ -0,0 +1,2 @@ +NAME READY RESTARTS AGE +felhom-webpage-68484d6485-6cw7k true,true,true 7,6,0 2026-10-19T09:00:02Z diff --git a/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioA.txt b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioA.txt new file mode 100644 index 00000000..a18cf439 --- /dev/null +++ b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioA.txt @@ -0,0 +1,26 @@ +NAME READY RESTARTS +felhom-webpage-869ff88bf7-fswbc true,true,true 0,0,0 +--- git-sync (last 5) +INFO: detected pid 1, running init handler +{"logger":"","ts":"2026-10-19 09:42:37.862037","caller":{"file":"main.go","line":626},"level":0,"msg":"starting up","version":"v4.4.0","pid":11,"uid":65534,"gid":65534,"home":"/tmp","flags":["--link=current","--max-failures=-1","--period=30s","--ref=main","--r +{"logger":"","ts":"2026-10-19 09:42:37.865072","caller":{"file":"main.go","line":731},"level":0,"msg":"git version","version":"git version 2.39.5"} +{"logger":"","ts":"2026-10-19 09:42:39.559074","caller":{"file":"main.go","line":1775},"level":0,"msg":"update required","ref":"main","local":"fd75003e173f31abf6bf5aed64cba8246d4f0076","remote":"fd75003e173f31abf6bf5aed64cba8246d4f0076","syncCount":0} +{"logger":"","ts":"2026-10-19 09:42:42.766595","caller":{"file":"main.go","line":1821},"level":0,"msg":"updated successfully","ref":"main","remote":"fd75003e173f31abf6bf5aed64cba8246d4f0076","syncCount":1} +--- git-sync-scripts (last 5) +INFO: detected pid 1, running init handler +{"logger":"","ts":"2026-10-19 09:42:38.097109","caller":{"file":"main.go","line":626},"level":0,"msg":"starting up","version":"v4.4.0","pid":11,"uid":65534,"gid":65534,"home":"/tmp","flags":["--link=current","--max-failures=-1","--period=30s","--ref=installer- +{"logger":"","ts":"2026-10-19 09:42:38.192285","caller":{"file":"main.go","line":731},"level":0,"msg":"git version","version":"git version 2.39.5"} +{"logger":"","ts":"2026-10-19 09:42:39.799361","caller":{"file":"main.go","line":1775},"level":0,"msg":"update required","ref":"installer-v1.34.0","local":"624bcdc37239cbb0b98678936a5c94182a8ba15d","remote":"624bcdc37239cbb0b98678936a5c94182a8ba15d","syncCount +{"logger":"","ts":"2026-10-19 09:42:41.900152","caller":{"file":"main.go","line":1821},"level":0,"msg":"updated successfully","ref":"installer-v1.34.0","remote":"624bcdc37239cbb0b98678936a5c94182a8ba15d","syncCount":1} +--- init containers +updated successfully (git-sync-init) +updated successfully (git-sync-scripts-init) +--- HTTP +site: 200 +script: 200 +SCRIPT_VERSION served: SCRIPT_VERSION="1.34.0" +manifest tag: installer-v +manifest --ref: --ref=installer-v1.34.0 +624bcdc37239cbb0b98678936a5c94182a8ba15d refs/tags/installer-v1.34.0^{} +git-sync start flag: --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git +git-sync-scripts start flag: --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git diff --git a/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioBC.txt b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioBC.txt new file mode 100644 index 00000000..96316937 --- /dev/null +++ b/documentation/audits/website-gitsync-incluster-2026-10-19/gitsync-scenarioBC.txt @@ -0,0 +1,16 @@ +NAME RESTARTS STATE +gs-with 0 map[running:map[startedAt:2026-10-19T09:44:19Z]] +gs-without 0 map[running:map[startedAt:2026-10-19T09:44:19Z]] +--- gs-with log (error lines): +--- gs-with previous run (if any): +--- gs-without log (error lines): +--- gs-without previous run (if any): +=== B/C re-read 09:48:05 (first fetch to 192.0.2.1 times out after ~133 s; the 95 s read above was too early) +NAME RESTARTS +gs-with 0 +gs-without 1 +--- gs-with: + 1 "failCount":1 + 1 "msg":"error syncing repo, will retry" +--- gs-without, previous run: +"msg":"too many failures, aborting" diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index aa698ebb..0545eaf7 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,12 @@ --- +## 2026-10-19 — the website keeps syncing during an internet outage (`fd75003e`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1163** | **The website's git-syncs reached Gitea over the internet and stopped on their first error.** All four read `https://gitea.dooplex.hu/…` (out and back in, although both run on DooPlex); git-sync v4.4.0's `--max-failures` defaults to 0, so on 2026-10-19 09:24 one failed fetch exited the sidecar (`too many failures, aborting … failCount:1`) and it restarted. A pod starting during an outage could not seed its pages at all. (P3) | CLOSED 2026-10-19 — the four git-syncs read `http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` (anonymous in-cluster fetch proven first: `main` and `installer-v1.34.0` resolve to the public hashes); the two sidecars carry `--max-failures=-1`; `hostinstall_gates.py` gate 6 keeps the four `--repo` values identical (red-proved). Deployed by a Deployment-only ArgoCD sync: new pod 0/0/0 restarts, both sidecars `updated successfully`, site and script 200, served `SCRIPT_VERSION` 1.34.0 = the tag. Proof of the mechanism on a dead address: with the flag `error syncing repo, will retry`, 0 restarts; without it `too many failures, aborting`, 1 restart. Ties the website to the DooPlex cluster → R-1164. | `audits/website-gitsync-incluster-2026-10-19/` | + ## 2026-10-19 (day) — agent 0.160.0, one sign-in for the file manager (controller 0.332.0), hub 0.159.0 (full text of each row: `git show e4f022a0:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 238eb286..ca9b77d0 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -264,7 +264,7 @@ stopping line that lies. | **R-1157** | Monitoring & notifications | P4 | **While the cadence-cap flag stays degraded (the agent's 24 h hold), further refused backups raise no new alarm:** a guest refused every day reads as one long episode, one mail. The same before hub 0.159.0. Found by the hub helper 2026-10-19. | OPEN — filed 2026-10-19 | — | Decide whether a refusal count in the event detail or a daily repeat is wanted | CC | -## Hub & operator — 12 rows (P3 6, P4 6) +## Hub & operator — 13 rows (P3 6, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -280,6 +280,7 @@ stopping line that lies. | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | | **R-1011** | Hub & operator | P3 | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** Seen 2026-10-14 (`/configuration`: two „0.156.1" entries, `ac83ea41…` is the step bundle). | **READY** — filed 2026-10-14 | — | CC: list only versions whose package holds the `felhom-agent` file (refuse a missing preferred file instead of falling back), with a test; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | +| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | ## Business & legal — 14 rows (P2 4, P3 2, P4 8)