Website git-sync in-cluster: R-1163 closed with evidence, R-1164 opened (P4); report

2026-10-19 11:51:54 +02:00
parent fd75003e17
commit e2c35007b5
8 changed files with 137 additions and 1 deletions
@@ -0,0 +1,81 @@
+# REPORT — the website keeps syncing during an internet outage (2026-10-19)
+
+## In plain words
+
+1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait
+   and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer
+   script both answer normally.
+2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the
+   address must go back to the public one — register row R-1164.
+
+## Part 1 — anonymous in-cluster fetch (before any commit)
+
+Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials:
+`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0`
+
+| ref | in-cluster | public (anonymous) |
+|---|---|---|
+| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal |
+| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` |
+
+Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`.
+
+## Part 2 — the change
+
+Commit **`fd75003e`** (no Co-Authored-By, per the brief):
+- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`;
+  the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and
+  `nginx-config-rev` unchanged.
+- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing
+  ref checks unchanged.
+
+Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0
+(`ok: manifest: all four git-syncs read the same repo (…)`).
+
+## Part 4 — deploy and proof
+
+**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e`
+(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod
+terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new
+ReplicaSet existed; re-run after the sync it waited for the old replica and finished.)
+
+**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and
+`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init
+containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh`
+**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`.
+
+**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after),
+`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`:
+- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**.
+- **C, without:** `too many failures, aborting`, restarts **1**.
+- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to
+  time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s.
+
+**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo
+values differ`, rc 1; restored byte-identical → rc 0.
+
+## Part 3 — docs changed
+
+- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/`
+  (main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the
+  DooPlex cluster.
+- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name
+  one repo (gate 6), and the sidecars carry `--max-failures=-1`.
+- `scripts/CHANGELOG.md`: one entry.
+
+## Register
+
+Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub &
+operator: move the website to Hetzner; then the git-syncs must use the public address again).
+
+## Observations
+
+- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) —
+  NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it.
+- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was
+  OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says.
+
+## Teardown
+
+Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0).
+No hub, controller, agent, customer box, demo box or ep0 touched.
@@ -0,0 +1,2 @@
+5d3cb5c132c2467e303442b15a86c37078b0136e	refs/heads/main
+add58eec9b2d8f0ab405862ddc827b64c86f1b56	refs/tags/installer-v1.34.0
@@ -0,0 +1,2 @@
+5d3cb5c132c2467e303442b15a86c37078b0136e	refs/heads/main
+add58eec9b2d8f0ab405862ddc827b64c86f1b56	refs/tags/installer-v1.34.0
@@ -0,0 +1,2 @@
+NAME                              READY            RESTARTS   AGE
+felhom-webpage-68484d6485-6cw7k   true,true,true   7,6,0      2026-10-19T09:00:02Z
@@ -0,0 +1,26 @@
+NAME                              READY            RESTARTS
+felhom-webpage-869ff88bf7-fswbc   true,true,true   0,0,0
+--- git-sync (last 5)
+INFO: detected pid 1, running init handler
+{"logger":"","ts":"2026-10-19 09:42:37.862037","caller":{"file":"main.go","line":626},"level":0,"msg":"starting up","version":"v4.4.0","pid":11,"uid":65534,"gid":65534,"home":"/tmp","flags":["--link=current","--max-failures=-1","--period=30s","--ref=main","--r
+{"logger":"","ts":"2026-10-19 09:42:37.865072","caller":{"file":"main.go","line":731},"level":0,"msg":"git version","version":"git version 2.39.5"}
+{"logger":"","ts":"2026-10-19 09:42:39.559074","caller":{"file":"main.go","line":1775},"level":0,"msg":"update required","ref":"main","local":"fd75003e173f31abf6bf5aed64cba8246d4f0076","remote":"fd75003e173f31abf6bf5aed64cba8246d4f0076","syncCount":0}
+{"logger":"","ts":"2026-10-19 09:42:42.766595","caller":{"file":"main.go","line":1821},"level":0,"msg":"updated successfully","ref":"main","remote":"fd75003e173f31abf6bf5aed64cba8246d4f0076","syncCount":1}
+--- git-sync-scripts (last 5)
+INFO: detected pid 1, running init handler
+{"logger":"","ts":"2026-10-19 09:42:38.097109","caller":{"file":"main.go","line":626},"level":0,"msg":"starting up","version":"v4.4.0","pid":11,"uid":65534,"gid":65534,"home":"/tmp","flags":["--link=current","--max-failures=-1","--period=30s","--ref=installer-
+{"logger":"","ts":"2026-10-19 09:42:38.192285","caller":{"file":"main.go","line":731},"level":0,"msg":"git version","version":"git version 2.39.5"}
+{"logger":"","ts":"2026-10-19 09:42:39.799361","caller":{"file":"main.go","line":1775},"level":0,"msg":"update required","ref":"installer-v1.34.0","local":"624bcdc37239cbb0b98678936a5c94182a8ba15d","remote":"624bcdc37239cbb0b98678936a5c94182a8ba15d","syncCount
+{"logger":"","ts":"2026-10-19 09:42:41.900152","caller":{"file":"main.go","line":1821},"level":0,"msg":"updated successfully","ref":"installer-v1.34.0","remote":"624bcdc37239cbb0b98678936a5c94182a8ba15d","syncCount":1}
+--- init containers
+updated successfully  (git-sync-init)
+updated successfully  (git-sync-scripts-init)
+--- HTTP
+site: 200
+script: 200
+SCRIPT_VERSION served: SCRIPT_VERSION="1.34.0"
+manifest tag: installer-v
+manifest --ref: --ref=installer-v1.34.0
+624bcdc37239cbb0b98678936a5c94182a8ba15d	refs/tags/installer-v1.34.0^{}
+git-sync start flag: --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
+git-sync-scripts start flag: --repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git
@@ -0,0 +1,16 @@
+NAME         RESTARTS   STATE
+gs-with      0          map[running:map[startedAt:2026-10-19T09:44:19Z]]
+gs-without   0          map[running:map[startedAt:2026-10-19T09:44:19Z]]
+--- gs-with log (error lines):
+--- gs-with previous run (if any):
+--- gs-without log (error lines):
+--- gs-without previous run (if any):
+=== B/C re-read 09:48:05 (first fetch to 192.0.2.1 times out after ~133 s; the 95 s read above was too early)
+NAME         RESTARTS
+gs-with      0
+gs-without   1
+--- gs-with:
+      1 "failCount":1
+      1 "msg":"error syncing repo, will retry"
+--- gs-without, previous run:
+"msg":"too many failures, aborting"
@@ -26,6 +26,12 @@

 ---

+## 2026-10-19 — the website keeps syncing during an internet outage (`fd75003e`)
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-1163** | **The website's git-syncs reached Gitea over the internet and stopped on their first error.** All four read `https://gitea.dooplex.hu/…` (out and back in, although both run on DooPlex); git-sync v4.4.0's `--max-failures` defaults to 0, so on 2026-10-19 09:24 one failed fetch exited the sidecar (`too many failures, aborting … failCount:1`) and it restarted. A pod starting during an outage could not seed its pages at all. (P3) | CLOSED 2026-10-19 — the four git-syncs read `http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` (anonymous in-cluster fetch proven first: `main` and `installer-v1.34.0` resolve to the public hashes); the two sidecars carry `--max-failures=-1`; `hostinstall_gates.py` gate 6 keeps the four `--repo` values identical (red-proved). Deployed by a Deployment-only ArgoCD sync: new pod 0/0/0 restarts, both sidecars `updated successfully`, site and script 200, served `SCRIPT_VERSION` 1.34.0 = the tag. Proof of the mechanism on a dead address: with the flag `error syncing repo, will retry`, 0 restarts; without it `too many failures, aborting`, 1 restart. Ties the website to the DooPlex cluster → R-1164. | `audits/website-gitsync-incluster-2026-10-19/` |
+
 ## 2026-10-19 (day) — agent 0.160.0, one sign-in for the file manager (controller 0.332.0), hub 0.159.0 (full text of each row: `git show e4f022a0:documentation/backlog/OPEN-ITEMS.md`)

 | Row | What | Closed | Evidence |
@@ -264,7 +264,7 @@ stopping line that lies.
 | **R-1157** | Monitoring & notifications | P4 | **While the cadence-cap flag stays degraded (the agent's 24 h hold), further refused backups raise no new alarm:** a guest refused every day reads as one long episode, one mail. The same before hub 0.159.0. Found by the hub helper 2026-10-19. | OPEN — filed 2026-10-19 | — | Decide whether a refusal count in the event detail or a daily repeat is wanted | CC |


-## Hub & operator — 12 rows (P3 6, P4 6)
+## Hub & operator — 13 rows (P3 6, P4 7)

 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -280,6 +280,7 @@ stopping line that lies.
 | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC |
 | **R-1011** | Hub & operator | P3 | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** Seen 2026-10-14 (`/configuration`: two „0.156.1" entries, `ac83ea41…` is the step bundle). | **READY** — filed 2026-10-14 | — | CC: list only versions whose package holds the `felhom-agent` file (refuse a missing preferred file instead of falling back), with a test; next hub release | CC |
 | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator |
+| **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |


 ## Business & legal — 14 rows (P2 4, P3 2, P4 8)