88 lines
5.2 KiB
Markdown
88 lines
5.2 KiB
Markdown
# REPORT — the website keeps syncing during an internet outage (2026-10-19)
|
||
|
||
## In plain words
|
||
|
||
1. **It worked.** The website's copy helpers now read Gitea inside the cluster, with no internet hop, and they wait
|
||
and try again instead of stopping. The new website pod runs with 0 restarts, and the site and the installer
|
||
script both answer normally.
|
||
2. **Note for later:** this ties the website to the DooPlex cluster. If it moves (for example to Hetzner), the
|
||
address must go back to the public one — register row R-1164.
|
||
|
||
## Part 1 — anonymous in-cluster fetch (before any commit)
|
||
|
||
Throwaway pod in a scratch namespace (`ccscratch-gitsync`, deleted after), no credentials:
|
||
`git ls-remote http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git main installer-v1.34.0`
|
||
|
||
| ref | in-cluster | public (anonymous) |
|
||
|---|---|---|
|
||
| `refs/heads/main` | `5d3cb5c1…` | `5d3cb5c1…` — equal |
|
||
| `refs/tags/installer-v1.34.0` | `add58eec…` (annotated tag object) | `add58eec…` — equal; peels to `624bcdc3…` |
|
||
|
||
Evidence: `audits/website-gitsync-incluster-2026-10-19/gitsync-part1-*.txt`.
|
||
|
||
## Part 2 — the change
|
||
|
||
Commit **`fd75003e`** (no Co-Authored-By, per the brief):
|
||
- `manifests/webpage.yaml`: all four git-syncs `--repo=http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git`;
|
||
the two sidecars `--max-failures=-1` (not the `--one-time` inits); one comment above the website sidecar. Refs and
|
||
`nginx-config-rev` unchanged.
|
||
- `scripts/hostinstall_gates.py` gate 6: fails unless there are four `--repo=` values and all are identical. Existing
|
||
ref checks unchanged.
|
||
|
||
Gates: `python3 scripts/repo_gates.py` → `all felhom.eu gates OK`; `python3 scripts/hostinstall_gates.py` → rc 0
|
||
(`ok: manifest: all four git-syncs read the same repo (…)`).
|
||
|
||
## Part 4 — deploy and proof
|
||
|
||
**Deploy:** hard refresh showed only `Deployment/felhom-webpage` OutOfSync; synced that resource alone at `fd75003e`
|
||
(the skill's Deployment-only sync). New pod `felhom-webpage-869ff88bf7-fswbc` ready at 09:42 UTC; the old pod
|
||
terminated. ArgoCD: Synced, operation Succeeded. (The first `rollout status` returned at once — it ran before the new
|
||
ReplicaSet existed; re-run after the sync it waited for the old replica and finished.)
|
||
|
||
**Scenario A** (`gitsync-scenarioA.txt`): restarts `0,0,0`; both sidecars started with the in-cluster `--repo` and
|
||
`--max-failures=-1`, then `updated successfully` (`main` → `fd75003e`, `installer-v1.34.0` → `624bcdc3`); both init
|
||
containers `updated successfully`; `https://felhom.eu/` **200**, `https://felhom.eu/scripts/felhom-host-install.sh`
|
||
**200**, served `SCRIPT_VERSION="1.34.0"` = manifest `--ref=installer-v1.34.0`.
|
||
|
||
**Scenarios B and C** (`gitsync-scenarioBC.txt`): scratch namespace `ccscratch-gitsync-bc` (deleted after),
|
||
`git-sync:v4.4.0 --repo=http://192.0.2.1/x.git --period=10s`:
|
||
- **B, with `--max-failures=-1`:** `error syncing repo, will retry` (`failCount:1`), restarts **0**.
|
||
- **C, without:** `too many failures, aborting`, restarts **1**.
|
||
- The first read at 95 s showed nothing for either: one fetch to an address that never answers takes **~133 s** to
|
||
time out. The re-read at 09:48 recorded both results. Instead of 90 s, a later run of this test needs about 150 s.
|
||
|
||
**Scenario D — gate red-proof:** one `--repo` changed to the public address → gate 6 `FAIL … the git-sync --repo
|
||
values differ`, rc 1; restored byte-identical → rc 0.
|
||
|
||
## Part 3 — docs changed
|
||
|
||
- `documentation/architecture/01-topology-and-trust.md` §7: a `[DESIGN]` paragraph — the website pulls `website/`
|
||
(main) and `scripts/` (installer tag) from Gitea in-cluster over plain HTTP, sidecars retry forever, tied to the
|
||
DooPlex cluster.
|
||
- `.claude/rules/manifests.md` „The two git-syncs": the fact that all four containers read Gitea in-cluster, must name
|
||
one repo (gate 6), and the sidecars carry `--max-failures=-1`.
|
||
- `scripts/CHANGELOG.md`: one entry.
|
||
|
||
## Register
|
||
|
||
Rows before **166**, after **167**. Closed **R-1163** (finding + fix, with evidence). Opened **R-1164** (P4, Hub &
|
||
operator: move the website to Hetzner; then the git-syncs must use the public address again).
|
||
|
||
## Observations
|
||
|
||
- The 95-second wait in the brief is too short for an address that never answers (~133 s TCP timeout per fetch) —
|
||
NOT-A-FINDING: recorded here as the correct wait for this test; nothing in the product depends on it.
|
||
- The skill says a whole-app sync would push git's view over de-gitted Secrets; today only the webpage Deployment was
|
||
OutOfSync — NOT-A-FINDING: the Deployment-only sync was used anyway, as the skill says.
|
||
|
||
- Around 09:50–09:55 UTC, DooPlex itself briefly could not resolve names (`Temporary failure in name resolution`;
|
||
`curl https://felhom.eu/` → `000`). By 09:56 DNS, the internet and the site (200) were back. The website pod kept
|
||
restarts `0,0,0` and synced `e2c35007` in-cluster at 09:54 with no error. NOT-A-FINDING: the same class of short
|
||
outage that triggered this task; it is not proof that the fix worked (whether the blip reached the cluster is unknown).
|
||
- CI: `fd75003e` job 2092 success, `e2c35007` job 2094 success.
|
||
|
||
## Teardown
|
||
|
||
Scratch namespaces `ccscratch-gitsync` and `ccscratch-gitsync-bc` deleted (`kubectl get ns | grep -c ccscratch` → 0).
|
||
No hub, controller, agent, customer box, demo box or ep0 touched.
|