admin
c718aad1bc
gates / gates (push) Successful in 7s
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state re-registers and orphans the old record) changed the design; P5 (a failed run signals NOTHING) is why the alarm exists at all. R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green run, as required: the class it opened is answered at both ends, the hook refusing locally and CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for the STATIC gate, while its original scope, the runtime gate, is deliberately still not automatic and should stay that way. NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both of which change how the operator works — so it is theirs to decide, and the row states the cost honestly rather than recommending it. CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout). CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
221 lines
10 KiB
Markdown
221 lines
10 KiB
Markdown
# SPIKE — a Gitea Actions runner for the felhom gate entry points (R-168, 2026-08-02)
|
|
|
|
**Verdict: the mechanism works and is SHIPPED.** All six probes answered; none produced a STOP.
|
|
The runner is unprivileged, host-mode, one registration for all four repos, and a failed run now
|
|
sends its own alarm — measured, not assumed.
|
|
|
|
**Context.** Session 1 (same day) gave every felhom repo one gate entry point and a
|
|
`.githooks/pre-push` that runs it and refuses a failing push. That hook is per-clone and
|
|
`git push --no-verify` skips it, so nothing independent of the person pushing ever saw whether the
|
|
gates passed. This spike built the independent half.
|
|
|
|
---
|
|
|
|
## Arrival state, re-confirmed live (all matched the anchor)
|
|
|
|
| Fact | Expected | Measured |
|
|
|---|---|---|
|
|
| Gitea version | 1.26.2 | **1.26.2** |
|
|
| Actions enabled | all five repos | **all five** (`repo_unit` type 10 present on each) |
|
|
| Runners registered | 0 | **0** (`action_runner` empty) |
|
|
| Workflow runs, ever | 0 | **0** (`action_run` empty) |
|
|
| Branch protections | 0 | **0** (`protected_branch` empty) |
|
|
| `.gitea/` directory | none | **none in any of the five** |
|
|
| ArgoCD `gitea` app | `path: gitea-system`, auto-sync off | **confirmed**, Synced at `420e819`, Healthy |
|
|
|
|
Repo visibility, which decided P3's design: the four product repos are **public**;
|
|
`homelab-manifests` is private.
|
|
|
|
> **A near-miss worth recording.** My first census query reported Actions enabled on only five
|
|
> *unrelated* repos, which looked like a baseline drift big enough to change the task. It was a
|
|
> `| tail -5` inside my own query helper truncating the result. The measurement was never wrong;
|
|
> the instrument was. Re-run without the pipe, all five target repos had the Actions unit. **A tool
|
|
> that silently drops rows is indistinguishable from a finding** — the same class as the
|
|
> `go test -run` filter that matches nothing and prints `ok`.
|
|
|
|
---
|
|
|
|
## P1 — does a registered runner pick up a job at all?
|
|
|
|
**Method.** Registered one runner at **owner scope** for `admin` (`repo_id=0`), label `felhom-gates`,
|
|
via a token minted with `gitea actions generate-runner-token --scope admin` and stored out-of-band as
|
|
`Secret/act-runner-registration`. Pushed a temporary `.gitea/workflows/probe.yml` with one
|
|
`run: echo` step to `felhom.eu`.
|
|
|
|
**Measured.** Run #1 appeared against the pushed commit, was claimed by `felhom-gates-runner`, and
|
|
finished **success**:
|
|
|
|
```
|
|
felhom-gates-runner(version:v0.6.1) received task 1 of job probe, be triggered by event: push
|
|
P1-OK runner picked up the job
|
|
🏁 Job succeeded
|
|
```
|
|
|
|
**Ruling: PASS.** No STOP.
|
|
|
|
---
|
|
|
|
## P2 — can the stock image run our checks without a container runtime?
|
|
|
|
**Method.** Inspected `gitea/act_runner:0.6.1` directly, then re-measured *inside a job* — because
|
|
what matters is what the **job** sees, not what the image contains, and in host mode those are the
|
|
same thing only if host mode is really in effect.
|
|
|
|
**Measured.** Stock image (Alpine Linux v3.23):
|
|
|
|
```
|
|
python3: sh: python3: not found ABSENT
|
|
git: git version 2.52.0 PRESENT
|
|
```
|
|
|
|
In-job, after building the minimal image:
|
|
|
|
```
|
|
P2 python3: Python 3.12.13
|
|
P2 git: git version 2.52.0
|
|
```
|
|
|
|
**Ruling: the known branch, not a failure.** Host mode works; only `python3` was missing. Built
|
|
`gitea.dooplex.hu/admin/felhom-act-runner:0.1.0` = stock (pinned) + `python3`, nothing else, from
|
|
`homelab-manifests/gitea-system/act-runner/Dockerfile`. Verified by deleting the local copy and
|
|
**re-pulling from the registry** rather than trusting the push's own output.
|
|
|
|
**Host mode was never in doubt, so the privileged/dind pattern was never reached for** (§5/§12 of the
|
|
task, and the reason is written into `act-runner.yaml`: DooPlex is Tier 2 and *is* the recovery
|
|
chain).
|
|
|
|
---
|
|
|
|
## P3 — can a workflow obtain the source without JavaScript actions?
|
|
|
|
**Method.** No `uses:` anywhere. A plain `run:` step clones from the **in-cluster** Gitea Service
|
|
(`http://gitea.gitea-system.svc.cluster.local:3000`) — no ingress, no TLS hop, no geo rule — and
|
|
checks out `$GITHUB_SHA`. The four product repos are public, so no credential is needed at all.
|
|
|
|
**Measured.**
|
|
|
|
```
|
|
P3 pushed sha = bbd62319096a1fb92f72218a28756e6fa101e87b
|
|
P3 checked-out sha = bbd62319096a1fb92f72218a28756e6fa101e87b
|
|
P3-OK checkout equals pushed commit
|
|
```
|
|
|
|
**Ruling: PASS.** The shipped workflows use the tighter form — `git init` + `git fetch --depth 1
|
|
origin $GITHUB_SHA` + `checkout FETCH_HEAD` — which is both shallow and pinned to the **exact pushed
|
|
commit** rather than the branch tip, so two racing pushes cannot make a run test the wrong tree. The
|
|
full clone in the probe took ~48 s for `felhom.eu`; the shallow fetch is materially faster.
|
|
|
|
---
|
|
|
|
## P4 — does one registration serve all four repos?
|
|
|
|
**Method.** The registration is owner-scoped (`owner_id=1`, `repo_id=0`). Pushed the real workflow to
|
|
all four repos and read which runner claimed each task.
|
|
|
|
**Measured.**
|
|
|
|
```
|
|
task repo runner_id runner
|
|
7 felhom.eu 2 felhom-gates-runner
|
|
8 felhom-controller 2 felhom-gates-runner
|
|
9 felhom-agent 2 felhom-gates-runner
|
|
10 app-catalog-felhom.eu 2 felhom-gates-runner
|
|
```
|
|
|
|
**Ruling: PASS.** One registration, four repos. No per-repo registration needed.
|
|
|
|
---
|
|
|
|
## P5 — does a failed run signal anything outside the UI? *(the probe that decided Part 4)*
|
|
|
|
**Method.** Broke a gate deliberately and pushed it **with `--no-verify`**, which is precisely the
|
|
bypass CI exists to catch. Run #3 concluded `failure`. Then looked for any outbound signal in the
|
|
ten minutes around it: Gitea pod logs filtered for mail/SMTP/notification activity, and the
|
|
`notification` table.
|
|
|
|
**Measured — nothing left the machine.**
|
|
|
|
```
|
|
gitea pod logs, mail/smtp/notif lines since the failure : (none)
|
|
notification rows created in the last 10 minutes : 0
|
|
```
|
|
|
|
Gitea's mailer is *configured and enabled* (`[mailer] ENABLED = true`, Gmail SMTP, FROM is the
|
|
operator's own address) and the sole user `admin` is active with
|
|
`email_notifications_preference = enabled` — so this is not a disabled-mailer artefact at the
|
|
config level.
|
|
|
|
**Honest limit on this measurement.** I did not independently prove that Gitea's SMTP path can
|
|
*deliver*, so "no mail" cannot be split with certainty between *"1.26.2 has no action-failure
|
|
notification"* and *"the mailer is broken"*. That distinction does not change the design: the alarm
|
|
is built on **Resend**, a different path entirely, and Scenario C proves that path end to end. It is
|
|
recorded here so nobody later reads this probe as a clean bill of health for Gitea's mailer.
|
|
|
|
**Ruling: Part 4 applies — build the alarm.** A red tick in a web UI nobody watches is exactly the
|
|
defect R-29 filed, rebuilt one layer up.
|
|
|
|
---
|
|
|
|
## P6 — does the runner need persistent state?
|
|
|
|
**Method.** Two measurements, not one, because "it survived a restart" and "the PVC is load-bearing"
|
|
are different claims.
|
|
|
|
- **P6a — restart with the PVC intact:** deleted the pod, let it come back.
|
|
- **P6b — restart with state lost:** removed `/data/.runner` (exactly what ephemeral storage would
|
|
do), deleted the pod, let it come back.
|
|
|
|
**Measured.**
|
|
|
|
| | runner list before | after | runner log |
|
|
|---|---|---|---|
|
|
| P6a | `1 felhom-gates-runner` | `1 felhom-gates-runner` | no re-registration; straight to `Starting runner daemon` |
|
|
| P6b | `1 felhom-gates-runner` | `1 …` **and** `2 …` | `Registering runner…` → `Runner registered successfully` |
|
|
|
|
**Ruling: the PVC is load-bearing.** Without persistence, every restart mints a new registration and
|
|
leaves the previous one behind as a permanently-offline record — the runner list would silently fill
|
|
with corpses. Shipped with a 5 Gi Longhorn PVC (largest repo checked out is ~125 MiB).
|
|
|
|
**The P6b casualty was cleaned up immediately**, not left for the teardown section:
|
|
`DELETE /api/v1/admin/actions/runners/1 → 204`, leaving exactly one runner.
|
|
|
|
---
|
|
|
|
## What the probes changed about the design
|
|
|
|
1. **A custom image** (P2) — stock + `python3` only, base pinned by tag.
|
|
2. **Shallow fetch of the exact SHA** rather than a clone of the branch tip (P3).
|
|
3. **A PVC** (P6), with the reason recorded in the manifest.
|
|
4. **A self-sent alarm** (P5) — the task's whole second half.
|
|
5. **A sibling clone for two of the four repos** — discovered while writing the workflows, not by a
|
|
probe. `controller_gates.py` and `agent_gates.py` invoke the shared `reuse_refs_check.py` that
|
|
lives in the `felhom.eu` clone next door and is deliberately never copied, and both repos'
|
|
`REUSE.md` files cite a path that lives in the hub. Without the sibling, CI would have failed
|
|
**closed** — correctly, but for the wrong reason. CI now reproduces the workspace's sibling
|
|
layout, and the resulting tallies match the local hook exactly (controller 126 exact / 6 suffix /
|
|
1 cross-repo; agent 88 / 1 / 1). **CI and the hook agree.**
|
|
|
|
## Two things that failed on the way, and why they are recorded
|
|
|
|
Both were caught because the step failed **loudly**; either would have shipped as a silent
|
|
non-alarm if the step had swallowed its exit code.
|
|
|
|
- **`curl: command not found`** — the first alarm used `curl`, which the deliberately minimal image
|
|
does not carry. Fixed by using `python3` + `urllib` rather than by growing the image: reaching for
|
|
a bigger base to send one HTTP request is the wrong trade, and every added tool becomes something
|
|
the next person assumes is load-bearing.
|
|
- **Cloudflare 403, error 1010** — `api.resend.com` sits behind Cloudflare, which blocks the default
|
|
`Python-urllib/3.x` User-Agent. **This failure looks exactly like an auth failure and is not one**,
|
|
which is the reason it is written down: the next person to see a 403 from Resend should check the
|
|
agent before rotating a key. Verified the fix from inside the runner image with a deliberately
|
|
invalid payload — with a User-Agent set, Resend answers `422 missing_required_field`, i.e. the
|
|
request reaches the API rather than the CDN.
|
|
|
|
## Standing limit, stated so it is not mistaken for something it is not
|
|
|
|
**CI here DETECTS. It does not BLOCK.** Every felhom repo pushes straight to `main` with no pull
|
|
request, so there is no merge for a status check to gate. This is not a gap in the runner — there is
|
|
no gate in the road. The refusing half is the local pre-push hook; this half notices when that hook
|
|
was skipped. Making CI blocking requires branch protection and a pull-request workflow, which is a
|
|
change to how the operator works and is **their** decision, not this task's.
|