admin
c718aad1bc
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state re-registers and orphans the old record) changed the design; P5 (a failed run signals NOTHING) is why the alarm exists at all. R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green run, as required: the class it opened is answered at both ends, the hook refusing locally and CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for the STATIC gate, while its original scope, the runtime gate, is deliberately still not automatic and should stay that way. NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both of which change how the operator works — so it is theirs to decide, and the row states the cost honestly rather than recommending it. CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout). CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test prints ok and exits 0, and an instrument that can silently drop results is not a measurement.
10 KiB
SPIKE — a Gitea Actions runner for the felhom gate entry points (R-168, 2026-08-02)
Verdict: the mechanism works and is SHIPPED. All six probes answered; none produced a STOP. The runner is unprivileged, host-mode, one registration for all four repos, and a failed run now sends its own alarm — measured, not assumed.
Context. Session 1 (same day) gave every felhom repo one gate entry point and a
.githooks/pre-push that runs it and refuses a failing push. That hook is per-clone and
git push --no-verify skips it, so nothing independent of the person pushing ever saw whether the
gates passed. This spike built the independent half.
Arrival state, re-confirmed live (all matched the anchor)
| Fact | Expected | Measured |
|---|---|---|
| Gitea version | 1.26.2 | 1.26.2 |
| Actions enabled | all five repos | all five (repo_unit type 10 present on each) |
| Runners registered | 0 | 0 (action_runner empty) |
| Workflow runs, ever | 0 | 0 (action_run empty) |
| Branch protections | 0 | 0 (protected_branch empty) |
.gitea/ directory |
none | none in any of the five |
ArgoCD gitea app |
path: gitea-system, auto-sync off |
confirmed, Synced at 420e819, Healthy |
Repo visibility, which decided P3's design: the four product repos are public;
homelab-manifests is private.
A near-miss worth recording. My first census query reported Actions enabled on only five unrelated repos, which looked like a baseline drift big enough to change the task. It was a
| tail -5inside my own query helper truncating the result. The measurement was never wrong; the instrument was. Re-run without the pipe, all five target repos had the Actions unit. A tool that silently drops rows is indistinguishable from a finding — the same class as thego test -runfilter that matches nothing and printsok.
P1 — does a registered runner pick up a job at all?
Method. Registered one runner at owner scope for admin (repo_id=0), label felhom-gates,
via a token minted with gitea actions generate-runner-token --scope admin and stored out-of-band as
Secret/act-runner-registration. Pushed a temporary .gitea/workflows/probe.yml with one
run: echo step to felhom.eu.
Measured. Run #1 appeared against the pushed commit, was claimed by felhom-gates-runner, and
finished success:
felhom-gates-runner(version:v0.6.1) received task 1 of job probe, be triggered by event: push
P1-OK runner picked up the job
🏁 Job succeeded
Ruling: PASS. No STOP.
P2 — can the stock image run our checks without a container runtime?
Method. Inspected gitea/act_runner:0.6.1 directly, then re-measured inside a job — because
what matters is what the job sees, not what the image contains, and in host mode those are the
same thing only if host mode is really in effect.
Measured. Stock image (Alpine Linux v3.23):
python3: sh: python3: not found ABSENT
git: git version 2.52.0 PRESENT
In-job, after building the minimal image:
P2 python3: Python 3.12.13
P2 git: git version 2.52.0
Ruling: the known branch, not a failure. Host mode works; only python3 was missing. Built
gitea.dooplex.hu/admin/felhom-act-runner:0.1.0 = stock (pinned) + python3, nothing else, from
homelab-manifests/gitea-system/act-runner/Dockerfile. Verified by deleting the local copy and
re-pulling from the registry rather than trusting the push's own output.
Host mode was never in doubt, so the privileged/dind pattern was never reached for (§5/§12 of the
task, and the reason is written into act-runner.yaml: DooPlex is Tier 2 and is the recovery
chain).
P3 — can a workflow obtain the source without JavaScript actions?
Method. No uses: anywhere. A plain run: step clones from the in-cluster Gitea Service
(http://gitea.gitea-system.svc.cluster.local:3000) — no ingress, no TLS hop, no geo rule — and
checks out $GITHUB_SHA. The four product repos are public, so no credential is needed at all.
Measured.
P3 pushed sha = bbd62319096a1fb92f72218a28756e6fa101e87b
P3 checked-out sha = bbd62319096a1fb92f72218a28756e6fa101e87b
P3-OK checkout equals pushed commit
Ruling: PASS. The shipped workflows use the tighter form — git init + git fetch --depth 1 origin $GITHUB_SHA + checkout FETCH_HEAD — which is both shallow and pinned to the exact pushed
commit rather than the branch tip, so two racing pushes cannot make a run test the wrong tree. The
full clone in the probe took ~48 s for felhom.eu; the shallow fetch is materially faster.
P4 — does one registration serve all four repos?
Method. The registration is owner-scoped (owner_id=1, repo_id=0). Pushed the real workflow to
all four repos and read which runner claimed each task.
Measured.
task repo runner_id runner
7 felhom.eu 2 felhom-gates-runner
8 felhom-controller 2 felhom-gates-runner
9 felhom-agent 2 felhom-gates-runner
10 app-catalog-felhom.eu 2 felhom-gates-runner
Ruling: PASS. One registration, four repos. No per-repo registration needed.
P5 — does a failed run signal anything outside the UI? (the probe that decided Part 4)
Method. Broke a gate deliberately and pushed it with --no-verify, which is precisely the
bypass CI exists to catch. Run #3 concluded failure. Then looked for any outbound signal in the
ten minutes around it: Gitea pod logs filtered for mail/SMTP/notification activity, and the
notification table.
Measured — nothing left the machine.
gitea pod logs, mail/smtp/notif lines since the failure : (none)
notification rows created in the last 10 minutes : 0
Gitea's mailer is configured and enabled ([mailer] ENABLED = true, Gmail SMTP, FROM is the
operator's own address) and the sole user admin is active with
email_notifications_preference = enabled — so this is not a disabled-mailer artefact at the
config level.
Honest limit on this measurement. I did not independently prove that Gitea's SMTP path can deliver, so "no mail" cannot be split with certainty between "1.26.2 has no action-failure notification" and "the mailer is broken". That distinction does not change the design: the alarm is built on Resend, a different path entirely, and Scenario C proves that path end to end. It is recorded here so nobody later reads this probe as a clean bill of health for Gitea's mailer.
Ruling: Part 4 applies — build the alarm. A red tick in a web UI nobody watches is exactly the defect R-29 filed, rebuilt one layer up.
P6 — does the runner need persistent state?
Method. Two measurements, not one, because "it survived a restart" and "the PVC is load-bearing" are different claims.
- P6a — restart with the PVC intact: deleted the pod, let it come back.
- P6b — restart with state lost: removed
/data/.runner(exactly what ephemeral storage would do), deleted the pod, let it come back.
Measured.
| runner list before | after | runner log | |
|---|---|---|---|
| P6a | 1 felhom-gates-runner |
1 felhom-gates-runner |
no re-registration; straight to Starting runner daemon |
| P6b | 1 felhom-gates-runner |
1 … and 2 … |
Registering runner… → Runner registered successfully |
Ruling: the PVC is load-bearing. Without persistence, every restart mints a new registration and leaves the previous one behind as a permanently-offline record — the runner list would silently fill with corpses. Shipped with a 5 Gi Longhorn PVC (largest repo checked out is ~125 MiB).
The P6b casualty was cleaned up immediately, not left for the teardown section:
DELETE /api/v1/admin/actions/runners/1 → 204, leaving exactly one runner.
What the probes changed about the design
- A custom image (P2) — stock +
python3only, base pinned by tag. - Shallow fetch of the exact SHA rather than a clone of the branch tip (P3).
- A PVC (P6), with the reason recorded in the manifest.
- A self-sent alarm (P5) — the task's whole second half.
- A sibling clone for two of the four repos — discovered while writing the workflows, not by a
probe.
controller_gates.pyandagent_gates.pyinvoke the sharedreuse_refs_check.pythat lives in thefelhom.euclone next door and is deliberately never copied, and both repos'REUSE.mdfiles cite a path that lives in the hub. Without the sibling, CI would have failed closed — correctly, but for the wrong reason. CI now reproduces the workspace's sibling layout, and the resulting tallies match the local hook exactly (controller 126 exact / 6 suffix / 1 cross-repo; agent 88 / 1 / 1). CI and the hook agree.
Two things that failed on the way, and why they are recorded
Both were caught because the step failed loudly; either would have shipped as a silent non-alarm if the step had swallowed its exit code.
curl: command not found— the first alarm usedcurl, which the deliberately minimal image does not carry. Fixed by usingpython3+urllibrather than by growing the image: reaching for a bigger base to send one HTTP request is the wrong trade, and every added tool becomes something the next person assumes is load-bearing.- Cloudflare 403, error 1010 —
api.resend.comsits behind Cloudflare, which blocks the defaultPython-urllib/3.xUser-Agent. This failure looks exactly like an auth failure and is not one, which is the reason it is written down: the next person to see a 403 from Resend should check the agent before rotating a key. Verified the fix from inside the runner image with a deliberately invalid payload — with a User-Agent set, Resend answers422 missing_required_field, i.e. the request reaches the API rather than the CDN.
Standing limit, stated so it is not mistaken for something it is not
CI here DETECTS. It does not BLOCK. Every felhom repo pushes straight to main with no pull
request, so there is no merge for a status check to gate. This is not a gap in the runner — there is
no gate in the road. The refusing half is the local pre-push hook; this half notices when that hook
was skipped. Making CI blocking requires branch protection and a pull-request workflow, which is a
change to how the operator works and is their decision, not this task's.