admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-167 + R-158 CLOSED, R-165 SPIKED, R-174..R-177 filed
gates / gates (push) Successful in 8s
Details

Browse Source 
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.

Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.

CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.

New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
This commit is contained in:
authentik Default Admin
2026-08-02 23:56:16 +02:00
parent 6d359a5360
commit 41dbecb264
8 changed files with 478 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+122 -230
View File
@@ -1,250 +1,142 @@
# REPORT — CI: a Gitea Actions runner, and a red run that reaches a person (R-168, 2026-08-02)
# REPORT — hub v0.89.0: the two halves of decision D-c, plus the R-165 merge spike (2026-08-02)
**Overwritten** per the standing rule. The prior contents (session 1's gate enforcement, same day)
have their durable record in `scripts/CHANGELOG.md`, `hub/CHANGELOG.md` and `CONTEXT.md` S-6/S-7.
**Overwritten** per the standing rule. The prior contents (R-168, the CI runner, same day) have their
durable record in `scripts/CHANGELOG.md` and `CONTEXT.md` S-8/S-9/S-10.
**Outcome: complete, and the half that matters is demonstrated.** All six probes answered; none
STOPped. The acceptance test was never a green pipeline — it was a **red run with a message in
hand**, and that is `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
**Companion report:** `felhom-controller/REPORT.md` holds the controller side (v0.191.0/.1/.2), the
full red-proof table, the Hungarian copy, and the live evidence for all three flows. This file covers
the hub change, the documentation coupling, and **Part 3's spike**.
---
## 1. Confirmed baselines — all matched §1 on arrival
## 1. Baseline drift — recorded, because the task's §1 was wrong
| Repo | `main` on arrival | matched? | version |
The task targeted hub **v0.87.0 → v0.88.0**. On arrival `main` was at `8ef92a3f` with hub **v0.88.0
already shipped** (R-172, the WAL fix), not `d5774d318941`/v0.87.0. Target corrected to **v0.89.0**.
Highest register ID in use was **R-173**, not R-171.
## 2. Hub change (v0.89.0)
**One new event type, not two.** The task called for a new customer-facing type *and* a new
operator-only one. Reconnaissance found `disk_warning`/`disk_critical` already allowlisted here, with
Hungarian `customerMessages`, in the controller's `DefaultEnabledEvents` and behind a UI checkbox —
**and with no producer in any repo.** The operator chose to wire that inert pair rather than mint a
near-duplicate, so only the operator type is new.
| Change | File | Why |
|---|---|---|
| `+ "recovery_unit_capture_failed"` | `internal/api/handler.go` (`allowedEventTypes`) | without it the controller's POST 400s and the event vanishes |
| `+ "recovery_unit_capture_failed"` | `internal/notify/dispatcher.go` (`operatorOnlyEvents`) | **this** is what makes it operator-only; the allowlist does not, and v0.78.0 claimed otherwise and shipped the defect |
| `- customerMessages["disk_warning"]`, `- ["disk_critical"]` | `internal/notify/templates.go` | `FormatCustomerEmail` PREFERS the entry over the message, so a static template would discard the drive label and the free-space figures the controller now sends. Same reason `offbox_enlarge_blocked` and `disk_health_degraded` have none |
| `+ func IsOperatorOnly` | `internal/notify/dispatcher.go` | lets the `api` package pin BOTH registers in ONE test; checked separately, allowlisted-but-not-operator-only is invisible. Read-only — the register stays unexported so nothing can widen it at runtime |
| `REUSE.md` §5 "new event type" rewritten | `REUSE.md` | it told readers to always add a `customerMessages` entry, which is **wrong** for operator-only types and **harmful** for dynamic-message ones |
**Tests 574 → 579**, full suite green (`go build ./... && go vet ./... && go test ./...`), all five
`repo_gates.py` gates OK.
**Red-proof (Scenario G), demonstrated not argued:** removing `recovery_unit_capture_failed` from
`operatorOnlyEvents` fails two tests, one reading *"a customer was emailed the OPERATOR-ONLY
recovery_unit_capture_failed (customer@example.com)"*. The dispatch test runs under the **breaking**
configuration — the customer has the event enabled and an email set — because that is the only
configuration in which the missing entry is visible.
**Live (guest 9201 → hub):** both event types accepted and stored; `operator | sent`; and the positive
observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from
`notification_log`. The customer half: `customer | disk_warning | sent` and `customer | disk_critical
| sent` with the dynamic Hungarian intact.
**Deploy:** GitOps only — `manifests/hub.yaml` bumped 0.88.0 → 0.89.0 (`6d359a5`), pushed, then a
deliberate ArgoCD hard-refresh + sync. Never `kubectl set image`. App `felhom` **Synced / Healthy**,
`deploy/hub` rolled out, running `gitea.dooplex.hu/admin/felhom-hub:0.89.0`, startup log clean.
## 3. Part 3 — the R-165 spike. **M1-M5 each answered; nothing was changed.**
Full document: `documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md`. No partition was created,
resized, moved or deleted; no golden rebuilt; no guest config edited. `ep0` and Peti's box were not
contacted (D-d, `runbooks/target-selection.md`).
**M1 — what is actually there. ANSWERED, and it contradicts the architecture doc.**
| | demo-felhom | demo-hp | golden default |
|---|---|---|---|
| `homelab-manifests` | `420e8197769b` | yes | n/a |
| `felhom.eu` | `af2d10388021` | yes | hub `v0.87.0` — **no bump** |
| `felhom-controller` | `eaded79b1894` | yes | `v0.188.0` — **no bump** |
| `felhom-agent` | `2c4efed5de32` | yes | `v0.119.0` — **no bump** |
| `app-catalog-felhom.eu` | `f16f29757e95` | yes | n/a |
| `mp0` `/var/lib/docker` | **200 G** (13 G used) | **50 G** (5.4 G used) | 16 G |
| `mp1` `/mnt/sys_drive` | **50 G** (2.0 G used, 5%) | **20 G** (92 M used, 1%) | 8 G |
Live Gitea re-confirmation, **all matched**: version **1.26.2**; Actions enabled on **all five**
repos; **0 runners**; **0 workflow runs ever**; **0 branch protections**; **no `.gitea/`** in any
repo. ArgoCD `gitea` app: `path: gitea-system`, auto-sync **off**, Synced at `420e819`, Healthy.
The only new artifact is the runner image, at its own `0.1.0`.
§7.5 documents the appliance as `mp0 50G / mp1 20G` — that is demo-hp exactly and **not** demo-felhom.
Any merge plan expressed as a fixed pair is already wrong for one of the two boxes that exist. §7.5's
headline bound (*"≈ 19 GB … ≈ 10 GB"*) is derived from `mp1 = 20 G` and is therefore one box's, not
the fleet's → **R-175**, filed and §7.5 annotated in this session.
## 2. All six probe results
**M2 — what lives on `mp1`. ANSWERED, and it is not only backups.** Four things would move:
Tier-1 units of driveless apps (269 M, ~30 apps on demo-felhom), **Tier-2 mirrors (1.7 G — i.e. the
MAJORITY is Tier 2, not Tier 1)**, the `userdata/import` drop zone which lives on the system drive by
**contract** (R-75), and the system-data userdata namespace. Observed fill is 5% / 1%: the constraint
is a **ceiling** problem, not a current-fill one.
Full method and measurement: `documentation/audits/SPIKE-ci-runner-2026-08-02.md`.
**M3 — which merge shapes exist. ANSWERED for three shapes, with ONE item explicitly unmeasured.**
The golden **fails closed on the split in four places**, not one (`build-golden.sh:126,130`
separate-mount asserts + `:315,319` vzdump-exclusion guards). The archive scope `rootfs+mp0+mp1` stays
complete after a merge (the data moves onto `mp0`). `mountParity` holds for new archives. **Unmeasured
and reported as such:** whether a *pre-merge* archive restore-tests into a *merged* guest — reading
`mountParity` says it should, but that is reasoning from source about an unvalidated mechanism, which
this project has got wrong four times → **R-176**.
| Probe | Ruling | Measurement |
|---|---|---|
| **P1** runner picks up a job | **PASS** | run #1 claimed by `felhom-gates-runner`, `P1-OK`, `🏁 Job succeeded` |
| **P2** stock image runs our checks | **known branch → build an image** | stock `act_runner:0.6.1` (Alpine 3.23): `git 2.52.0` PRESENT, `python3` **ABSENT**. Host mode itself fine → privileged/dind never reached for |
| **P3** source without JS actions | **PASS** | `pushed sha == checked-out sha` (`bbd62319…`), plain `git clone`/`fetch` from the in-cluster Service, no `uses:` |
| **P4** one registration, four repos | **PASS** | tasks 7,8,9,10 across four repos, all `runner_id=2 felhom-gates-runner` |
| **P5** does a failure signal anything | **NOTHING SENT → build the alarm** | 0 mail/SMTP/notification log lines, **0** notification rows, after a real `failure` run |
| **P6** persistent state needed | **PVC is load-bearing** | PVC intact → same registration; `/data/.runner` removed → **second record** appears and the first is orphaned |
**M4 — the bulkhead. ANSWERED, and it is the important one.** `mp1` is not only a ceiling: today an
overflow is refused per app with the last good unit byte-identical **and cannot reach
`/var/lib/docker`**. After the merge it can, and a full Docker data-root is a stopped box, not a slow
one. Four replacements costed — a reserved block percentage, **a refusal threshold in the capture
path**, a project quota, or deeming R-167's warnings sufficient — with the trade-off of each.
**Deliberately not chosen: this is the operator's ruling.**
**P5's honest limit, stated rather than glossed:** I did not independently prove Gitea's SMTP path
can *deliver*, so "no mail" cannot be split with certainty between *no action-failure notification in
1.26.2* and *a broken mailer*. It does not change the design — the alarm rides Resend, a different
path — and Scenario C proves that path end to end.
**M5 — existing boxes. ANSWERED for the measurable population; one part honestly UNMEASURED.** The
hub's `/hosts` register holds four hosts, **two ONLINE**, both demo boxes — and both are **Tier 0,
therefore reinstallable rather than migratable (D-d)**, so migration cost for the measurable population
is **zero**. **D-a's condition (1) — "before any external install" — is currently SATISFIED**, which
makes this the cheapest this decision will ever be. **`peti-felhom` exists as a customer with NO host
in the register**, so its layout is not knowable from the hub and the box was not contacted; whether it
needs converting or reinstalling is the operator's information. The in-place migration procedure has
**never been rehearsed**, so "is the box restorable at every point of it?" is currently unknown → also
**R-176**.
## 3. Files created / modified
**Ranked options and recommendation:** (1) **S1 — one volume with the two paths as directories — plus
B2, a refusal threshold in the capture path**, shipped as a fresh-install shape with the demo boxes
reinstalled; (2) S1 + warnings only; (3) S3, grow `mp1` and keep the split (D-a's rejected baseline,
measured for comparison); (4) S2, two mounts on one pool — **not recommended at all**, it satisfies
every assertion while delivering none of the benefit and converts a clean per-app refusal into a
shared-pool exhaustion neither `df` can see coming.
**`homelab-manifests`** — `gitea-system/act-runner.yaml` (NEW: PVC + Deployment),
`gitea-system/act-runner/Dockerfile` (NEW), `CHANGELOG.md`, `REPORT.md`.
**STOPPED at the operator's question**, per the task. The merge is next session's supervised work.
**`felhom.eu`** — `.gitea/workflows/gates.yml` (NEW),
`documentation/audits/SPIKE-ci-runner-2026-08-02.md` (NEW), `CLAUDE.md`, `CONTEXT.md` (S-8…S-11),
`scripts/CHANGELOG.md`, `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
`STATUS.md`, `REPORT.md`. *(`hub/internal/web/configs.go` was broken and restored within the session
for Scenario B; it is byte-identical to its starting state — verified by the gate that objects to it.)*
## 4. Documentation coupling
**`felhom-controller`**, **`felhom-agent`**, **`app-catalog-felhom.eu`** —
`.gitea/workflows/gates.yml` (NEW), `CHANGELOG.md`, `REPORT.md`.
## 4. Commits pushed to `main`
| Repo | Hash | What |
|---|---|---|
| `homelab-manifests` | `c51b77c` | the runner manifest + Dockerfile |
| `felhom.eu` | `bbd6231` | the temporary Part 0 probe workflow |
| | `666a34d` | real `gates.yml`, probe workflow **deleted** |
| | `3252d51` | **Scenario B** deliberate breakage (`--no-verify`) |
| | `dd13f63` | alarm step v1 — died on `curl: command not found` |
| | `f7dbc33` | alarm rewritten in python3/urllib |
| | `9530de7` | alarm User-Agent (Cloudflare 1010) — **the run that produced the accepted-id** |
| | `4cc1238` | **breakage reverted**, main green |
| `felhom-controller` | `dcc400e` | `gates.yml` (with sibling clone) |
| `felhom-agent` | `eb99144` | `gates.yml` (with sibling clone) |
| `app-catalog-felhom.eu` | `aa57588` | `gates.yml` (`--fast`, no sibling needed) |
Plus the documentation commits listed at the end of this session.
## 5. Scenario results A-G
| # | Result | Positive observable |
|---|---|---|
| **A** | **PASS** | run #2 (`666a34d`) SUCCESS. **Every member gate's own stdout in the CI log**, not just the summary: `site gates OK — BOM, emoji=0…`, `hostinstall gates: ALL PASS`, `hub confirm gate OK…`, `manifest bearer gate OK…`, `OK [hostexecutor]: 61 cited paths — exact 60, suffix 1…`, then `all felhom.eu gates OK` |
| **B** | **PASS** | run #3 (`3252d51`) **FAILURE**. The failing gate's own text: `FAIL: hub/internal/web/configs.go:34 carries const hostInstallVersion…`, `hostinstall gates: 3 FAILURE(S)`, `CONVICTED: hostinstall`, `🏁 Job failed` |
| **C** | **PASS** | run #6 FAILURE → **`RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`**. See §6 |
| **D** | **PASS** | four repos, four runs, **one runner**: tasks 7/8/9/10 all `runner_id=2 felhom-gates-runner`; conclusions `success` ×4 |
| **E** | **PASS** | live pod spec (§8) **and** the refusal in a job log: `/root/.cache/act/…/3.sh: line 2: docker: command not found` → `E-OK docker refused` |
| **F** | **PASS** | catalogue run: `image-pin gate OK — 53 templates, 0 unpinned images`; `--fast SKIPPED: image-resolvable, volume-persistence…`; resolvability/volume-persistence/canary output lines **0**, image-pull lines **0** |
| **G** | **PASS** | P6: PVC intact → runner list unchanged (`1 felhom-gates-runner`); state removed → **`1` and `2`**, i.e. a second registration. Orphan deleted, `204` |
**Scenario A note worth keeping:** the controller and agent runs prove more than "it ran" — their
reuse tallies in CI (**126 exact / 6 suffix / 1 cross-repo**; **88 / 1 / 1**) are *identical* to the
local pre-push hook's. **CI and the hook agree.** Per §12 a disagreement would have outranked this
whole task; there is none.
## 6. Scenario C evidence in full
The run was genuinely red for a genuinely broken gate, and the message is the observable — not a
tick in a UI:
```
CONVICTED: hostinstall
RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab
🏁 Job failed
```
The Resend id is the **provider's accepted-id**, returned by `api.resend.com` for a message
`monitoring@felhom.eu → admin@felhom.eu`, subject `[felhom CI] gates FAILED in admin/felhom.eu`,
carrying the repo, the commit, a link to the run, and the line that matters most: *if the local
pre-push hook was green for this commit, CI and the hook disagree, and that outranks whatever the
push was for.*
**The key never appeared in the log** — Gitea masked it (`RESEND_API_KEY: ***`) and a grep for the
key prefix across the run log returns **0 lines**.
## 7. Runner deployment
- **GitOps only.** `act-runner.yaml` committed to `homelab-manifests` `main` (`c51b77c`), pushed,
then a **deliberate ArgoCD sync** of the `gitea` Application. No `kubectl apply`, no
`kubectl set image` at any point.
- **ArgoCD:** `sync=Synced health=Healthy`, revision `c51b77c68f10…`.
- **Pod:** `act-runner-…` `1/1 Running`; PVC `act-runner-data` **Bound**, 5Gi, `longhorn`.
- **Running image:** `gitea.dooplex.hu/admin/felhom-act-runner:0.1.0`, verified by deleting the local
copy and **re-pulling from the registry** (`Python 3.12.13`, `git version 2.52.0`) rather than
trusting the push's own output.
- **Online in Gitea's runner list**, with its label:
```
id=2 name=felhom-gates-runner status=online labels=[felhom-gates]
```
Owner-scoped (`owner_id=1`, `repo_id=0`) — which is why one registration serves four repos.
## 8. Security posture — shown, not asserted
From the **live** pod spec:
```
privileged : False
allowPrivilegeEscalation : False
capabilities.drop : ['ALL']
automountServiceAccountToken : False
hostNetwork/hostPID/hostIPC : None None None
nodeSelector : None
volumes : [('data', ['persistentVolumeClaim'])]
hostPath volumes : NONE
docker socket mounts : NONE
serviceaccount token mounts : NONE
resources : requests 50m/128Mi, limits 500m/512Mi
```
And the negative proven rather than inferred — from a job log:
```
/root/.cache/act/187d434f3fdbb3b4/act/workflow/3.sh: line 2: docker: command not found
E-OK docker refused (output above is the refusal)
```
Resources are **half of Gitea's limits** (Gitea: 1 CPU / 1Gi) and a fifth of its requests,
deliberately: Gitea shares this node and holds every repository. The heaviest entry point was
measured at **11.9 MiB RSS, 0.98 s wall**.
## 9. Teardown — all three layers
1. **Workload** — nothing to remove: the runner is the deliverable and stays. No scratch Deployment
or PVC was created beyond it.
2. **Host** — no scratch Longhorn volume was created beyond the runner's own 5Gi PVC, which is
intended and in git. Local Docker images (`gitea/act_runner:0.6.1` base and the built
`felhom-act-runner:0.1.0`) remain in DooPlex's local Docker store; they are ordinary build
artifacts and **no global prune was run** (forbidden on this host).
3. **The Gitea-side registration records — the layer this project has missed four times.**
- The **P6b orphan** (`id=1`) was deleted **immediately when it was created**, not left for the
end: `DELETE /api/v1/admin/actions/runners/1 → 204`.
- **Final runner list: exactly one, the intended live runner** —
`id=2 felhom-gates-runner status=online labels=[felhom-gates]`, and the live pod's own
`/data/.runner` carries `id=2`, so the record and the process agree.
**A trap worth recording, because it briefly looked like a failed teardown:** a raw
`select id,name from action_runner` returns **two** rows. Gitea **soft-deletes** runners — the
deleted row is tombstoned (`deleted` set), not removed, and the admin API correctly reports one.
The query without the tombstone filter was the misleading instrument, not the teardown. Same
class as the `| tail -5` census bug and the `go test -run` filter: **a query that does not model
the deletion semantics is not a check.**
- The **temporary probe workflow** `.gitea/workflows/probe.yml` was deleted from `main` in
`666a34d`, the same commit that added the real one.
- The **admin API access token** minted for this session (used to read run conclusions and to
delete the orphan runner) is **revoked**. The clean route — `DELETE /api/v1/users/admin/tokens/
{id}` — needs HTTP **Basic** auth with the operator's Gitea password, not the token itself;
tried `GET /api/v1/users/admin/tokens` with the `PASSWORD` entry from `~/.config/credentials`
and got **401**, so that is not the Gitea admin password. Revoked instead by deleting the single
`access_token` row I created, matched on its `cc-ci-session-*` name: **1 row before, 0 after**,
6 unrelated tokens untouched. The scratch file holding it was removed. It appears in no
committed file, log or report.
## 10. `OPEN-ITEMS.md` rows opened / closed / re-ranked
- **R-168 → SHIPPED**, with the evidence in the row: the runner, the one-registration-four-repos
measurement, the two probes that changed the design (P2, P6), and the demonstrated alarm with its
accepted-id.
- **R-29 → CLOSED**, both halves — and **closed on the demonstrated alarm, not on a green run**, as
the task required. The class it opened (*a gate that exists, is red, and is invoked by nothing*) is
answered at both ends.
- **R-161 → updated**: its automatic half now exists for the static gate. Its *original* scope, the
runtime volume-persistence gate, is deliberately still **not** automatic and should stay that way.
- **R-169 → NEW.** `grep -oE "R-[0-9]+" | sort -u -t- -k2 -n | tail` established **R-168 as the
highest in use**, so R-169 was free. *CI can only report, because there is no gate in the road* —
making it blocking needs branch protection plus a PR workflow, both of which change how the
operator works. **WAITING-ON-OPERATOR**, owner: operator. The row states the cost honestly (for a
single-operator project a PR workflow may be worse than the disease) and names the real gap: the
window between a `--no-verify` push landing and the operator reading the alarm.
- `ROADMAP.md`: R-168 collapsed to its one-liner, R-29 closed, R-169 added.
- **`architecture/00-capability-map.md` — N/A, stated rather than silently skipped.** No
customer-visible or platform capability changed; this is developer tooling.
## 11. Was `--no-verify` used?
**Yes — three times, all on `felhom.eu`, all deliberate and all disclosed.**
| Commit | Why |
| File | Change |
|---|---|
| `3252d51` | **Scenario B itself.** The commit had to carry a broken gate, and pushing it with `--no-verify` *is* the bypass CI exists to catch. The local hook would have refused it — that is the point. |
| `dd13f63` | The tree was still deliberately broken; pushing the alarm step required bypassing the hook that was correctly objecting. |
| `f7dbc33`, `9530de7` | Same, iterating on the alarm while the tree was still red on purpose. |
| `documentation/backlog/OPEN-ITEMS.md` | **R-158** closed (by R-167 — *no second row for the same wire*); **R-167** closed; **R-165** updated with M1-M5 + the operator question, stays open; **4 new rows** R-174/175/176/177 |
| `documentation/backlog/ROADMAP.md` | R-158 collapsed to a shipped one-liner; R-167 added as shipped; R-165 added as spiked/waiting-on-operator |
| `documentation/architecture/00-capability-map.md` | **two new rows**, both **PROVEN-LIVE** with live citations |
| `documentation/architecture/07-backup-architecture.md` §7.5 | **S-1: the contract changed in the same session.** The section's closing claim *"nothing warns when an app crosses the line"* is now false; the alerting is written in, and the one-box-vs-fleet caveat added |
| `CONTEXT.md` | **S-11** (D-c's routing, and why R-158's own `backup_failed` proposal was overruled) and **S-12** (the monitoring landed *before* the merge, not with it) |
| `STATUS.md` | new plain-language section; the merge decision added to *Waiting on you*; **two older entries trimmed** so the page did not grow — one screen, per its own rule |
| `REUSE.md` | the "new event type" extension point rewritten (see §2) |
Every other push in this session ran its pre-push hook and passed. The breakage was reverted in
`4cc1238`, and the hook passed on that push.
## 5. Register IDs
## 12. Observations — noticed, recorded, NOT acted on
**Opened:** R-174, R-175, R-176, R-177. Each established free by
`grep -ro "R-17n\b" documentation/ *.md` → **0 hits**, run before minting.
**Closed:** R-158, R-167, R-174. **Updated, still open:** R-165, R-163 (unchanged — it stays the
record of the constraint until the merge lands).
1. **CI and the pre-push hook never disagreed.** §12 said a disagreement would outrank the CI work;
there was none. The controller's and agent's reuse tallies in CI match the local run exactly. The
one place they *would* have disagreed was designed out rather than discovered: without a sibling
`felhom.eu` clone, both entry points fail closed with `gate is MISSING`.
2. **A near-miss that outranks a probe result.** My first census query reported Actions enabled on
five *unrelated* repos — a baseline drift big enough to change the task. It was a `| tail -5`
inside my own helper truncating the output. The measurement was never wrong; the instrument was.
This is the same class as the `go test -run` filter from session 1, and it is now a `CLAUDE.md`
rule: **an instrument that can silently drop results is not a measurement.**
3. **Two failures on the way to the alarm, both worth their write-up** — `curl: command not found`
(the image is minimal on purpose; fixed with `urllib`, not a bigger image) and **Cloudflare 403
error 1010** on `api.resend.com`, which blocks the default `Python-urllib` User-Agent. The second
matters because **it looks exactly like an auth failure**: the next person to see a 403 from
Resend should check the User-Agent before rotating a key. Both are in the spike doc and in
`CONTEXT.md` S-9.
4. **A full clone is slow enough to notice** — the P3 probe's `git clone` of `felhom.eu` took ~48 s.
The shipped workflows use `git fetch --depth 1 <sha>` instead, which is both faster and pinned to
the exact pushed commit. Not a finding, but the reason the shipped form differs from the probe's.
5. **Not acted on, out of scope:** Gitea's own mailer is configured and enabled but was never proven
to deliver (§2). If it *is* broken, nothing else in Gitea that relies on email works either —
worth a five-minute test email from the admin UI at some point. It does not affect the CI alarm,
which does not use it.
6. **Not acted on, out of scope:** `homelab-manifests` has no gate entry point and no pre-push hook,
unlike the four product repos, and now no CI workflow either. It is the repo that deploys the
cluster, so that asymmetry may deserve a look — but adding one was not in this task, and the repo
has no gates to run yet.
## 6. CI — run ids and conclusions
| Repo | Final commit | Run | Conclusion |
|---|---|---|---|
| `felhom-controller` | `9a3c485` | see below | see below |
| `felhom.eu` | (docs commit) | see below | see below |
*(Filled in from `…/actions/tasks?limit=3` after the final push — see the closing section of the
session output; a green CI that was not looked at is an assumption, not an observation.)*
## 7. `--no-verify`
**Not used anywhere.** Every push in this session ran `.githooks/pre-push` (`repo_gates.py --fast` /
`controller_gates.py --fast`) and passed.