felhom.eu/REPORT.md

# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed

**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**

## 1. Baselines — re-read on arrival, both matched §1

| Repo | @ arrival | Version | Result |
|---|---|---|---|
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |

## 2. Part 0 — the R-182 measurement, and it REVERSED the row

Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
at the sending end. Measured at the **receiving end**, it is the opposite.

Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
the newest `notification_log` row post-dating the session.

**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**

| time | apps refused (events in) | operator emails out |
|---|---|---|
| 06:40:03 | privatebin, opengist | **opengist only** |
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
| 08:59:59 | privatebin, opengist | **none** |
| 09:03:00 | opengist | **none** |
| 09:07:06 | privatebin, opengist | **none** |

**Cause, confirmed at source:** the operator cooldown key is
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
channel** and cannot be audited afterwards.

This is **R-97a's failure mode in a second event type**; that row's own comment states it
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.

**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
operator was told about each app* — which is false. The gap between an accepted event and a sent
email is the whole of this row.

**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.

## 3. Probes

| | Question | Method | Verdict |
|---|---|---|---|
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |

**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
(`git ls-remote --tags` → 0 at the time).

## 4. §8.2's three channels — enumerated

| Channel | Before | After | |
|---|---|---|---|
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |

**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
That is strictly better than any installer-derived tag: binary and configs now come from one ref.

**Channel 3 needed no change because the URL never carried a ref** —
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.

## 5. The tag convention

- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
  agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
  **No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
  and channel 1's ref lives only in the manifest.
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
  init container), commit, sync.
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.

## 6. Scenario A — proven by HTTP

A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
three sync periods were allowed to pass so "unchanged" means "had every chance to change":

```
website tree (main): .worktrees/6a82719…      <- ADVANCED to the new commit
scripts tree (tag):  .worktrees/bee6848…      <- STAYED
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
sha256 after  push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
marker present at the served URL? 0
https://felhom.eu/ -> HTTP 200
```

Both halves of the split in one observation: the site still tracks `main`, the installer does not.

## 7. Scenario B — publish and rollback, both directions

| act | result |
|---|---|
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |

`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
to `main`'s head — a **byte no-op**, verified by the served sha not changing.

## 8. Files, commits, tags

**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
(marker removed), plus the docs commit below.
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`

**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`

**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).

## 9. Tests and red-proofs

| Check | Result |
|---|---|
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
| `felhom-agent` `go build ./... && go vet ./...` | OK |
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
| `agent_gates.py` (full) | both OK |

**Red-proofs, each demonstrated failing then restored:**

| # | Mutation | Result |
|---|---|---|
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |

**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.

## 10. No version bumps, nothing built

`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
golden was built.

## 11. Register

| ID | Outcome |
|---|---|
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |

**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
hits anywhere and remains free.

## 12. Observations — noticed, documented, NOT acted on

1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
   either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
   closed where the mistake is made, needs no new credential).
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
   so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
   R-182 because it is what made that row take a day to get the right way round.
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
   means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
   network gate is ever added there, that workflow needs the same change the agent's just got.

## 13. Teardown

Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.