felhom.eu/REPORT.md

REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed

Date: 2026-08-03 · Repos: felhom.eu (installer v1.22.0 → v1.23.0), felhom-agent (no bump) Nothing was built — no image, no binary, no golden. Hub stays v0.89.0.

1. Baselines — re-read on arrival, both matched §1

Repo	@ arrival	Version	Result
felhom.eu	8360f940bfb2	hub v0.89.0, SCRIPT_VERSION="1.22.0", 0 tags (confirmed)	installer v1.23.0, first tag installer-v1.23.0
felhom-agent	9dfd89cb947e	v0.120.0	unchanged — scripts and gates only

2. Part 0 — the R-182 measurement, and it REVERSED the row

Filed yesterday as "the reserve re-alerts on every status refresh" — too many alerts, observed at the sending end. Measured at the receiving end, it is the opposite.

Method: the hub's SQLite copied with its -wal (4 MB and newer than the db — copying hub.db alone would have read stale data, the exact trap this project recorded before), freshness confirmed by the newest notification_log row post-dating the session.

9 recovery_unit_capture_failed events received today → 2 operator emails sent.

time	apps refused (events in)	operator emails out
06:40:03	privatebin, opengist	opengist only
08:59:46/47	opengist, privatebin	privatebin only
08:59:59	privatebin, opengist	none
09:03:00	opengist	none
09:07:06	privatebin, opengist	none

Cause, confirmed at source: the operator cooldown key is customerID + ":" + eventType + cooldownTierSuffix(details) (dispatcher.go:268, 1 hour hardcoded). RecoveryUnitFailureDetails carries app and no tier, so the suffix is empty and the key holds no app identifier. The first refused app takes the slot; every other app's refusal for the next hour is dropped — and dropped before LogNotification, so it leaves no row on any channel and cannot be audited afterwards.

This is R-97a's failure mode in a second event type; that row's own comment states it ("felhom-pbs failing at 09:00 would swallow local failing at 09:20"). cooldownTierSuffix was written narrow on purpose; recovery_unit_capture_failed simply never opted in.

A correction I owe on yesterday's report. It said "one recovery_unit_capture_failed per app, HTTP 200". That was true of what the controller pushed, and a reader would take it as the operator was told about each app — which is false. The gap between an accepted event and a sent email is the whole of this row.

Nothing was changed (§8.5). R-182 is re-scoped with the evidence and the fix shape.

3. Probes

	Question	Method	Verdict
P-A	does git-sync v4.4.0 follow a tag, and notice a moved one?	throwaway docker run git-sync against this repo, tag moved under it	PASS both halves — update required … local:fb65202 remote:8360f94 → updated successfully, one period (~20 s)
P-B	does Gitea serve raw/tag/<tag>/<path>?	one fetch on a throwaway tag	PASS — HTTP 200, byte-identical to raw/branch/main
P-C	can CI read the package registry?	anonymous fetches	PARTIAL, and it changed the gate's design — package download 200 (and 404 for a fake version, so it discriminates), tags api 200; package listing api 401, hub artifact manifest 401

Publish model P-A implies: publishing is moving the tag; rollback is moving it back, in ~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone (git ls-remote --tags → 0 at the time).

4. §8.2's three channels — enumerated

Channel	Before	After
1. the served script	main, 30 s	installer-v1.23.0	MOVED — webpage.yaml split into two syncs
2. the run-time fetches	raw/branch/main	raw/tag/v$ART_AGENT_VER	MOVED — but see below
3. the URL producers	main	unchanged	NO CHANGE NEEDED — and that is a finding, not an omission

Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable. There are sixteen fetches, not nine, and they come from felhom-agent, not felhom.eu — so no tag on this repo could ever have covered them, and §8.1's "derive the tag from SCRIPT_VERSION" was impossible for them. Raised before building; operator ruled to pin them to the agent version being installed, which the installer already resolves from the hub manifest and already sha-verifies. That is strictly better than any installer-derived tag: binary and configs now come from one ref.

Channel 3 needed no change because the URL never carried a ref — https://felhom.eu/scripts/felhom-host-install.sh is path-based; the ref lives in the manifest. So felhom-bootstrap.sh and the hub's day-0 command follow the tag automatically. No hub template change ⇒ no hub bump, so §1's rule was never in tension and the STOP it anticipated never arose.

5. The tag convention

• Shape: installer-v<SCRIPT_VERSION> in felhom.eu (prefixed so it cannot be read as a hub, agent, controller or golden version); v<semver> in felhom-agent (that repo versions one thing). No new constant in the installer — channel 2 derives its ref from $ART_AGENT_VER at run time, and channel 1's ref lives only in the manifest.
• Publish: cut installer-v<new SCRIPT_VERSION>, bump the --ref in webpage.yaml (sidecar and init container), commit, sync.
• Roll back: move the tag back to the previous commit — takes ~30 s, no ArgoCD sync, no deploy.

6. Scenario A — proven by HTTP

A real commit was pushed to main (a marker comment in the installer) without moving the tag, and three sync periods were allowed to pass so "unchanged" means "had every chance to change":

website tree (main): .worktrees/6a82719…      <- ADVANCED to the new commit
scripts tree (tag):  .worktrees/bee6848…      <- STAYED
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
sha256 after  push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
marker present at the served URL? 0
https://felhom.eu/ -> HTTP 200

Both halves of the split in one observation: the site still tracks main, the installer does not.

7. Scenario B — publish and rollback, both directions

act	result
tag moved bee6848 → 6a82719	scripts tree moved in ~40 s; served sha256 ea2b4aa9…; marker present
tag moved back → bee6848	scripts tree back in ~40 s; served sha256 2f859555… — exactly the pre-publish sha; marker gone

https://felhom.eu/ returned 200 throughout. The marker commit was then reverted, and the tag moved to main's head — a byte no-op, verified by the served sha not changing.

8. Files, commits, tags

felhom.eu — bee6848 (installer + gate + manifest), 6a82719 (Scenario A marker), e79a20b (marker removed), plus the docs commit below. scripts/felhom-host-install.sh · scripts/hostinstall_gates.py · scripts/CHANGELOG.md · manifests/webpage.yaml · CLAUDE.md · CONTEXT.md · STATUS.md · REPORT.md · documentation/backlog/{OPEN-ITEMS,ROADMAP}.md · documentation/architecture/00-capability-map.md

felhom-agent — dd2d1fe (release path + gate + CI), 0db7766 (REPORT). scripts/release-agent.sh (new) · scripts/check-published-versions.py (new) · scripts/agent_gates.py · .gitea/workflows/gates.yml · CLAUDE.md · CHANGELOG.md · REPORT.md

Tags created: felhom.eu/installer-v1.23.0 (the first tag this repo has ever had) and felhom-agent/v0.120.0 (retroactive, at cd6e267 — the commit the published binary was built from; configs/ is byte-identical there and at main, so nothing depended on the choice).

9. Tests and red-proofs

Check	Result
felhom.eu repo_gates.py --fast	all 5 gates OK
felhom-agent go build ./... && go vet ./...	OK
felhom-agent go test ./...	29 packages ok, rc=0 (read separately from any commit)
agent_gates.py --fast	published correctly SKIPPED (hook must not fail on a network blip)
agent_gates.py (full)	both OK

Red-proofs, each demonstrated failing then restored:

#	Mutation	Result
C	one of the sixteen fetches reverted to raw/branch/main	RED — gate 6a and 6b both fired
D	assertions 6a and 6b removed (every guard the test covers), same bad installer	zero mentions of the regression — the guards are what catch it
6c	the manifest before the split	RED on its own, before I fixed it — the gate was demonstrated red by the real pre-change state
F	v9.9.9 tagged and not published	RED, binary NOT downloadable (HTTP 404 …), rc=1
F′	the gate deregistered from agent_gates.py, same bad state	rc=0, "all agent gates OK" — restored → CONVICTED: published, rc=1

Scenario F measured on real CI, not inferred. Runs 69 and 70 are on the same commit 0db7766: success before v9.9.9 existed, failure after pushing it. One variable. This also retrospectively explains runs 67/68. One deliberate CI failure email reached the operator — that was this proof, not an incident. I could not read CI's own step log: the jobs endpoint needs a Gitea API token, and the only credential available (~/.docker/config.json) is a registry password that the API rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.

10. No version bumps, nothing built

felhom-agent v0.120.0 unchanged (no Go code changed). Hub v0.89.0 unchanged (no hub file touched). The installer's SCRIPT_VERSION did go 1.22.0 → 1.23.0 — the installer is not in §12's no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or golden was built.

11. Register

ID	Outcome
R-110	CLOSED — SHIPPED (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed
R-115	CLOSED — SHIPPED (release-agent.sh + check-published-versions.py, no bump)
R-182	RE-SCOPED — the direction reversed by Part 0's measurement; still open, now correctly described
R-183	NEW, and CLOSED the same session — binary and configs came from two different refs
R-184	NEW, open — nothing stops the hub vouching a version that was never released

IDs established free: ^| \*\*R-183\*\* / ^| \*\*R-184\*\* in OPEN-ITEMS.md → 0 rows each; all other hits are this session's own code and changelogs (forward references I wrote). R-185 → 0 hits anywhere and remains free.

12. Observations — noticed, documented, NOT acted on

1. The gate cannot see what is vouched — filed as R-184 rather than papered over. Closing it needs either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails closed where the mistake is made, needs no new credential).
2. A suppressed operator alert leaves no row at all. The cooldown returns before LogNotification, so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside R-182 because it is what made that row take a day to get the right way round.
3. on: [push] fires CI for tag pushes too. Useful (it is how Scenario F was measured), but it means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
4. felhom.eu CI still runs --fast. Correct today, since all its gates are network-free; if a network gate is ever added there, that workflow needs the same change the agent's just got.

13. Teardown

Probe container, probe sync tree and probe tag (probe-r110-delete-me) removed; the red-proof tag v9.9.9 deleted (git ls-remote --tags → only v0.120.0); the Scenario A marker reverted from main and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.