Found by reading the first REAL digest, not by design. Every app row ended with
the same usage clause the mail already prints once on its own Filesystem line.
On a two-app box that is untidy; down a list of a dozen it is the same forty
characters twelve times, pushing the part that DIFFERS off a phone screen at
07:00 — the only moment this mail has to work.
The reserve's refusal message is authored for a single-app alert where naming
the filesystem is right, so the message is unchanged; the digest trims the
duplicate when rendering. trimRepeatedUsage removes ONLY an exact
"— <target path>:" suffix, so an unrelated reason is untouched and a reason that
is nothing but the usage clause is left alone rather than emptied.
Also updates TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer, which
required the OPERATOR to be emailed a per-app capture failure. That was correct
when the event was the only signal and is wrong now that it is the record and
the digest is the notification. Its customer-safety claim is unchanged and is
why the test still exists; the operator assertion is inverted with the reasoning
written in place, and R-158's guarantee is shown to have MOVED, not weakened.
processOperator's cooldown no longer returns bare. It dropped the event BEFORE
LogNotification, so a suppressed operator alert and an event that never happened
were indistinguishable — from the operator's side and from the hub's own records.
Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were
mailed, seven left no row anywhere. That is why the defect took a day to get the
right way round: there was nothing to read.
A suppressed operator event now writes a `suppressed` row carrying the message
and the key that suppressed it. This applies to EVERY operator event, not only
the one that exposed it. It does NOT change the cooldown's duration or semantics.
backup_run_failures: the per-run digest. In allowedEventTypes AND in
operatorOnlyEvents — allowlisting alone does not make an event operator-only,
and FormatCustomerEmail falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type enabled receiving nothing.
recordOnlyEvents: a third routing class — stored and recorded, never mailed.
recovery_unit_capture_failed moves here: it is the record, the digest is the
notification. A register rather than downgrading severity to info, which would
relabel a genuine failure as informational everywhere it is queried.
cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so
tier keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, which is the intent — a digest is
already rate-limited by construction; the refresh sweep sends no run_id and so
stays under the ordinary hourly cooldown.
The email renders as a list, not a JSON blob. An absent space reading renders as
unavailable, never as zeros.
Two channels moved off main in the same change, because either one left behind
makes the other cosmetic.
Channel 1 — the served script. webpage.yaml git-synced /scripts/ from
--branch=main every 30s and nginx served that tree, so pushing this file WAS
publishing it: within half a minute it was what every new machine downloaded and
ran as root, with no staging and no rollback but another push. The sync is now
SPLIT: the website keeps tracking main at the same cadence (a copy edit must
never need a release) and /scripts/ tracks the tag installer-v<SCRIPT_VERSION>.
PROVEN before the manifest was touched: git-sync v4.4.0 follows a tag AND
notices a MOVED one — measured on a throwaway sync against this repo,
"update required ... local:<old> remote:<new>" -> "updated successfully",
within one period. The moved-tag half is what the publish model rests on.
Channel 2 — the sixteen files fetched at run time. fetch_raw pulled from
$AGENT_REPO/raw/branch/main; it now pulls raw/tag/v$ART_AGENT_VER. That is a
correctness fix, not only a channel one (R-183): a fresh install fetched the
vouched agent BINARY while taking its unit file, sudoers and guarded wrappers
from whatever main held. Two refs, one install, nothing compared them. Their
correct ref was never SCRIPT_VERSION — they do not live in this repo.
No fallback to a branch: a vouched version whose tag is missing fails loudly
rather than quietly serving main.
Channel 3 — the URL — needed no change, recorded rather than left silent:
https://felhom.eu/scripts/felhom-host-install.sh never carried a ref, so both
producers follow the tag with no edit. No hub change, no hub version bump.
Gate 6 in hostinstall_gates.py pins all three structurally with no network, so
it stays in --fast and runs in CI. It deliberately does NOT assert "a tag exists
for the current SCRIPT_VERSION": that would go red on the very push that bumps
the version, before publishing — and publishing being separate is the ruling.
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the
forward-compat their comment advertised: a section an emitter adds is silently
discarded until it is named in both the shape struct and AssembledRecipe. No
error, no log, no failing test.
R-122 (found this session): that already happened and shipped. The controller
has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the
hub stored it for all three real customers, and appHalfShape never listed the
key, so no delivered recipe has ever contained it. It stayed green because the
fixture drAppHalf is hand-written and omits the field.
R-109: the agent's new backup_target is a new top-level host-half section and
would have been dropped identically, making the fix read as shipped while
changing nothing an operator can see.
3 tests built on halves read verbatim out of the live dr_recipe table, plus
2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok.
Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with
no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling.
The golden's version IS the controller it bakes (build-golden.sh:345 defaults
GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed
controller means every FRESH install lands on stale application code. On the R-120
occurrence that stale code shipped a customer-facing falsehood: a box from the
0.185.1 golden told a customer whose backup drive had fallen out that the backup was
on the same disk as the system -- false, the drive was gone -- and offered a
different drive as the remedy.
WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17
releases behind), R-115 (agent built and deployed, never published), R-120 (this).
The first two were closed by re-baking and remembering; remembering then failed
again. R-29 is the standing proof that a check nobody runs is worse than none because
it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three
version bumps and hub_confirm_gate.py has never run at all. So the property that
matters is not whether a check exists but whether it BLOCKS.
- Wired into handleSetArtifacts (internal/web/configs.go), immediately before the
only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch
without anyone choosing to. A script in scripts/ would have been a fourth orphan.
- It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy.
- Signal: store.NewestReportedControllerVersion() over reports.controller_version,
SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this
fleet has shipped. No outbound call, no new credential.
- Fail-open in exactly two deliberate cases: an empty golden field (clearing the
manifest is legitimate) and an unknown fleet version (a new hub must vouch its
first golden).
NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that
exists in the schema and that NOTHING writes -- it would always have seen "" and
failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a
writer before trusting the column.
Blind spot stated rather than papered over: a controller no box has ever run is
invisible to this signal. Not the failure that has bitten -- all three instances were
deployed-newer-than-baked.
4 tests through the PRODUCTION handler over httptest, never an injected seam. The
refusal asserts both the flash and that the manifest was NOT written, because a gate
that redirects and saves anyway reads as enforcement while providing none. Red-proof:
deleting the block makes the stale golden vouchable and both assertions fail.
ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the
contrast with its three orphans is kept rather than lost. The orphans are unchanged.
Suite rc=0 read separately from this commit.
The manifest tag is what ArgoCD deploys; the code commit and CHANGELOG bump
deploy nothing on their own. Ships BEFORE the controller: an event type the hub
does not allowlist is answered 400 and the event vanishes.
v0.78.0 asserted in a comment that a type with no customerMessages entry cannot
reach a customer. It can: templates.go falls back to the raw message when the
entry is missing, and the only customer gate is prefs.EnabledEvents — pure
configuration. A customer with whole_guest_backup_failed enabled would have been
emailed raw English operator text about a backup they cannot act on. The new test
proves it against the v0.78.0 shape.
operatorOnlyEvents is now an explicit register checked before prefs, logging a
skipped/operator_only row so the skip is visible. NOT implemented as 'missing
customerMessages blocks delivery' — several types rely on that fallback on
purpose. The handler comment now names the real mechanism.
internal/quiesce had no route to the hub at all: three failed whole-guest backups
on 2026-07-27 produced zero events. Hub half of the fix.
whole_guest_backup_failed / _recovered are allowlisted with NO customerMessages
entry. Deliberately not backup_failed/backup_completed — those have customer
Hungarian templates AND sit in demo-felhom's live enabled_events, so reusing them
would email the customer that their backup failed while it is still retrying
behind the R-88 breaker.
The recovery joins recoveredPairedDownTypes because it is severity info and
severityNotifies drops info — otherwise the operator hears it break and never
hears it heal. Its customer leg is pairing-gated and can never fire.
Operator cooldown gains a per-tier dimension from the event details, so one tier
cannot mask another for an hour. Narrow: empty suffix unless a tier is sent, so
no existing event type changes.
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown
event_type, so controller v0.173.0's new drift alert would be silently inert
without it. Shipped with the controller that emits it.
Docs:
- RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the
step everyone will want to skip (establish which value is CORRECT from what
the agent is actually bound to, rather than assuming bootstrap.json wins) and
what success looks like (SILENCE, not a "recovered" line, because a fresh
controller's healthy first observation is not logged). Records both
2026-07-26 repairs.
- ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the
clobber-a-working-channel risk spelled out in BOTH directions so it is not
resolved opportunistically; R-79 the whole-surface English-strings sweep;
R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3
days of stale backup materially exceeds the ~1.5-day channel outage, so the
causal link the DIAG hedged on cannot be the whole story.
- Capability map: note against the drive-wizard row (every agent-backed
capability rides this channel) that a silent drift class is now detected.
NO row status flips — detection is not prevention.
Found validating v0.69.0 against the live hub. demo-vm-felhom was deleted
on 07-18 and was still on the Customers list AND still raising offsite_stale
(10 events, latest 07-21 17:34, operator email at 19:34) — because
GetCustomers() is report-derived and no lifecycle tier ever deleted a report.
New leg 3 (residue), before the record purge: reports, app_telemetry,
app_log_tails, log_tail_requests, customer_notifications, plus the
credential-bearing appliance_registrations and selfbind_tokens. Audit
(events, notification_log) and F-14 provenance still survive.
Ghost customers are now deletable: 404 means "nothing here", not "no config
row". With no config row the offsite descriptor is unknowable, so the Hetzner
and descriptor legs record skipped_no_config rather than a bare "skipped".
Two more red-proofs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J55BQE1gE2V4ffud5jweGS
Deploys hub v0.67.0 (auto-minted self-bind link, post-RESET staleness banner,
unprovisioned-offsite warning, pbsdr_reissued flash text). The manifest is the
truth — the code push and image build deploy nothing until this tag moves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Deploy the PBS-DR self-heal reconciler scoped to the drill guest only for the
first supervised run; widen to the fleet (remove the env) in a follow-up sync
once the converged-host no-op is confirmed live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
admin
Claude Code