MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.
The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.
internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.
The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.
Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.
A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.
THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.
A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.
Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.
7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.
R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.
R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.
Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called
from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given
a Hungarian message on the hub -- and never invoked. A drive going absent produced
apps stopped, a WARN log and a UI badge, then silence on every channel. Verified
against the gitignored-cmd/ trap with a positive control. Fifth instance of this
class, found by E-2 Phase 0 rather than by a failure.
A drive that is ONLY a backup target has no apps to stop, so it was silent twice.
ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest
backup target it raises the more specific backup_target_absent (error) instead --
never both, since two mails for one event trains people to ignore the channel --
and recovers as backup_target_restored (info, the existing pairing-gated pattern;
severityNotifies NOT widened). The recovery mirrors the alarm's choice or the
operator cannot match them.
Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0),
not from our StoragePath.BackupTarget: that is customer INTENT, and on the two
boxes migrated by hand in E-1 the intent was never recorded while the drive really
is the target. An older agent omits the field -> false -> generic alarm, never a
wrong one.
Before this an absent backup target had NO prompt signal: the tier stays DUE
(targetStoragePresent checks name presence, never reachability), so the only
evidence was its own failure at the next due cycle, up to ~24h away. The R-100
shape.
Tests observe the WIRE, not a mock, because the failure class is "nothing
arrives": a real Notifier posts to an httptest hub and the test asserts the event
type and severity that actually went out.
MinAgent: 0.112.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a
licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY
agent, and refuses it for AgeStateUnknown. An unreadable storage no longer
masquerades as a first-ever backup and no longer quiesces apps outside the window.
A missing wire field means legacy, not unknown — deliberately. Treating it as
unknown would stop the valve firing on un-upgraded boxes and starve genuinely new
ones. Degrade logged once; unrecognised future values also map to legacy.
Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the
signature change compiled and vetted clean while quiesceBackend silently stopped
satisfying it — which would have degraded every box to the single-tier path with
no error. Added a compile-time witness.
Also corrects the notifier comment that claimed operator-only came from a missing
customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0).
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.
R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md
bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json
local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths,
its own event type local_api_endpoint_drift, and its own Hungarian banner shown
ABOVE the channel banner because drift is the cause and "agent unreachable" the
symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a
correct controller.yaml on any half-provisioned or hand-repaired guest, so the
authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/
incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing
path is untouched). Fingerprint compared as a BOOLEAN only; token never
compared, logged or exposed.
EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of
reconcileSambaAt's early returns, and the doc comment is corrected in the same
change -- it claimed "detection and deployment agree in both directions" while
citing only !smb.Enabled, an assertion that went false when !smb.UserSet was
added. Not over-suppressed: sharing on WITH a password and a dead container
still alarms.
Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded")
instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded
stops being dead code. Logging only -- the placeholder is still matched in the
re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all
nine pre-existing channelhealth tests pass.
Tests 951 -> 959, all green. Red-proofs A (both directions), E and F.
MinAgent unchanged; felhom-agent untouched.
Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect
by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard
card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm);
unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits
disk_health_degraded on a degradation only vs an in-memory baseline (first run
silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner
(deliberate). Pairs with the hub allowlist bump.
Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent
(red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical,
nil-smart card, TTL cache.
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.
Rests on two invariants (recorded at the seam, README, CONTEXT):
I1 StopStack = compose down => zero containers => StateStopped
I2 P2 census: all catalog services unless-stopped => faults never rest
at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
New internal/channelhealth Checker: ~60s probe via the PRODUCTION memoized client
(Server.ProbeAgentChannel, GET /storage), classifies failures (spike Q1 map), debounces transient
reasons (N>=2; construction error latches distinctly), seeds first obs, alerts operator+dashboard on
transition. Notifier.NotifyAgentChannelDown/Recovered (English, operator-only), AlertManager dashboard
banner (Hungarian). No agent/hub change. Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Controller half of Phase-1 Infra Backup retirement (hub v0.12.0;
SPIKE-infra-backup-2026-06-15). Pure dead-code removal, no behaviour change.
- Remove Pusher.PushInfraBackup (caller-less; hub endpoint gone).
- Remove Notifier.NotifyBackupCompleted / backup_completed event (caller-less
since slice 8C; hub deadline check now reads agent host-report PBS snapshots).
- Remove report.BackupReport.ResticPassword — builder never sets it post-8C
(confirmed in source + live), but it historically leaked the restic password
into the hub's plaintext reports store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Debug-mode-only dashboard (/debug) with 8 collapsible sections:
system diagnostics, notification testing, backup triggers, storage
simulation, hub & connectivity, self-update dry-run, DR/setup wizard,
and in-memory log viewer. Migrates debug dump from API router to web
server. Adds ring buffer log capture, storage disconnect simulation,
event history tracking, and cross-drive/self-update test methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After successful config apply, immediately push infra backup to Hub
so the config sync status updates right away. Also fix startup event
message that showed "vv0.21.2" instead of "v0.21.3".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SyncPreferences() method on Notifier: POST to hub /api/v1/preferences
- IsEnabled() getter for hub connectivity check
- settingsNotificationsHandler: sync to hub after local save (3 flash message variants)
- Startup sync: non-blocking goroutine pushes prefs to hub on boot (DB rebuild recovery)
- Updated CONTEXT.md, README.md with v0.7.2 changes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Monitoring page: "Távoli monitoring" section showing healthcheck ping UUID
configuration status (configured/not configured) for each of the 5 pings
- Alert manager: persistent dashboard banners on all pages generated from
health check results, missing pings, and backup status
- Notification system: controller-side notifier sends events to hub relay,
with cooldown tracking and event-type filtering
- Notification preferences UI: email, event checkboxes, cooldown settings
on the settings page with test email functionality
- Settings refactored: shared settingsData() helper, NotificationPrefs
struct with getter/setter and defaults
New files:
- controller/internal/web/alerts.go (AlertManager)
- controller/internal/notify/notifier.go (hub notification client)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
admin