Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.
Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.
R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
Found by checking v0.91.0 against the live box, not by review. demo-felhom's two
retained PBS snapshots sit 8h54m apart (one is a healing artefact), so the
mean-gap estimator reads a WEEKLY tier as nine-hourly: x4 = 36h, the 7-day floor
lifts it to 168h, and a weekly tier proved weekly reaches ~8.25d of proof age.
The false alarm this task exists to prevent would have returned within a week, on
the box it had just shipped to.
restoreProvenWindow now takes max(observed, declared). A gap SHORTER than the
declared rhythm is routine and means nothing (a retry, a manual run, a heal, a
catch-up); a gap LONGER than it is real information. Cost stated: a tier running
faster than its declared rhythm gets a slower stale signal — the right direction
for a signal that means 'unverified', since 'broken now' is a different event.
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
isStale counted from last_run, written unconditionally on failure, so a nightly-failing
tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade
(logged once) and the never-ran branch untouched. emitStale states the real reason.
A failed restore-test was a [WARN] line in the ingest handler and nothing else —
no event, no notification, no gauge. True for the LOCAL tier that was already
being tested, so the loudest DR signal this system produces was inaudible.
Rotating tiers without this would only mean two tiers can fail silently
instead of one.
Two signals, deliberately NOT merged:
restore_test_failed (error) — a run completed and did NOT pass
restore_test_stale (warning) — a tier not PROVEN within its interval
Merging them collapses 'your DR is broken' into 'your DR is unverified', and
the second is the one that quietly becomes the first. The staleness wording
says 'unverified, not known-broken' and a test asserts that phrasing.
Anchored per R-81, not re-derived: a never-proven tier on a newborn box is
UNKNOWN, not FAILED, until the window elapses. This family has made the
opposite mistake three times; this monitor was written straight after the third,
so it copies R-81's structure rather than inventing a fourth shape.
restoreProvenStaleAfter = 7d is derived: oldest-first over two tiers at a 24h
cadence proves each ~every 2 days, so 7d tolerates ~3 missed opportunities and
sits inside the 2-week offsite retention.
Per-tier proof comes from the hub's retained host-report window — the agent
reports only its latest run, so the latest report alone cannot answer 'when was
the OTHER tier last proven?'. Reused R-81's mechanism instead of a wire change.
Both types registered in allowedEventTypes (R-77's inert-seam lesson) and
operator-tier only — no customerMessages entry.
FIXED a time bomb I introduced in Slice C: the restart-blind-window test
hard-coded 2026-07-18T18:31:06Z while comparing against the real clock. Harmless
under one 26h threshold; once the offsite tier got an 8-day limit it passed all
day and began failing at 18:31 UTC, exactly 8 days later. Now relative.
Red-proofs B and D observed. Full suite green (17 packages, rc=0).
R-81 merged every backup signal into one 'newest' against a single 26h limit.
backupStaleAfter's own comment recorded why that stops being right under a
weekly offsite tier. Each tier is now judged against its own threshold;
R-81's structure (three verdicts, anchored absence, distinct reasons) and its
boundary test are preserved intact.
- offsiteBackupStaleAfter = 8d (7d cadence + headroom); backupStaleAfter keeps
26h and now names the HOST tier only
- splitTiers / assessTier / newestBackupEvidenceByTier
Slice-A.4 rule implemented: a PBS-targeted vzdump appears in BOTH arrays, so
classification is by TARGET TYPE (target_id -> storage_targets[].name -> type),
never by array membership — otherwise a PBS backup makes a stale host tier look
fresh. storage_targets is used rather than pbs_dr.storage_id because the latter
is null on a box with a PBS storage but no DR descriptor.
A tier is only judged when the box HAS it, else every box without an offsite
tier would alarm once the anchor elapsed — R-81's mistake one level down. With
neither tier identifiable (old agent) the pre-Slice-C path runs unchanged.
Intended behaviour change: a 30h offsite snapshot no longer alarms. Three
fixtures asserted the merged threshold; each still asserts an alarm at the
correct limit. No assertion was weakened.
RECORDED LIMITATION: the hub infers 'PBS => weekly' from storage type.
defaultBackupTarget is felhom-pbs, so a box that never sets local_backup_target
would run PBS as its DAILY tier and be judged against 8 days — 7 days of
blindness. No box is in that shape today; the real fix is the agent reporting
per-tier cadences. Own task.
Red-proof observed. Replayed live: demo-felhom OK, demo-hp UNKNOWN (defers
correctly), drill-r50 MISSED (true positive). No customer email would be sent.
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".
- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
retained history (bounded 7-day lookback, early-exit on fresh evidence) —
"when did I last SEE evidence of a backup?" The anchor was free: the hub
already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
alarm six days in seven) and owned by R-82.
Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.
Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
Sibling checker over the controller report's offsite object: quota-fill
warn/crit + the silently-stuck staleness detector (escrowed-only,
red-proofed; nil-safe on pre-v0.109 reports; same-second tie-guard).
SetOffsiteFrozen flips ONLY readonly on the exactly-1 labelled sub-account
(SSH preserved); Freeze/Unfreeze buttons — manual only, never automatic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
store.GetHostOOBStates parses the agent oob heartbeat stanza. monitor/host_oob:
transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the
operator peer is configured, OR config invalid) — proactive "can the operator get
in right now" signal; unconfigured OOB never alerts. Wired into the 60s sweep.
Non-hollow tests + transitions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A heal marker is an event, not a baseline: construction seeds pre-existing markers
(startup false-alarm guard) but a newly-observed marker now raises the warning, so
the first auto-heal surfaces (matches the live drill). Added tests for both halves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
ProcessEvent routed only warning/error; a critical-severity event was silently dropped.
Now routes warning/error/critical, logs unrecognized severities, and guards a nil
GetNotificationPrefs (which would panic/crash the hub). host_disk_critical emits its
natural critical severity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Constructors seed only healthy hosts; an already-degraded/stale host is left unseeded so the first
Check() emits once (cooldown dedups on hub restart). Born-degraded red-proof + staleness test updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
Companion to felhom-agent v0.44.0. New monitor.HostCapabilityChecker (sibling of
HostStalenessChecker) reads the capabilities snapshot from the latest host report and emits
agent_capability_degraded/recovered (operator-only, 1h cooldown) on ok<->degraded transitions
for any Critical capability. store.GetHostCapabilities (MAX(id), no migration). Goldens mirror
the new capabilities field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin
Claude Code