felhom.eu/REPORT-r100.md

REPORT — R-100: a failing offsite tier must go stale (2026-07-28)

Hub v0.79.0 → v0.80.0; companion felhom-controller v0.180.0 → v0.181.0 (the producer, shipped first). Written as REPORT-r100.md so the shared REPORT.md is not clobbered.

Baselines (reconfirmed, not copied)

felhom.eu 6369570, felhom-controller 4056fec, felhom-agent d5c7691 — all = origin/main. The only dirt in felhom.eu was a foreign documentation/PROMPT-TEMPLATE.md (shared worktree, untouched). Hub manifest and running pod both 0.79.0; staleAfter = 48h; controller 0.180.0 and agent 0.110.0 live on both boxes.

---

The premise was wrong, and it was mine

R-100 was filed yesterday claiming "the operator's fleet-wide alarm plane is silent". Phase 0 refuted that, twice:

1. A failing offsite run does alarm. main.go:655 wires SetOffboxNotify → NotifyBackupFailed; the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB: backup_failed | operator | sent | 5, latest 2026-07-27 17:42. The isStale doc comment — "a recent-but-failing run is NOT stale (backup_failed owns that signal)" — was accurate.
2. The orphaned-repo path I expected to be an indefinite hole is already covered. The scheduled run returns early at offbox.go:606, before the LastRun write at :716, so LastRun freezes and offsite_stale fires normally.

I could find no failure mode that both advances LastRun and produces no operator signal.

The real defect — defeated defence in depth. offsite_stale is the hub-side, pull-based net that exists to be independent of controller-pushed events. Anchoring it on LastRun made it depend on the very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller keeps refreshing the field the net reads. F-HUB — this campaign's own finding, the hub dropping an event under SQLITE_BUSY with no retry — is exactly that loss.

Honest severity: MEDIUM, not the top-ranked item. The fix is unchanged; the justification is not.

---

Phase 0 answers

P0.1 — a last-success timestamp did not exist. OffboxTarget carried LastRun/LastStatus/ LastError/LastDuration only. Recording one is a new field, not a transmission of something known.

P0.2 — LastStatus on the wire, from 4000 live reports (not from source alone):

value	count	paired with
ok	2269	last_run set
absent/null	541	last_run empty — never-ran
error	27	last_run set
running	7	a report captured mid-run

Plus 1156 reports with no offsite object at all. The legacy trap — status absent with a real last_run — occurs 0 times, because LastStatus="running" is written the moment a run starts. It is still handled explicitly, but it is not a live shape. running being real is why the verdict ignores status entirely.

P0.3 — sweep

tier	LastRun written on failure?	read as success by a verdict?
Offsite restic	YES (offbox.go:716)	YES — hub isStale. The defect
Tier 2 cross-drive	YES (recordTier2Failure)	No hub verdict; UI only → R-101, filed
Tier 1 recovery units	NO — derived from an actual artifact	structurally immune
Shares offsite leg	YES	sharing.html:180 shows the time only when status=="ok" — honest
DB dump	n/a — event-based (db_dump_completed/db_dump_failed)	immune by design

offsite.go is the only hub verdict anchored on a LastRun-shaped field. The deadline checker already uses distinct success/failure events — the pattern this converges on.

P0.4 — the customer is NOT shown a failed offsite run as successful. backups_remote.html:34-36 leads with the status (✓ Rendben / ✗ Hiba / Fut…). Two narrower Tier-2 instances → R-101.

---

The fix

Controller v0.181.0 (producer, shipped first). OffboxTarget.LastSuccess, carried on the report as last_success. The rule is a pure function called unconditionally beside the LastRun write:

func offboxAnchorAfterRun(prev, at string, runErr error) string {
	if runErr != nil { return prev } // failures neither advance nor clear
	return at
}

Both directions are separate bugs: a failure must not advance it (the original defect) and must not clear it (one bad night making an established tier read as never-succeeded).

Two silent-wipe sites found and closed — the "seam built but never wired" shape, where the field exists, the writer sets it, and an unrelated routine path zeroes it:

• offboxConfigHandler rebuilds the target from the form and copies runtime status field by field, so an ordinary settings save would have erased the anchor;
• ApplyOffsiteTarget does the same on a hub re-apply.

Neither would have surfaced until the verdict changed, days later. The first was proven live — see below.

Hub v0.80.0. Three deliberate branches:

• never ran — unchanged v0.73.0 anchored behaviour, still keyed on last_run on purpose: that field answers "has anything ever happened here", and a box whose first run failed is a run, not a newborn.
• legacy (last_run set, no last_success) — degrades explicitly to the old behaviour, logged once per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success keeps the bug. Same degrade direction as R-88 Part 2's age_state.
• anchored — counts from last_success; last_status is deliberately not consulted, because "error ⇒ stale" pages on every blip (the F-A1 noise path).

The alarm text had to move with the verdict. emitStale still said last run 8h ago while firing on a six-day-old success — a true alarm that reads as false. staleAge now separates "runs are happening and failing — check the error, not the schedule" from "the offsite leg is silently not running".

---

Red-proofs — all observed failing

#	red-proof	observed failure
A	restore the LastRun anchor	a tier that has not succeeded in 6 days reads as FRESH — that is R-100
B	delete the never-ran branch	a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed
C	collapse to LastStatus == "error"	a single transient failure alarmed — 20h ... well inside the 48h threshold
D	delete the legacy degrade	a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet
+	drop the runErr guard (controller)	a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image
+	always return prev	a successful run did not advance the anchor
+	drop the wire field	OffboxReportStatus dropped LastSuccess — the hub would degrade forever
+	drop the handler preservation	a settings save erased LastSuccess

A hollow test of my own, caught by red-proofing it. The first version of the controller test re-implemented the rule in a local closure — mutating production code left it green. That is why offboxAnchorAfterRun was extracted: the test now calls the real rule.

Fixtures are the real wire shapes from P0.2, not invented JSON. go build/go vet/go test green in both repos (hub 17 pkgs, controller 27 pkgs), run separately from every commit.

---

§6 — LIVE, on demo-hp (disposable; peti-felhom never touched)

A genuine restic failure was induced by pointing the target at a closed port (23 → 2) — it creates nothing, touches no data, and is exactly reversible.

success run  →  last_status=ok     last_run=11:24:20Z   last_success=11:24:20Z
INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z   ← the wipe-site fix, live
failing run  →  last_status=error  last_run=11:25:48Z   last_success=11:24:20Z    ← ANCHOR HELD

As the hub received it:

box	status	last_run	last_success	anchor
demo-hp (induced failure)	error	11:25:48Z	11:24:20Z	HELD
demo-felhom (healthy)	ok	11:29:22Z	11:29:22Z	advanced

Also observed live, unplanned: Scenario E. Both boxes were still on the old controller at hub startup, and the degrade logged exactly once per customer — [WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT anchor. Two lines, two customers, same second.

No spurious alarms: 0 offsite_stale events since deploy (correct — both tiers succeeded minutes ago). backup_failed fired for demo-hp at 11:25:48 from the induced failure, confirming the pre-existing channel is intact and re-confirming the Phase 0 correction.

Config restored and verified field by field: host=u629488-sub3.your-storagebox.de port=23 user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed.

Proven live vs. proven by injected clock — stated plainly

• Live: the anchor does not advance on failure; it does on success; it survives a settings save; last_success reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
• Unit, injected clock only: the 48h threshold behaviour itself — Scenarios A/B/C/D turning on elapsed time. A live threshold test would take days. The threshold was NOT proven live.

---

Part 2 — the rule

"Presence is not success" added to CLAUDE.md and its versioned copy, with both instances (F-CRIT-2's phantom ctime, R-100's LastRun) and the corollary R-100's fix produced: when a verdict changes which field it counts from, the alarm text must change with it. // R-100 notes sit at isStale and at the controller write site, each naming the test that pins it.

Filed, not fixed

• R-101 — Tier-2 LastRun is also written on failure, and three customer surfaces render it without a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.

NOT yet live-validated (carried forward)

• The 48h staleness threshold itself (see above) — and with it Scenario A end-to-end: no offsite_stale event has yet been observed firing from a genuinely stale success anchor, because that needs 48h of failure.
• Fault 4 — restic transport interruption; four injection approaches defeated by guest-bridged networking. (This task's closed-port injection sidesteps it rather than solving it.)
• R-99 — prune never removes phantom snapshots.
• R-101 — filed today, unvalidated.
• contentionAlarmAfter (3h) — injected clock only.