From 3db8bfb9534ff2819c1f15c855e012ee7c6eda04 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 13:36:00 +0200 Subject: [PATCH] REPORT + CONTEXT: R-100 producer half (v0.181.0), proven live --- CONTEXT.md | 16 +++++- REPORT.md | 154 ++++++++++++++++++++++------------------------------- 2 files changed, 79 insertions(+), 91 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index a16458f..5edb748 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,21 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-28 (v0.180.0 — F-OBS: the dead-app check gets a positive observable) +Last updated: 2026-07-28 (v0.181.0 — R-100: the offsite last-SUCCESS anchor) + +> **2026-07-28 — v0.181.0 (R-100).** `OffboxTarget.LastSuccess` + wire field `last_success`; the hub +> (v0.80.0) anchors offsite staleness on it. **`LastRun` is written unconditionally on every run +> INCLUDING failures** — it records an ATTEMPT — so the hub's "how long since LastRun" verdict read a +> nightly-failing tier as perfectly fresh forever. The rule is the pure `offboxAnchorAfterRun(prev, at, +> runErr)`: a failure neither ADVANCES nor CLEARS the anchor (both are distinct bugs; clearing it would +> make one bad night look like never-succeeded). `LastStatus == "error" ⇒ stale` was rejected — it pages +> on every blip, the F-A1 noise mode. **TWO SILENT-WIPE SITES CLOSED** (`offboxConfigHandler` and +> `ApplyOffsiteTarget` both rebuild the target and copy runtime status field-by-field — omitting +> LastSuccess would erase the anchor on any settings save or hub re-apply). **LESSON: my first test +> modelled the rule in a local closure and stayed GREEN when production was mutated — hollow; the +> extraction to a pure function is what made the red-proof bite.** Live on demo-hp: failing run advanced +> `last_run` to 11:25:48Z while `last_success` HELD at 11:24:20Z; demo-felhom healthy → advanced. The +> settings-save preservation was proven live too. Detail: `REPORT.md` + `felhom.eu/REPORT-r100.md`. > **2026-07-28 — v0.180.0 (F-OBS).** Source: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. > On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had diff --git a/REPORT.md b/REPORT.md index 12c9529..a47db91 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,111 +1,85 @@ -# REPORT — F-OBS: the dead-app check gets a positive observable (v0.180.0) (2026-07-28) +# REPORT — R-100: record the last SUCCESS, not just the last attempt (v0.181.0) (2026-07-28) -**Overwritten** per the standing rule. Controller **v0.179.0 → v0.180.0**. One Campaign 8 finding, -the smallest of the three in this task's scope — and the one the task marked droppable. It was not -dropped, because it is what makes the *other two* fixes trustworthy in the field. - -Companions this session: `felhom-agent` v0.107.0 → v0.109.0 (F-REBOOT) and `felhom.eu` host-install -v1.21.0 (F-LEAK). No wire/contract change here; no agent change required. +**Overwritten** per the standing rule. Controller **v0.180.0 → v0.181.0** — the **producer** half of +R-100, shipped and deployed BEFORE the hub so the consumer never reads a field nothing sends. +Consumer: hub **v0.80.0**. Full arc, Phase 0 answers and the live proof: `felhom.eu/REPORT-r100.md`. ## Baselines (reconfirmed, not copied) -Controller `0.179.0` live on both boxes at start; agent `0.106.0`; `felhom-controller fb91c8d`, -`felhom-agent af1c21a`, `felhom.eu d0cec9d`, all clean. +`felhom-controller 4056fec` = origin/main, clean; controller 0.180.0 live on both boxes; hub 0.79.0 +(manifest and running pod); agent 0.110.0. ---- +## What changed here -## The finding +`OffboxTarget` gains **`LastSuccess`** (RFC3339), carried to the hub on the offsite report as +`last_success`. `LastRun` is written **unconditionally** at the end of every run including failures — it +records an **attempt**, so "how long since `LastRun`" answers "how long since we last TRIED", which is +not the question the hub's freshness verdict was asking of it. -On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had -run**. Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on -`logging.level == debug` — so on a default box the line was never *produced* and therefore could not -even reach the always-DEBUG ring. A 30 s interval also puts the job on the scheduler's quiet path. +The rule is a pure function, called unconditionally beside the `LastRun` write: -So **"no alarms" was indistinguishable from "the detector never ran"**. That is exactly the fallacy -this project has a standing rule against (rule 3: an absent log line is not evidence of correct -behaviour), and it directly undermines confidence in F-CRIT-1's fix — which is a fix to *this same -detector*. Without an observable, the only field evidence that F-CRIT-1 works would be silence. - -## The fix - -`noteDeadAppScan(logger, scans, evaluated, down)` — a pure, separately testable function emitting an -**INFO** line every **20th** scan (`deadAppHeartbeatEvery`), i.e. every 10 minutes at the job's 30 s -cadence: - -``` -[INFO] [deadapp] check alive — N scans since boot, M evaluated, D currently down +```go +func offboxAnchorAfterRun(prev, at string, runErr error) string { + if runErr != nil { return prev } // failures neither advance nor clear the anchor + return at +} ``` -Two things it deliberately does: +Both directions are distinct bugs and both are pinned: a failure must not **advance** it (the original +defect) and must not **clear** it (one bad night making an established tier read as never-succeeded — +the mirror-image over-correction, and on the hub side the newborn-box path). -- **Carries what it saw, not that it ran.** An operator needs to separate "running, everything up" - from "running, 2 apps down". A bare liveness ping cannot express the second, and the second is the - interesting state. -- **Summarises rather than floods.** One line per run is 2880/day, which is precisely the pressure - that made silence attractive to the original author — so a fix that floods is not a fix. +**Why not simply `LastStatus == "error" ⇒ stale`:** that is the tempting one-line fix and it pages on +every transient blip — the F-A1 noise mode that trains an operator to ignore the alarm. Anchoring on the +last success tolerates one bad night and catches a persistent one, using the threshold that already +exists. No new knob. -The function is pure and takes the logger as a parameter specifically so the *emitted line* can be -asserted. The original mistake was invisible because it lived inline with nothing pinning its output. +## Two silent-wipe sites, found and closed + +Both are the "seam built but never wired" shape — the field exists, the writer sets it, and an unrelated +routine path zeroes it. Neither would have surfaced until the hub's verdict changed, days later. + +- **`offboxConfigHandler`** rebuilds the target from the form and copies runtime status across field by + field, so an ordinary settings save (edit the host, edit the path) would have erased the anchor. +- **`ApplyOffsiteTarget`** does the same on a hub re-apply — an established tier reset to "never + succeeded" every time the hub re-pushed its descriptor. + +## A hollow test of my own, caught by red-proofing it + +The first version of `TestOffboxLastSuccess_OnlyAdvancesOnSuccess` re-implemented the rule in a local +closure. Mutating the production code left it **green** — it asserted my model, not the system. That is +what the extraction to `offboxAnchorAfterRun` is for: the test now calls the real rule and the red-proof +bites. ## Red-proofs — all observed failing +| red-proof | observed failure | +|---|---| +| drop the `runErr` guard | `a FAILED run advanced LastSuccess to "2026-07-21T02:15:00Z" — that is the R-100 defect in mirror image` | +| always return `prev` | `a successful run did not advance the anchor` | +| drop `LastSuccess` from the wire | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever on a controller that has it` | +| drop the handler preservation | `a settings save erased LastSuccess (got "") — the tier would read as never-succeeded` | -| # | red-proof | observed failure | -|---|---|---| -| 1 | delete the `logger.Printf` (or the whole call from the job closure) | `no observable emitted at scan 20 — silence is indistinguishable from not running` | -| 2 | change the guard to `scans%1 != 0` (emit every run) | `emitted 60 lines across 60 scans — that is the flood that made silence attractive` | - -Two further tests are tripwires rather than red-proofs: the observable must be at **INFO** (a -`logging.level: info` box would otherwise never see it) and the cadence must stay inside 15 minutes, -so a stalled detector is obvious well within the 180 s alarm grace this check feeds. Off-cadence scans -must stay quiet, and a nil logger must not panic — the job closure has no recovery. - -`go build`, `go vet ./...`, `go test ./...` — green, run separately from the commit. - ---- - -## Live, on demo-hp - -Endpoint-level/journal method (no browser on DooPlex, per the standing constraint): +`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`, run separately from the commit. +## Live, on demo-hp and demo-felhom ``` -[INFO] [deadapp] check alive — 20 scans since boot, 5 evaluated, 0 currently down +demo-hp success run → ok last_run=11:24:20Z last_success=11:24:20Z +demo-hp settings save (port 23→2) → last_success PRESERVED 11:24:20Z ← the wipe-site fix, live +demo-hp failing run → error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD +demo-felhom healthy run→ ok last_run=11:29:22Z last_success=11:29:22Z ← advanced ``` +Both shapes reached the hub. The failure was induced with a **closed port** (creates nothing, touches no +data) and the config was restored field by field. `peti-felhom` was never touched. -Present at **INFO** on a box whose `logging.level` is the default `info` — which is the whole point, -and the thing the previous behaviour could not do. The same observable is what let F-CRIT-1's Scenario -B replay earlier today assert *"9 dead-app scans during the silent window"* rather than merely -*"nothing was logged"*: the silence was proven to be suppression, not a dead detector. +**Not proven live:** the hub's 48h staleness threshold itself — that is unit-tested with an injected +clock and would take days to observe live. -**demo-felhom** received the same deploy with no injection and is quiet apart from its own heartbeat — -the fix is not chattering on an untouched box. - ---- - -## The same shape, found in my own new code - -Applying this finding's check to the agent's brand-new guest-power watchdog (shipped hours earlier in -`felhom-agent` v0.107.0) showed it had **exactly this defect**: it logged at startup and when it -*acted*, and was otherwise silent, so its health could only be inferred from absence. Fixed in agent -**v0.109.0** with the same pattern (INFO summary every 10th sweep, carrying what it saw, aborted sweeps -not counted). Detail in `felhom-agent/REPORT.md`. - -That is the argument for not dropping this part: the anti-pattern reproduces itself, and the fix is -what makes it visible. - ---- - -## Deployed & fleet state -`felhom-controller` **0.180.0** on **demo-hp** (`Up, healthy`) and **demo-felhom**, both healthy. -Cadences untouched. Agent **0.109.0** on both boxes. +## Deployed +`felhom-controller` **0.181.0** on **demo-hp** and **demo-felhom**, both `Up (healthy)`. ## NOT yet live-validated (carried forward) -- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged - networking. Still the most valuable follow-up. -- **R-100** (new, filed today) — a restic offsite tier that fails nightly never goes stale on the hub: - `isStale` reads only `LastRun` (`hub/internal/monitor/offsite.go:120`), which the controller writes - **unconditionally** on failure (`controller/internal/backup/offbox.go:716`). F-CRIT-2's defect class, - one layer up. Investigated, deliberately **not** fixed. -- **R-99** — prune never removes phantom snapshots (no retention bug; they accumulate). -- `contentionAlarmAfter` (3h) — unit-tested with an injected clock, **not** live; a live proof needs a - gate wedged for three hours. -- Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR - bring-up path (`ModeDRGuestLoss`), still read-only-verified. +- The 48h threshold, and with it an actually-observed `offsite_stale` firing from a stale anchor. +- **R-101** (filed today) — Tier-2 `LastRun` is also written on failure and three customer surfaces + render it without a status, incl. the restore-confirm dialog. +- **Fault 4** — restic transport interruption; the closed-port injection sidesteps it, not solves it. +- **R-99** — prune never removes phantom snapshots. +