admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

REPORT + CONTEXT: R-100 producer half (v0.181.0), proven live

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-28 13:36:00 +02:00
parent e000e201af
commit 3db8bfb953
2 changed files with 79 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+15 -1
View File
@@ -7,7 +7,21 @@
>
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-07-28 (v0.180.0 — F-OBS: the dead-app check gets a positive observable)
Last updated: 2026-07-28 (v0.181.0 — R-100: the offsite last-SUCCESS anchor)
> **2026-07-28 — v0.181.0 (R-100).** `OffboxTarget.LastSuccess` + wire field `last_success`; the hub
> (v0.80.0) anchors offsite staleness on it. **`LastRun` is written unconditionally on every run
> INCLUDING failures** — it records an ATTEMPT — so the hub's "how long since LastRun" verdict read a
> nightly-failing tier as perfectly fresh forever. The rule is the pure `offboxAnchorAfterRun(prev, at,
> runErr)`: a failure neither ADVANCES nor CLEARS the anchor (both are distinct bugs; clearing it would
> make one bad night look like never-succeeded). `LastStatus == "error" ⇒ stale` was rejected — it pages
> on every blip, the F-A1 noise mode. **TWO SILENT-WIPE SITES CLOSED** (`offboxConfigHandler` and
> `ApplyOffsiteTarget` both rebuild the target and copy runtime status field-by-field — omitting
> LastSuccess would erase the anchor on any settings save or hub re-apply). **LESSON: my first test
> modelled the rule in a local closure and stayed GREEN when production was mutated — hollow; the
> extraction to a pure function is what made the red-proof bite.** Live on demo-hp: failing run advanced
> `last_run` to 11:25:48Z while `last_success` HELD at 11:24:20Z; demo-felhom healthy → advanced. The
> settings-save preservation was proven live too. Detail: `REPORT.md` + `felhom.eu/REPORT-r100.md`.
> **2026-07-28 — v0.180.0 (F-OBS).** Source: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
> On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had
REPORT.md
+64 -90
View File
@@ -1,111 +1,85 @@
# REPORT — F-OBS: the dead-app check gets a positive observable (v0.180.0) (2026-07-28)
# REPORT — R-100: record the last SUCCESS, not just the last attempt (v0.181.0) (2026-07-28)
**Overwritten** per the standing rule. Controller **v0.179.0 → v0.180.0**. One Campaign 8 finding,
the smallest of the three in this task's scope — and the one the task marked droppable. It was not
dropped, because it is what makes the *other two* fixes trustworthy in the field.
Companions this session: `felhom-agent` v0.107.0 → v0.109.0 (F-REBOOT) and `felhom.eu` host-install
v1.21.0 (F-LEAK). No wire/contract change here; no agent change required.
**Overwritten** per the standing rule. Controller **v0.180.0 → v0.181.0** — the **producer** half of
R-100, shipped and deployed BEFORE the hub so the consumer never reads a field nothing sends.
Consumer: hub **v0.80.0**. Full arc, Phase 0 answers and the live proof: `felhom.eu/REPORT-r100.md`.
## Baselines (reconfirmed, not copied)
Controller `0.179.0` live on both boxes at start; agent `0.106.0`; `felhom-controller fb91c8d`,
`felhom-agent af1c21a`, `felhom.eu d0cec9d`, all clean.
`felhom-controller 4056fec` = origin/main, clean; controller 0.180.0 live on both boxes; hub 0.79.0
(manifest and running pod); agent 0.110.0.
---
## What changed here
## The finding
`OffboxTarget` gains **`LastSuccess`** (RFC3339), carried to the hub on the offsite report as
`last_success`. `LastRun` is written **unconditionally** at the end of every run including failures — it
records an **attempt**, so "how long since `LastRun`" answers "how long since we last TRIED", which is
not the question the hub's freshness verdict was asking of it.
On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had
run**. Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on
`logging.level == debug` — so on a default box the line was never *produced* and therefore could not
even reach the always-DEBUG ring. A 30 s interval also puts the job on the scheduler's quiet path.
The rule is a pure function, called unconditionally beside the `LastRun` write:
So **"no alarms" was indistinguishable from "the detector never ran"**. That is exactly the fallacy
this project has a standing rule against (rule 3: an absent log line is not evidence of correct
behaviour), and it directly undermines confidence in F-CRIT-1's fix — which is a fix to *this same
detector*. Without an observable, the only field evidence that F-CRIT-1 works would be silence.
## The fix
`noteDeadAppScan(logger, scans, evaluated, down)` — a pure, separately testable function emitting an
**INFO** line every **20th** scan (`deadAppHeartbeatEvery`), i.e. every 10 minutes at the job's 30 s
cadence:
```
[INFO] [deadapp] check alive — N scans since boot, M evaluated, D currently down
```go
func offboxAnchorAfterRun(prev, at string, runErr error) string {
if runErr != nil { return prev } // failures neither advance nor clear the anchor
return at
}
```
Two things it deliberately does:
Both directions are distinct bugs and both are pinned: a failure must not **advance** it (the original
defect) and must not **clear** it (one bad night making an established tier read as never-succeeded —
the mirror-image over-correction, and on the hub side the newborn-box path).
- **Carries what it saw, not that it ran.** An operator needs to separate "running, everything up"
from "running, 2 apps down". A bare liveness ping cannot express the second, and the second is the
interesting state.
- **Summarises rather than floods.** One line per run is 2880/day, which is precisely the pressure
that made silence attractive to the original author — so a fix that floods is not a fix.
**Why not simply `LastStatus == "error" ⇒ stale`:** that is the tempting one-line fix and it pages on
every transient blip — the F-A1 noise mode that trains an operator to ignore the alarm. Anchoring on the
last success tolerates one bad night and catches a persistent one, using the threshold that already
exists. No new knob.
The function is pure and takes the logger as a parameter specifically so the *emitted line* can be
asserted. The original mistake was invisible because it lived inline with nothing pinning its output.
## Two silent-wipe sites, found and closed
Both are the "seam built but never wired" shape — the field exists, the writer sets it, and an unrelated
routine path zeroes it. Neither would have surfaced until the hub's verdict changed, days later.
- **`offboxConfigHandler`** rebuilds the target from the form and copies runtime status across field by
field, so an ordinary settings save (edit the host, edit the path) would have erased the anchor.
- **`ApplyOffsiteTarget`** does the same on a hub re-apply — an established tier reset to "never
succeeded" every time the hub re-pushed its descriptor.
## A hollow test of my own, caught by red-proofing it
The first version of `TestOffboxLastSuccess_OnlyAdvancesOnSuccess` re-implemented the rule in a local
closure. Mutating the production code left it **green** — it asserted my model, not the system. That is
what the extraction to `offboxAnchorAfterRun` is for: the test now calls the real rule and the red-proof
bites.
## Red-proofs — all observed failing
| red-proof | observed failure |
|---|---|
| drop the `runErr` guard | `a FAILED run advanced LastSuccess to "2026-07-21T02:15:00Z" — that is the R-100 defect in mirror image` |
| always return `prev` | `a successful run did not advance the anchor` |
| drop `LastSuccess` from the wire | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever on a controller that has it` |
| drop the handler preservation | `a settings save erased LastSuccess (got "") — the tier would read as never-succeeded` |
| # | red-proof | observed failure |
|---|---|---|
| 1 | delete the `logger.Printf` (or the whole call from the job closure) | `no observable emitted at scan 20 — silence is indistinguishable from not running` |
| 2 | change the guard to `scans%1 != 0` (emit every run) | `emitted 60 lines across 60 scans — that is the flood that made silence attractive` |
Two further tests are tripwires rather than red-proofs: the observable must be at **INFO** (a
`logging.level: info` box would otherwise never see it) and the cadence must stay inside 15 minutes,
so a stalled detector is obvious well within the 180 s alarm grace this check feeds. Off-cadence scans
must stay quiet, and a nil logger must not panic — the job closure has no recovery.
`go build`, `go vet ./...`, `go test ./...` — green, run separately from the commit.
---
## Live, on demo-hp
Endpoint-level/journal method (no browser on DooPlex, per the standing constraint):
`go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`, run separately from the commit.
## Live, on demo-hp and demo-felhom
```
[INFO] [deadapp] check alive — 20 scans since boot, 5 evaluated, 0 currently down
demo-hp success run → ok last_run=11:24:20Z last_success=11:24:20Z
demo-hp settings save (port 23→2) → last_success PRESERVED 11:24:20Z ← the wipe-site fix, live
demo-hp failing run → error last_run=11:25:48Z last_success=11:24:20Z ← ANCHOR HELD
demo-felhom healthy run→ ok last_run=11:29:22Z last_success=11:29:22Z ← advanced
```
Both shapes reached the hub. The failure was induced with a **closed port** (creates nothing, touches no
data) and the config was restored field by field. `peti-felhom` was never touched.
Present at **INFO** on a box whose `logging.level` is the default `info` — which is the whole point,
and the thing the previous behaviour could not do. The same observable is what let F-CRIT-1's Scenario
B replay earlier today assert *"9 dead-app scans during the silent window"* rather than merely
*"nothing was logged"*: the silence was proven to be suppression, not a dead detector.
**Not proven live:** the hub's 48h staleness threshold itself — that is unit-tested with an injected
clock and would take days to observe live.
**demo-felhom** received the same deploy with no injection and is quiet apart from its own heartbeat —
the fix is not chattering on an untouched box.
---
## The same shape, found in my own new code
Applying this finding's check to the agent's brand-new guest-power watchdog (shipped hours earlier in
`felhom-agent` v0.107.0) showed it had **exactly this defect**: it logged at startup and when it
*acted*, and was otherwise silent, so its health could only be inferred from absence. Fixed in agent
**v0.109.0** with the same pattern (INFO summary every 10th sweep, carrying what it saw, aborted sweeps
not counted). Detail in `felhom-agent/REPORT.md`.
That is the argument for not dropping this part: the anti-pattern reproduces itself, and the fix is
what makes it visible.
---
## Deployed & fleet state
`felhom-controller` **0.180.0** on **demo-hp** (`Up, healthy`) and **demo-felhom**, both healthy.
Cadences untouched. Agent **0.109.0** on both boxes.
## Deployed
`felhom-controller` **0.181.0** on **demo-hp** and **demo-felhom**, both `Up (healthy)`.
## NOT yet live-validated (carried forward)
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
networking. Still the most valuable follow-up.
- **R-100** (new, filed today) — a restic offsite tier that fails nightly never goes stale on the hub:
`isStale` reads only `LastRun` (`hub/internal/monitor/offsite.go:120`), which the controller writes
**unconditionally** on failure (`controller/internal/backup/offbox.go:716`). F-CRIT-2's defect class,
one layer up. Investigated, deliberately **not** fixed.
- **R-99** — prune never removes phantom snapshots (no retention bug; they accumulate).
- `contentionAlarmAfter` (3h) — unit-tested with an injected clock, **not** live; a live proof needs a
gate wedged for three hours.
- Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR
bring-up path (`ModeDRGuestLoss`), still read-only-verified.
- The 48h threshold, and with it an actually-observed `offsite_stale` firing from a stale anchor.
- **R-101** (filed today) — Tier-2 `LastRun` is also written on failure and three customer surfaces
render it without a status, incl. the restore-confirm dialog.
- **Fault 4** — restic transport interruption; the closed-port injection sidesteps it, not solves it.
- **R-99** — prune never removes phantom snapshots.