Compare commits

...

3 Commits

Author SHA1 Message Date
admin 18d03bd437 v0.132.0: the slow crash-loop counter (R-539, operator ruling 3 of 2026-09-16)
gates / gates (push) Successful in 12s
Beside the unchanged 3-in-15-minutes brake, a second counter: restarts the
supervisor performed in the last 24 hours. At the fifth the heartbeat stanza
sets slow_crashloop_since (moving at most once per 24 h), slow_crashloop and
restarts_24h; hub v0.117.0 mints controller_slow_crashloop (warning,
operator-only) when the timestamp moves. It never stops restarting.

Persisted per guest (tmp+rename, 0600) so an agent restart or reboot does not
reset it - unlike the fast record, whose reason for staying in memory (a
persisted give-up outliving the fix) does not apply to a counter that only
warns. Deliberate kills count. The startup line prints the new limits.

Red-proofs seen failing: no counter; the once-per-24h guard removed ('the
operator would be mailed per restart'); the save removed ('Restarts24h:1'
after an agent restart). Negative control: restarts 7 h apart never raise it.
go build/vet/test ./... green, 30 packages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-17 10:26:42 +02:00
admin e98b857684 REPORT: v0.131.0 supervisor + per-tier status, delivery and live validation
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-15 11:37:07 +02:00
admin dcdeb3d16d CHANGELOG: v0.131.0 released (tag + package verified by download)
gates / gates (push) Successful in 12s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-15 09:50:47 +02:00
5 changed files with 271 additions and 46 deletions
+29 -1
View File
@@ -1,4 +1,32 @@
## Unreleased (to become v0.131.0) — a dead controller comes back by itself; the backup status speaks per tier (2026-09-15, R-523 / R-517 / R-518) ## Unreleased (to become v0.132.0) — a controller that dies slowly is reported, not just restarted (2026-09-17, R-539)
**MinAgent impact:** none required by any controller. Hub **v0.117.0** turns the new fields into
`controller_slow_crashloop`; an older hub ignores them.
- **R-539 (operator ruling 3 of 2026-09-16) — the slow crash-loop counter.** The 3-restarts-in-15-minutes
brake cannot see a controller that dies every 20 minutes (measured 2026-09-16, R-531: four restarts,
none accumulating, only an `info` event that mails nobody). Beside it, unchanged, a second counter:
restarts the supervisor performed in the last **24 hours**; at the **fifth**, the heartbeat's
`controller_supervisor` stanza sets `slow_crashloop_since` (and `slow_crashloop: true`,
`restarts_24h`). The hub mails on that timestamp MOVING; it moves **at most once per 24 hours**. It
does **not** stop restarting — the fast brake remains the only brake.
- **Persisted per guest** at `/var/lib/felhom-agent/guests/<vmid>/controller-restarts-24h.json`
(tmp + rename, 0600), so an agent restart or a host reboot does not reset it. The fast record stays
in memory; the reason it does (a persisted "give up" could outlive the fix) does not apply to a
counter that only warns. Unreadable or corrupt → WARN and a clean start, never a blocked supervisor.
- **Deliberate kills count.** The supervisor cannot tell an operator's `docker kill` from a crash
(measured 2026-09-15); a controller killed five times a day is worth a line either way.
- The supervisor's startup line now prints `slow_crashloop_max=5 slow_crashloop_window=24h0m0s`.
**Red-proofs, each seen failing:** five restarts 20 minutes apart raise it
(`TestControllerSupervisor_SlowCrashloop` — fails with no counter; fails again with the once-per-24-hours
guard removed, "the operator would be mailed per restart"); the counter survives an agent restart
(`…SlowCounterSurvivesAgentRestart` — fails with the save removed, `Restarts24h:1`); restarts seven hours
apart never raise it (the negative control). Wire shape extended with `restarts_24h` and `slow_crashloop`.
## v0.131.0 — a dead controller comes back by itself; the backup status speaks per tier (2026-09-15, R-523 / R-517 / R-518)
> **RELEASED 2026-09-15** by `scripts/release-agent.sh` — tag `v0.131.0`, sha256 `1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c`. Delivered to boxes by the controller v0.243.0 floor (declared MinAgent), not by hand.
- **R-523 (P1) — the in-guest controller supervisor.** BIGNIGHT F9: `docker kill felhom-controller` - **R-523 (P1) — the in-guest controller supervisor.** BIGNIGHT F9: `docker kill felhom-controller`
left the household's dashboard on 502 for 33 minutes, because nothing watched the container. left the household's dashboard on 502 for 33 minutes, because nothing watched the container.
+39 -42
View File
@@ -1,50 +1,47 @@
# REPORT — agent v0.129.0: a correct code for an earlier package (R-311, 2026-08-12) # REPORT — agent v0.131.0: the controller comes back by itself; backup status per tier (2026-09-15)
## What changed and why Task: *before the volunteer — the big night's P1 fixes*, Parts A and C (agent half). Architecture: `03-host-agent.md`
§4 (the "healing a crashed controller" sentence), `07-backup-architecture.md` §6.
Yesterday's drill proved a retained escrow package **works** — unsealed with the old recovery code, it ## Measured first (A.1)
opened a set-aside store and restored planted files byte-identical — while this agent answered that Docker 29.8.0, throwaway containers on scratch 9202: after `docker kill`, **both** `--restart unless-stopped` and
same correct code with *"the recovery code did not open the sealed bundle"*. Nothing had ever tried `--restart always` stayed `exited (137)` 60 s later. The task's claim was right; a policy change is not a fix.
the retained packages, so a correct-but-earlier code and a mistype were genuinely indistinguishable.
- `internal/hub/client.go` — `FetchRetainedIdentityEscrow` → `GET /api/v1/hosts/<id>/escrow/retained` ## What shipped
(hub ≥ v0.103.0). **A 404 is a clean "none"**, not a fault: an older hub must not turn into a failed - **Controller supervisor** (`internal/localapi/controllersupervisor.go`): every 30 s, for provisioned felhom-pool guests
recovery. that are running, restart `felhom-controller-bootstrap.service` on the second not-running observation. Guards: swap
- `internal/escrow/recover.go` — optional `FetchRetained`, `ErrCodeOpensRetained` + in flight, host-side park marker, locked / vzdump-busy / stopped guest, unknown docker answer, unprovisioned guest,
`RetainedOpenedError{SupersededAt, KeyFingerprint, Index, HasResticPassword}`. Consulted **only** 3 restarts in 15 min → 30 min pause. Record rides the host report as `controller_supervisor`; hub v0.114.0 mints the
after the current package refuses. events. Same GuestExecutor and sudoers grants as the swap — no new privilege.
- `internal/localapi/escrow_recover.go` — a **fifth** case on the R-224 switch: **422**, with - **Per-tier backup status**: `GET /backup/status` (untargeted) gains `tiers[]` — newest success (record, or storage
`opens_retained`, `superseded_at`, `retained_has_restic_pw`. Added to the switch, not a restructure. after a restart), last attempt kept apart, storage presence; `GET /backup/tiers` gains `storage`.
- `cmd/felhom-agent/main.go` — the retained fetcher wired on the same self-scoped hub client. - **Golden script**: `--restart always`. No golden baked (R-468); existing boxes keep `unless-stopped`.
## Fail-safe, in every direction ## Red-proofs (each seen failing, then restored)
- remove the restart call → `the killed controller was NOT restarted — this is R-523 (restarts=0)`
- remove the backoff block → `crash-looping controller restarted 10 times in 10 minutes — want exactly 3`
- `last_success` from the newest attempt → `pbs tier reports a failed attempt as its last success`
nil fetcher · hub without the route (404) · transport failure · malformed package → **the original ## Release and delivery
refusal stands, unchanged**. The worst outcome of this feature breaking is the behaviour before it. `scripts/release-agent.sh 0.131.0`: tag `v0.131.0`, sha256 `1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c`,
Attempts bounded (`MaxRetainedTried`, default 6) — each unwrap is ~1 s of scrypt, so an unbounded loop verified by download. **The task's "the floor delivers the agent" was wrong** (R-530): the hub holds a floor above the
would turn one wrong code into a minutes-long hang. box's agent; agents update only by an operator-signed job. On the operator's keys: `felhom-opsign -op agent_update` for
`demo-hp-bb76ea` only → authorized 08:44:16Z, committed 08:45:21Z, `controller-supervisor: started`. demo-felhom and
Peti's box stay on 0.130.0.
## Tests — 7, with REAL age crypto ## Live validation on demo-hp guest 9201 (agent 0.131.0, controller 0.243.0)
| moment | result |
|---|---|
| idle kill 08:53:27Z | restarted 08:54:22Z; dashboard 200 **59 s** after the kill |
| parked + kill | stayed dead 100 s, `the guest is PARKED — leaving it` every sweep; unpark → 200 in **25 s** |
| kill 10 s into a swap | `during a controller SWAP — the swap owns it` ×3; the swap rolled back itself, healthy 09:02:06Z |
| kill 5 s into a deploy | **not measured**: the three test restarts had filled the budget, so the guard paused (as designed) and the hub mailed `controller_crashloop` |
| resume after the pause | pause held to 09:35:52Z; restarted 09:36:24Z; dashboard 200 at 09:36:30Z |
Two invalid attempts, both marked in the evidence: a swap POST sent over plain HTTP (400, no swap), and a "health 200"
line during the pause that the container state contradicts.
Real crypto because the two situations are indistinguishable **at the unwrap**; a faked unwrap would ## Teardown
prove nothing about what was broken. Full suite green (`go build`/`vet`/`test ./...`), agent gates OK. Machine: 9201 back on its controller (see resume); the throwaway homebox deploy left no container. Host: park marker
removed; nothing else changed. Hub: demo-hp floor override 0.243.0 kept (it delivers this release).
**Red-proof, mutation asserted applied before the run:** remove the `tryRetained` block from Evidence: `felhom.eu/documentation/audits/evidence-p1fixes-2026-09-15/A*`.
`RecoverOffsiteRepoPassword` →
`err = escrow: the recovery code did not unwrap the identity escrow (wrong recovery code…)` →
`TestRecover_CodeOpensRetainedPackage_IsNotAWrongCode` FAILS. **The lie returns, in those words.**
That is the layer the lie actually lives in: removing the *controller's* case yields the neutral
message instead, because R-224's safe default catches it.
## Released and deployed
`release-agent.sh 0.129.0` — tagged `v0.129.0`, published, **verified by independent download**,
sha256 `53a54f0620afbd6d…`. Installed on `felhom-pve`, `felhom-agent --version` = 0.129.0, unit active,
journal clean (normal PBS verify cycle). **NOT VOUCHED** — that stays the operator's act.
## Bypass, stated as required
`git push --no-verify` was used **once** for the code push. The `release-complete` gate refuses a
CHANGELOG entry whose tag and package do not exist, and `release-agent.sh` refuses a tree that is not
pushed — circular by construction. The bypass was immediately followed by the real release; gates were
re-run afterwards and are **green**, and the tag+package now exist.
+7
View File
@@ -150,6 +150,13 @@ type ControllerSupervisorGuest struct {
Crashloop bool `json:"crashloop"` Crashloop bool `json:"crashloop"`
CrashloopSince string `json:"crashloop_since,omitempty"` // RFC3339; the last crash-loop, kept after it ends CrashloopSince string `json:"crashloop_since,omitempty"` // RFC3339; the last crash-loop, kept after it ends
Parked bool `json:"parked"` Parked bool `json:"parked"`
// R-539 (v0.132.0) — the SLOW crash loop. Restarts24h counts restarts the supervisor performed in
// the last 24 hours (persisted, so an agent restart does not reset it); SlowCrashloop is true while
// the last raise is under 24 hours old; SlowCrashloopSince is the raise itself, which the hub keys on
// MOVING (hub v0.117.0 controller_slow_crashloop). It moves at most once per 24 hours.
Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
} }
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States: // PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
+96 -2
View File
@@ -2,6 +2,7 @@ package localapi
import ( import (
"context" "context"
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
@@ -68,6 +69,20 @@ const (
ControllerParkedMarker = "controller-parked" ControllerParkedMarker = "controller-parked"
defaultGuestsStateDir = "/var/lib/felhom-agent/guests" defaultGuestsStateDir = "/var/lib/felhom-agent/guests"
// R-539 (operator ruling 3 of 2026-09-16) — the SLOW crash loop. The 3-in-15-minutes brake above
// cannot see a controller that dies every 20 minutes: no two restarts share its window, so it is
// restarted for ever and the only trace is an info event that mails nobody (measured 2026-09-16,
// R-531). A second counter over 24 hours raises a WARNING at the fifth restart. It does NOT stop
// restarting — the fast brake stays the only brake, unchanged. Every restart the supervisor
// performs counts, including one that follows a deliberate operator `docker kill` (measured
// 2026-09-15: the supervisor cannot tell a kill from a crash, and a controller that is killed five
// times a day is worth a line to the operator either way).
controllerSlowCrashloopWindow = 24 * time.Hour
controllerSlowCrashloopMax = 5
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
// the parked marker.
controllerSlowCounterFile = "controller-restarts-24h.json"
) )
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power // controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -81,6 +96,13 @@ type controllerSupState struct {
lastReason string lastReason string
crashloopSince time.Time // zero = not in a crash-loop pause crashloopSince time.Time // zero = not in a crash-loop pause
parked bool parked bool
// R-539 — the slow counter. PERSISTED, unlike everything above, and the precedent's reason does not
// apply to it: persisting the fast record could carry a stale "give up" across the restart that
// fixed it, but this record never gives anything up — it only warns. Losing it on an agent restart,
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
restarts24h []time.Time
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
} }
type controllerSupervisor struct { type controllerSupervisor struct {
@@ -98,7 +120,9 @@ func (s *Server) WatchControllers(ctx context.Context) {
} }
s.logger.Info("controller-supervisor: started", "interval", controllerSupervisorInterval.String(), s.logger.Info("controller-supervisor: started", "interval", controllerSupervisorInterval.String(),
"confirm_sweeps", controllerSupervisorConfirm, "crashloop_max", controllerCrashloopMax, "confirm_sweeps", controllerSupervisorConfirm, "crashloop_max", controllerCrashloopMax,
"crashloop_window", controllerCrashloopWindow.String(), "guests_dir", s.guestsStateDir()) "crashloop_window", controllerCrashloopWindow.String(),
"slow_crashloop_max", controllerSlowCrashloopMax, "slow_crashloop_window", controllerSlowCrashloopWindow.String(),
"guests_dir", s.guestsStateDir())
t := time.NewTicker(controllerSupervisorInterval) t := time.NewTicker(controllerSupervisorInterval)
defer t.Stop() defer t.Stop()
for { for {
@@ -139,11 +163,61 @@ func (s *Server) supState(vmid int) *controllerSupState {
st := s.ctrlSup.guests[vmid] st := s.ctrlSup.guests[vmid]
if st == nil { if st == nil {
st = &controllerSupState{} st = &controllerSupState{}
s.loadSlowCounter(vmid, st)
s.ctrlSup.guests[vmid] = st s.ctrlSup.guests[vmid] = st
} }
return st return st
} }
// slowCounterRecord is the on-disk shape of the R-539 counter.
type slowCounterRecord struct {
Restarts []time.Time `json:"restarts"`
SlowCrashloopSince time.Time `json:"slow_crashloop_since,omitempty"`
}
func (s *Server) slowCounterPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), controllerSlowCounterFile)
}
// loadSlowCounter restores the persisted counter into a fresh state. Absent = a clean start; unreadable
// or corrupt = a clean start with a WARN (a warning counter must never block supervision).
func (s *Server) loadSlowCounter(vmid int, st *controllerSupState) {
b, err := os.ReadFile(s.slowCounterPath(vmid))
if err != nil {
if !os.IsNotExist(err) {
s.logger.Warn("controller-supervisor: slow counter unreadable — starting it from zero", "vmid", vmid, "err", err)
}
return
}
var rec slowCounterRecord
if err := json.Unmarshal(b, &rec); err != nil {
s.logger.Warn("controller-supervisor: slow counter corrupt — starting it from zero", "vmid", vmid, "err", err)
return
}
st.restarts24h = pruneBefore(rec.Restarts, s.clock().Add(-controllerSlowCrashloopWindow))
st.slowCrashloopSince = rec.SlowCrashloopSince
if len(st.restarts24h) > 0 || !st.slowCrashloopSince.IsZero() {
s.logger.Info("controller-supervisor: slow counter restored from disk", "vmid", vmid,
"restarts_24h", len(st.restarts24h), "slow_crashloop_since", st.slowCrashloopSince.Format(time.RFC3339))
}
}
// saveSlowCounter writes the counter atomically (tmp + rename, 0600). A failure is logged and the
// in-memory counter carries on — the next restart retries the write.
func (s *Server) saveSlowCounter(vmid int, rec slowCounterRecord) {
path := s.slowCounterPath(vmid)
b, err := json.Marshal(rec)
if err == nil {
tmp := path + ".tmp"
if err = os.WriteFile(tmp, b, 0o600); err == nil {
err = os.Rename(tmp, path)
}
}
if err != nil {
s.logger.Warn("controller-supervisor: could not persist the slow counter (kept in memory)", "vmid", vmid, "path", path, "err", err)
}
}
// ControllerSupervisorTick performs one sweep. Exported so a test (and a live check) can drive one // ControllerSupervisorTick performs one sweep. Exported so a test (and a live check) can drive one
// cycle without waiting on the ticker. // cycle without waiting on the ticker.
func (s *Server) ControllerSupervisorTick(ctx context.Context) { func (s *Server) ControllerSupervisorTick(ctx context.Context) {
@@ -299,8 +373,22 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
st.lastRestartAt = now st.lastRestartAt = now
st.lastReason = reason st.lastReason = reason
st.notRunningSeen = 0 st.notRunningSeen = 0
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
n24 := len(st.restarts24h)
raised := false
if n24 >= controllerSlowCrashloopMax && (st.slowCrashloopSince.IsZero() || now.Sub(st.slowCrashloopSince) >= controllerSlowCrashloopWindow) {
st.slowCrashloopSince = now
raised = true
}
rec := slowCounterRecord{Restarts: append([]time.Time(nil), st.restarts24h...), SlowCrashloopSince: st.slowCrashloopSince}
s.ctrlSup.mu.Unlock() s.ctrlSup.mu.Unlock()
s.logger.Warn("controller-supervisor: RESTARTED the controller", "vmid", vmid, "reason", reason) s.saveSlowCounter(vmid, rec)
s.logger.Warn("controller-supervisor: RESTARTED the controller", "vmid", vmid, "reason", reason, "restarts_24h", n24)
if raised {
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
}
return false return false
} }
@@ -352,6 +440,12 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
if !st.crashloopSince.IsZero() { if !st.crashloopSince.IsZero() {
g.CrashloopSince = st.crashloopSince.UTC().Format(time.RFC3339) g.CrashloopSince = st.crashloopSince.UTC().Format(time.RFC3339)
} }
now := s.clock()
g.Restarts24h = len(pruneBefore(append([]time.Time(nil), st.restarts24h...), now.Add(-controllerSlowCrashloopWindow)))
if !st.slowCrashloopSince.IsZero() {
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
}
out.Guests = append(out.Guests, g) out.Guests = append(out.Guests, g)
} }
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID }) sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
+100 -1
View File
@@ -243,9 +243,108 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
g := m["guests"][0] g := m["guests"][0]
for _, k := range []string{"vmid", "restarts_total", "last_restart_at", "last_reason", "crashloop", "parked"} { for _, k := range []string{"vmid", "restarts_total", "last_restart_at", "last_reason", "crashloop", "parked", "restarts_24h", "slow_crashloop"} {
if _, ok := g[k]; !ok { if _, ok := g[k]; !ok {
t.Fatalf("stanza lacks %q — the hub keys on it: %s", k, b) t.Fatalf("stanza lacks %q — the hub keys on it: %s", k, b)
} }
} }
} }
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
// moves the clock on by gap. The container comes back "running", so each restart is a separate act.
func supKillOnce(t *testing.T, s *Server, ex *supExec, clk *supClock, gap time.Duration) {
t.Helper()
before := ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
clk.t = clk.t.Add(controllerSupervisorInterval)
s.ControllerSupervisorTick(context.Background())
if ex.count(9201) != before+1 {
t.Fatalf("kill was not followed by exactly one restart (restarts %d → %d)", before, ex.count(9201))
}
clk.t = clk.t.Add(gap)
}
// The consequence: a controller that dies every 20 minutes — never three times inside the 15-minute
// brake — raises slow_crashloop on the FIFTH restart in 24 hours, and the raise does not move again on
// the sixth (the hub mails on movement; once per 24 hours is the ruling).
//
// RED-PROOF: without the slow counter the stanza never sets slow_crashloop → "five restarts 20 minutes
// apart did not raise slow_crashloop — this is R-539".
func TestControllerSupervisor_SlowCrashloop(t *testing.T) {
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
s, clk, _ := supServer(t, ex, runningGuest(9201), 9201)
ctx := context.Background()
for i := 1; i <= 4; i++ {
supKillOnce(t, s, ex, clk, 20*time.Minute)
}
g := s.ControllerSupervisorStatus(ctx).Guests[0]
if g.Crashloop {
t.Fatalf("the 15-minute brake fired on restarts 20 minutes apart — the fixture is wrong: %+v", g)
}
if g.SlowCrashloop || g.SlowCrashloopSince != "" {
t.Fatalf("slow_crashloop raised after only 4 restarts: %+v", g)
}
supKillOnce(t, s, ex, clk, 20*time.Minute)
g = s.ControllerSupervisorStatus(ctx).Guests[0]
if !g.SlowCrashloop || g.SlowCrashloopSince == "" || g.Restarts24h != 5 {
t.Fatalf("five restarts 20 minutes apart did not raise slow_crashloop — this is R-539: %+v", g)
}
first := g.SlowCrashloopSince
supKillOnce(t, s, ex, clk, 20*time.Minute)
g = s.ControllerSupervisorStatus(ctx).Guests[0]
if g.SlowCrashloopSince != first {
t.Fatalf("the raise moved again on the 6th restart (%q → %q) — the operator would be mailed per restart", first, g.SlowCrashloopSince)
}
if !g.SlowCrashloop {
t.Fatalf("slow_crashloop cleared while the loop continues: %+v", g)
}
}
// The negative control: restarts that never reach five inside any 24 hours never raise it.
func TestControllerSupervisor_SpreadRestartsNeverSlowCrashloop(t *testing.T) {
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
s, clk, _ := supServer(t, ex, runningGuest(9201), 9201)
for i := 0; i < 8; i++ { // eight restarts, 7 hours apart: at most 4 inside any 24 hours
supKillOnce(t, s, ex, clk, 7*time.Hour)
}
g := s.ControllerSupervisorStatus(context.Background()).Guests[0]
if g.SlowCrashloop || g.SlowCrashloopSince != "" {
t.Fatalf("restarts 7 hours apart raised slow_crashloop: %+v", g)
}
if g.Restarts24h > 4 {
t.Fatalf("restarts_24h=%d — the 24-hour window is not pruning", g.Restarts24h)
}
}
// An agent restart must not reset the slow counter (the ruling; a box whose AGENT also restarts would
// otherwise never reach five). The same state directory, a fresh Server.
//
// RED-PROOF: keep the counter in memory only → the second Server starts at 0 → "the agent restart
// reset the slow counter".
func TestControllerSupervisor_SlowCounterSurvivesAgentRestart(t *testing.T) {
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
s, clk, dir := supServer(t, ex, runningGuest(9201), 9201)
for i := 0; i < 4; i++ {
supKillOnce(t, s, ex, clk, 20*time.Minute)
}
s2 := &Server{
staleLock: s.staleLock,
guestExec: ex,
guestsDir: dir,
swapInFlight: map[int]bool{},
logger: s.logger,
now: clk.now,
}
supKillOnce(t, s2, ex, clk, 20*time.Minute)
g := s2.ControllerSupervisorStatus(context.Background()).Guests[0]
if g.Restarts24h != 5 || !g.SlowCrashloop {
t.Fatalf("the agent restart reset the slow counter: %+v", g)
}
}