Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 18d03bd437 | |||
| e98b857684 | |||
| dcdeb3d16d |
+29
-1
@@ -1,4 +1,32 @@
|
|||||||
## Unreleased (to become v0.131.0) — a dead controller comes back by itself; the backup status speaks per tier (2026-09-15, R-523 / R-517 / R-518)
|
## Unreleased (to become v0.132.0) — a controller that dies slowly is reported, not just restarted (2026-09-17, R-539)
|
||||||
|
|
||||||
|
**MinAgent impact:** none required by any controller. Hub **v0.117.0** turns the new fields into
|
||||||
|
`controller_slow_crashloop`; an older hub ignores them.
|
||||||
|
|
||||||
|
- **R-539 (operator ruling 3 of 2026-09-16) — the slow crash-loop counter.** The 3-restarts-in-15-minutes
|
||||||
|
brake cannot see a controller that dies every 20 minutes (measured 2026-09-16, R-531: four restarts,
|
||||||
|
none accumulating, only an `info` event that mails nobody). Beside it, unchanged, a second counter:
|
||||||
|
restarts the supervisor performed in the last **24 hours**; at the **fifth**, the heartbeat's
|
||||||
|
`controller_supervisor` stanza sets `slow_crashloop_since` (and `slow_crashloop: true`,
|
||||||
|
`restarts_24h`). The hub mails on that timestamp MOVING; it moves **at most once per 24 hours**. It
|
||||||
|
does **not** stop restarting — the fast brake remains the only brake.
|
||||||
|
- **Persisted per guest** at `/var/lib/felhom-agent/guests/<vmid>/controller-restarts-24h.json`
|
||||||
|
(tmp + rename, 0600), so an agent restart or a host reboot does not reset it. The fast record stays
|
||||||
|
in memory; the reason it does (a persisted "give up" could outlive the fix) does not apply to a
|
||||||
|
counter that only warns. Unreadable or corrupt → WARN and a clean start, never a blocked supervisor.
|
||||||
|
- **Deliberate kills count.** The supervisor cannot tell an operator's `docker kill` from a crash
|
||||||
|
(measured 2026-09-15); a controller killed five times a day is worth a line either way.
|
||||||
|
- The supervisor's startup line now prints `slow_crashloop_max=5 slow_crashloop_window=24h0m0s`.
|
||||||
|
|
||||||
|
**Red-proofs, each seen failing:** five restarts 20 minutes apart raise it
|
||||||
|
(`TestControllerSupervisor_SlowCrashloop` — fails with no counter; fails again with the once-per-24-hours
|
||||||
|
guard removed, "the operator would be mailed per restart"); the counter survives an agent restart
|
||||||
|
(`…SlowCounterSurvivesAgentRestart` — fails with the save removed, `Restarts24h:1`); restarts seven hours
|
||||||
|
apart never raise it (the negative control). Wire shape extended with `restarts_24h` and `slow_crashloop`.
|
||||||
|
|
||||||
|
## v0.131.0 — a dead controller comes back by itself; the backup status speaks per tier (2026-09-15, R-523 / R-517 / R-518)
|
||||||
|
|
||||||
|
> **RELEASED 2026-09-15** by `scripts/release-agent.sh` — tag `v0.131.0`, sha256 `1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c`. Delivered to boxes by the controller v0.243.0 floor (declared MinAgent), not by hand.
|
||||||
|
|
||||||
- **R-523 (P1) — the in-guest controller supervisor.** BIGNIGHT F9: `docker kill felhom-controller`
|
- **R-523 (P1) — the in-guest controller supervisor.** BIGNIGHT F9: `docker kill felhom-controller`
|
||||||
left the household's dashboard on 502 for 33 minutes, because nothing watched the container.
|
left the household's dashboard on 502 for 33 minutes, because nothing watched the container.
|
||||||
|
|||||||
@@ -1,50 +1,47 @@
|
|||||||
# REPORT — agent v0.129.0: a correct code for an earlier package (R-311, 2026-08-12)
|
# REPORT — agent v0.131.0: the controller comes back by itself; backup status per tier (2026-09-15)
|
||||||
|
|
||||||
## What changed and why
|
Task: *before the volunteer — the big night's P1 fixes*, Parts A and C (agent half). Architecture: `03-host-agent.md`
|
||||||
|
§4 (the "healing a crashed controller" sentence), `07-backup-architecture.md` §6.
|
||||||
|
|
||||||
Yesterday's drill proved a retained escrow package **works** — unsealed with the old recovery code, it
|
## Measured first (A.1)
|
||||||
opened a set-aside store and restored planted files byte-identical — while this agent answered that
|
Docker 29.8.0, throwaway containers on scratch 9202: after `docker kill`, **both** `--restart unless-stopped` and
|
||||||
same correct code with *"the recovery code did not open the sealed bundle"*. Nothing had ever tried
|
`--restart always` stayed `exited (137)` 60 s later. The task's claim was right; a policy change is not a fix.
|
||||||
the retained packages, so a correct-but-earlier code and a mistype were genuinely indistinguishable.
|
|
||||||
|
|
||||||
- `internal/hub/client.go` — `FetchRetainedIdentityEscrow` → `GET /api/v1/hosts/<id>/escrow/retained`
|
## What shipped
|
||||||
(hub ≥ v0.103.0). **A 404 is a clean "none"**, not a fault: an older hub must not turn into a failed
|
- **Controller supervisor** (`internal/localapi/controllersupervisor.go`): every 30 s, for provisioned felhom-pool guests
|
||||||
recovery.
|
that are running, restart `felhom-controller-bootstrap.service` on the second not-running observation. Guards: swap
|
||||||
- `internal/escrow/recover.go` — optional `FetchRetained`, `ErrCodeOpensRetained` +
|
in flight, host-side park marker, locked / vzdump-busy / stopped guest, unknown docker answer, unprovisioned guest,
|
||||||
`RetainedOpenedError{SupersededAt, KeyFingerprint, Index, HasResticPassword}`. Consulted **only**
|
3 restarts in 15 min → 30 min pause. Record rides the host report as `controller_supervisor`; hub v0.114.0 mints the
|
||||||
after the current package refuses.
|
events. Same GuestExecutor and sudoers grants as the swap — no new privilege.
|
||||||
- `internal/localapi/escrow_recover.go` — a **fifth** case on the R-224 switch: **422**, with
|
- **Per-tier backup status**: `GET /backup/status` (untargeted) gains `tiers[]` — newest success (record, or storage
|
||||||
`opens_retained`, `superseded_at`, `retained_has_restic_pw`. Added to the switch, not a restructure.
|
after a restart), last attempt kept apart, storage presence; `GET /backup/tiers` gains `storage`.
|
||||||
- `cmd/felhom-agent/main.go` — the retained fetcher wired on the same self-scoped hub client.
|
- **Golden script**: `--restart always`. No golden baked (R-468); existing boxes keep `unless-stopped`.
|
||||||
|
|
||||||
## Fail-safe, in every direction
|
## Red-proofs (each seen failing, then restored)
|
||||||
|
- remove the restart call → `the killed controller was NOT restarted — this is R-523 (restarts=0)`
|
||||||
|
- remove the backoff block → `crash-looping controller restarted 10 times in 10 minutes — want exactly 3`
|
||||||
|
- `last_success` from the newest attempt → `pbs tier reports a failed attempt as its last success`
|
||||||
|
|
||||||
nil fetcher · hub without the route (404) · transport failure · malformed package → **the original
|
## Release and delivery
|
||||||
refusal stands, unchanged**. The worst outcome of this feature breaking is the behaviour before it.
|
`scripts/release-agent.sh 0.131.0`: tag `v0.131.0`, sha256 `1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c`,
|
||||||
Attempts bounded (`MaxRetainedTried`, default 6) — each unwrap is ~1 s of scrypt, so an unbounded loop
|
verified by download. **The task's "the floor delivers the agent" was wrong** (R-530): the hub holds a floor above the
|
||||||
would turn one wrong code into a minutes-long hang.
|
box's agent; agents update only by an operator-signed job. On the operator's keys: `felhom-opsign -op agent_update` for
|
||||||
|
`demo-hp-bb76ea` only → authorized 08:44:16Z, committed 08:45:21Z, `controller-supervisor: started`. demo-felhom and
|
||||||
|
Peti's box stay on 0.130.0.
|
||||||
|
|
||||||
## Tests — 7, with REAL age crypto
|
## Live validation on demo-hp guest 9201 (agent 0.131.0, controller 0.243.0)
|
||||||
|
| moment | result |
|
||||||
|
|---|---|
|
||||||
|
| idle kill 08:53:27Z | restarted 08:54:22Z; dashboard 200 **59 s** after the kill |
|
||||||
|
| parked + kill | stayed dead 100 s, `the guest is PARKED — leaving it` every sweep; unpark → 200 in **25 s** |
|
||||||
|
| kill 10 s into a swap | `during a controller SWAP — the swap owns it` ×3; the swap rolled back itself, healthy 09:02:06Z |
|
||||||
|
| kill 5 s into a deploy | **not measured**: the three test restarts had filled the budget, so the guard paused (as designed) and the hub mailed `controller_crashloop` |
|
||||||
|
| resume after the pause | pause held to 09:35:52Z; restarted 09:36:24Z; dashboard 200 at 09:36:30Z |
|
||||||
|
Two invalid attempts, both marked in the evidence: a swap POST sent over plain HTTP (400, no swap), and a "health 200"
|
||||||
|
line during the pause that the container state contradicts.
|
||||||
|
|
||||||
Real crypto because the two situations are indistinguishable **at the unwrap**; a faked unwrap would
|
## Teardown
|
||||||
prove nothing about what was broken. Full suite green (`go build`/`vet`/`test ./...`), agent gates OK.
|
Machine: 9201 back on its controller (see resume); the throwaway homebox deploy left no container. Host: park marker
|
||||||
|
removed; nothing else changed. Hub: demo-hp floor override 0.243.0 kept (it delivers this release).
|
||||||
|
|
||||||
**Red-proof, mutation asserted applied before the run:** remove the `tryRetained` block from
|
Evidence: `felhom.eu/documentation/audits/evidence-p1fixes-2026-09-15/A*`.
|
||||||
`RecoverOffsiteRepoPassword` →
|
|
||||||
`err = escrow: the recovery code did not unwrap the identity escrow (wrong recovery code…)` →
|
|
||||||
`TestRecover_CodeOpensRetainedPackage_IsNotAWrongCode` FAILS. **The lie returns, in those words.**
|
|
||||||
That is the layer the lie actually lives in: removing the *controller's* case yields the neutral
|
|
||||||
message instead, because R-224's safe default catches it.
|
|
||||||
|
|
||||||
## Released and deployed
|
|
||||||
|
|
||||||
`release-agent.sh 0.129.0` — tagged `v0.129.0`, published, **verified by independent download**,
|
|
||||||
sha256 `53a54f0620afbd6d…`. Installed on `felhom-pve`, `felhom-agent --version` = 0.129.0, unit active,
|
|
||||||
journal clean (normal PBS verify cycle). **NOT VOUCHED** — that stays the operator's act.
|
|
||||||
|
|
||||||
## Bypass, stated as required
|
|
||||||
|
|
||||||
`git push --no-verify` was used **once** for the code push. The `release-complete` gate refuses a
|
|
||||||
CHANGELOG entry whose tag and package do not exist, and `release-agent.sh` refuses a tree that is not
|
|
||||||
pushed — circular by construction. The bypass was immediately followed by the real release; gates were
|
|
||||||
re-run afterwards and are **green**, and the tag+package now exist.
|
|
||||||
|
|||||||
@@ -150,6 +150,13 @@ type ControllerSupervisorGuest struct {
|
|||||||
Crashloop bool `json:"crashloop"`
|
Crashloop bool `json:"crashloop"`
|
||||||
CrashloopSince string `json:"crashloop_since,omitempty"` // RFC3339; the last crash-loop, kept after it ends
|
CrashloopSince string `json:"crashloop_since,omitempty"` // RFC3339; the last crash-loop, kept after it ends
|
||||||
Parked bool `json:"parked"`
|
Parked bool `json:"parked"`
|
||||||
|
// R-539 (v0.132.0) — the SLOW crash loop. Restarts24h counts restarts the supervisor performed in
|
||||||
|
// the last 24 hours (persisted, so an agent restart does not reset it); SlowCrashloop is true while
|
||||||
|
// the last raise is under 24 hours old; SlowCrashloopSince is the raise itself, which the hub keys on
|
||||||
|
// MOVING (hub v0.117.0 controller_slow_crashloop). It moves at most once per 24 hours.
|
||||||
|
Restarts24h int `json:"restarts_24h"`
|
||||||
|
SlowCrashloop bool `json:"slow_crashloop"`
|
||||||
|
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
||||||
}
|
}
|
||||||
|
|
||||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package localapi
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
@@ -68,6 +69,20 @@ const (
|
|||||||
ControllerParkedMarker = "controller-parked"
|
ControllerParkedMarker = "controller-parked"
|
||||||
|
|
||||||
defaultGuestsStateDir = "/var/lib/felhom-agent/guests"
|
defaultGuestsStateDir = "/var/lib/felhom-agent/guests"
|
||||||
|
|
||||||
|
// R-539 (operator ruling 3 of 2026-09-16) — the SLOW crash loop. The 3-in-15-minutes brake above
|
||||||
|
// cannot see a controller that dies every 20 minutes: no two restarts share its window, so it is
|
||||||
|
// restarted for ever and the only trace is an info event that mails nobody (measured 2026-09-16,
|
||||||
|
// R-531). A second counter over 24 hours raises a WARNING at the fifth restart. It does NOT stop
|
||||||
|
// restarting — the fast brake stays the only brake, unchanged. Every restart the supervisor
|
||||||
|
// performs counts, including one that follows a deliberate operator `docker kill` (measured
|
||||||
|
// 2026-09-15: the supervisor cannot tell a kill from a crash, and a controller that is killed five
|
||||||
|
// times a day is worth a line to the operator either way).
|
||||||
|
controllerSlowCrashloopWindow = 24 * time.Hour
|
||||||
|
controllerSlowCrashloopMax = 5
|
||||||
|
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
||||||
|
// the parked marker.
|
||||||
|
controllerSlowCounterFile = "controller-restarts-24h.json"
|
||||||
)
|
)
|
||||||
|
|
||||||
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
||||||
@@ -81,6 +96,13 @@ type controllerSupState struct {
|
|||||||
lastReason string
|
lastReason string
|
||||||
crashloopSince time.Time // zero = not in a crash-loop pause
|
crashloopSince time.Time // zero = not in a crash-loop pause
|
||||||
parked bool
|
parked bool
|
||||||
|
|
||||||
|
// R-539 — the slow counter. PERSISTED, unlike everything above, and the precedent's reason does not
|
||||||
|
// apply to it: persisting the fast record could carry a stale "give up" across the restart that
|
||||||
|
// fixed it, but this record never gives anything up — it only warns. Losing it on an agent restart,
|
||||||
|
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
||||||
|
restarts24h []time.Time
|
||||||
|
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
||||||
}
|
}
|
||||||
|
|
||||||
type controllerSupervisor struct {
|
type controllerSupervisor struct {
|
||||||
@@ -98,7 +120,9 @@ func (s *Server) WatchControllers(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
s.logger.Info("controller-supervisor: started", "interval", controllerSupervisorInterval.String(),
|
s.logger.Info("controller-supervisor: started", "interval", controllerSupervisorInterval.String(),
|
||||||
"confirm_sweeps", controllerSupervisorConfirm, "crashloop_max", controllerCrashloopMax,
|
"confirm_sweeps", controllerSupervisorConfirm, "crashloop_max", controllerCrashloopMax,
|
||||||
"crashloop_window", controllerCrashloopWindow.String(), "guests_dir", s.guestsStateDir())
|
"crashloop_window", controllerCrashloopWindow.String(),
|
||||||
|
"slow_crashloop_max", controllerSlowCrashloopMax, "slow_crashloop_window", controllerSlowCrashloopWindow.String(),
|
||||||
|
"guests_dir", s.guestsStateDir())
|
||||||
t := time.NewTicker(controllerSupervisorInterval)
|
t := time.NewTicker(controllerSupervisorInterval)
|
||||||
defer t.Stop()
|
defer t.Stop()
|
||||||
for {
|
for {
|
||||||
@@ -139,11 +163,61 @@ func (s *Server) supState(vmid int) *controllerSupState {
|
|||||||
st := s.ctrlSup.guests[vmid]
|
st := s.ctrlSup.guests[vmid]
|
||||||
if st == nil {
|
if st == nil {
|
||||||
st = &controllerSupState{}
|
st = &controllerSupState{}
|
||||||
|
s.loadSlowCounter(vmid, st)
|
||||||
s.ctrlSup.guests[vmid] = st
|
s.ctrlSup.guests[vmid] = st
|
||||||
}
|
}
|
||||||
return st
|
return st
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// slowCounterRecord is the on-disk shape of the R-539 counter.
|
||||||
|
type slowCounterRecord struct {
|
||||||
|
Restarts []time.Time `json:"restarts"`
|
||||||
|
SlowCrashloopSince time.Time `json:"slow_crashloop_since,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) slowCounterPath(vmid int) string {
|
||||||
|
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), controllerSlowCounterFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadSlowCounter restores the persisted counter into a fresh state. Absent = a clean start; unreadable
|
||||||
|
// or corrupt = a clean start with a WARN (a warning counter must never block supervision).
|
||||||
|
func (s *Server) loadSlowCounter(vmid int, st *controllerSupState) {
|
||||||
|
b, err := os.ReadFile(s.slowCounterPath(vmid))
|
||||||
|
if err != nil {
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
s.logger.Warn("controller-supervisor: slow counter unreadable — starting it from zero", "vmid", vmid, "err", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var rec slowCounterRecord
|
||||||
|
if err := json.Unmarshal(b, &rec); err != nil {
|
||||||
|
s.logger.Warn("controller-supervisor: slow counter corrupt — starting it from zero", "vmid", vmid, "err", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
st.restarts24h = pruneBefore(rec.Restarts, s.clock().Add(-controllerSlowCrashloopWindow))
|
||||||
|
st.slowCrashloopSince = rec.SlowCrashloopSince
|
||||||
|
if len(st.restarts24h) > 0 || !st.slowCrashloopSince.IsZero() {
|
||||||
|
s.logger.Info("controller-supervisor: slow counter restored from disk", "vmid", vmid,
|
||||||
|
"restarts_24h", len(st.restarts24h), "slow_crashloop_since", st.slowCrashloopSince.Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// saveSlowCounter writes the counter atomically (tmp + rename, 0600). A failure is logged and the
|
||||||
|
// in-memory counter carries on — the next restart retries the write.
|
||||||
|
func (s *Server) saveSlowCounter(vmid int, rec slowCounterRecord) {
|
||||||
|
path := s.slowCounterPath(vmid)
|
||||||
|
b, err := json.Marshal(rec)
|
||||||
|
if err == nil {
|
||||||
|
tmp := path + ".tmp"
|
||||||
|
if err = os.WriteFile(tmp, b, 0o600); err == nil {
|
||||||
|
err = os.Rename(tmp, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("controller-supervisor: could not persist the slow counter (kept in memory)", "vmid", vmid, "path", path, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ControllerSupervisorTick performs one sweep. Exported so a test (and a live check) can drive one
|
// ControllerSupervisorTick performs one sweep. Exported so a test (and a live check) can drive one
|
||||||
// cycle without waiting on the ticker.
|
// cycle without waiting on the ticker.
|
||||||
func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
||||||
@@ -299,8 +373,22 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
|||||||
st.lastRestartAt = now
|
st.lastRestartAt = now
|
||||||
st.lastReason = reason
|
st.lastReason = reason
|
||||||
st.notRunningSeen = 0
|
st.notRunningSeen = 0
|
||||||
|
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
|
||||||
|
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
|
||||||
|
n24 := len(st.restarts24h)
|
||||||
|
raised := false
|
||||||
|
if n24 >= controllerSlowCrashloopMax && (st.slowCrashloopSince.IsZero() || now.Sub(st.slowCrashloopSince) >= controllerSlowCrashloopWindow) {
|
||||||
|
st.slowCrashloopSince = now
|
||||||
|
raised = true
|
||||||
|
}
|
||||||
|
rec := slowCounterRecord{Restarts: append([]time.Time(nil), st.restarts24h...), SlowCrashloopSince: st.slowCrashloopSince}
|
||||||
s.ctrlSup.mu.Unlock()
|
s.ctrlSup.mu.Unlock()
|
||||||
s.logger.Warn("controller-supervisor: RESTARTED the controller", "vmid", vmid, "reason", reason)
|
s.saveSlowCounter(vmid, rec)
|
||||||
|
s.logger.Warn("controller-supervisor: RESTARTED the controller", "vmid", vmid, "reason", reason, "restarts_24h", n24)
|
||||||
|
if raised {
|
||||||
|
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
|
||||||
|
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
|
||||||
|
}
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -352,6 +440,12 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
|
|||||||
if !st.crashloopSince.IsZero() {
|
if !st.crashloopSince.IsZero() {
|
||||||
g.CrashloopSince = st.crashloopSince.UTC().Format(time.RFC3339)
|
g.CrashloopSince = st.crashloopSince.UTC().Format(time.RFC3339)
|
||||||
}
|
}
|
||||||
|
now := s.clock()
|
||||||
|
g.Restarts24h = len(pruneBefore(append([]time.Time(nil), st.restarts24h...), now.Add(-controllerSlowCrashloopWindow)))
|
||||||
|
if !st.slowCrashloopSince.IsZero() {
|
||||||
|
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
||||||
|
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
||||||
|
}
|
||||||
out.Guests = append(out.Guests, g)
|
out.Guests = append(out.Guests, g)
|
||||||
}
|
}
|
||||||
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
||||||
|
|||||||
@@ -243,9 +243,108 @@ func TestControllerSupervisorStanza_WireShape(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
g := m["guests"][0]
|
g := m["guests"][0]
|
||||||
for _, k := range []string{"vmid", "restarts_total", "last_restart_at", "last_reason", "crashloop", "parked"} {
|
for _, k := range []string{"vmid", "restarts_total", "last_restart_at", "last_reason", "crashloop", "parked", "restarts_24h", "slow_crashloop"} {
|
||||||
if _, ok := g[k]; !ok {
|
if _, ok := g[k]; !ok {
|
||||||
t.Fatalf("stanza lacks %q — the hub keys on it: %s", k, b)
|
t.Fatalf("stanza lacks %q — the hub keys on it: %s", k, b)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
// ---- R-539 (operator ruling 3 of 2026-09-16): the SLOW crash loop ----------------------------------
|
||||||
|
|
||||||
|
// supKillOnce kills the controller and lets the supervisor restart it (two confirming sweeps), then
|
||||||
|
// moves the clock on by gap. The container comes back "running", so each restart is a separate act.
|
||||||
|
func supKillOnce(t *testing.T, s *Server, ex *supExec, clk *supClock, gap time.Duration) {
|
||||||
|
t.Helper()
|
||||||
|
before := ex.count(9201)
|
||||||
|
ex.mu.Lock()
|
||||||
|
ex.status[9201] = "exited"
|
||||||
|
ex.mu.Unlock()
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
clk.t = clk.t.Add(controllerSupervisorInterval)
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
if ex.count(9201) != before+1 {
|
||||||
|
t.Fatalf("kill was not followed by exactly one restart (restarts %d → %d)", before, ex.count(9201))
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(gap)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The consequence: a controller that dies every 20 minutes — never three times inside the 15-minute
|
||||||
|
// brake — raises slow_crashloop on the FIFTH restart in 24 hours, and the raise does not move again on
|
||||||
|
// the sixth (the hub mails on movement; once per 24 hours is the ruling).
|
||||||
|
//
|
||||||
|
// RED-PROOF: without the slow counter the stanza never sets slow_crashloop → "five restarts 20 minutes
|
||||||
|
// apart did not raise slow_crashloop — this is R-539".
|
||||||
|
func TestControllerSupervisor_SlowCrashloop(t *testing.T) {
|
||||||
|
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
|
||||||
|
s, clk, _ := supServer(t, ex, runningGuest(9201), 9201)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
for i := 1; i <= 4; i++ {
|
||||||
|
supKillOnce(t, s, ex, clk, 20*time.Minute)
|
||||||
|
}
|
||||||
|
g := s.ControllerSupervisorStatus(ctx).Guests[0]
|
||||||
|
if g.Crashloop {
|
||||||
|
t.Fatalf("the 15-minute brake fired on restarts 20 minutes apart — the fixture is wrong: %+v", g)
|
||||||
|
}
|
||||||
|
if g.SlowCrashloop || g.SlowCrashloopSince != "" {
|
||||||
|
t.Fatalf("slow_crashloop raised after only 4 restarts: %+v", g)
|
||||||
|
}
|
||||||
|
supKillOnce(t, s, ex, clk, 20*time.Minute)
|
||||||
|
g = s.ControllerSupervisorStatus(ctx).Guests[0]
|
||||||
|
if !g.SlowCrashloop || g.SlowCrashloopSince == "" || g.Restarts24h != 5 {
|
||||||
|
t.Fatalf("five restarts 20 minutes apart did not raise slow_crashloop — this is R-539: %+v", g)
|
||||||
|
}
|
||||||
|
first := g.SlowCrashloopSince
|
||||||
|
supKillOnce(t, s, ex, clk, 20*time.Minute)
|
||||||
|
g = s.ControllerSupervisorStatus(ctx).Guests[0]
|
||||||
|
if g.SlowCrashloopSince != first {
|
||||||
|
t.Fatalf("the raise moved again on the 6th restart (%q → %q) — the operator would be mailed per restart", first, g.SlowCrashloopSince)
|
||||||
|
}
|
||||||
|
if !g.SlowCrashloop {
|
||||||
|
t.Fatalf("slow_crashloop cleared while the loop continues: %+v", g)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The negative control: restarts that never reach five inside any 24 hours never raise it.
|
||||||
|
func TestControllerSupervisor_SpreadRestartsNeverSlowCrashloop(t *testing.T) {
|
||||||
|
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
|
||||||
|
s, clk, _ := supServer(t, ex, runningGuest(9201), 9201)
|
||||||
|
for i := 0; i < 8; i++ { // eight restarts, 7 hours apart: at most 4 inside any 24 hours
|
||||||
|
supKillOnce(t, s, ex, clk, 7*time.Hour)
|
||||||
|
}
|
||||||
|
g := s.ControllerSupervisorStatus(context.Background()).Guests[0]
|
||||||
|
if g.SlowCrashloop || g.SlowCrashloopSince != "" {
|
||||||
|
t.Fatalf("restarts 7 hours apart raised slow_crashloop: %+v", g)
|
||||||
|
}
|
||||||
|
if g.Restarts24h > 4 {
|
||||||
|
t.Fatalf("restarts_24h=%d — the 24-hour window is not pruning", g.Restarts24h)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An agent restart must not reset the slow counter (the ruling; a box whose AGENT also restarts would
|
||||||
|
// otherwise never reach five). The same state directory, a fresh Server.
|
||||||
|
//
|
||||||
|
// RED-PROOF: keep the counter in memory only → the second Server starts at 0 → "the agent restart
|
||||||
|
// reset the slow counter".
|
||||||
|
func TestControllerSupervisor_SlowCounterSurvivesAgentRestart(t *testing.T) {
|
||||||
|
ex := &supExec{status: map[int]string{9201: "running"}, onRestart: "running"}
|
||||||
|
s, clk, dir := supServer(t, ex, runningGuest(9201), 9201)
|
||||||
|
for i := 0; i < 4; i++ {
|
||||||
|
supKillOnce(t, s, ex, clk, 20*time.Minute)
|
||||||
|
}
|
||||||
|
s2 := &Server{
|
||||||
|
staleLock: s.staleLock,
|
||||||
|
guestExec: ex,
|
||||||
|
guestsDir: dir,
|
||||||
|
swapInFlight: map[int]bool{},
|
||||||
|
logger: s.logger,
|
||||||
|
now: clk.now,
|
||||||
|
}
|
||||||
|
supKillOnce(t, s2, ex, clk, 20*time.Minute)
|
||||||
|
g := s2.ControllerSupervisorStatus(context.Background()).Guests[0]
|
||||||
|
if g.Restarts24h != 5 || !g.SlowCrashloop {
|
||||||
|
t.Fatalf("the agent restart reset the slow counter: %+v", g)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user