Hub v0.149.0: a box the household switched off is not an outage for 14 days (R-998)
gates / gates (push) Successful in 6m40s
gates / gates (push) Successful in 6m40s
hosts.deliberate_off_at follows the agent's deliberate_off field (set, then cleared by the first report of a new boot; one info event each way). While held, every silence alarm is held: host and node staleness, the 05:00 backup deadlines, off-site and restore-test freshness, the OS alarms and the kernel notice. After 14 days the usual alarms fire once, naming the power-off. Pages show KIKAPCSOLVA and the time. Consequence tests with controls; each guard red-proofed.
This commit is contained in:
@@ -1,3 +1,26 @@
|
||||
## v0.149.0 — a box the household switched off is not an outage, for 14 days (R-998) (2026-10-13)
|
||||
|
||||
Needs **agent ≥ 0.155.0** to see anything: the agent's new `deliberate_off.requested_at` report field (household
|
||||
„A gép kikapcsolása", doc 14). Older agents never send it — nothing changes for them.
|
||||
|
||||
- **Ingest:** `hosts.deliberate_off_at` (new column) follows the field: set while the reports carry it, cleared by the
|
||||
first report without it (the box is in a new boot). One `info` event each way — `host_deliberate_off`,
|
||||
`host_deliberate_off_ended` — on the customer's list, never mailed.
|
||||
- **The hold (14 days, `store.DeliberateOffHold`):** while a silent box is held, NOTHING about its silence fires —
|
||||
`host_stale`/`host_down` (state `off`), `node_stale`/`node_down` (state `deliberately_off`, also read by the 05:00
|
||||
check), `expected_backup_missed`/`expected_dbdump_missed` (also skipped from the host row, so a box switched off at
|
||||
04:50 is covered), `offsite_stale` and `offsite_escrow_pending`, `restore_test_stale`, the OS alarms
|
||||
(`os_update_stale`, `agent_behind`, `os_config_bundle_behind`, `os_reboot_needed`, `os_not_covered`, `os_ring0_stalled`)
|
||||
and the household's kernel notice mail. **After the hold**, still silent: the usual `host_down` + `node_down`, once,
|
||||
with a text naming the household power-off. Back on inside the hold: no `*_recovered` mail.
|
||||
- **Pages:** hosts list, host detail and the customer/dashboard roll-up show **KIKAPCSOLVA** (grey) and „Kikapcsolva a
|
||||
háztartás által, T — riasztás szünetel eddig: T+14 nap" instead of DOWN. Delete logic still reads the raw status.
|
||||
- Tests (consequence, each with a control that must alarm): `monitor/deliberate_off_test.go` (no alarm in the hold;
|
||||
alarms after 14 days with the right text, once; back on is quiet; the 05:00 deadline skips),
|
||||
`api/deliberate_off_test.go` (set, one event, cleared), `web/deliberate_off_view_test.go`. Red-proofed: the hold
|
||||
removed from each checker, a hold that never ends, the deadline skip removed, the ingest unwired, an event per
|
||||
report, the hosts list on the raw status — each convicted.
|
||||
|
||||
## v0.148.0 — SECURITY: the customer list and box reports need the operator key (2026-10-12)
|
||||
|
||||
Security review 2026-10-12 (`documentation/audits/security-review-2026-10-12/`), R-951, P1.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// Event types of the household power-off (R-998, doc 14). Both are info: recorded on the customer's event list,
|
||||
// never mailed (the dispatcher mails warning and above).
|
||||
const (
|
||||
eventHostDeliberateOff = "host_deliberate_off"
|
||||
eventHostDeliberateOffEnded = "host_deliberate_off_ended"
|
||||
)
|
||||
|
||||
// checkDeliberateOff keeps the host row's deliberate-off time in step with the report: set while the report
|
||||
// carries the field, cleared by the first report without it (the box is in a new boot — switched back on).
|
||||
func (h *Handler) checkDeliberateOff(hostID, custID string, rep *hostReportPayload) {
|
||||
var at *time.Time
|
||||
if rep.DeliberateOff != nil && rep.DeliberateOff.RequestedAt != "" {
|
||||
t, err := time.Parse(time.RFC3339, rep.DeliberateOff.RequestedAt)
|
||||
if err != nil {
|
||||
h.logger.Printf("[WARN] host-report from %s: unreadable deliberate_off.requested_at %q — ignored", hostID, rep.DeliberateOff.RequestedAt)
|
||||
return
|
||||
}
|
||||
t = t.UTC()
|
||||
at = &t
|
||||
}
|
||||
prev, err := h.store.SetHostDeliberateOff(hostID, at)
|
||||
if err != nil {
|
||||
h.logger.Printf("[WARN] host-report from %s: deliberate-off not stored: %v", hostID, err)
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case at != nil && (prev == nil || !prev.Equal(*at)):
|
||||
msg := "Switched off by the household at " + at.Format("2006-01-02 15:04 UTC") +
|
||||
" — silence alarms held for " + holdDays() + " days"
|
||||
h.logger.Printf("[INFO] host %s: deliberate power-off recorded (%s)", hostID, at.Format(time.RFC3339))
|
||||
h.saveInfoEvent(custID, eventHostDeliberateOff, msg)
|
||||
case at == nil && prev != nil:
|
||||
msg := "Reporting again after the household's power-off of " + prev.Format("2006-01-02 15:04 UTC")
|
||||
h.logger.Printf("[INFO] host %s: deliberate power-off ended (was %s)", hostID, prev.Format(time.RFC3339))
|
||||
h.saveInfoEvent(custID, eventHostDeliberateOffEnded, msg)
|
||||
}
|
||||
}
|
||||
|
||||
func holdDays() string {
|
||||
d := int(store.DeliberateOffHold / (24 * time.Hour))
|
||||
return itoa(d)
|
||||
}
|
||||
|
||||
func itoa(n int) string {
|
||||
if n == 0 {
|
||||
return "0"
|
||||
}
|
||||
var b []byte
|
||||
for n > 0 {
|
||||
b = append([]byte{byte('0' + n%10)}, b...)
|
||||
n /= 10
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (h *Handler) saveInfoEvent(custID, eventType, msg string) {
|
||||
if _, err := h.store.SaveEvent(custID, eventType, "info", msg, "{}", "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] save %s event: %v", eventType, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-998: the agent's deliberate_off field sets the host row's hold; the next report without it (a new boot — the box
|
||||
// was switched back on) clears it. Each change leaves one info event on the customer's list; a repeat leaves none.
|
||||
func TestHostReport_DeliberateOffSetThenCleared(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "p"})
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
withOff := strings.TrimSuffix(validReportBody("h1"), "}") + `,"deliberate_off":{"requested_at":"2026-10-13T18:00:00Z"}}`
|
||||
|
||||
for i := 0; i < 2; i++ { // the field rides every report until the box is down; the second must not re-record
|
||||
if rr := do(h, http.MethodPost, "/host-report", "HKEY", withOff); rr.Code != 200 {
|
||||
t.Fatalf("report %d: %d %s", i, rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
at := st.HostDeliberateOffAt("h1")
|
||||
if at == nil || !at.Equal(time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("hold = %v, want 2026-10-13 18:00 UTC", at)
|
||||
}
|
||||
evs, _ := st.GetEventsByType("c1", eventHostDeliberateOff, time.Now().Add(-time.Hour))
|
||||
if len(evs) != 1 || evs[0].Severity != "info" {
|
||||
t.Fatalf("set events = %+v, want exactly one info event", evs)
|
||||
}
|
||||
|
||||
if rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")); rr.Code != 200 {
|
||||
t.Fatalf("plain report: %d", rr.Code)
|
||||
}
|
||||
if st.HostDeliberateOffAt("h1") != nil {
|
||||
t.Fatal("a report without the field must clear the hold")
|
||||
}
|
||||
ended, _ := st.GetEventsByType("c1", eventHostDeliberateOffEnded, time.Now().Add(-time.Hour))
|
||||
if len(ended) != 1 {
|
||||
t.Fatalf("ended events = %d, want 1", len(ended))
|
||||
}
|
||||
}
|
||||
@@ -744,6 +744,11 @@ type hostReportPayload struct {
|
||||
CollectedAt time.Time `json:"collected_at"`
|
||||
Lines []string `json:"lines"`
|
||||
} `json:"log_tail"`
|
||||
// DeliberateOff (R-998, agent >= 0.155.0): the household asked this box to power off, in its current boot.
|
||||
// Absent = not (or an older agent). deliberate_off.go turns it into the host row's hold.
|
||||
DeliberateOff *struct {
|
||||
RequestedAt string `json:"requested_at"`
|
||||
} `json:"deliberate_off"`
|
||||
}
|
||||
|
||||
// hostForeignKeyArchives mirrors the agent's hub.ForeignKeyArchives (R-366 slice 2).
|
||||
@@ -921,6 +926,7 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
h.checkTunnel(hostID, custID, rep.Cloudflared.Status, rep.Cloudflared.Detail)
|
||||
h.checkCrash(hostID, custID, body)
|
||||
h.checkDeliberateOff(hostID, custID, &rep)
|
||||
|
||||
for _, g := range rep.Guests {
|
||||
status := g.Status
|
||||
|
||||
@@ -430,7 +430,13 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
// to be quiet. That is R-195's shape exactly — a skip keyed off the wrong fact missing the
|
||||
// customer it would most obviously cover — which is why both doors are closed together.
|
||||
st := stalenessState(staleness, id)
|
||||
if st == StateDisabled {
|
||||
if st == StateDisabled || st == StateDeliberateOff { // R-998: a household power-off is not a missed backup
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
// R-998, read from the host row too: a box switched off minutes before 05:00 is not yet past the staleness
|
||||
// threshold, so its staleness state is still "ok" — the hold must not depend on that clock.
|
||||
if _, held := s.CustomerDeliberatelyOff(id, time.Now()); held {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package monitor
|
||||
|
||||
// R-998 (doc 14) — A BOX THE HOUSEHOLD SWITCHED OFF IS NOT A BOX THAT DIED, FOR 14 DAYS.
|
||||
//
|
||||
// The consequence is asserted, not the mechanism: does the alarm fire? Each "held" case runs beside a CONTROL
|
||||
// with the same silence and no power-off, which must alarm — a quiet result alone could mean a broken fixture.
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
type offRig struct {
|
||||
st *store.Store
|
||||
path string
|
||||
node *StalenessChecker
|
||||
host *HostStalenessChecker
|
||||
events []string
|
||||
msgs []string
|
||||
}
|
||||
|
||||
// newOffRig: one customer with one host, both reporting NOW, both checkers having seen them healthy.
|
||||
func newOffRig(t *testing.T) *offRig {
|
||||
t.Helper()
|
||||
st, path := seedStalenessCustomer(t, "ok", 0)
|
||||
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(`{}`), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := &offRig{st: st, path: path}
|
||||
on := func(cid, et, sev, msg, det, src string) {
|
||||
r.events = append(r.events, et)
|
||||
r.msgs = append(r.msgs, msg)
|
||||
}
|
||||
r.node = NewStalenessChecker(st, 30*time.Minute, on, log.New(io.Discard, "", 0))
|
||||
r.host = NewHostStalenessChecker(st, 30*time.Minute, on, log.New(io.Discard, "", 0))
|
||||
r.node.Check()
|
||||
r.host.Check()
|
||||
if len(r.events) != 0 {
|
||||
t.Fatalf("setup emitted %v", r.events)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// silent ages both the controller's and the agent's last report by d.
|
||||
func (r *offRig) silent(t *testing.T, d time.Duration) {
|
||||
t.Helper()
|
||||
db, err := sql.Open("sqlite", r.path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
at := time.Now().UTC().Add(-d).Format("2006-01-02 15:04:05")
|
||||
if _, err := db.Exec(`UPDATE reports SET received_at = ? WHERE customer_id = 'c1'`, at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := db.Exec(`UPDATE hosts SET last_report_at = ? WHERE host_id = 'h1'`, at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *offRig) offAt(t *testing.T, ago time.Duration) {
|
||||
t.Helper()
|
||||
at := time.Now().UTC().Add(-ago).Truncate(time.Second)
|
||||
if _, err := r.st.SetHostDeliberateOff("h1", &at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *offRig) check() { r.node.Check(); r.host.Check() }
|
||||
|
||||
func hasEvent(evs []string, want string) bool {
|
||||
for _, e := range evs {
|
||||
if e == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Silent for 3 hours: the control alarms node_down + host_down; the switched-off box raises nothing.
|
||||
func TestDeliberateOff_NoSilenceAlarmDuringTheHold(t *testing.T) {
|
||||
ctl := newOffRig(t)
|
||||
ctl.silent(t, 3*time.Hour)
|
||||
ctl.check()
|
||||
if !hasEvent(ctl.events, "node_down") || !hasEvent(ctl.events, "host_down") {
|
||||
t.Fatalf("CONTROL: a box silent 3 h must alarm, got %v — the fixture is broken", ctl.events)
|
||||
}
|
||||
|
||||
r := newOffRig(t)
|
||||
r.offAt(t, 3*time.Hour)
|
||||
r.silent(t, 3*time.Hour)
|
||||
r.check()
|
||||
r.check()
|
||||
if len(r.events) != 0 {
|
||||
t.Fatalf("a box the household switched off alarmed: %v", r.events)
|
||||
}
|
||||
if r.node.GetState("c1") != StateDeliberateOff || r.host.GetState("h1") != "off" {
|
||||
t.Fatalf("states node=%q host=%q, want %q / off", r.node.GetState("c1"), r.host.GetState("h1"), StateDeliberateOff)
|
||||
}
|
||||
}
|
||||
|
||||
// After the 14-day hold, still silent → the usual alarms fire, and their text names the power-off.
|
||||
func TestDeliberateOff_AlarmsAfterFourteenDays(t *testing.T) {
|
||||
r := newOffRig(t)
|
||||
r.offAt(t, 13*24*time.Hour)
|
||||
r.silent(t, 13*24*time.Hour)
|
||||
r.check()
|
||||
if len(r.events) != 0 {
|
||||
t.Fatalf("day 13: %v", r.events)
|
||||
}
|
||||
r.offAt(t, 14*24*time.Hour+time.Minute) // the hold ran out
|
||||
r.silent(t, 14*24*time.Hour+time.Minute)
|
||||
r.check()
|
||||
if !hasEvent(r.events, "node_down") || !hasEvent(r.events, "host_down") {
|
||||
t.Fatalf("after 14 days: events %v, want node_down + host_down", r.events)
|
||||
}
|
||||
for _, m := range r.msgs {
|
||||
if !strings.Contains(m, "household") {
|
||||
t.Fatalf("the alarm does not say the household switched it off: %q", m)
|
||||
}
|
||||
}
|
||||
r.check() // once, not every minute
|
||||
if len(r.events) != 2 {
|
||||
t.Fatalf("repeated: %v", r.events)
|
||||
}
|
||||
}
|
||||
|
||||
// Switched back on inside the hold: reports resume → no recovery mail for an outage that never was.
|
||||
func TestDeliberateOff_BackOnIsQuiet(t *testing.T) {
|
||||
r := newOffRig(t)
|
||||
r.offAt(t, 2*time.Hour)
|
||||
r.silent(t, 2*time.Hour)
|
||||
r.check()
|
||||
r.silent(t, 0)
|
||||
if _, err := r.st.SetHostDeliberateOff("h1", nil); err != nil { // the new boot's first report
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.check()
|
||||
if len(r.events) != 0 {
|
||||
t.Fatalf("back on: %v", r.events)
|
||||
}
|
||||
if r.node.GetState("c1") != "ok" || r.host.GetState("h1") != "ok" {
|
||||
t.Fatalf("states node=%q host=%q, want ok", r.node.GetState("c1"), r.host.GetState("h1"))
|
||||
}
|
||||
}
|
||||
|
||||
// The 05:00 deadline: Tester 2's shape (R-872) alarms; the same box switched off by the household does not.
|
||||
func TestDeliberateOff_DeadlineSkips(t *testing.T) {
|
||||
st, _, sc := downBox(t, 5*24*time.Hour)
|
||||
if got := deadlineEvents(st, sc); got["expected_backup_missed"] != 1 {
|
||||
t.Fatalf("CONTROL: %v — the down box must raise its missed-backup alarm", got)
|
||||
}
|
||||
st2, _, sc2 := downBox(t, 5*24*time.Hour)
|
||||
at := time.Now().UTC().Add(-2 * time.Hour)
|
||||
if _, err := st2.SetHostDeliberateOff("h1", &at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := deadlineEvents(st2, sc2); len(got) != 0 {
|
||||
t.Fatalf("a switched-off box got deadline alarms: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,9 @@ func (sc *HostStalenessChecker) Check() {
|
||||
age := time.Since(row.LastReportAt)
|
||||
var newState string
|
||||
switch {
|
||||
case age > sc.threshold && store.DeliberateOffHeld(row.DeliberateOffAt, time.Now()):
|
||||
// R-998: the household switched the box off on purpose — silent, for the 14-day hold.
|
||||
newState = "off"
|
||||
case age > sc.downAfter:
|
||||
newState = "down"
|
||||
case age > sc.threshold:
|
||||
@@ -105,6 +108,20 @@ func (sc *HostStalenessChecker) Check() {
|
||||
}
|
||||
|
||||
oldState := sc.states[row.HostID]
|
||||
if newState == "off" || oldState == "off" {
|
||||
// Into the hold, or out of it back to ok: no event (the report ingest records the info events).
|
||||
// Out of it into stale/down (the hold ran out with the box still silent): the usual alarm, with a
|
||||
// message that says why it comes now.
|
||||
sc.states[row.HostID] = newState
|
||||
if newState == "stale" || newState == "down" {
|
||||
sc.downtimeStart[row.HostID] = time.Now()
|
||||
sc.emitHoldEnded(row, newState, age)
|
||||
} else if newState == "off" && oldState != "off" {
|
||||
sc.logger.Printf("[INFO] Host staleness: %s deliberately off since %s — alarms held until %s",
|
||||
row.HostID, row.DeliberateOffAt.Format(time.RFC3339), row.DeliberateOffAt.Add(store.DeliberateOffHold).Format(time.RFC3339))
|
||||
}
|
||||
continue
|
||||
}
|
||||
if oldState == "" {
|
||||
// F2: first observation (incl. a hub restart). Healthy seeds silently; a host already
|
||||
// stale/down at first sight emits once (else a box that went silent while the hub was
|
||||
@@ -181,3 +198,23 @@ func (sc *HostStalenessChecker) emitTransition(hostID, customerID, oldState, new
|
||||
sc.onEvent(customerID, eventType, severity, message, "{}", "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// emitHoldEnded raises the alarm a deliberate-off hold was holding (R-998): the box was switched off by the household
|
||||
// and is still silent when the hold runs out. The text names the power-off — „no report for 14d" alone reads as an
|
||||
// outage nobody saw coming.
|
||||
func (sc *HostStalenessChecker) emitHoldEnded(row store.HostStaleRow, newState string, age time.Duration) {
|
||||
eventType, severity := "host_stale", "warning"
|
||||
if newState == "down" {
|
||||
eventType, severity = "host_down", "error"
|
||||
}
|
||||
message := "Host " + row.HostID + ": switched off by the household on " + row.DeliberateOffAt.UTC().Format("2006-01-02 15:04 UTC") +
|
||||
" and still silent after the " + formatDuration(store.DeliberateOffHold) + " hold (no report for " + formatDuration(age) + ")"
|
||||
sc.logger.Printf("[INFO] Host staleness: %s off → %s (%s, the deliberate-off hold ended)", row.HostID, newState, eventType)
|
||||
if _, err := sc.store.SaveEvent(row.CustomerID, eventType, severity, message, "{}", "hub"); err != nil {
|
||||
sc.logger.Printf("[WARN] Failed to save host staleness event for %s: %v", row.HostID, err)
|
||||
return
|
||||
}
|
||||
if sc.onEvent != nil {
|
||||
sc.onEvent(row.CustomerID, eventType, severity, message, "{}", "hub")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -468,6 +468,11 @@ func (oc *OffsiteChecker) Check() {
|
||||
delete(oc.usedWindows, c.CustomerID)
|
||||
continue
|
||||
}
|
||||
// R-998: a box the household switched off has no fresh off-site copy by design — held for the 14 days.
|
||||
// States are kept, so the edge does not re-fire when it comes back.
|
||||
if _, held := oc.store.CustomerDeliberatelyOff(c.CustomerID, time.Now()); held {
|
||||
continue
|
||||
}
|
||||
if oc.store.IsCustomerBlocked(c.CustomerID) {
|
||||
delete(oc.fillStates, c.CustomerID)
|
||||
delete(oc.staleStates, c.CustomerID)
|
||||
|
||||
@@ -188,6 +188,9 @@ func (c *RestoreTestChecker) Check() {
|
||||
}
|
||||
now := c.now()
|
||||
for _, id := range ids {
|
||||
if _, held := c.store.CustomerDeliberatelyOff(id, now); held {
|
||||
continue // R-998: switched off by the household — no restore test can run; held for the 14 days
|
||||
}
|
||||
latest, rerr := c.store.GetLatestHostReportJSON(id)
|
||||
if rerr != nil || latest == "" {
|
||||
continue // no agent host-report → nothing to judge (the R-81 legacy-shape branch)
|
||||
|
||||
@@ -35,6 +35,12 @@ type EventNotifyFunc func(customerID, eventType, severity, message, detailsJSON,
|
||||
// second door.
|
||||
const StateDisabled = "disabled"
|
||||
|
||||
// StateDeliberateOff (R-998, doc 14) is the state of a customer whose box the HOUSEHOLD switched off with
|
||||
// „A gép kikapcsolása" — the host row carries the time (store.DeliberateOffHeld). Like StateDisabled it is a
|
||||
// recorded state, so CheckBackupDeadlines can skip it; unlike StateDisabled it ENDS: after the 14-day hold the
|
||||
// usual stale/down alarm fires, with a message naming the power-off. Pinned by deliberate_off_test.go.
|
||||
const StateDeliberateOff = "deliberately_off"
|
||||
|
||||
// StalenessChecker monitors customer report freshness and generates
|
||||
// node_stale / node_down / node_recovered events on state transitions.
|
||||
type StalenessChecker struct {
|
||||
@@ -139,7 +145,10 @@ func (sc *StalenessChecker) Check() {
|
||||
|
||||
age := time.Since(c.ReceivedAt)
|
||||
var newState string
|
||||
_, offHeld := sc.store.CustomerDeliberatelyOff(c.CustomerID, time.Now())
|
||||
switch {
|
||||
case age > sc.threshold && offHeld:
|
||||
newState = StateDeliberateOff
|
||||
case age > sc.downAfter:
|
||||
newState = "down"
|
||||
case age > sc.threshold:
|
||||
@@ -149,6 +158,22 @@ func (sc *StalenessChecker) Check() {
|
||||
}
|
||||
|
||||
oldState := sc.states[c.CustomerID]
|
||||
// R-998: into the hold, or back to ok from it — silent. Out of it into stale/down — the usual alarm,
|
||||
// with the power-off named in the text.
|
||||
if newState == StateDeliberateOff || (oldState == StateDeliberateOff && newState == "ok") {
|
||||
if oldState != newState {
|
||||
sc.logger.Printf("[INFO] Staleness: %s %s → %s (household power-off, no event)", c.CustomerID, oldState, newState)
|
||||
}
|
||||
sc.states[c.CustomerID] = newState
|
||||
delete(sc.downtimeStart, c.CustomerID)
|
||||
continue
|
||||
}
|
||||
if oldState == StateDeliberateOff {
|
||||
sc.states[c.CustomerID] = newState
|
||||
sc.downtimeStart[c.CustomerID] = time.Now()
|
||||
sc.emitHoldEnded(c.CustomerID, newState, age)
|
||||
continue
|
||||
}
|
||||
if oldState == "" || oldState == StateDisabled {
|
||||
// New customer — set state without event.
|
||||
//
|
||||
@@ -275,3 +300,21 @@ func (sc *StalenessChecker) isNewBox(customerID string) bool {
|
||||
}
|
||||
return time.Since(h.CreatedAt) < sc.threshold
|
||||
}
|
||||
|
||||
// emitHoldEnded raises the node alarm a household power-off was holding, once the 14-day hold ran out (R-998).
|
||||
func (sc *StalenessChecker) emitHoldEnded(customerID, newState string, age time.Duration) {
|
||||
eventType, severity := "node_stale", "warning"
|
||||
if newState == "down" {
|
||||
eventType, severity = "node_down", "error"
|
||||
}
|
||||
message := "No report received for " + formatDuration(age) + " — the box was switched off by the household and the " +
|
||||
formatDuration(store.DeliberateOffHold) + " hold has ended"
|
||||
sc.logger.Printf("[INFO] Staleness: %s %s → %s (%s, the deliberate-off hold ended)", customerID, StateDeliberateOff, newState, eventType)
|
||||
if _, err := sc.store.SaveEvent(customerID, eventType, severity, message, "{}", "hub"); err != nil {
|
||||
sc.logger.Printf("[WARN] Failed to save staleness event for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
if sc.onEvent != nil {
|
||||
sc.onEvent(customerID, eventType, severity, message, "{}", "hub")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package osupdates
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// notDeliberatelyOff drops the hosts the household switched off with „A gép kikapcsolása" (R-998, doc 14) while the
|
||||
// 14-day hold lasts: a box that is off cannot run its OS leg, take a new agent or bundle, or restart for a kernel —
|
||||
// alarming about it, or mailing „tonight your box restarts", would be noise. After the hold they count again.
|
||||
func notDeliberatelyOff(st *store.Store, hosts []store.Host, now time.Time) []store.Host {
|
||||
out := hosts[:0:0]
|
||||
for _, h := range hosts {
|
||||
if store.DeliberateOffHeld(st.HostDeliberateOffAt(h.HostID), now) {
|
||||
continue
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -198,6 +198,7 @@ func (s *Service) KernelNotify() []string {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
hosts = notDeliberatelyOff(s.Store, hosts, now) // R-998: no „tonight your box restarts" mail to a box that is off
|
||||
var told []string
|
||||
for _, h := range hosts {
|
||||
kver, _ := s.KernelDue(h.HostID)
|
||||
|
||||
@@ -1059,6 +1059,7 @@ func (s *Service) Alarms() ([]string, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hosts = notDeliberatelyOff(s.Store, hosts, now) // R-998: a switched-off box cannot update — held for the 14 days
|
||||
var sent []string
|
||||
for _, h := range hosts {
|
||||
st := s.Store.GetOSHostSettings(h.HostID)
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── The household's deliberate power-off (R-998, documentation/architecture/14-host-power-off.md) ──────
|
||||
//
|
||||
// The agent reports `deliberate_off.requested_at` from the moment the household pressed „A gép kikapcsolása"
|
||||
// until the box goes down; the next boot's reports do not carry it. The hub keeps the time on the host row.
|
||||
// While it is set and younger than DeliberateOffHold, every silence alarm about that box is held: host and
|
||||
// controller staleness, the backup deadlines, off-site and restore-test freshness, the OS alarms and the
|
||||
// kernel notice. After the hold, the alarms fire as usual — a box nobody switched back on in two weeks is
|
||||
// worth a look. Pinned by monitor/deliberate_off_test.go, api/deliberate_off_test.go and web/deliberate_off_view_test.go.
|
||||
|
||||
// DeliberateOffHold is how long a deliberate power-off silences the box's alarms (operator default, 14 days).
|
||||
const DeliberateOffHold = 14 * 24 * time.Hour
|
||||
|
||||
// DeliberateOffHeld reports whether a deliberate-off time still holds the alarms at now.
|
||||
func DeliberateOffHeld(at *time.Time, now time.Time) bool {
|
||||
return at != nil && !at.IsZero() && now.Sub(*at) < DeliberateOffHold
|
||||
}
|
||||
|
||||
func parseDeliberateOff(v string) *time.Time {
|
||||
if v == "" {
|
||||
return nil
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339, v)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return &t
|
||||
}
|
||||
|
||||
// SetHostDeliberateOff records the report's deliberate-off field. at == nil clears it (a report without the
|
||||
// field: the box is in a new boot). It returns the previous value, so the caller can emit an event on a change.
|
||||
func (s *Store) SetHostDeliberateOff(hostID string, at *time.Time) (prev *time.Time, err error) {
|
||||
var old sql.NullString
|
||||
if err := s.db.QueryRow(`SELECT deliberate_off_at FROM hosts WHERE host_id = ?`, hostID).Scan(&old); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
prev = parseDeliberateOff(old.String)
|
||||
var val any
|
||||
if at != nil {
|
||||
val = at.UTC().Format(time.RFC3339)
|
||||
}
|
||||
if (prev == nil && at == nil) || (prev != nil && at != nil && prev.Equal(*at)) {
|
||||
return prev, nil
|
||||
}
|
||||
_, err = s.db.Exec(`UPDATE hosts SET deliberate_off_at = ?, updated_at = datetime('now') WHERE host_id = ?`, val, hostID)
|
||||
return prev, err
|
||||
}
|
||||
|
||||
// HostDeliberateOffAt returns the host's deliberate-off time (nil = none).
|
||||
func (s *Store) HostDeliberateOffAt(hostID string) *time.Time {
|
||||
var v sql.NullString
|
||||
if err := s.db.QueryRow(`SELECT deliberate_off_at FROM hosts WHERE host_id = ?`, hostID).Scan(&v); err != nil {
|
||||
return nil
|
||||
}
|
||||
return parseDeliberateOff(v.String)
|
||||
}
|
||||
|
||||
// CustomerDeliberatelyOff reports whether any of the customer's hosts is deliberately off and still within the
|
||||
// hold at now — the checkers keyed by customer use it. The time is the newest such request.
|
||||
func (s *Store) CustomerDeliberatelyOff(customerID string, now time.Time) (time.Time, bool) {
|
||||
rows, err := s.db.Query(`SELECT deliberate_off_at FROM hosts WHERE customer_id = ? AND deliberate_off_at IS NOT NULL`, customerID)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
defer rows.Close()
|
||||
var newest time.Time
|
||||
held := false
|
||||
for rows.Next() {
|
||||
var v string
|
||||
if rows.Scan(&v) != nil {
|
||||
continue
|
||||
}
|
||||
if t := parseDeliberateOff(v); DeliberateOffHeld(t, now) {
|
||||
held = true
|
||||
if t.After(newest) {
|
||||
newest = *t
|
||||
}
|
||||
}
|
||||
}
|
||||
return newest, held
|
||||
}
|
||||
@@ -471,6 +471,9 @@ func (s *Store) migrate() error {
|
||||
// expected key fingerprint, tunnel id) — the hub serves these only in recovery mode; no usable
|
||||
// secret is hub-held (the blobs need R, which the hub never has).
|
||||
s.db.Exec(`ALTER TABLE hosts ADD COLUMN recovery_mode_until DATETIME`)
|
||||
// R-998 (doc 14): the household switched the box off on purpose at this time (RFC3339 UTC, from the agent's
|
||||
// deliberate_off report field). NULL = not deliberately off. deliberateoff.go reads and writes it.
|
||||
s.db.Exec(`ALTER TABLE hosts ADD COLUMN deliberate_off_at TEXT`)
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
||||
|
||||
@@ -2823,6 +2826,7 @@ type Host struct {
|
||||
DesiredJSON string
|
||||
DesiredGeneration int64
|
||||
RecoveryModeUntil *time.Time // slice 10D: recovery mode active until this time (nil/past = off)
|
||||
DeliberateOffAt *time.Time // R-998: switched off by the household at this time (nil = not); deliberateoff.go
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
@@ -2866,6 +2870,8 @@ type HostStaleRow struct {
|
||||
HostID string
|
||||
CustomerID string
|
||||
LastReportAt time.Time
|
||||
// DeliberateOffAt (R-998): when the household switched the box off on purpose; nil = not.
|
||||
DeliberateOffAt *time.Time
|
||||
}
|
||||
|
||||
// GuestID derives the interim guest primary key from host + vmid. The hub owns the
|
||||
@@ -2890,10 +2896,10 @@ func (s *Store) scanHost(scan func(dest ...any) error) (*Host, error) {
|
||||
|
||||
func scanHostRaw(scan func(dest ...any) error) (*Host, error) {
|
||||
var h Host
|
||||
var lastReport, recoveryUntil sql.NullString
|
||||
var lastReport, recoveryUntil, deliberateOff sql.NullString
|
||||
var createdAt, updatedAt string
|
||||
err := scan(&h.HostID, &h.CustomerID, &h.APIKey, &h.AgentVersion, &lastReport,
|
||||
&h.DesiredJSON, &h.DesiredGeneration, &recoveryUntil, &createdAt, &updatedAt)
|
||||
&h.DesiredJSON, &h.DesiredGeneration, &recoveryUntil, &createdAt, &updatedAt, &deliberateOff)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -2905,13 +2911,14 @@ func scanHostRaw(scan func(dest ...any) error) (*Host, error) {
|
||||
t := parseSQLiteTime(recoveryUntil.String)
|
||||
h.RecoveryModeUntil = &t
|
||||
}
|
||||
h.DeliberateOffAt = parseDeliberateOff(deliberateOff.String)
|
||||
h.CreatedAt = parseSQLiteTime(createdAt)
|
||||
h.UpdatedAt = parseSQLiteTime(updatedAt)
|
||||
return &h, nil
|
||||
}
|
||||
|
||||
const hostSelectCols = `host_id, customer_id, api_key, agent_version, last_report_at,
|
||||
desired_json, desired_generation, recovery_mode_until, created_at, updated_at`
|
||||
desired_json, desired_generation, recovery_mode_until, created_at, updated_at, deliberate_off_at`
|
||||
|
||||
// GetHostByAPIKey looks up a host by its per-host hub key. Returns nil (no error)
|
||||
// if no match — parallels GetCustomerConfigByAPIKey.
|
||||
@@ -3847,7 +3854,7 @@ func (s *Store) ListGuestsForHost(hostID string) ([]Guest, error) {
|
||||
// have never reported (NULL last_report_at) are skipped — a freshly-minted host is
|
||||
// not "down" until it has checked in at least once.
|
||||
func (s *Store) GetHostStaleness() ([]HostStaleRow, error) {
|
||||
rows, err := s.db.Query(`SELECT host_id, customer_id, last_report_at FROM hosts WHERE last_report_at IS NOT NULL`)
|
||||
rows, err := s.db.Query(`SELECT host_id, customer_id, last_report_at, COALESCE(deliberate_off_at, '') FROM hosts WHERE last_report_at IS NOT NULL`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -3855,11 +3862,12 @@ func (s *Store) GetHostStaleness() ([]HostStaleRow, error) {
|
||||
var out []HostStaleRow
|
||||
for rows.Next() {
|
||||
var r HostStaleRow
|
||||
var last string
|
||||
if err := rows.Scan(&r.HostID, &r.CustomerID, &last); err != nil {
|
||||
var last, off string
|
||||
if err := rows.Scan(&r.HostID, &r.CustomerID, &last, &off); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.LastReportAt = parseSQLiteTime(last)
|
||||
r.DeliberateOffAt = parseDeliberateOff(off)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-998: a box the household switched off reads KIKAPCSOLVA with the power-off line on the hosts page — not DOWN —
|
||||
// exactly while the alarms are held; after the 14-day hold it reads DOWN again.
|
||||
func TestHostsPage_DeliberatelyOffBox(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "t.db")
|
||||
st, err := store.New(path, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
s := New(st, "", "", "test", 30*time.Minute, log.New(io.Discard, "", 0))
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"})
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"})
|
||||
st.SaveHostReport("h1", "c1", []byte(`{}`), store.HostReportDenorm{})
|
||||
age := func(d time.Duration) {
|
||||
db, _ := sql.Open("sqlite", path)
|
||||
defer db.Close()
|
||||
db.Exec(`UPDATE hosts SET last_report_at = ? WHERE host_id = 'h1'`, time.Now().UTC().Add(-d).Format("2006-01-02 15:04:05"))
|
||||
}
|
||||
page := func() string {
|
||||
rr := httptest.NewRecorder()
|
||||
s.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/hosts", nil))
|
||||
if rr.Code != 200 {
|
||||
t.Fatalf("/hosts: %d", rr.Code)
|
||||
}
|
||||
return rr.Body.String()
|
||||
}
|
||||
|
||||
age(3 * time.Hour)
|
||||
if p := page(); !strings.Contains(p, ">DOWN<") || strings.Contains(p, "data-deliberate-off") {
|
||||
t.Fatal("CONTROL: a silent box without a power-off must read DOWN")
|
||||
}
|
||||
at := time.Now().UTC().Add(-3 * time.Hour).Truncate(time.Second)
|
||||
st.SetHostDeliberateOff("h1", &at)
|
||||
p := page()
|
||||
if !strings.Contains(p, ">KIKAPCSOLVA<") || strings.Contains(p, ">DOWN<") {
|
||||
t.Fatal("a box switched off by the household must read KIKAPCSOLVA, not DOWN")
|
||||
}
|
||||
if !strings.Contains(p, "Kikapcsolva a háztartás által, "+at.Format("2006-01-02 15:04")) {
|
||||
t.Fatal("the hosts page does not name the household power-off and its time")
|
||||
}
|
||||
old := time.Now().UTC().Add(-15 * 24 * time.Hour)
|
||||
st.SetHostDeliberateOff("h1", &old)
|
||||
age(15 * 24 * time.Hour)
|
||||
if p := page(); !strings.Contains(p, ">DOWN<") {
|
||||
t.Fatal("after the 14-day hold the box must read DOWN again")
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,26 @@ func (s *Server) hostStatus(lastReport *time.Time) string {
|
||||
}
|
||||
}
|
||||
|
||||
// hostDisplayStatus is hostStatus for the PAGES (R-998): a silent host the household switched off on purpose, inside
|
||||
// the 14-day hold, reads "off" — not stale/down — exactly while the checkers hold its alarms (the same
|
||||
// store.DeliberateOffHeld). Delete logic keeps reading hostStatus. Pinned by deliberate_off_view_test.go.
|
||||
func (s *Server) hostDisplayStatus(h *store.Host) string {
|
||||
st := s.hostStatus(h.LastReportAt)
|
||||
if (st == "stale" || st == "down") && store.DeliberateOffHeld(h.DeliberateOffAt, time.Now()) {
|
||||
return "off"
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// deliberateOffLine is the operator line for a household power-off ("" when none is held).
|
||||
func deliberateOffLine(h *store.Host) string {
|
||||
if h == nil || !store.DeliberateOffHeld(h.DeliberateOffAt, time.Now()) {
|
||||
return ""
|
||||
}
|
||||
return "Kikapcsolva a háztartás által, " + h.DeliberateOffAt.UTC().Format("2006-01-02 15:04") + " UTC — riasztás szünetel eddig: " +
|
||||
h.DeliberateOffAt.Add(store.DeliberateOffHold).UTC().Format("2006-01-02 15:04") + " UTC"
|
||||
}
|
||||
|
||||
// hostStatusClass maps the internal status to the existing status-badge-* CSS class.
|
||||
// "stale" reuses the amber -warn class (there is no dedicated -stale class), keeping the
|
||||
// styling consistent with the rest of the console.
|
||||
@@ -55,6 +75,8 @@ func hostStatusClass(status string) string {
|
||||
return "status-badge-warn"
|
||||
case "down":
|
||||
return "status-badge-down"
|
||||
case "off":
|
||||
return "status-badge-pending" // grey: deliberate, not an incident (R-998)
|
||||
default:
|
||||
return "status-badge-pending"
|
||||
}
|
||||
@@ -69,6 +91,8 @@ func hostStatusLabel(status string) string {
|
||||
return "STALE"
|
||||
case "down":
|
||||
return "DOWN"
|
||||
case "off":
|
||||
return "KIKAPCSOLVA" // R-998: switched off by the household
|
||||
default:
|
||||
return "NO REPORT"
|
||||
}
|
||||
@@ -511,8 +535,9 @@ type hostListRow struct {
|
||||
CustomerID string
|
||||
CustomerName string
|
||||
AgentVersion string
|
||||
Status string // ok | stale | down | pending
|
||||
Status string // ok | stale | down | pending | off (R-998)
|
||||
StatusLabel string
|
||||
OffLine string // R-998: the household power-off line while the hold lasts
|
||||
StatusClass string
|
||||
LastReportAt *time.Time
|
||||
HasReport bool
|
||||
@@ -563,8 +588,9 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
rows := make([]hostListRow, 0, len(hosts))
|
||||
for _, h := range hosts {
|
||||
status := s.hostStatus(h.LastReportAt)
|
||||
status := s.hostDisplayStatus(&h)
|
||||
row := hostListRow{
|
||||
OffLine: deliberateOffLine(&h),
|
||||
HostID: h.HostID,
|
||||
CustomerID: h.CustomerID,
|
||||
CustomerName: s.customerName(h.CustomerID),
|
||||
@@ -826,8 +852,9 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
|
||||
"AgentVersion": host.AgentVersion,
|
||||
"CreatedAt": host.CreatedAt,
|
||||
"Status": status,
|
||||
"StatusLabel": hostStatusLabel(status),
|
||||
"StatusClass": hostStatusClass(status),
|
||||
"StatusLabel": hostStatusLabel(s.hostDisplayStatus(host)),
|
||||
"StatusClass": hostStatusClass(s.hostDisplayStatus(host)),
|
||||
"OffLine": deliberateOffLine(host),
|
||||
"LastReportAt": host.LastReportAt,
|
||||
"HasReport": host.LastReportAt != nil,
|
||||
"RecoveryMode": host.InRecoveryMode(time.Now()),
|
||||
|
||||
@@ -66,6 +66,10 @@ func (s *Server) foldHostStatus(customerID, base string, customerHasReported boo
|
||||
}
|
||||
worst, worstHost := "", ""
|
||||
for i := range hosts {
|
||||
// R-998: a host the household switched off (silent, inside the hold) is the customer's state, not an outage.
|
||||
if s.hostDisplayStatus(&hosts[i]) == "off" {
|
||||
return "off", deliberateOffLine(&hosts[i])
|
||||
}
|
||||
hs := s.hostStatus(hosts[i].LastReportAt)
|
||||
if hs == "pending" && !customerHasReported {
|
||||
continue
|
||||
|
||||
@@ -1154,7 +1154,7 @@ func statusColor(status string) string {
|
||||
return "crit"
|
||||
case "blocked":
|
||||
return "warn"
|
||||
case "disabled", "pending":
|
||||
case "disabled", "pending", "off":
|
||||
return "neutral"
|
||||
default:
|
||||
return "neutral"
|
||||
|
||||
@@ -65,7 +65,7 @@
|
||||
<span class="status-badge status-badge-{{.OverallStatus}}">
|
||||
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "down"}}DOWN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}{{.OverallStatus}}{{end}}
|
||||
</span>
|
||||
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
|
||||
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
|
||||
{{else}}
|
||||
<span class="text-muted">—</span>
|
||||
{{end}}
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
<h1>
|
||||
<span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span>
|
||||
{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}
|
||||
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
|
||||
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
|
||||
</h1>
|
||||
{{if .HasReports}}
|
||||
<p class="subtitle">
|
||||
@@ -107,7 +107,7 @@
|
||||
<div class="summary-strip">
|
||||
<span class="strip-name">{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</span>
|
||||
<span class="strip-item"><span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span> {{.OverallStatus}}</span>
|
||||
{{if .HostCause}}<span class="strip-item"><span class="status-badge status-badge-warn">{{.HostCause}}</span></span>{{end}}
|
||||
{{if .HostCause}}<span class="strip-item"><span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span></span>{{end}}
|
||||
{{if .HasReports}}
|
||||
<span class="strip-item">Controller <code>{{.Customer.ControllerVersion}}</code></span>
|
||||
<span class="strip-item">Last report {{timeAgo .Customer.ReceivedAt}}</span>
|
||||
|
||||
@@ -75,7 +75,7 @@
|
||||
<span class="status-badge status-badge-{{.OverallStatus}}">
|
||||
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}DOWN{{end}}
|
||||
</span>
|
||||
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
|
||||
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
|
||||
</td>
|
||||
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{if gt (add (add .EventCriticals .EventErrors) .EventWarnings) 0}}{{if gt .EventCriticals 0}}<span class="severity-badge severity-critical">{{.EventCriticals}}</span>{{end}}{{if gt .EventErrors 0}}<span class="severity-badge severity-error">{{.EventErrors}}</span>{{end}}{{if gt .EventWarnings 0}}<span class="severity-badge severity-warning">{{.EventWarnings}}</span>{{end}}{{else}}<span class="text-muted">—</span>{{end}}{{end}}</td>
|
||||
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}</td>
|
||||
|
||||
@@ -52,6 +52,10 @@
|
||||
<span class="label">Last Report</span>
|
||||
<span class="value">{{if .HasReport}}{{timeAgoPtr .LastReportAt}}{{else}}waiting for first report{{end}}</span>
|
||||
</div>
|
||||
{{if .OffLine}}<div class="info-item">
|
||||
<span class="label">Power</span>
|
||||
<span class="value" data-deliberate-off>{{.OffLine}}</span>
|
||||
</div>{{end}}
|
||||
<!-- R-30 slice 1: the box's wait-channel presence (in memory; ~4-min cadence), beside the
|
||||
15-min report clock. Every branch pinned by r30_presence_delete_test.go. -->
|
||||
<div class="info-item">
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
<td>{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</td>
|
||||
<td>{{if .AgentVersion}}<code>{{.AgentVersion}}</code>{{else}}—{{end}}{{if .FloorHeld}} <span class="status-badge status-warn" title="{{.HeldReason}}">floor held</span>{{else if eq .FloorSource "declared"}} <span class="status-badge" title="served above the vouched golden: the agent requirement was declared with the floor (R-472)">floor: declared MinAgent</span>{{end}}</td>
|
||||
<td style="font-size: 0.85em;">{{.PVEVersion}}<br><span class="text-muted">{{.KernelRunning}}</span></td>
|
||||
<td><span class="status-badge {{.StatusClass}}">{{.StatusLabel}}</span></td>
|
||||
<td><span class="status-badge {{.StatusClass}}">{{.StatusLabel}}</span>{{if .OffLine}}<br><span class="text-muted" style="font-size: 0.85em;" data-deliberate-off>{{.OffLine}}</span>{{end}}</td>
|
||||
<td>{{if .HasReport}}{{.GuestRunning}}/{{.GuestTotal}}{{else}}—{{end}}</td>
|
||||
<td>{{if .HasReport}}{{formatFloat .Vitals.CPUPercent}}%{{else}}—{{end}}</td>
|
||||
<td>{{if .HasReport}}{{formatFloat .Vitals.MemoryPercent}}%{{else}}—{{end}}</td>
|
||||
|
||||
Reference in New Issue
Block a user