Hub v0.149.0: a box the household switched off is not an outage for 14 days (R-998)
gates / gates (push) Successful in 6m40s

hosts.deliberate_off_at follows the agent's deliberate_off field (set, then cleared by the first
report of a new boot; one info event each way). While held, every silence alarm is held: host and
node staleness, the 05:00 backup deadlines, off-site and restore-test freshness, the OS alarms and
the kernel notice. After 14 days the usual alarms fire once, naming the power-off. Pages show
KIKAPCSOLVA and the time. Consequence tests with controls; each guard red-proofed.
This commit is contained in:
2026-10-13 19:47:02 +02:00
parent ed59bf3cbd
commit 50e47e29af
24 changed files with 638 additions and 17 deletions
+23
View File
@@ -1,3 +1,26 @@
## v0.149.0 — a box the household switched off is not an outage, for 14 days (R-998) (2026-10-13)
Needs **agent ≥ 0.155.0** to see anything: the agent's new `deliberate_off.requested_at` report field (household
„A gép kikapcsolása", doc 14). Older agents never send it — nothing changes for them.
- **Ingest:** `hosts.deliberate_off_at` (new column) follows the field: set while the reports carry it, cleared by the
first report without it (the box is in a new boot). One `info` event each way — `host_deliberate_off`,
`host_deliberate_off_ended` — on the customer's list, never mailed.
- **The hold (14 days, `store.DeliberateOffHold`):** while a silent box is held, NOTHING about its silence fires —
`host_stale`/`host_down` (state `off`), `node_stale`/`node_down` (state `deliberately_off`, also read by the 05:00
check), `expected_backup_missed`/`expected_dbdump_missed` (also skipped from the host row, so a box switched off at
04:50 is covered), `offsite_stale` and `offsite_escrow_pending`, `restore_test_stale`, the OS alarms
(`os_update_stale`, `agent_behind`, `os_config_bundle_behind`, `os_reboot_needed`, `os_not_covered`, `os_ring0_stalled`)
and the household's kernel notice mail. **After the hold**, still silent: the usual `host_down` + `node_down`, once,
with a text naming the household power-off. Back on inside the hold: no `*_recovered` mail.
- **Pages:** hosts list, host detail and the customer/dashboard roll-up show **KIKAPCSOLVA** (grey) and „Kikapcsolva a
háztartás által, T — riasztás szünetel eddig: T+14 nap" instead of DOWN. Delete logic still reads the raw status.
- Tests (consequence, each with a control that must alarm): `monitor/deliberate_off_test.go` (no alarm in the hold;
alarms after 14 days with the right text, once; back on is quiet; the 05:00 deadline skips),
`api/deliberate_off_test.go` (set, one event, cleared), `web/deliberate_off_view_test.go`. Red-proofed: the hold
removed from each checker, a hold that never ends, the deadline skip removed, the ingest unwired, an event per
report, the hosts list on the raw status — each convicted.
## v0.148.0 — SECURITY: the customer list and box reports need the operator key (2026-10-12)
Security review 2026-10-12 (`documentation/audits/security-review-2026-10-12/`), R-951, P1.
+68
View File
@@ -0,0 +1,68 @@
package api
import (
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// Event types of the household power-off (R-998, doc 14). Both are info: recorded on the customer's event list,
// never mailed (the dispatcher mails warning and above).
const (
eventHostDeliberateOff = "host_deliberate_off"
eventHostDeliberateOffEnded = "host_deliberate_off_ended"
)
// checkDeliberateOff keeps the host row's deliberate-off time in step with the report: set while the report
// carries the field, cleared by the first report without it (the box is in a new boot — switched back on).
func (h *Handler) checkDeliberateOff(hostID, custID string, rep *hostReportPayload) {
var at *time.Time
if rep.DeliberateOff != nil && rep.DeliberateOff.RequestedAt != "" {
t, err := time.Parse(time.RFC3339, rep.DeliberateOff.RequestedAt)
if err != nil {
h.logger.Printf("[WARN] host-report from %s: unreadable deliberate_off.requested_at %q — ignored", hostID, rep.DeliberateOff.RequestedAt)
return
}
t = t.UTC()
at = &t
}
prev, err := h.store.SetHostDeliberateOff(hostID, at)
if err != nil {
h.logger.Printf("[WARN] host-report from %s: deliberate-off not stored: %v", hostID, err)
return
}
switch {
case at != nil && (prev == nil || !prev.Equal(*at)):
msg := "Switched off by the household at " + at.Format("2006-01-02 15:04 UTC") +
" — silence alarms held for " + holdDays() + " days"
h.logger.Printf("[INFO] host %s: deliberate power-off recorded (%s)", hostID, at.Format(time.RFC3339))
h.saveInfoEvent(custID, eventHostDeliberateOff, msg)
case at == nil && prev != nil:
msg := "Reporting again after the household's power-off of " + prev.Format("2006-01-02 15:04 UTC")
h.logger.Printf("[INFO] host %s: deliberate power-off ended (was %s)", hostID, prev.Format(time.RFC3339))
h.saveInfoEvent(custID, eventHostDeliberateOffEnded, msg)
}
}
func holdDays() string {
d := int(store.DeliberateOffHold / (24 * time.Hour))
return itoa(d)
}
func itoa(n int) string {
if n == 0 {
return "0"
}
var b []byte
for n > 0 {
b = append([]byte{byte('0' + n%10)}, b...)
n /= 10
}
return string(b)
}
func (h *Handler) saveInfoEvent(custID, eventType, msg string) {
if _, err := h.store.SaveEvent(custID, eventType, "info", msg, "{}", "hub"); err != nil {
h.logger.Printf("[WARN] save %s event: %v", eventType, err)
}
}
+44
View File
@@ -0,0 +1,44 @@
package api
import (
"net/http"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-998: the agent's deliberate_off field sets the host row's hold; the next report without it (a new boot — the box
// was switched back on) clears it. Each change leaves one info event on the customer's list; a repeat leaves none.
func TestHostReport_DeliberateOffSetThenCleared(t *testing.T) {
h, st, _ := newTestHandler(t)
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "p"})
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
withOff := strings.TrimSuffix(validReportBody("h1"), "}") + `,"deliberate_off":{"requested_at":"2026-10-13T18:00:00Z"}}`
for i := 0; i < 2; i++ { // the field rides every report until the box is down; the second must not re-record
if rr := do(h, http.MethodPost, "/host-report", "HKEY", withOff); rr.Code != 200 {
t.Fatalf("report %d: %d %s", i, rr.Code, rr.Body.String())
}
}
at := st.HostDeliberateOffAt("h1")
if at == nil || !at.Equal(time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)) {
t.Fatalf("hold = %v, want 2026-10-13 18:00 UTC", at)
}
evs, _ := st.GetEventsByType("c1", eventHostDeliberateOff, time.Now().Add(-time.Hour))
if len(evs) != 1 || evs[0].Severity != "info" {
t.Fatalf("set events = %+v, want exactly one info event", evs)
}
if rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")); rr.Code != 200 {
t.Fatalf("plain report: %d", rr.Code)
}
if st.HostDeliberateOffAt("h1") != nil {
t.Fatal("a report without the field must clear the hold")
}
ended, _ := st.GetEventsByType("c1", eventHostDeliberateOffEnded, time.Now().Add(-time.Hour))
if len(ended) != 1 {
t.Fatalf("ended events = %d, want 1", len(ended))
}
}
+6
View File
@@ -744,6 +744,11 @@ type hostReportPayload struct {
CollectedAt time.Time `json:"collected_at"`
Lines []string `json:"lines"`
} `json:"log_tail"`
// DeliberateOff (R-998, agent >= 0.155.0): the household asked this box to power off, in its current boot.
// Absent = not (or an older agent). deliberate_off.go turns it into the host row's hold.
DeliberateOff *struct {
RequestedAt string `json:"requested_at"`
} `json:"deliberate_off"`
}
// hostForeignKeyArchives mirrors the agent's hub.ForeignKeyArchives (R-366 slice 2).
@@ -921,6 +926,7 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
h.checkTunnel(hostID, custID, rep.Cloudflared.Status, rep.Cloudflared.Detail)
h.checkCrash(hostID, custID, body)
h.checkDeliberateOff(hostID, custID, &rep)
for _, g := range rep.Guests {
status := g.Status
+7 -1
View File
@@ -430,7 +430,13 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
// to be quiet. That is R-195's shape exactly — a skip keyed off the wrong fact missing the
// customer it would most obviously cover — which is why both doors are closed together.
st := stalenessState(staleness, id)
if st == StateDisabled {
if st == StateDisabled || st == StateDeliberateOff { // R-998: a household power-off is not a missed backup
skipped++
continue
}
// R-998, read from the host row too: a box switched off minutes before 05:00 is not yet past the staleness
// threshold, so its staleness state is still "ok" — the hold must not depend on that clock.
if _, held := s.CustomerDeliberatelyOff(id, time.Now()); held {
skipped++
continue
}
+170
View File
@@ -0,0 +1,170 @@
package monitor
// R-998 (doc 14) — A BOX THE HOUSEHOLD SWITCHED OFF IS NOT A BOX THAT DIED, FOR 14 DAYS.
//
// The consequence is asserted, not the mechanism: does the alarm fire? Each "held" case runs beside a CONTROL
// with the same silence and no power-off, which must alarm — a quiet result alone could mean a broken fixture.
import (
"database/sql"
"io"
"log"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
type offRig struct {
st *store.Store
path string
node *StalenessChecker
host *HostStalenessChecker
events []string
msgs []string
}
// newOffRig: one customer with one host, both reporting NOW, both checkers having seen them healthy.
func newOffRig(t *testing.T) *offRig {
t.Helper()
st, path := seedStalenessCustomer(t, "ok", 0)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatal(err)
}
if err := st.SaveHostReport("h1", "c1", []byte(`{}`), store.HostReportDenorm{}); err != nil {
t.Fatal(err)
}
r := &offRig{st: st, path: path}
on := func(cid, et, sev, msg, det, src string) {
r.events = append(r.events, et)
r.msgs = append(r.msgs, msg)
}
r.node = NewStalenessChecker(st, 30*time.Minute, on, log.New(io.Discard, "", 0))
r.host = NewHostStalenessChecker(st, 30*time.Minute, on, log.New(io.Discard, "", 0))
r.node.Check()
r.host.Check()
if len(r.events) != 0 {
t.Fatalf("setup emitted %v", r.events)
}
return r
}
// silent ages both the controller's and the agent's last report by d.
func (r *offRig) silent(t *testing.T, d time.Duration) {
t.Helper()
db, err := sql.Open("sqlite", r.path)
if err != nil {
t.Fatal(err)
}
defer db.Close()
at := time.Now().UTC().Add(-d).Format("2006-01-02 15:04:05")
if _, err := db.Exec(`UPDATE reports SET received_at = ? WHERE customer_id = 'c1'`, at); err != nil {
t.Fatal(err)
}
if _, err := db.Exec(`UPDATE hosts SET last_report_at = ? WHERE host_id = 'h1'`, at); err != nil {
t.Fatal(err)
}
}
func (r *offRig) offAt(t *testing.T, ago time.Duration) {
t.Helper()
at := time.Now().UTC().Add(-ago).Truncate(time.Second)
if _, err := r.st.SetHostDeliberateOff("h1", &at); err != nil {
t.Fatal(err)
}
}
func (r *offRig) check() { r.node.Check(); r.host.Check() }
func hasEvent(evs []string, want string) bool {
for _, e := range evs {
if e == want {
return true
}
}
return false
}
// Silent for 3 hours: the control alarms node_down + host_down; the switched-off box raises nothing.
func TestDeliberateOff_NoSilenceAlarmDuringTheHold(t *testing.T) {
ctl := newOffRig(t)
ctl.silent(t, 3*time.Hour)
ctl.check()
if !hasEvent(ctl.events, "node_down") || !hasEvent(ctl.events, "host_down") {
t.Fatalf("CONTROL: a box silent 3 h must alarm, got %v — the fixture is broken", ctl.events)
}
r := newOffRig(t)
r.offAt(t, 3*time.Hour)
r.silent(t, 3*time.Hour)
r.check()
r.check()
if len(r.events) != 0 {
t.Fatalf("a box the household switched off alarmed: %v", r.events)
}
if r.node.GetState("c1") != StateDeliberateOff || r.host.GetState("h1") != "off" {
t.Fatalf("states node=%q host=%q, want %q / off", r.node.GetState("c1"), r.host.GetState("h1"), StateDeliberateOff)
}
}
// After the 14-day hold, still silent → the usual alarms fire, and their text names the power-off.
func TestDeliberateOff_AlarmsAfterFourteenDays(t *testing.T) {
r := newOffRig(t)
r.offAt(t, 13*24*time.Hour)
r.silent(t, 13*24*time.Hour)
r.check()
if len(r.events) != 0 {
t.Fatalf("day 13: %v", r.events)
}
r.offAt(t, 14*24*time.Hour+time.Minute) // the hold ran out
r.silent(t, 14*24*time.Hour+time.Minute)
r.check()
if !hasEvent(r.events, "node_down") || !hasEvent(r.events, "host_down") {
t.Fatalf("after 14 days: events %v, want node_down + host_down", r.events)
}
for _, m := range r.msgs {
if !strings.Contains(m, "household") {
t.Fatalf("the alarm does not say the household switched it off: %q", m)
}
}
r.check() // once, not every minute
if len(r.events) != 2 {
t.Fatalf("repeated: %v", r.events)
}
}
// Switched back on inside the hold: reports resume → no recovery mail for an outage that never was.
func TestDeliberateOff_BackOnIsQuiet(t *testing.T) {
r := newOffRig(t)
r.offAt(t, 2*time.Hour)
r.silent(t, 2*time.Hour)
r.check()
r.silent(t, 0)
if _, err := r.st.SetHostDeliberateOff("h1", nil); err != nil { // the new boot's first report
t.Fatal(err)
}
r.check()
if len(r.events) != 0 {
t.Fatalf("back on: %v", r.events)
}
if r.node.GetState("c1") != "ok" || r.host.GetState("h1") != "ok" {
t.Fatalf("states node=%q host=%q, want ok", r.node.GetState("c1"), r.host.GetState("h1"))
}
}
// The 05:00 deadline: Tester 2's shape (R-872) alarms; the same box switched off by the household does not.
func TestDeliberateOff_DeadlineSkips(t *testing.T) {
st, _, sc := downBox(t, 5*24*time.Hour)
if got := deadlineEvents(st, sc); got["expected_backup_missed"] != 1 {
t.Fatalf("CONTROL: %v — the down box must raise its missed-backup alarm", got)
}
st2, _, sc2 := downBox(t, 5*24*time.Hour)
at := time.Now().UTC().Add(-2 * time.Hour)
if _, err := st2.SetHostDeliberateOff("h1", &at); err != nil {
t.Fatal(err)
}
if got := deadlineEvents(st2, sc2); len(got) != 0 {
t.Fatalf("a switched-off box got deadline alarms: %v", got)
}
}
+37
View File
@@ -96,6 +96,9 @@ func (sc *HostStalenessChecker) Check() {
age := time.Since(row.LastReportAt)
var newState string
switch {
case age > sc.threshold && store.DeliberateOffHeld(row.DeliberateOffAt, time.Now()):
// R-998: the household switched the box off on purpose — silent, for the 14-day hold.
newState = "off"
case age > sc.downAfter:
newState = "down"
case age > sc.threshold:
@@ -105,6 +108,20 @@ func (sc *HostStalenessChecker) Check() {
}
oldState := sc.states[row.HostID]
if newState == "off" || oldState == "off" {
// Into the hold, or out of it back to ok: no event (the report ingest records the info events).
// Out of it into stale/down (the hold ran out with the box still silent): the usual alarm, with a
// message that says why it comes now.
sc.states[row.HostID] = newState
if newState == "stale" || newState == "down" {
sc.downtimeStart[row.HostID] = time.Now()
sc.emitHoldEnded(row, newState, age)
} else if newState == "off" && oldState != "off" {
sc.logger.Printf("[INFO] Host staleness: %s deliberately off since %s — alarms held until %s",
row.HostID, row.DeliberateOffAt.Format(time.RFC3339), row.DeliberateOffAt.Add(store.DeliberateOffHold).Format(time.RFC3339))
}
continue
}
if oldState == "" {
// F2: first observation (incl. a hub restart). Healthy seeds silently; a host already
// stale/down at first sight emits once (else a box that went silent while the hub was
@@ -181,3 +198,23 @@ func (sc *HostStalenessChecker) emitTransition(hostID, customerID, oldState, new
sc.onEvent(customerID, eventType, severity, message, "{}", "hub")
}
}
// emitHoldEnded raises the alarm a deliberate-off hold was holding (R-998): the box was switched off by the household
// and is still silent when the hold runs out. The text names the power-off — „no report for 14d" alone reads as an
// outage nobody saw coming.
func (sc *HostStalenessChecker) emitHoldEnded(row store.HostStaleRow, newState string, age time.Duration) {
eventType, severity := "host_stale", "warning"
if newState == "down" {
eventType, severity = "host_down", "error"
}
message := "Host " + row.HostID + ": switched off by the household on " + row.DeliberateOffAt.UTC().Format("2006-01-02 15:04 UTC") +
" and still silent after the " + formatDuration(store.DeliberateOffHold) + " hold (no report for " + formatDuration(age) + ")"
sc.logger.Printf("[INFO] Host staleness: %s off → %s (%s, the deliberate-off hold ended)", row.HostID, newState, eventType)
if _, err := sc.store.SaveEvent(row.CustomerID, eventType, severity, message, "{}", "hub"); err != nil {
sc.logger.Printf("[WARN] Failed to save host staleness event for %s: %v", row.HostID, err)
return
}
if sc.onEvent != nil {
sc.onEvent(row.CustomerID, eventType, severity, message, "{}", "hub")
}
}
+5
View File
@@ -468,6 +468,11 @@ func (oc *OffsiteChecker) Check() {
delete(oc.usedWindows, c.CustomerID)
continue
}
// R-998: a box the household switched off has no fresh off-site copy by design — held for the 14 days.
// States are kept, so the edge does not re-fire when it comes back.
if _, held := oc.store.CustomerDeliberatelyOff(c.CustomerID, time.Now()); held {
continue
}
if oc.store.IsCustomerBlocked(c.CustomerID) {
delete(oc.fillStates, c.CustomerID)
delete(oc.staleStates, c.CustomerID)
+3
View File
@@ -188,6 +188,9 @@ func (c *RestoreTestChecker) Check() {
}
now := c.now()
for _, id := range ids {
if _, held := c.store.CustomerDeliberatelyOff(id, now); held {
continue // R-998: switched off by the household — no restore test can run; held for the 14 days
}
latest, rerr := c.store.GetLatestHostReportJSON(id)
if rerr != nil || latest == "" {
continue // no agent host-report → nothing to judge (the R-81 legacy-shape branch)
+43
View File
@@ -35,6 +35,12 @@ type EventNotifyFunc func(customerID, eventType, severity, message, detailsJSON,
// second door.
const StateDisabled = "disabled"
// StateDeliberateOff (R-998, doc 14) is the state of a customer whose box the HOUSEHOLD switched off with
// „A gép kikapcsolása" — the host row carries the time (store.DeliberateOffHeld). Like StateDisabled it is a
// recorded state, so CheckBackupDeadlines can skip it; unlike StateDisabled it ENDS: after the 14-day hold the
// usual stale/down alarm fires, with a message naming the power-off. Pinned by deliberate_off_test.go.
const StateDeliberateOff = "deliberately_off"
// StalenessChecker monitors customer report freshness and generates
// node_stale / node_down / node_recovered events on state transitions.
type StalenessChecker struct {
@@ -139,7 +145,10 @@ func (sc *StalenessChecker) Check() {
age := time.Since(c.ReceivedAt)
var newState string
_, offHeld := sc.store.CustomerDeliberatelyOff(c.CustomerID, time.Now())
switch {
case age > sc.threshold && offHeld:
newState = StateDeliberateOff
case age > sc.downAfter:
newState = "down"
case age > sc.threshold:
@@ -149,6 +158,22 @@ func (sc *StalenessChecker) Check() {
}
oldState := sc.states[c.CustomerID]
// R-998: into the hold, or back to ok from it — silent. Out of it into stale/down — the usual alarm,
// with the power-off named in the text.
if newState == StateDeliberateOff || (oldState == StateDeliberateOff && newState == "ok") {
if oldState != newState {
sc.logger.Printf("[INFO] Staleness: %s %s → %s (household power-off, no event)", c.CustomerID, oldState, newState)
}
sc.states[c.CustomerID] = newState
delete(sc.downtimeStart, c.CustomerID)
continue
}
if oldState == StateDeliberateOff {
sc.states[c.CustomerID] = newState
sc.downtimeStart[c.CustomerID] = time.Now()
sc.emitHoldEnded(c.CustomerID, newState, age)
continue
}
if oldState == "" || oldState == StateDisabled {
// New customer — set state without event.
//
@@ -275,3 +300,21 @@ func (sc *StalenessChecker) isNewBox(customerID string) bool {
}
return time.Since(h.CreatedAt) < sc.threshold
}
// emitHoldEnded raises the node alarm a household power-off was holding, once the 14-day hold ran out (R-998).
func (sc *StalenessChecker) emitHoldEnded(customerID, newState string, age time.Duration) {
eventType, severity := "node_stale", "warning"
if newState == "down" {
eventType, severity = "node_down", "error"
}
message := "No report received for " + formatDuration(age) + " — the box was switched off by the household and the " +
formatDuration(store.DeliberateOffHold) + " hold has ended"
sc.logger.Printf("[INFO] Staleness: %s %s → %s (%s, the deliberate-off hold ended)", customerID, StateDeliberateOff, newState, eventType)
if _, err := sc.store.SaveEvent(customerID, eventType, severity, message, "{}", "hub"); err != nil {
sc.logger.Printf("[WARN] Failed to save staleness event for %s: %v", customerID, err)
return
}
if sc.onEvent != nil {
sc.onEvent(customerID, eventType, severity, message, "{}", "hub")
}
}
+21
View File
@@ -0,0 +1,21 @@
package osupdates
import (
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// notDeliberatelyOff drops the hosts the household switched off with „A gép kikapcsolása" (R-998, doc 14) while the
// 14-day hold lasts: a box that is off cannot run its OS leg, take a new agent or bundle, or restart for a kernel —
// alarming about it, or mailing „tonight your box restarts", would be noise. After the hold they count again.
func notDeliberatelyOff(st *store.Store, hosts []store.Host, now time.Time) []store.Host {
out := hosts[:0:0]
for _, h := range hosts {
if store.DeliberateOffHeld(st.HostDeliberateOffAt(h.HostID), now) {
continue
}
out = append(out, h)
}
return out
}
+1
View File
@@ -198,6 +198,7 @@ func (s *Service) KernelNotify() []string {
if err != nil {
return nil
}
hosts = notDeliberatelyOff(s.Store, hosts, now) // R-998: no „tonight your box restarts" mail to a box that is off
var told []string
for _, h := range hosts {
kver, _ := s.KernelDue(h.HostID)
+1
View File
@@ -1059,6 +1059,7 @@ func (s *Service) Alarms() ([]string, error) {
if err != nil {
return nil, err
}
hosts = notDeliberatelyOff(s.Store, hosts, now) // R-998: a switched-off box cannot update — held for the 14 days
var sent []string
for _, h := range hosts {
st := s.Store.GetOSHostSettings(h.HostID)
+87
View File
@@ -0,0 +1,87 @@
package store
import (
"database/sql"
"time"
)
// ── The household's deliberate power-off (R-998, documentation/architecture/14-host-power-off.md) ──────
//
// The agent reports `deliberate_off.requested_at` from the moment the household pressed „A gép kikapcsolása"
// until the box goes down; the next boot's reports do not carry it. The hub keeps the time on the host row.
// While it is set and younger than DeliberateOffHold, every silence alarm about that box is held: host and
// controller staleness, the backup deadlines, off-site and restore-test freshness, the OS alarms and the
// kernel notice. After the hold, the alarms fire as usual — a box nobody switched back on in two weeks is
// worth a look. Pinned by monitor/deliberate_off_test.go, api/deliberate_off_test.go and web/deliberate_off_view_test.go.
// DeliberateOffHold is how long a deliberate power-off silences the box's alarms (operator default, 14 days).
const DeliberateOffHold = 14 * 24 * time.Hour
// DeliberateOffHeld reports whether a deliberate-off time still holds the alarms at now.
func DeliberateOffHeld(at *time.Time, now time.Time) bool {
return at != nil && !at.IsZero() && now.Sub(*at) < DeliberateOffHold
}
func parseDeliberateOff(v string) *time.Time {
if v == "" {
return nil
}
t, err := time.Parse(time.RFC3339, v)
if err != nil {
return nil
}
return &t
}
// SetHostDeliberateOff records the report's deliberate-off field. at == nil clears it (a report without the
// field: the box is in a new boot). It returns the previous value, so the caller can emit an event on a change.
func (s *Store) SetHostDeliberateOff(hostID string, at *time.Time) (prev *time.Time, err error) {
var old sql.NullString
if err := s.db.QueryRow(`SELECT deliberate_off_at FROM hosts WHERE host_id = ?`, hostID).Scan(&old); err != nil {
return nil, err
}
prev = parseDeliberateOff(old.String)
var val any
if at != nil {
val = at.UTC().Format(time.RFC3339)
}
if (prev == nil && at == nil) || (prev != nil && at != nil && prev.Equal(*at)) {
return prev, nil
}
_, err = s.db.Exec(`UPDATE hosts SET deliberate_off_at = ?, updated_at = datetime('now') WHERE host_id = ?`, val, hostID)
return prev, err
}
// HostDeliberateOffAt returns the host's deliberate-off time (nil = none).
func (s *Store) HostDeliberateOffAt(hostID string) *time.Time {
var v sql.NullString
if err := s.db.QueryRow(`SELECT deliberate_off_at FROM hosts WHERE host_id = ?`, hostID).Scan(&v); err != nil {
return nil
}
return parseDeliberateOff(v.String)
}
// CustomerDeliberatelyOff reports whether any of the customer's hosts is deliberately off and still within the
// hold at now — the checkers keyed by customer use it. The time is the newest such request.
func (s *Store) CustomerDeliberatelyOff(customerID string, now time.Time) (time.Time, bool) {
rows, err := s.db.Query(`SELECT deliberate_off_at FROM hosts WHERE customer_id = ? AND deliberate_off_at IS NOT NULL`, customerID)
if err != nil {
return time.Time{}, false
}
defer rows.Close()
var newest time.Time
held := false
for rows.Next() {
var v string
if rows.Scan(&v) != nil {
continue
}
if t := parseDeliberateOff(v); DeliberateOffHeld(t, now) {
held = true
if t.After(newest) {
newest = *t
}
}
}
return newest, held
}
+14 -6
View File
@@ -471,6 +471,9 @@ func (s *Store) migrate() error {
// expected key fingerprint, tunnel id) — the hub serves these only in recovery mode; no usable
// secret is hub-held (the blobs need R, which the hub never has).
s.db.Exec(`ALTER TABLE hosts ADD COLUMN recovery_mode_until DATETIME`)
// R-998 (doc 14): the household switched the box off on purpose at this time (RFC3339 UTC, from the agent's
// deliberate_off report field). NULL = not deliberately off. deliberateoff.go reads and writes it.
s.db.Exec(`ALTER TABLE hosts ADD COLUMN deliberate_off_at TEXT`)
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
@@ -2823,6 +2826,7 @@ type Host struct {
DesiredJSON string
DesiredGeneration int64
RecoveryModeUntil *time.Time // slice 10D: recovery mode active until this time (nil/past = off)
DeliberateOffAt *time.Time // R-998: switched off by the household at this time (nil = not); deliberateoff.go
CreatedAt time.Time
UpdatedAt time.Time
}
@@ -2866,6 +2870,8 @@ type HostStaleRow struct {
HostID string
CustomerID string
LastReportAt time.Time
// DeliberateOffAt (R-998): when the household switched the box off on purpose; nil = not.
DeliberateOffAt *time.Time
}
// GuestID derives the interim guest primary key from host + vmid. The hub owns the
@@ -2890,10 +2896,10 @@ func (s *Store) scanHost(scan func(dest ...any) error) (*Host, error) {
func scanHostRaw(scan func(dest ...any) error) (*Host, error) {
var h Host
var lastReport, recoveryUntil sql.NullString
var lastReport, recoveryUntil, deliberateOff sql.NullString
var createdAt, updatedAt string
err := scan(&h.HostID, &h.CustomerID, &h.APIKey, &h.AgentVersion, &lastReport,
&h.DesiredJSON, &h.DesiredGeneration, &recoveryUntil, &createdAt, &updatedAt)
&h.DesiredJSON, &h.DesiredGeneration, &recoveryUntil, &createdAt, &updatedAt, &deliberateOff)
if err != nil {
return nil, err
}
@@ -2905,13 +2911,14 @@ func scanHostRaw(scan func(dest ...any) error) (*Host, error) {
t := parseSQLiteTime(recoveryUntil.String)
h.RecoveryModeUntil = &t
}
h.DeliberateOffAt = parseDeliberateOff(deliberateOff.String)
h.CreatedAt = parseSQLiteTime(createdAt)
h.UpdatedAt = parseSQLiteTime(updatedAt)
return &h, nil
}
const hostSelectCols = `host_id, customer_id, api_key, agent_version, last_report_at,
desired_json, desired_generation, recovery_mode_until, created_at, updated_at`
desired_json, desired_generation, recovery_mode_until, created_at, updated_at, deliberate_off_at`
// GetHostByAPIKey looks up a host by its per-host hub key. Returns nil (no error)
// if no match — parallels GetCustomerConfigByAPIKey.
@@ -3847,7 +3854,7 @@ func (s *Store) ListGuestsForHost(hostID string) ([]Guest, error) {
// have never reported (NULL last_report_at) are skipped — a freshly-minted host is
// not "down" until it has checked in at least once.
func (s *Store) GetHostStaleness() ([]HostStaleRow, error) {
rows, err := s.db.Query(`SELECT host_id, customer_id, last_report_at FROM hosts WHERE last_report_at IS NOT NULL`)
rows, err := s.db.Query(`SELECT host_id, customer_id, last_report_at, COALESCE(deliberate_off_at, '') FROM hosts WHERE last_report_at IS NOT NULL`)
if err != nil {
return nil, err
}
@@ -3855,11 +3862,12 @@ func (s *Store) GetHostStaleness() ([]HostStaleRow, error) {
var out []HostStaleRow
for rows.Next() {
var r HostStaleRow
var last string
if err := rows.Scan(&r.HostID, &r.CustomerID, &last); err != nil {
var last, off string
if err := rows.Scan(&r.HostID, &r.CustomerID, &last, &off); err != nil {
return nil, err
}
r.LastReportAt = parseSQLiteTime(last)
r.DeliberateOffAt = parseDeliberateOff(off)
out = append(out, r)
}
return out, rows.Err()
@@ -0,0 +1,63 @@
package web
import (
"database/sql"
"io"
"log"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-998: a box the household switched off reads KIKAPCSOLVA with the power-off line on the hosts page — not DOWN —
// exactly while the alarms are held; after the 14-day hold it reads DOWN again.
func TestHostsPage_DeliberatelyOffBox(t *testing.T) {
path := filepath.Join(t.TempDir(), "t.db")
st, err := store.New(path, log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { st.Close() })
s := New(st, "", "", "test", 30*time.Minute, log.New(io.Discard, "", 0))
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"})
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"})
st.SaveHostReport("h1", "c1", []byte(`{}`), store.HostReportDenorm{})
age := func(d time.Duration) {
db, _ := sql.Open("sqlite", path)
defer db.Close()
db.Exec(`UPDATE hosts SET last_report_at = ? WHERE host_id = 'h1'`, time.Now().UTC().Add(-d).Format("2006-01-02 15:04:05"))
}
page := func() string {
rr := httptest.NewRecorder()
s.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/hosts", nil))
if rr.Code != 200 {
t.Fatalf("/hosts: %d", rr.Code)
}
return rr.Body.String()
}
age(3 * time.Hour)
if p := page(); !strings.Contains(p, ">DOWN<") || strings.Contains(p, "data-deliberate-off") {
t.Fatal("CONTROL: a silent box without a power-off must read DOWN")
}
at := time.Now().UTC().Add(-3 * time.Hour).Truncate(time.Second)
st.SetHostDeliberateOff("h1", &at)
p := page()
if !strings.Contains(p, ">KIKAPCSOLVA<") || strings.Contains(p, ">DOWN<") {
t.Fatal("a box switched off by the household must read KIKAPCSOLVA, not DOWN")
}
if !strings.Contains(p, "Kikapcsolva a háztartás által, "+at.Format("2006-01-02 15:04")) {
t.Fatal("the hosts page does not name the household power-off and its time")
}
old := time.Now().UTC().Add(-15 * 24 * time.Hour)
st.SetHostDeliberateOff("h1", &old)
age(15 * 24 * time.Hour)
if p := page(); !strings.Contains(p, ">DOWN<") {
t.Fatal("after the 14-day hold the box must read DOWN again")
}
}
+31 -4
View File
@@ -44,6 +44,26 @@ func (s *Server) hostStatus(lastReport *time.Time) string {
}
}
// hostDisplayStatus is hostStatus for the PAGES (R-998): a silent host the household switched off on purpose, inside
// the 14-day hold, reads "off" — not stale/down — exactly while the checkers hold its alarms (the same
// store.DeliberateOffHeld). Delete logic keeps reading hostStatus. Pinned by deliberate_off_view_test.go.
func (s *Server) hostDisplayStatus(h *store.Host) string {
st := s.hostStatus(h.LastReportAt)
if (st == "stale" || st == "down") && store.DeliberateOffHeld(h.DeliberateOffAt, time.Now()) {
return "off"
}
return st
}
// deliberateOffLine is the operator line for a household power-off ("" when none is held).
func deliberateOffLine(h *store.Host) string {
if h == nil || !store.DeliberateOffHeld(h.DeliberateOffAt, time.Now()) {
return ""
}
return "Kikapcsolva a háztartás által, " + h.DeliberateOffAt.UTC().Format("2006-01-02 15:04") + " UTC — riasztás szünetel eddig: " +
h.DeliberateOffAt.Add(store.DeliberateOffHold).UTC().Format("2006-01-02 15:04") + " UTC"
}
// hostStatusClass maps the internal status to the existing status-badge-* CSS class.
// "stale" reuses the amber -warn class (there is no dedicated -stale class), keeping the
// styling consistent with the rest of the console.
@@ -55,6 +75,8 @@ func hostStatusClass(status string) string {
return "status-badge-warn"
case "down":
return "status-badge-down"
case "off":
return "status-badge-pending" // grey: deliberate, not an incident (R-998)
default:
return "status-badge-pending"
}
@@ -69,6 +91,8 @@ func hostStatusLabel(status string) string {
return "STALE"
case "down":
return "DOWN"
case "off":
return "KIKAPCSOLVA" // R-998: switched off by the household
default:
return "NO REPORT"
}
@@ -511,8 +535,9 @@ type hostListRow struct {
CustomerID string
CustomerName string
AgentVersion string
Status string // ok | stale | down | pending
Status string // ok | stale | down | pending | off (R-998)
StatusLabel string
OffLine string // R-998: the household power-off line while the hold lasts
StatusClass string
LastReportAt *time.Time
HasReport bool
@@ -563,8 +588,9 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
rows := make([]hostListRow, 0, len(hosts))
for _, h := range hosts {
status := s.hostStatus(h.LastReportAt)
status := s.hostDisplayStatus(&h)
row := hostListRow{
OffLine: deliberateOffLine(&h),
HostID: h.HostID,
CustomerID: h.CustomerID,
CustomerName: s.customerName(h.CustomerID),
@@ -826,8 +852,9 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
"AgentVersion": host.AgentVersion,
"CreatedAt": host.CreatedAt,
"Status": status,
"StatusLabel": hostStatusLabel(status),
"StatusClass": hostStatusClass(status),
"StatusLabel": hostStatusLabel(s.hostDisplayStatus(host)),
"StatusClass": hostStatusClass(s.hostDisplayStatus(host)),
"OffLine": deliberateOffLine(host),
"LastReportAt": host.LastReportAt,
"HasReport": host.LastReportAt != nil,
"RecoveryMode": host.InRecoveryMode(time.Now()),
+4
View File
@@ -66,6 +66,10 @@ func (s *Server) foldHostStatus(customerID, base string, customerHasReported boo
}
worst, worstHost := "", ""
for i := range hosts {
// R-998: a host the household switched off (silent, inside the hold) is the customer's state, not an outage.
if s.hostDisplayStatus(&hosts[i]) == "off" {
return "off", deliberateOffLine(&hosts[i])
}
hs := s.hostStatus(hosts[i].LastReportAt)
if hs == "pending" && !customerHasReported {
continue
+1 -1
View File
@@ -1154,7 +1154,7 @@ func statusColor(status string) string {
return "crit"
case "blocked":
return "warn"
case "disabled", "pending":
case "disabled", "pending", "off":
return "neutral"
default:
return "neutral"
+1 -1
View File
@@ -65,7 +65,7 @@
<span class="status-badge status-badge-{{.OverallStatus}}">
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "down"}}DOWN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}{{.OverallStatus}}{{end}}
</span>
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
{{else}}
<span class="text-muted">—</span>
{{end}}
@@ -26,7 +26,7 @@
<h1>
<span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span>
{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
</h1>
{{if .HasReports}}
<p class="subtitle">
@@ -107,7 +107,7 @@
<div class="summary-strip">
<span class="strip-name">{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</span>
<span class="strip-item"><span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span> {{.OverallStatus}}</span>
{{if .HostCause}}<span class="strip-item"><span class="status-badge status-badge-warn">{{.HostCause}}</span></span>{{end}}
{{if .HostCause}}<span class="strip-item"><span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span></span>{{end}}
{{if .HasReports}}
<span class="strip-item">Controller <code>{{.Customer.ControllerVersion}}</code></span>
<span class="strip-item">Last report {{timeAgo .Customer.ReceivedAt}}</span>
+1 -1
View File
@@ -75,7 +75,7 @@
<span class="status-badge status-badge-{{.OverallStatus}}">
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}DOWN{{end}}
</span>
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
{{if .HostCause}}<span class="status-badge {{if eq .OverallStatus "off"}}status-badge-pending{{else}}status-badge-warn{{end}}">{{.HostCause}}</span>{{end}}
</td>
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{if gt (add (add .EventCriticals .EventErrors) .EventWarnings) 0}}{{if gt .EventCriticals 0}}<span class="severity-badge severity-critical">{{.EventCriticals}}</span>{{end}}{{if gt .EventErrors 0}}<span class="severity-badge severity-error">{{.EventErrors}}</span>{{end}}{{if gt .EventWarnings 0}}<span class="severity-badge severity-warning">{{.EventWarnings}}</span>{{end}}{{else}}<span class="text-muted">—</span>{{end}}{{end}}</td>
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}</td>
@@ -52,6 +52,10 @@
<span class="label">Last Report</span>
<span class="value">{{if .HasReport}}{{timeAgoPtr .LastReportAt}}{{else}}waiting for first report{{end}}</span>
</div>
{{if .OffLine}}<div class="info-item">
<span class="label">Power</span>
<span class="value" data-deliberate-off>{{.OffLine}}</span>
</div>{{end}}
<!-- R-30 slice 1: the box's wait-channel presence (in memory; ~4-min cadence), beside the
15-min report clock. Every branch pinned by r30_presence_delete_test.go. -->
<div class="info-item">
+1 -1
View File
@@ -104,7 +104,7 @@
<td>{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</td>
<td>{{if .AgentVersion}}<code>{{.AgentVersion}}</code>{{else}}—{{end}}{{if .FloorHeld}} <span class="status-badge status-warn" title="{{.HeldReason}}">floor held</span>{{else if eq .FloorSource "declared"}} <span class="status-badge" title="served above the vouched golden: the agent requirement was declared with the floor (R-472)">floor: declared MinAgent</span>{{end}}</td>
<td style="font-size: 0.85em;">{{.PVEVersion}}<br><span class="text-muted">{{.KernelRunning}}</span></td>
<td><span class="status-badge {{.StatusClass}}">{{.StatusLabel}}</span></td>
<td><span class="status-badge {{.StatusClass}}">{{.StatusLabel}}</span>{{if .OffLine}}<br><span class="text-muted" style="font-size: 0.85em;" data-deliberate-off>{{.OffLine}}</span>{{end}}</td>
<td>{{if .HasReport}}{{.GuestRunning}}/{{.GuestTotal}}{{else}}—{{end}}</td>
<td>{{if .HasReport}}{{formatFloat .Vitals.CPUPercent}}%{{else}}—{{end}}</td>
<td>{{if .HasReport}}{{formatFloat .Vitals.MemoryPercent}}%{{else}}—{{end}}</td>