hub v0.12.0: retire Infra Backup, purge its plaintext secrets, fix backup-deadline email
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead since slice 8C yet stored plaintext customer secrets at rest (app-secret key, restic password, Cloudflare tokens) — a zero-knowledge violation — and its absence made the daily expected_backup_missed email fire for healthy customers. - Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed. Keep the db_dump half. No host-report → no backup alarm (liveness owns that). New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix. - Remove the infra-backup endpoints, store methods/types, and operator panel; /recovery now returns config_yaml only. - migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint) to physically reclaim the plaintext pages, gated on table existence. Flagged out-of-scope: exposed creds need operator rotation; legacy reports table holds historical plaintext restic_password rows (separate leak, not purged here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,48 @@
|
|||||||
# Felhom Hub — Changelog
|
# Felhom Hub — Changelog
|
||||||
|
|
||||||
|
## v0.12.0 — retire Infra Backup + purge its plaintext secrets + fix the daily backup-deadline email (2026-06-16)
|
||||||
|
|
||||||
|
Phase-1 of the Infra Backup retirement (per `documentation/audits/SPIKE-infra-backup-2026-06-15.md`).
|
||||||
|
The mechanism had been dead since slice 8C, yet the hub still stored each version as a **plaintext
|
||||||
|
JSON blob at rest** containing the customer's app-secret encryption key, restic password, and
|
||||||
|
Cloudflare tokens — a zero-knowledge violation. Its absence was also the root cause of the daily
|
||||||
|
`expected_backup_missed` false-alarm email.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **Backup-deadline check repointed to PBS freshness.** `monitor.CheckBackupDeadlines` no longer
|
||||||
|
looks for a `backup_completed` event (no component emits it anymore — the disk-tier backup moved to
|
||||||
|
the agent in slice 8C, so the check fired daily for every healthy customer). It now reads the
|
||||||
|
customer's **latest agent host-report** and raises `expected_backup_missed` only on positive
|
||||||
|
evidence: no PBS snapshot / successful vzdump at all, the newest backup older than **26h**, or the
|
||||||
|
newest PBS snapshot's `verify_state == "failed"`. A fresh-but-not-yet-verified snapshot is **not** a
|
||||||
|
failure (PBS verifies on its own cadence) — alarming on it would just re-create the false alarm. The
|
||||||
|
**db-dump half is unchanged** (the in-guest controller still emits `db_dump_completed`). A customer
|
||||||
|
with **no host-report** (legacy/defunct) gets no backup alarm here — liveness is the
|
||||||
|
host-staleness checker's job. New store accessor `GetLatestHostReportJSON`. Tests:
|
||||||
|
`internal/monitor/deadline_test.go` (fresh+verified→quiet, stale→alarm, failed-verify→alarm,
|
||||||
|
no-report→quiet, db-dump half preserved, plus a pure `assessBackupFreshness` table). The
|
||||||
|
fresh+verified→quiet test is the **companion**: it fails against the old event-based check.
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
- **The Infra Backup feature**: ingest endpoint `POST /api/v1/infra-backup`, getters
|
||||||
|
`GET /api/v1/infra-backup/{id}[/versions]` and their handlers; store methods
|
||||||
|
`SaveInfraBackup` / `GetInfraBackup` / `GetInfraBackupByID` / `GetInfraBackupMeta` /
|
||||||
|
`ListInfraBackupVersions` / `pruneInfraBackups` + the `InfraBackupMeta` / `InfraBackupVersion`
|
||||||
|
types; the operator **"Infra Backup" panel** (`customer_unified.html`, `customer.html`). The
|
||||||
|
`GET /api/v1/recovery/{id}` endpoint is kept but now returns **only** the generated `config_yaml`
|
||||||
|
(no infra-backup payload). The customer-page **config-drift badge** that diffed against the stored
|
||||||
|
controller.yaml is hidden (its at-rest source is gone); the live **"Show Diff"** path is unaffected.
|
||||||
|
|
||||||
|
### Security / migration
|
||||||
|
- **Plaintext secret purge.** `migrate()` now `DROP`s `infra_backup_versions` + `infra_backups` and
|
||||||
|
runs **`VACUUM`** (+ `wal_checkpoint(TRUNCATE)`) so the freed pages holding the plaintext keys/
|
||||||
|
tokens are **physically reclaimed**, not merely delinked. Gated on table existence so normal
|
||||||
|
restarts don't pay the VACUUM cost.
|
||||||
|
- **Out of scope (flagged for the operator):** the exposed Cloudflare / hub / session credentials in
|
||||||
|
the dropped blobs remain valid until rotated (operator step). Separately, the legacy `reports`
|
||||||
|
table holds thousands of historical rows with a plaintext `restic_password` value from old
|
||||||
|
controller versions — a distinct leak, not purged here (the live controller no longer sends it).
|
||||||
|
|
||||||
## v0.11.0 — slice 10D: DR capstone — recovery mode + re-enroll + directive serving (2026-06-10)
|
## v0.11.0 — slice 10D: DR capstone — recovery mode + re-enroll + directive serving (2026-06-10)
|
||||||
|
|
||||||
The hub half of the slice-10 DR capstone (closes slice 10). The hub ORCHESTRATES recovery but holds
|
The hub half of the slice-10 DR capstone (closes slice 10). The hub ORCHESTRATES recovery but holds
|
||||||
|
|||||||
+16
-27
@@ -18,7 +18,7 @@ A lightweight Go service that receives periodic reports and structured events fr
|
|||||||
│ │ │ ┌─────────────────┐ │
|
│ │ │ ┌─────────────────┐ │
|
||||||
│ POST /api/v1/ │ │ │ API Handler │ │
|
│ POST /api/v1/ │ │ │ API Handler │ │
|
||||||
│ report │ │ │ (ingest reports, │ │
|
│ report │ │ │ (ingest reports, │ │
|
||||||
│ infra-backup │◀── config push ────│ │ infra backups, │ │
|
│ host-report │◀── config push ────│ │ host reports, │ │
|
||||||
│ notify │ (YAML body) │ │ config push, │ │
|
│ notify │ (YAML body) │ │ config push, │ │
|
||||||
│ │ │ │ asset serving) │ │
|
│ │ │ │ asset serving) │ │
|
||||||
│ GET /api/v1/ │ │ └────────┬────────┘ │
|
│ GET /api/v1/ │ │ └────────┬────────┘ │
|
||||||
@@ -27,7 +27,7 @@ A lightweight Go service that receives periodic reports and structured events fr
|
|||||||
│ │ SQLite Store │ │
|
│ │ SQLite Store │ │
|
||||||
Operator browser │ │ (reports, │ │
|
Operator browser │ │ (reports, │ │
|
||||||
┌─────────────────┐ │ │ assets, │ │
|
┌─────────────────┐ │ │ assets, │ │
|
||||||
│ Web Dashboard │◀── HTML pages ──────│ │ infra_backups, │ │
|
│ Web Dashboard │◀── HTML pages ──────│ │ host_reports, │ │
|
||||||
│ (hub.felhom.eu) │ (bcrypt auth) │ │ configs, │ │
|
│ (hub.felhom.eu) │ (bcrypt auth) │ │ configs, │ │
|
||||||
└─────────────────┘ │ │ notifications) │ │
|
└─────────────────┘ │ │ notifications) │ │
|
||||||
│ └─────────────────┘ │
|
│ └─────────────────┘ │
|
||||||
@@ -62,44 +62,34 @@ All API endpoints require `Authorization: Bearer <api_key>` (except `/healthz` a
|
|||||||
|
|
||||||
The `POST /api/v1/report` handler (v0.4.0+) automatically parses the optional `app_telemetry` JSON array from the request body and stores it in `app_telemetry` / `app_log_issues` tables. Old controllers (no `app_telemetry` key) continue to work unchanged.
|
The `POST /api/v1/report` handler (v0.4.0+) automatically parses the optional `app_telemetry` JSON array from the request body and stores it in `app_telemetry` / `app_log_issues` tables. Old controllers (no `app_telemetry` key) continue to work unchanged.
|
||||||
|
|
||||||
### Infrastructure Backup (Disaster Recovery)
|
### Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)
|
||||||
|
|
||||||
| Method | Path | Description |
|
The Infra Backup mechanism (`POST/GET /api/v1/infra-backup`, the operator panel, the
|
||||||
|--------|------|-------------|
|
`infra_backup_versions` / `infra_backups` tables) has been **removed**. It had been dead since
|
||||||
| `POST` | `/api/v1/infra-backup` | Controller pushes infrastructure snapshot |
|
slice 8C (the disk-tier backup moved to the host agent), and it stored each version as a **plaintext
|
||||||
| `GET` | `/api/v1/infra-backup/{customer_id}` | Fresh controller pulls backup for restore |
|
JSON blob** holding the customer's app-secret encryption key, restic password, and Cloudflare tokens —
|
||||||
|
a zero-knowledge violation. The retirement migration `DROP`s both tables and `VACUUM`s the DB to
|
||||||
The infra-backup payload contains everything needed to restore a customer deployment:
|
physically reclaim the plaintext pages. Disaster recovery now rests on the agent's **PBS whole-CT
|
||||||
- `controller.yaml` (base64, full config including secrets)
|
snapshot** (the data bytes) plus the generated `controller.yaml` from the recovery endpoint (the
|
||||||
- `settings.json` (base64, backup preferences, storage paths)
|
config); a secret-free **DR recipe** is the later DR slice's job. See
|
||||||
- Disk layout (UUIDs, labels, mount points, fstab options, bind-mount topology)
|
`documentation/audits/SPIKE-infra-backup-2026-06-15.md`.
|
||||||
- Deployed stacks manifest (app names, HDD paths, display names)
|
|
||||||
- Restic passwords (primary + cross-drive, for encrypted backup access)
|
|
||||||
|
|
||||||
**Disaster recovery flow:**
|
|
||||||
1. Customer's system drive fails → replaced with fresh Debian install
|
|
||||||
2. `docker-setup.sh` deploys controller with minimal config (domain only)
|
|
||||||
3. Controller enters setup wizard → user chooses restore from local drive or Hub
|
|
||||||
4. For Hub restore: calls `GET /api/v1/recovery/{customer_id}` (gets config + infra backup)
|
|
||||||
5. Controller uses disk UUIDs to auto-mount surviving drives
|
|
||||||
6. Controller restores apps from local backups on those drives
|
|
||||||
|
|
||||||
### Recovery (Disaster Recovery)
|
### Recovery (Disaster Recovery)
|
||||||
|
|
||||||
| Method | Path | Description |
|
| Method | Path | Description |
|
||||||
|--------|------|-------------|
|
|--------|------|-------------|
|
||||||
| `GET` | `/api/v1/recovery/{customer_id}` | Combined recovery: returns generated controller.yaml + infra backup in one response |
|
| `GET` | `/api/v1/recovery/{customer_id}` | Returns the generated controller.yaml for a customer |
|
||||||
|
|
||||||
Auth: `X-Retrieval-Password` header (same per-customer password as config retrieval). Response:
|
Auth: `X-Retrieval-Password` header (same per-customer password as config retrieval). Response:
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"customer_id": "example",
|
"customer_id": "example",
|
||||||
"config_yaml": "customer:\n id: example\n ...",
|
"config_yaml": "customer:\n id: example\n ...",
|
||||||
"infra_backup": { ... },
|
"has_infra_backup": false
|
||||||
"has_infra_backup": true
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
If no infra backup exists yet, `infra_backup` is null and `has_infra_backup` is false.
|
The `has_infra_backup` field is retained as `false` so any old client degrades gracefully to the
|
||||||
|
config-only path.
|
||||||
|
|
||||||
### Report Response
|
### Report Response
|
||||||
|
|
||||||
@@ -228,7 +218,6 @@ SQLite with WAL mode. Tables:
|
|||||||
|-------|---------|
|
|-------|---------|
|
||||||
| `reports` | Full JSON reports with denormalized fields for dashboard queries |
|
| `reports` | Full JSON reports with denormalized fields for dashboard queries |
|
||||||
| `events` | Structured events from controllers and Hub (type, severity, message, details, source) |
|
| `events` | Structured events from controllers and Hub (type, severity, message, details, source) |
|
||||||
| `infra_backups` | Per-customer infrastructure snapshots for disaster recovery |
|
|
||||||
| `customer_notifications` | Email, enabled event types, cooldown hours per customer |
|
| `customer_notifications` | Email, enabled event types, cooldown hours per customer |
|
||||||
| `notification_log` | Send/skip/fail history for notifications with channel (operator/customer) |
|
| `notification_log` | Send/skip/fail history for notifications with channel (operator/customer) |
|
||||||
| `customer_configs` | Pre-configured customer settings, retrieval passwords, per-customer API keys, status (active/blocked) |
|
| `customer_configs` | Pre-configured customer settings, retrieval passwords, per-customer API keys, status (active/blocked) |
|
||||||
|
|||||||
+14
-130
@@ -169,14 +169,6 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
h.handleEvent(w, r)
|
h.handleEvent(w, r)
|
||||||
case r.Method == http.MethodPost && path == "/notify":
|
case r.Method == http.MethodPost && path == "/notify":
|
||||||
h.handleNotify(w, r)
|
h.handleNotify(w, r)
|
||||||
case r.Method == http.MethodPost && path == "/infra-backup":
|
|
||||||
h.handleInfraBackupPush(w, r)
|
|
||||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/infra-backup/") && strings.HasSuffix(path, "/versions"):
|
|
||||||
customerID := strings.TrimPrefix(path, "/infra-backup/")
|
|
||||||
customerID = strings.TrimSuffix(customerID, "/versions")
|
|
||||||
h.handleInfraBackupVersions(w, r, customerID)
|
|
||||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/infra-backup/"):
|
|
||||||
h.handleInfraBackupGet(w, r, strings.TrimPrefix(path, "/infra-backup/"))
|
|
||||||
case r.Method == http.MethodPost && path == "/preferences":
|
case r.Method == http.MethodPost && path == "/preferences":
|
||||||
h.handleSavePreferences(w, r)
|
h.handleSavePreferences(w, r)
|
||||||
case r.Method == http.MethodGet && path == "/customers":
|
case r.Method == http.MethodGet && path == "/customers":
|
||||||
@@ -1244,94 +1236,14 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
|
|||||||
w.Write([]byte(`{"status":"ok"}`))
|
w.Write([]byte(`{"status":"ok"}`))
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleInfraBackupPush stores an infrastructure snapshot from a controller.
|
// handleRecovery returns the generated controller.yaml for disaster recovery.
|
||||||
func (h *Handler) handleInfraBackupPush(w http.ResponseWriter, r *http.Request) {
|
|
||||||
if !h.checkAuth(r) {
|
|
||||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20)) // 1MB limit
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, "Bad request", http.StatusBadRequest)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
var payload struct {
|
|
||||||
CustomerID string `json:"customer_id"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(body, &payload); err != nil || payload.CustomerID == "" {
|
|
||||||
http.Error(w, "Invalid payload: customer_id required", http.StatusBadRequest)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := h.store.SaveInfraBackup(payload.CustomerID, body); err != nil {
|
|
||||||
h.logger.Printf("[ERROR] Failed to save infra backup for %s: %v", payload.CustomerID, err)
|
|
||||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
h.logger.Printf("[INFO] Infra backup saved for %s (%d bytes)", payload.CustomerID, len(body))
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
w.Write([]byte(`{"status":"ok"}`))
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleInfraBackupGet returns the infrastructure backup for a customer.
|
|
||||||
func (h *Handler) handleInfraBackupGet(w http.ResponseWriter, r *http.Request, customerID string) {
|
|
||||||
if !h.checkAuth(r) {
|
|
||||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if customerID == "" {
|
|
||||||
http.Error(w, "Missing customer_id", http.StatusBadRequest)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
data, err := h.store.GetInfraBackup(customerID)
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Printf("[ERROR] Failed to get infra backup for %s: %v", customerID, err)
|
|
||||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if data == nil {
|
|
||||||
http.NotFound(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
w.Write(data)
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleInfraBackupVersions returns a list of backup versions for a customer.
|
|
||||||
// Auth: Bearer token.
|
|
||||||
func (h *Handler) handleInfraBackupVersions(w http.ResponseWriter, r *http.Request, customerID string) {
|
|
||||||
if !h.checkAuth(r) {
|
|
||||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if customerID == "" {
|
|
||||||
http.Error(w, "Missing customer_id", http.StatusBadRequest)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
versions, err := h.store.ListInfraBackupVersions(customerID)
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Printf("[ERROR] Failed to list infra backup versions for %s: %v", customerID, err)
|
|
||||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if versions == nil {
|
|
||||||
versions = []store.InfraBackupVersion{}
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
json.NewEncoder(w).Encode(versions)
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleRecovery returns both the generated controller.yaml and the infra backup for disaster recovery.
|
|
||||||
// Auth: X-Retrieval-Password header (same as config retrieval).
|
// Auth: X-Retrieval-Password header (same as config retrieval).
|
||||||
|
//
|
||||||
|
// The infra-backup payload was retired (Phase-1, 2026-06-16): it pushed plaintext
|
||||||
|
// customer secrets to the hub (a zero-knowledge violation) and had been dead since
|
||||||
|
// slice 8C. DR config now comes from the generated controller.yaml here; the data
|
||||||
|
// bytes come from the agent's PBS whole-CT snapshot. A secret-free DR recipe is the
|
||||||
|
// later DR slice's job.
|
||||||
func (h *Handler) handleRecovery(w http.ResponseWriter, r *http.Request, customerID string) {
|
func (h *Handler) handleRecovery(w http.ResponseWriter, r *http.Request, customerID string) {
|
||||||
if customerID == "" {
|
if customerID == "" {
|
||||||
http.Error(w, "Missing customer_id", http.StatusBadRequest)
|
http.Error(w, "Missing customer_id", http.StatusBadRequest)
|
||||||
@@ -1372,47 +1284,19 @@ func (h *Handler) handleRecovery(w http.ResponseWriter, r *http.Request, custome
|
|||||||
configYAML = yamlOutput
|
configYAML = yamlOutput
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fetch infra backup (optional — may not exist for new customers)
|
// infra_backup retired: the response keeps has_infra_backup=false so any old client
|
||||||
var infraBackup json.RawMessage
|
// degrades gracefully to the config_yaml-only path.
|
||||||
hasInfraBackup := false
|
|
||||||
|
|
||||||
// Support ?version=ID for selecting a specific backup version
|
|
||||||
if versionStr := r.URL.Query().Get("version"); versionStr != "" {
|
|
||||||
var versionID int64
|
|
||||||
if _, err := fmt.Sscanf(versionStr, "%d", &versionID); err == nil {
|
|
||||||
if data, err := h.store.GetInfraBackupByID(versionID); err == nil && data != nil {
|
|
||||||
infraBackup = data
|
|
||||||
hasInfraBackup = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if data, err := h.store.GetInfraBackup(customerID); err == nil && data != nil {
|
|
||||||
infraBackup = data
|
|
||||||
hasInfraBackup = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Include version list for version picker
|
|
||||||
var backupVersions []store.InfraBackupVersion
|
|
||||||
if versions, err := h.store.ListInfraBackupVersions(customerID); err == nil {
|
|
||||||
backupVersions = versions
|
|
||||||
}
|
|
||||||
|
|
||||||
resp := struct {
|
resp := struct {
|
||||||
CustomerID string `json:"customer_id"`
|
CustomerID string `json:"customer_id"`
|
||||||
ConfigYAML string `json:"config_yaml"`
|
ConfigYAML string `json:"config_yaml"`
|
||||||
InfraBackup json.RawMessage `json:"infra_backup"`
|
HasInfraBackup bool `json:"has_infra_backup"`
|
||||||
HasInfraBackup bool `json:"has_infra_backup"`
|
|
||||||
BackupVersions []store.InfraBackupVersion `json:"backup_versions,omitempty"`
|
|
||||||
}{
|
}{
|
||||||
CustomerID: customerID,
|
CustomerID: customerID,
|
||||||
ConfigYAML: configYAML,
|
ConfigYAML: configYAML,
|
||||||
InfraBackup: infraBackup,
|
HasInfraBackup: false,
|
||||||
HasInfraBackup: hasInfraBackup,
|
|
||||||
BackupVersions: backupVersions,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
h.logger.Printf("[INFO] Recovery data downloaded for customer %s (has_infra_backup=%v, versions=%d)", customerID, hasInfraBackup, len(backupVersions))
|
h.logger.Printf("[INFO] Recovery data downloaded for customer %s (config only; infra-backup retired)", customerID)
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
json.NewEncoder(w).Encode(resp)
|
json.NewEncoder(w).Encode(resp)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,120 @@
|
|||||||
package monitor
|
package monitor
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// backupStaleAfter is the maximum age of the newest offsite backup before the daily
|
||||||
|
// deadline check raises expected_backup_missed. 26h covers an evening backup schedule
|
||||||
|
// (e.g. ~18:00–22:00) plus headroom, so a healthy once-daily cadence never trips the
|
||||||
|
// early-morning check.
|
||||||
|
const backupStaleAfter = 26 * time.Hour
|
||||||
|
|
||||||
|
// hostReportBackups is the minimal slice of an agent host-report the deadline check
|
||||||
|
// reads to judge backup freshness (pbs_snapshots is the offsite-DR signal; backups is
|
||||||
|
// the local vzdump fallback). Mirrors the agent's hub.PBSSnapshot / hub.Backup wire
|
||||||
|
// contract for just the fields we need.
|
||||||
|
type hostReportBackups struct {
|
||||||
|
PBSSnapshots []struct {
|
||||||
|
BackupTime string `json:"backup_time"`
|
||||||
|
VerifyState string `json:"verify_state"`
|
||||||
|
} `json:"pbs_snapshots"`
|
||||||
|
Backups []struct {
|
||||||
|
StartedAt string `json:"started_at"`
|
||||||
|
Success bool `json:"success"`
|
||||||
|
} `json:"backups"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// backupAssessment is the verdict for one customer's offsite backup health.
|
||||||
|
type backupAssessment struct {
|
||||||
|
missed bool // raise expected_backup_missed
|
||||||
|
reason string // human-readable cause (event message + logs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assessBackupFreshness decides whether a customer's latest host-report shows a healthy,
|
||||||
|
// recent backup. Pure (now is injected) so the policy is unit-tested. Only POSITIVE
|
||||||
|
// evidence of a problem fires an alarm:
|
||||||
|
// - no PBS snapshot AND no successful vzdump in the report → missed ("no backup recorded")
|
||||||
|
// - newest backup older than backupStaleAfter → missed ("stale")
|
||||||
|
// - the newest PBS snapshot's verify_state is "failed" → missed ("verify failed")
|
||||||
|
//
|
||||||
|
// A fresh-but-not-yet-verified snapshot (verify_state "none"/"") is NOT treated as a
|
||||||
|
// failure: PBS verification runs on its own cadence, so a snapshot taken hours before the
|
||||||
|
// 03:00 check may legitimately be unverified. Alarming on that would re-introduce exactly
|
||||||
|
// the daily false alarm this repoint removes (hence "failed" only, not "≠ ok").
|
||||||
|
func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment {
|
||||||
|
var hr hostReportBackups
|
||||||
|
if err := json.Unmarshal([]byte(reportJSON), &hr); err != nil {
|
||||||
|
// Unparseable report → can't confirm a backup. Surface it rather than swallow it.
|
||||||
|
return backupAssessment{missed: true, reason: "latest host-report could not be parsed"}
|
||||||
|
}
|
||||||
|
|
||||||
|
var newestPBS time.Time
|
||||||
|
var newestPBSVerify string
|
||||||
|
havePBS := false
|
||||||
|
for _, ps := range hr.PBSSnapshots {
|
||||||
|
t, ok := parseBackupTime(ps.BackupTime)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !havePBS || t.After(newestPBS) {
|
||||||
|
havePBS = true
|
||||||
|
newestPBS = t
|
||||||
|
newestPBSVerify = strings.ToLower(strings.TrimSpace(ps.VerifyState))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var newestVzdump time.Time
|
||||||
|
haveVzdump := false
|
||||||
|
for _, b := range hr.Backups {
|
||||||
|
if !b.Success {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
t, ok := parseBackupTime(b.StartedAt)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !haveVzdump || t.After(newestVzdump) {
|
||||||
|
haveVzdump = true
|
||||||
|
newestVzdump = t
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !havePBS && !haveVzdump {
|
||||||
|
return backupAssessment{missed: true, reason: "no PBS snapshot or successful backup in the latest host-report"}
|
||||||
|
}
|
||||||
|
|
||||||
|
newest := newestPBS
|
||||||
|
if haveVzdump && (!havePBS || newestVzdump.After(newest)) {
|
||||||
|
newest = newestVzdump
|
||||||
|
}
|
||||||
|
if age := now.Sub(newest); age > backupStaleAfter {
|
||||||
|
return backupAssessment{missed: true, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)}
|
||||||
|
}
|
||||||
|
if havePBS && newestPBSVerify == "failed" {
|
||||||
|
return backupAssessment{missed: true, reason: "newest PBS snapshot failed verification"}
|
||||||
|
}
|
||||||
|
return backupAssessment{missed: false}
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseBackupTime parses an RFC3339 timestamp from a host-report and normalizes to UTC.
|
||||||
|
func parseBackupTime(s string) (time.Time, bool) {
|
||||||
|
s = strings.TrimSpace(s)
|
||||||
|
if s == "" {
|
||||||
|
return time.Time{}, false
|
||||||
|
}
|
||||||
|
if t, err := time.Parse(time.RFC3339, s); err == nil {
|
||||||
|
return t.UTC(), true
|
||||||
|
}
|
||||||
|
return time.Time{}, false
|
||||||
|
}
|
||||||
|
|
||||||
// budapest returns the Europe/Budapest timezone (cached).
|
// budapest returns the Europe/Budapest timezone (cached).
|
||||||
var budapest *time.Location
|
var budapest *time.Location
|
||||||
|
|
||||||
@@ -20,11 +128,16 @@ func init() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// CheckBackupDeadlines checks whether active customers had their expected
|
// CheckBackupDeadlines checks whether active customers had their expected
|
||||||
// daily backups and DB dumps. Runs once daily at 05:00 Budapest time.
|
// daily backups and DB dumps. Runs once daily (early morning, Budapest time).
|
||||||
//
|
//
|
||||||
// For each active customer, it checks for backup_completed and db_dump_completed
|
// Backup half: read the agent's latest host-report and raise expected_backup_missed
|
||||||
// events since Budapest midnight. If neither success nor failure events exist,
|
// only when its PBS snapshots / vzdump backups show no fresh, verified backup (see
|
||||||
// it inserts expected_backup_missed / expected_dbdump_missed events.
|
// assessBackupFreshness). This replaced the old backup_completed-event check, which
|
||||||
|
// fired daily for every healthy customer because no component emits that event anymore
|
||||||
|
// (the controller's disk-tier backup moved to the agent in slice 8C).
|
||||||
|
//
|
||||||
|
// DB-dump half: unchanged — the in-guest controller still emits db_dump_completed, so
|
||||||
|
// the event-based check there is correct.
|
||||||
//
|
//
|
||||||
// Customers whose nodes are "down" (no report in >1h) are skipped — they
|
// Customers whose nodes are "down" (no report in >1h) are skipped — they
|
||||||
// already have staleness events.
|
// already have staleness events.
|
||||||
@@ -54,17 +167,27 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check backup_completed / backup_failed since midnight
|
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
||||||
backupOK, _ := s.GetEventsByType(id, "backup_completed", sinceUTC)
|
// the authoritative offsite-backup signal post-slice-8C.
|
||||||
backupFailed, _ := s.GetEventsByType(id, "backup_failed", sinceUTC)
|
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
||||||
if len(backupOK) == 0 && len(backupFailed) == 0 {
|
switch {
|
||||||
msg := "No backup completed or failed since midnight"
|
case rerr != nil:
|
||||||
if _, err := s.SaveEvent(id, "expected_backup_missed", "error", msg, "{}", "hub"); err != nil {
|
logger.Printf("[WARN] Deadline check: failed to read host-report for %s: %v", id, rerr)
|
||||||
logger.Printf("[WARN] Failed to save expected_backup_missed for %s: %v", id, err)
|
case reportJSON == "":
|
||||||
} else if onEvent != nil {
|
// No agent host-report at all (legacy/defunct controller-only customer).
|
||||||
onEvent(id, "expected_backup_missed", "error", msg, "{}", "hub")
|
// Liveness is owned by the host-staleness checker; the backup deadline check
|
||||||
|
// has no PBS data to judge here and must not emit a daily backup alarm of its
|
||||||
|
// own. (The DB-dump half below still applies.)
|
||||||
|
default:
|
||||||
|
if a := assessBackupFreshness(reportJSON, time.Now().UTC()); a.missed {
|
||||||
|
msg := "No fresh verified backup: " + a.reason
|
||||||
|
if _, err := s.SaveEvent(id, "expected_backup_missed", "error", msg, "{}", "hub"); err != nil {
|
||||||
|
logger.Printf("[WARN] Failed to save expected_backup_missed for %s: %v", id, err)
|
||||||
|
} else if onEvent != nil {
|
||||||
|
onEvent(id, "expected_backup_missed", "error", msg, "{}", "hub")
|
||||||
|
}
|
||||||
|
backupMissed++
|
||||||
}
|
}
|
||||||
backupMissed++
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check db_dump_completed / db_dump_failed since midnight
|
// Check db_dump_completed / db_dump_failed since midnight
|
||||||
|
|||||||
@@ -0,0 +1,201 @@
|
|||||||
|
package monitor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newDeadlineStore creates an isolated store with one active customer + a host row.
|
||||||
|
func newDeadlineStore(t *testing.T) *store.Store {
|
||||||
|
t.Helper()
|
||||||
|
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("store.New: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { st.Close() })
|
||||||
|
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
|
||||||
|
t.Fatalf("SaveCustomerConfig: %v", err)
|
||||||
|
}
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
return st
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostReportJSON builds a host-report payload with the given PBS snapshots and vzdump
|
||||||
|
// backups. Each snapshot is {backup_time, verify_state}; each backup is {started_at, success}.
|
||||||
|
func hostReportJSON(t *testing.T, pbs [][2]string, backups []struct {
|
||||||
|
at string
|
||||||
|
ok bool
|
||||||
|
}) string {
|
||||||
|
t.Helper()
|
||||||
|
type snap struct {
|
||||||
|
BackupTime string `json:"backup_time"`
|
||||||
|
VerifyState string `json:"verify_state"`
|
||||||
|
}
|
||||||
|
type bk struct {
|
||||||
|
StartedAt string `json:"started_at"`
|
||||||
|
Success bool `json:"success"`
|
||||||
|
}
|
||||||
|
payload := struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||||
|
Backups []bk `json:"backups"`
|
||||||
|
}{HostID: "h1"}
|
||||||
|
for _, p := range pbs {
|
||||||
|
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{BackupTime: p[0], VerifyState: p[1]})
|
||||||
|
}
|
||||||
|
for _, b := range backups {
|
||||||
|
payload.Backups = append(payload.Backups, bk{StartedAt: b.at, Success: b.ok})
|
||||||
|
}
|
||||||
|
out, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal report: %v", err)
|
||||||
|
}
|
||||||
|
return string(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// runDeadline runs CheckBackupDeadlines and returns the list of emitted event types for c1.
|
||||||
|
func runDeadline(t *testing.T, st *store.Store) []string {
|
||||||
|
t.Helper()
|
||||||
|
var got []string
|
||||||
|
onEvent := func(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||||
|
if customerID == "c1" {
|
||||||
|
got = append(got, eventType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// nil staleness → no "down" skip; the check evaluates c1.
|
||||||
|
CheckBackupDeadlines(st, nil, onEvent, log.New(io.Discard, "", 0))
|
||||||
|
return got
|
||||||
|
}
|
||||||
|
|
||||||
|
func has(events []string, t string) bool {
|
||||||
|
for _, e := range events {
|
||||||
|
if e == t {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func rfc(d time.Duration) string {
|
||||||
|
return time.Now().UTC().Add(d).Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_FreshVerifiedPBS_NoBackupAlarm is the COMPANION test.
|
||||||
|
// It FAILS against the pre-fix (event-based) check: with a fresh, verified PBS snapshot
|
||||||
|
// but no backup_completed event, the old code emitted expected_backup_missed anyway.
|
||||||
|
// The repoint reads the host-report instead, so a healthy customer raises no alarm.
|
||||||
|
func TestCheckBackupDeadlines_FreshVerifiedPBS_NoBackupAlarm(t *testing.T) {
|
||||||
|
st := newDeadlineStore(t)
|
||||||
|
// Make the db-dump half pass too, so the only thing under test is the backup half.
|
||||||
|
if _, err := st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-3 * time.Hour), "ok"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("fresh+verified PBS must NOT raise expected_backup_missed; got %v", got)
|
||||||
|
}
|
||||||
|
if has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("db_dump_completed present → no dbdump alarm expected; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_StalePBS_Alarms: newest backup older than 26h → alarm.
|
||||||
|
func TestCheckBackupDeadlines_StalePBS_Alarms(t *testing.T) {
|
||||||
|
st := newDeadlineStore(t)
|
||||||
|
st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller")
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-30 * time.Hour), "ok"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("stale (>26h) PBS snapshot must raise expected_backup_missed; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_FailedVerify_Alarms: fresh snapshot but verify failed → alarm.
|
||||||
|
func TestCheckBackupDeadlines_FailedVerify_Alarms(t *testing.T) {
|
||||||
|
st := newDeadlineStore(t)
|
||||||
|
st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller")
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-2 * time.Hour), "failed"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("failed PBS verify must raise expected_backup_missed; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_NoHostReport_NoBackupAlarm: a customer with no host-report
|
||||||
|
// (legacy/defunct controller-only) must NOT get a backup alarm from this check —
|
||||||
|
// liveness is the staleness checker's job.
|
||||||
|
func TestCheckBackupDeadlines_NoHostReport_NoBackupAlarm(t *testing.T) {
|
||||||
|
st := newDeadlineStore(t)
|
||||||
|
st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller")
|
||||||
|
// No SaveHostReport call.
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("no host-report → no backup alarm; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_DbDumpHalfPreserved: fresh backup (no backup alarm) but a
|
||||||
|
// missing db_dump_completed event must still raise expected_dbdump_missed.
|
||||||
|
func TestCheckBackupDeadlines_DbDumpHalfPreserved(t *testing.T) {
|
||||||
|
st := newDeadlineStore(t)
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-2 * time.Hour), "ok"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("fresh backup → no backup alarm; got %v", got)
|
||||||
|
}
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("missing db_dump_completed must still raise expected_dbdump_missed; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAssessBackupFreshness exercises the pure freshness policy directly.
|
||||||
|
func TestAssessBackupFreshness(t *testing.T) {
|
||||||
|
now := time.Date(2026, 6, 16, 3, 0, 0, 0, time.UTC)
|
||||||
|
at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) }
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
report string
|
||||||
|
wantMissed bool
|
||||||
|
}{
|
||||||
|
{"fresh verified PBS", `{"pbs_snapshots":[{"backup_time":"` + at(-3*time.Hour) + `","verify_state":"ok"}]}`, false},
|
||||||
|
{"fresh unverified (none) is not a failure", `{"pbs_snapshots":[{"backup_time":"` + at(-3*time.Hour) + `","verify_state":"none"}]}`, false},
|
||||||
|
{"stale verified", `{"pbs_snapshots":[{"backup_time":"` + at(-30*time.Hour) + `","verify_state":"ok"}]}`, true},
|
||||||
|
{"fresh but verify failed", `{"pbs_snapshots":[{"backup_time":"` + at(-2*time.Hour) + `","verify_state":"failed"}]}`, true},
|
||||||
|
{"no snapshots and no backups", `{"pbs_snapshots":[],"backups":[]}`, true},
|
||||||
|
{"vzdump fallback fresh success", `{"backups":[{"started_at":"` + at(-4*time.Hour) + `","success":true}]}`, false},
|
||||||
|
{"vzdump only, failed → counts as none", `{"backups":[{"started_at":"` + at(-4*time.Hour) + `","success":false}]}`, true},
|
||||||
|
{"newest vzdump fresh rescues stale PBS", `{"pbs_snapshots":[{"backup_time":"` + at(-40*time.Hour) + `","verify_state":"ok"}],"backups":[{"started_at":"` + at(-2*time.Hour) + `","success":true}]}`, false},
|
||||||
|
{"unparseable report", `not json`, true},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
got := assessBackupFreshness(c.report, now)
|
||||||
|
if got.missed != c.wantMissed {
|
||||||
|
t.Fatalf("missed=%v want=%v (reason=%q)", got.missed, c.wantMissed, got.reason)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
+45
-291
@@ -94,12 +94,6 @@ func (s *Store) migrate() error {
|
|||||||
CREATE INDEX IF NOT EXISTS idx_notification_log_customer
|
CREATE INDEX IF NOT EXISTS idx_notification_log_customer
|
||||||
ON notification_log(customer_id, created_at DESC);
|
ON notification_log(customer_id, created_at DESC);
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS infra_backups (
|
|
||||||
customer_id TEXT PRIMARY KEY,
|
|
||||||
backup_json TEXT NOT NULL,
|
|
||||||
updated_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS customer_configs (
|
CREATE TABLE IF NOT EXISTS customer_configs (
|
||||||
customer_id TEXT PRIMARY KEY,
|
customer_id TEXT PRIMARY KEY,
|
||||||
customer_name TEXT NOT NULL DEFAULT '',
|
customer_name TEXT NOT NULL DEFAULT '',
|
||||||
@@ -196,27 +190,31 @@ func (s *Store) migrate() error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.7.0: versioned infra backups with GFS retention
|
// Phase-1 retire (2026-06-16): the Infra Backup mechanism is gone. It pushed
|
||||||
_, err = s.db.Exec(`
|
// plaintext customer secrets to the hub (the app-secret encryption key, restic
|
||||||
CREATE TABLE IF NOT EXISTS infra_backup_versions (
|
// password, Cloudflare tokens — a zero-knowledge violation) and had been dead since
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
// slice 8C. Drop both tables and VACUUM so the freed pages — which still hold the
|
||||||
customer_id TEXT NOT NULL,
|
// plaintext — are physically reclaimed from the DB file, not merely delinked.
|
||||||
backup_json TEXT NOT NULL,
|
// Gated on existence so ordinary restarts don't pay the VACUUM cost.
|
||||||
created_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
var infraTables int
|
||||||
);
|
s.db.QueryRow(`SELECT COUNT(*) FROM sqlite_master WHERE type='table'
|
||||||
CREATE INDEX IF NOT EXISTS idx_ibv_customer_time
|
AND name IN ('infra_backup_versions','infra_backups')`).Scan(&infraTables)
|
||||||
ON infra_backup_versions(customer_id, created_at DESC);
|
if infraTables > 0 {
|
||||||
`)
|
if _, err = s.db.Exec(`DROP TABLE IF EXISTS infra_backup_versions;
|
||||||
if err != nil {
|
DROP TABLE IF EXISTS infra_backups;`); err != nil {
|
||||||
return err
|
return fmt.Errorf("dropping retired infra_backup tables: %w", err)
|
||||||
|
}
|
||||||
|
// VACUUM rewrites the database file, discarding the freed (plaintext) pages.
|
||||||
|
if _, err = s.db.Exec(`VACUUM`); err != nil {
|
||||||
|
return fmt.Errorf("vacuum after infra_backup drop: %w", err)
|
||||||
|
}
|
||||||
|
// WAL checkpoint(TRUNCATE) so no plaintext lingers in the -wal sidecar either.
|
||||||
|
s.db.Exec(`PRAGMA wal_checkpoint(TRUNCATE)`)
|
||||||
|
if s.logger != nil {
|
||||||
|
s.logger.Printf("[INFO] Retired infra-backup: dropped %d table(s) and VACUUMed to reclaim plaintext pages", infraTables)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// One-time migration: copy existing single-row backups to versioned table
|
|
||||||
s.db.Exec(`INSERT INTO infra_backup_versions (customer_id, backup_json, created_at)
|
|
||||||
SELECT customer_id, backup_json, updated_at FROM infra_backups
|
|
||||||
WHERE NOT EXISTS (SELECT 1 FROM infra_backup_versions
|
|
||||||
WHERE infra_backup_versions.customer_id = infra_backups.customer_id)`)
|
|
||||||
|
|
||||||
// v0.7.0: host-domain (slice 3). Purely additive — the controller path
|
// v0.7.0: host-domain (slice 3). Purely additive — the controller path
|
||||||
// (reports/customer_configs) is untouched; the schema cutover is slice 10.
|
// (reports/customer_configs) is untouched; the schema cutover is slice 10.
|
||||||
// Columns marked INERT exist now so slice 10 needs no ALTER; nothing reads or
|
// Columns marked INERT exist now so slice 10 needs no ALTER; nothing reads or
|
||||||
@@ -633,272 +631,8 @@ func (s *Store) GetCustomerHistory(customerID string, since time.Duration) ([]Cu
|
|||||||
return history, rows.Err()
|
return history, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
// SaveInfraBackup inserts a new infra backup version and prunes old ones (GFS retention).
|
// (Infra-backup store methods retired 2026-06-16. The tables are dropped + VACUUMed
|
||||||
func (s *Store) SaveInfraBackup(customerID string, backupJSON []byte) error {
|
// in migrate(); the push/get/versions handlers and the operator panel are gone.)
|
||||||
_, err := s.db.Exec(`
|
|
||||||
INSERT INTO infra_backup_versions (customer_id, backup_json, created_at)
|
|
||||||
VALUES (?, ?, datetime('now'))`,
|
|
||||||
customerID, string(backupJSON),
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Also maintain legacy table for backward compatibility during rollback window
|
|
||||||
s.db.Exec(`INSERT INTO infra_backups (customer_id, backup_json, updated_at)
|
|
||||||
VALUES (?, ?, datetime('now'))
|
|
||||||
ON CONFLICT(customer_id) DO UPDATE SET
|
|
||||||
backup_json = excluded.backup_json,
|
|
||||||
updated_at = datetime('now')`,
|
|
||||||
customerID, string(backupJSON))
|
|
||||||
|
|
||||||
s.pruneInfraBackups(customerID)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetInfraBackup returns the latest infra backup JSON for a customer, or nil if not found.
|
|
||||||
func (s *Store) GetInfraBackup(customerID string) ([]byte, error) {
|
|
||||||
var data string
|
|
||||||
err := s.db.QueryRow(
|
|
||||||
"SELECT backup_json FROM infra_backup_versions WHERE customer_id = ? ORDER BY created_at DESC LIMIT 1",
|
|
||||||
customerID,
|
|
||||||
).Scan(&data)
|
|
||||||
if err == sql.ErrNoRows {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return []byte(data), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetInfraBackupByID returns the infra backup JSON for a specific version ID, or nil if not found.
|
|
||||||
func (s *Store) GetInfraBackupByID(id int64) ([]byte, error) {
|
|
||||||
var data string
|
|
||||||
err := s.db.QueryRow(
|
|
||||||
"SELECT backup_json FROM infra_backup_versions WHERE id = ?", id,
|
|
||||||
).Scan(&data)
|
|
||||||
if err == sql.ErrNoRows {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return []byte(data), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// InfraBackupMeta holds summary info for the dashboard (avoids parsing full JSON).
|
|
||||||
type InfraBackupMeta struct {
|
|
||||||
UpdatedAt time.Time
|
|
||||||
StackCount int
|
|
||||||
DiskCount int
|
|
||||||
VersionCount int
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetInfraBackupMeta returns summary metadata for a customer's latest infra backup.
|
|
||||||
func (s *Store) GetInfraBackupMeta(customerID string) (*InfraBackupMeta, error) {
|
|
||||||
var backupJSON, createdAt string
|
|
||||||
err := s.db.QueryRow(
|
|
||||||
"SELECT backup_json, created_at FROM infra_backup_versions WHERE customer_id = ? ORDER BY created_at DESC LIMIT 1",
|
|
||||||
customerID,
|
|
||||||
).Scan(&backupJSON, &createdAt)
|
|
||||||
if err == sql.ErrNoRows {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
meta := &InfraBackupMeta{
|
|
||||||
UpdatedAt: parseSQLiteTime(createdAt),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Count total versions
|
|
||||||
s.db.QueryRow("SELECT COUNT(*) FROM infra_backup_versions WHERE customer_id = ?", customerID).Scan(&meta.VersionCount)
|
|
||||||
|
|
||||||
// Parse just the fields we need
|
|
||||||
parseInfraBackupCounts(backupJSON, &meta.StackCount, &meta.DiskCount, nil, s.logger, customerID)
|
|
||||||
|
|
||||||
return meta, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// InfraBackupVersion holds summary info for a single backup version.
|
|
||||||
type InfraBackupVersion struct {
|
|
||||||
ID int64 `json:"id"`
|
|
||||||
CreatedAt time.Time `json:"created_at"`
|
|
||||||
StackCount int `json:"stack_count"`
|
|
||||||
DiskCount int `json:"disk_count"`
|
|
||||||
StackNames []string `json:"stack_names,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// ListInfraBackupVersions returns metadata for all retained versions of a customer's backup.
|
|
||||||
func (s *Store) ListInfraBackupVersions(customerID string) ([]InfraBackupVersion, error) {
|
|
||||||
rows, err := s.db.Query(
|
|
||||||
"SELECT id, backup_json, created_at FROM infra_backup_versions WHERE customer_id = ? ORDER BY created_at DESC",
|
|
||||||
customerID,
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
defer rows.Close()
|
|
||||||
|
|
||||||
var versions []InfraBackupVersion
|
|
||||||
for rows.Next() {
|
|
||||||
var v InfraBackupVersion
|
|
||||||
var backupJSON, createdAt string
|
|
||||||
if err := rows.Scan(&v.ID, &backupJSON, &createdAt); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
v.CreatedAt = parseSQLiteTime(createdAt)
|
|
||||||
parseInfraBackupCounts(backupJSON, &v.StackCount, &v.DiskCount, &v.StackNames, nil, "")
|
|
||||||
versions = append(versions, v)
|
|
||||||
}
|
|
||||||
return versions, rows.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
// pruneInfraBackups applies GFS retention: keep all from last 24h, latest per day (7d),
|
|
||||||
// latest per week (4w), latest per month (3mo). Delete everything else.
|
|
||||||
func (s *Store) pruneInfraBackups(customerID string) {
|
|
||||||
rows, err := s.db.Query(
|
|
||||||
"SELECT id, created_at FROM infra_backup_versions WHERE customer_id = ? ORDER BY created_at DESC",
|
|
||||||
customerID,
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
defer rows.Close()
|
|
||||||
|
|
||||||
type entry struct {
|
|
||||||
id int64
|
|
||||||
createdAt time.Time
|
|
||||||
}
|
|
||||||
var all []entry
|
|
||||||
for rows.Next() {
|
|
||||||
var e entry
|
|
||||||
var ts string
|
|
||||||
if err := rows.Scan(&e.id, &ts); err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
e.createdAt = parseSQLiteTime(ts)
|
|
||||||
all = append(all, e)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(all) <= 1 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
now := time.Now().UTC()
|
|
||||||
keep := make(map[int64]bool)
|
|
||||||
seenDays := make(map[string]bool)
|
|
||||||
seenWeeks := make(map[string]bool)
|
|
||||||
seenMonths := make(map[string]bool)
|
|
||||||
|
|
||||||
for _, e := range all {
|
|
||||||
age := now.Sub(e.createdAt)
|
|
||||||
|
|
||||||
// Keep all from last 24h
|
|
||||||
if age < 24*time.Hour {
|
|
||||||
keep[e.id] = true
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Latest per calendar day for last 7 days
|
|
||||||
if age < 7*24*time.Hour {
|
|
||||||
day := e.createdAt.Format("2006-01-02")
|
|
||||||
if !seenDays[day] {
|
|
||||||
seenDays[day] = true
|
|
||||||
keep[e.id] = true
|
|
||||||
}
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Latest per ISO week for last 4 weeks
|
|
||||||
if age < 28*24*time.Hour {
|
|
||||||
year, week := e.createdAt.ISOWeek()
|
|
||||||
wk := fmt.Sprintf("%d-W%02d", year, week)
|
|
||||||
if !seenWeeks[wk] {
|
|
||||||
seenWeeks[wk] = true
|
|
||||||
keep[e.id] = true
|
|
||||||
}
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Latest per calendar month for last 3 months
|
|
||||||
if age < 90*24*time.Hour {
|
|
||||||
month := e.createdAt.Format("2006-01")
|
|
||||||
if !seenMonths[month] {
|
|
||||||
seenMonths[month] = true
|
|
||||||
keep[e.id] = true
|
|
||||||
}
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// Older than 3 months — don't keep
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build delete list
|
|
||||||
var deleteIDs []interface{}
|
|
||||||
var placeholders []string
|
|
||||||
for _, e := range all {
|
|
||||||
if !keep[e.id] {
|
|
||||||
deleteIDs = append(deleteIDs, e.id)
|
|
||||||
placeholders = append(placeholders, "?")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(deleteIDs) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
query := "DELETE FROM infra_backup_versions WHERE id IN (" + joinStrings(placeholders, ",") + ")"
|
|
||||||
_, err = s.db.Exec(query, deleteIDs...)
|
|
||||||
if err != nil {
|
|
||||||
s.logger.Printf("[WARN] Failed to prune infra backup versions for %s: %v", customerID, err)
|
|
||||||
} else {
|
|
||||||
s.logger.Printf("[INFO] Pruned %d old infra backup version(s) for %s (kept %d)", len(deleteIDs), customerID, len(keep))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseInfraBackupCounts extracts stack/disk counts and optionally stack names from backup JSON.
|
|
||||||
func parseInfraBackupCounts(backupJSON string, stackCount, diskCount *int, stackNames *[]string, logger *log.Logger, customerID string) {
|
|
||||||
var parsed struct {
|
|
||||||
DeployedStacks []struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
DisplayName string `json:"display_name"`
|
|
||||||
} `json:"deployed_stacks"`
|
|
||||||
DiskLayout struct {
|
|
||||||
Mounts []json.RawMessage `json:"mounts"`
|
|
||||||
} `json:"disk_layout"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal([]byte(backupJSON), &parsed); err != nil {
|
|
||||||
if logger != nil {
|
|
||||||
logger.Printf("[WARN] Failed to parse infra backup metadata for %s: %v", customerID, err)
|
|
||||||
}
|
|
||||||
return
|
|
||||||
}
|
|
||||||
*stackCount = len(parsed.DeployedStacks)
|
|
||||||
*diskCount = len(parsed.DiskLayout.Mounts)
|
|
||||||
if stackNames != nil {
|
|
||||||
for _, s := range parsed.DeployedStacks {
|
|
||||||
name := s.DisplayName
|
|
||||||
if name == "" {
|
|
||||||
name = s.Name
|
|
||||||
}
|
|
||||||
*stackNames = append(*stackNames, name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func joinStrings(ss []string, sep string) string {
|
|
||||||
if len(ss) == 0 {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
result := ss[0]
|
|
||||||
for _, s := range ss[1:] {
|
|
||||||
result += sep + s
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
|
|
||||||
// Prune deletes reports older than the given number of days.
|
// Prune deletes reports older than the given number of days.
|
||||||
func (s *Store) Prune(maxDays int) (int64, error) {
|
func (s *Store) Prune(maxDays int) (int64, error) {
|
||||||
@@ -1645,6 +1379,26 @@ func (s *Store) SaveHostReport(hostID, customerID string, reportJSON []byte, d H
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GetLatestHostReportJSON returns the most recent host-report payload (report_json)
|
||||||
|
// for a customer, by received_at, or ("", nil) if the customer has no host-report.
|
||||||
|
// The backup deadline check uses it to read the agent's own backup reality
|
||||||
|
// (pbs_snapshots + vzdump backups) — the authoritative offsite-backup signal
|
||||||
|
// post-slice-8C, replacing the no-longer-emitted backup_completed event.
|
||||||
|
func (s *Store) GetLatestHostReportJSON(customerID string) (string, error) {
|
||||||
|
var j string
|
||||||
|
err := s.db.QueryRow(
|
||||||
|
`SELECT report_json FROM host_reports WHERE customer_id = ? ORDER BY received_at DESC LIMIT 1`,
|
||||||
|
customerID,
|
||||||
|
).Scan(&j)
|
||||||
|
if err == sql.ErrNoRows {
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return j, nil
|
||||||
|
}
|
||||||
|
|
||||||
// UpsertGuestFromReport upserts the REALITY columns of a guest. On conflict it
|
// UpsertGuestFromReport upserts the REALITY columns of a guest. On conflict it
|
||||||
// must NOT clobber the inert columns (api_key / desired_spec_json).
|
// must NOT clobber the inert columns (api_key / desired_spec_json).
|
||||||
func (s *Store) UpsertGuestFromReport(g *Guest) error {
|
func (s *Store) UpsertGuestFromReport(g *Guest) error {
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package web
|
package web
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/base64"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"html/template"
|
"html/template"
|
||||||
@@ -186,34 +185,13 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Config value comparison (parse both YAMLs, compare actual values)
|
// Config drift badge: the at-rest comparison source (infra-backup) was retired
|
||||||
var configSyncStatus string // "in_sync", "mismatch", "unknown"
|
// 2026-06-16. With no stored controller.yaml to diff against, the passive badge is
|
||||||
|
// left empty (the template hides it when ConfigSyncStatus == ""). The live "Show
|
||||||
|
// Diff" path (handleCompareConfig, which fetches the controller's config over HTTP)
|
||||||
|
// is unaffected and remains the way to check drift on demand.
|
||||||
|
var configSyncStatus string // "" hides the badge; "in_sync"/"mismatch" reserved for a future live source
|
||||||
var configDiffCount int
|
var configDiffCount int
|
||||||
if cfg != nil {
|
|
||||||
infraData, _ := s.store.GetInfraBackup(customerID)
|
|
||||||
if infraData != nil {
|
|
||||||
controllerYAML := extractControllerYAML(infraData)
|
|
||||||
if controllerYAML != "" {
|
|
||||||
templateYAML := defaultControllerTemplate
|
|
||||||
if s.templateFetcher != nil {
|
|
||||||
templateYAML = s.templateFetcher.Template()
|
|
||||||
}
|
|
||||||
if hubYAML, err := configgen.Generate(templateYAML, cfg); err == nil {
|
|
||||||
diffs := compareYAMLValues(hubYAML, controllerYAML)
|
|
||||||
configDiffCount = len(diffs)
|
|
||||||
if configDiffCount == 0 {
|
|
||||||
configSyncStatus = "in_sync"
|
|
||||||
} else {
|
|
||||||
configSyncStatus = "mismatch"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
configSyncStatus = "unknown"
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
configSyncStatus = "unknown"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Version check
|
// Version check
|
||||||
var latestVersion string
|
var latestVersion string
|
||||||
@@ -225,13 +203,10 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// History, notifications, events, infra backup
|
// History, notifications, events
|
||||||
var history []store.CustomerSummary
|
var history []store.CustomerSummary
|
||||||
var notifPrefs *store.NotificationPrefs
|
var notifPrefs *store.NotificationPrefs
|
||||||
var recentNotifs []store.NotificationLogEntry
|
var recentNotifs []store.NotificationLogEntry
|
||||||
var infraMeta *store.InfraBackupMeta
|
|
||||||
var infraBackupAge string
|
|
||||||
var infraBackupVersions []store.InfraBackupVersion
|
|
||||||
var events []store.Event
|
var events []store.Event
|
||||||
var eventCounts map[string]int
|
var eventCounts map[string]int
|
||||||
|
|
||||||
@@ -241,11 +216,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
|||||||
history, _ = s.store.GetCustomerHistory(customerID, 24*time.Hour)
|
history, _ = s.store.GetCustomerHistory(customerID, 24*time.Hour)
|
||||||
notifPrefs, _ = s.store.GetNotificationPrefs(customerID)
|
notifPrefs, _ = s.store.GetNotificationPrefs(customerID)
|
||||||
recentNotifs, _ = s.store.GetRecentNotifications(customerID, 10)
|
recentNotifs, _ = s.store.GetRecentNotifications(customerID, 10)
|
||||||
infraMeta, _ = s.store.GetInfraBackupMeta(customerID)
|
|
||||||
if infraMeta != nil {
|
|
||||||
infraBackupAge = timeAgo(infraMeta.UpdatedAt)
|
|
||||||
}
|
|
||||||
infraBackupVersions, _ = s.store.ListInfraBackupVersions(customerID)
|
|
||||||
events, _ = s.store.GetRecentEvents(customerID, 50)
|
events, _ = s.store.GetRecentEvents(customerID, 50)
|
||||||
eventCounts, _ = s.store.CountEventsBySeverity(customerID, time.Now().Add(-24*time.Hour))
|
eventCounts, _ = s.store.CountEventsBySeverity(customerID, time.Now().Add(-24*time.Hour))
|
||||||
appTelemetry, _ = s.store.GetCustomerAppSummary(customerID, time.Now().Add(-7*24*time.Hour))
|
appTelemetry, _ = s.store.GetCustomerAppSummary(customerID, time.Now().Add(-7*24*time.Hour))
|
||||||
@@ -274,9 +244,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
|||||||
ConfigSyncStatus string // "in_sync", "mismatch", "unknown"
|
ConfigSyncStatus string // "in_sync", "mismatch", "unknown"
|
||||||
ConfigDiffCount int
|
ConfigDiffCount int
|
||||||
|
|
||||||
InfraBackup *store.InfraBackupMeta
|
|
||||||
InfraBackupAge string
|
|
||||||
InfraBackupVersions []store.InfraBackupVersion
|
|
||||||
NotifPrefs *store.NotificationPrefs
|
NotifPrefs *store.NotificationPrefs
|
||||||
RecentNotifications []store.NotificationLogEntry
|
RecentNotifications []store.NotificationLogEntry
|
||||||
History []store.CustomerSummary
|
History []store.CustomerSummary
|
||||||
@@ -316,9 +283,6 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
|||||||
ConfigSyncStatus: configSyncStatus,
|
ConfigSyncStatus: configSyncStatus,
|
||||||
ConfigDiffCount: configDiffCount,
|
ConfigDiffCount: configDiffCount,
|
||||||
|
|
||||||
InfraBackup: infraMeta,
|
|
||||||
InfraBackupAge: infraBackupAge,
|
|
||||||
InfraBackupVersions: infraBackupVersions,
|
|
||||||
NotifPrefs: notifPrefs,
|
NotifPrefs: notifPrefs,
|
||||||
RecentNotifications: recentNotifs,
|
RecentNotifications: recentNotifs,
|
||||||
History: history,
|
History: history,
|
||||||
@@ -734,22 +698,7 @@ func buildConfigJSON(r *http.Request) string {
|
|||||||
return string(data)
|
return string(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Config comparison helpers ---
|
// --- Config comparison helpers (used by the live "Show Diff" handler) ---
|
||||||
|
|
||||||
// extractControllerYAML decodes the controller.yaml from an infra backup JSON payload.
|
|
||||||
func extractControllerYAML(infraData []byte) string {
|
|
||||||
var parsed struct {
|
|
||||||
ControllerConfigB64 string `json:"controller_config_b64"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(infraData, &parsed); err != nil || parsed.ControllerConfigB64 == "" {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
data, err := base64.StdEncoding.DecodeString(parsed.ControllerConfigB64)
|
|
||||||
if err != nil {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
return string(data)
|
|
||||||
}
|
|
||||||
|
|
||||||
// volatileKeys are YAML keys ignored during config comparison (always differ or deprecated).
|
// volatileKeys are YAML keys ignored during config comparison (always differ or deprecated).
|
||||||
var volatileKeys = map[string]bool{
|
var volatileKeys = map[string]bool{
|
||||||
|
|||||||
@@ -133,29 +133,6 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<!-- Infra Backup (Disaster Recovery) -->
|
|
||||||
<section class="card">
|
|
||||||
<h2>Infra Backup</h2>
|
|
||||||
{{if .InfraBackup}}
|
|
||||||
<div class="info-grid">
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Last Updated</span>
|
|
||||||
<span class="value">{{.InfraBackupAge}}</span>
|
|
||||||
</div>
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Deployed Stacks</span>
|
|
||||||
<span class="value">{{.InfraBackup.StackCount}}</span>
|
|
||||||
</div>
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Disks</span>
|
|
||||||
<span class="value">{{.InfraBackup.DiskCount}}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
{{else}}
|
|
||||||
<p style="color: #facc15">No infra backup received yet</p>
|
|
||||||
{{end}}
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- Health -->
|
<!-- Health -->
|
||||||
<section class="card">
|
<section class="card">
|
||||||
<h2>Health</h2>
|
<h2>Health</h2>
|
||||||
|
|||||||
@@ -302,55 +302,6 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
<!-- Infra Backup -->
|
|
||||||
<section class="card">
|
|
||||||
<h2>Infra Backup</h2>
|
|
||||||
{{if .InfraBackup}}
|
|
||||||
<div class="info-grid">
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Last Updated</span>
|
|
||||||
<span class="value">{{.InfraBackupAge}}</span>
|
|
||||||
</div>
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Deployed Stacks</span>
|
|
||||||
<span class="value">{{.InfraBackup.StackCount}}</span>
|
|
||||||
</div>
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Disks</span>
|
|
||||||
<span class="value">{{.InfraBackup.DiskCount}}</span>
|
|
||||||
</div>
|
|
||||||
<div class="info-item">
|
|
||||||
<span class="label">Versions</span>
|
|
||||||
<span class="value">{{.InfraBackup.VersionCount}}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
{{if .InfraBackupVersions}}
|
|
||||||
<details style="margin-top: 0.75rem;">
|
|
||||||
<summary style="cursor: pointer; color: var(--text-secondary, #94a3b8); font-size: 0.85em;">Backup History ({{len .InfraBackupVersions}} versions)</summary>
|
|
||||||
<table style="width: 100%; margin-top: 0.5rem; font-size: 0.85em;">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th style="text-align: left; padding: 0.25rem 0.5rem;">Date</th>
|
|
||||||
<th style="text-align: left; padding: 0.25rem 0.5rem;">Apps</th>
|
|
||||||
<th style="text-align: right; padding: 0.25rem 0.5rem;">Disks</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
{{range .InfraBackupVersions}}
|
|
||||||
<tr>
|
|
||||||
<td style="padding: 0.25rem 0.5rem;">{{.CreatedAt.Format "2006-01-02 15:04"}}</td>
|
|
||||||
<td style="padding: 0.25rem 0.5rem;">{{.StackCount}}{{if .StackNames}}: {{range $i, $n := .StackNames}}{{if $i}}, {{end}}{{$n}}{{end}}{{end}}</td>
|
|
||||||
<td style="text-align: right; padding: 0.25rem 0.5rem;">{{.DiskCount}}</td>
|
|
||||||
</tr>
|
|
||||||
{{end}}
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</details>
|
|
||||||
{{end}}
|
|
||||||
{{else}}
|
|
||||||
<p style="color: #facc15">No infra backup received yet</p>
|
|
||||||
{{end}}
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- Health -->
|
<!-- Health -->
|
||||||
<section class="card">
|
<section class="card">
|
||||||
@@ -489,7 +440,7 @@
|
|||||||
{{if eq .ConfigSyncStatus "in_sync"}}<span style="color: #22c55e;">✓ In sync</span>
|
{{if eq .ConfigSyncStatus "in_sync"}}<span style="color: #22c55e;">✓ In sync</span>
|
||||||
{{else if eq .ConfigSyncStatus "mismatch"}}<span style="color: #f59e0b;">⚠ Config mismatch — {{.ConfigDiffCount}} difference{{if gt .ConfigDiffCount 1}}s{{end}}</span>
|
{{else if eq .ConfigSyncStatus "mismatch"}}<span style="color: #f59e0b;">⚠ Config mismatch — {{.ConfigDiffCount}} difference{{if gt .ConfigDiffCount 1}}s{{end}}</span>
|
||||||
<button class="btn btn-outline btn-sm" style="margin-left: 0.5em; font-size: 0.8em;" onclick="showConfigDiff('{{.CustomerID}}')">Show Diff</button>
|
<button class="btn btn-outline btn-sm" style="margin-left: 0.5em; font-size: 0.8em;" onclick="showConfigDiff('{{.CustomerID}}')">Show Diff</button>
|
||||||
{{else}}<span style="color: #94a3b8;">Unknown — no infra backup available yet</span>
|
{{else}}<span style="color: #94a3b8;">Unknown — use "Show Diff" to compare live</span>
|
||||||
{{end}}
|
{{end}}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
+1
-1
@@ -117,7 +117,7 @@ spec:
|
|||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: hub
|
- name: hub
|
||||||
image: gitea.dooplex.hu/admin/felhom-hub:v0.11.0
|
image: gitea.dooplex.hu/admin/felhom-hub:v0.12.0
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8080
|
- containerPort: 8080
|
||||||
name: http
|
name: http
|
||||||
|
|||||||
Reference in New Issue
Block a user