84 lines
5.5 KiB
Markdown
84 lines
5.5 KiB
Markdown
# felhom.eu — task reports
|
|
|
|
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
|
|
|
|
---
|
|
|
|
# Hub v0.12.0 — retire Infra Backup, purge plaintext secrets, fix backup-deadline email
|
|
|
|
**Date:** 2026-06-16
|
|
**Scope:** Phase-1 of `documentation/audits/SPIKE-infra-backup-2026-06-15.md` (hub side).
|
|
**Deployed:** `felhom-hub:v0.12.0` live on k3s (ArgoCD `felhom` app Synced); commit `0635640`.
|
|
**Companion repo:** `felhom-controller` v0.69.0 (own REPORT there).
|
|
|
|
## What shipped
|
|
|
|
1. **Backup-deadline check repointed to PBS freshness** (`internal/monitor/deadline.go`).
|
|
The backup half no longer queries for a `backup_completed` event (nothing emits it post-slice-8C,
|
|
so it fired daily for every healthy customer). It now reads the customer's **latest agent
|
|
host-report** (`store.GetLatestHostReportJSON`) and raises `expected_backup_missed` only on
|
|
positive evidence: no PBS snapshot / successful vzdump at all, newest backup older than **26h**, or
|
|
the newest PBS snapshot's `verify_state == "failed"`. A fresh-but-unverified snapshot is **not** a
|
|
failure (PBS verifies on its own cadence — alarming on it would recreate the false alarm). The
|
|
**db-dump half is unchanged**. A customer with **no host-report** gets no backup alarm here
|
|
(liveness is the host-staleness checker's job).
|
|
|
|
2. **Infra Backup feature removed** (`api/handler.go`, `store/store.go`, `web/configs.go`,
|
|
`templates/customer_unified.html`, `templates/customer.html`): the `POST/GET /api/v1/infra-backup[…]`
|
|
endpoints + handlers, the store methods/types (`SaveInfraBackup`/`GetInfraBackup`/`GetInfraBackupByID`/
|
|
`GetInfraBackupMeta`/`ListInfraBackupVersions`/`pruneInfraBackups`, `InfraBackupMeta`/`InfraBackupVersion`),
|
|
and the operator "Infra Backup" panel. `GET /api/v1/recovery/{id}` now returns **config_yaml only**.
|
|
The customer-page config-drift badge (diffed against the stored controller.yaml) is hidden; the live
|
|
"Show Diff" path is unaffected.
|
|
|
|
3. **Plaintext secret purge** (`store/store.go migrate()`): `DROP infra_backup_versions; DROP
|
|
infra_backups; VACUUM; wal_checkpoint(TRUNCATE)` — gated on table existence so normal restarts skip
|
|
it. VACUUM physically reclaims the freed pages so the plaintext keys/tokens are not merely delinked.
|
|
|
|
## Gate (STEP 1a) — PASSED before any change
|
|
|
|
The demo's latest host-report (id 699, agent v0.36.7) carried **5 PBS snapshots, all `verify=ok`,
|
|
newest `2026-06-15T18:41:29Z` (13.9 h old, < 26 h)** plus a matching successful vzdump. The repoint's
|
|
data source is present and fresh, so the repoint alone clears the email.
|
|
|
|
## Verification (live, non-hollow)
|
|
|
|
- **Build + tests:** `go build ./... && go test ./...` green. `deadline_test.go` covers fresh+verified→
|
|
quiet (the **companion**), stale→alarm, failed-verify→alarm, no-host-report→quiet, db-dump half
|
|
preserved, and a pure `assessBackupFreshness` table. Red-proof: with the old event-based half
|
|
temporarily restored, the companion + no-report + db-dump-preserved tests **fail**, while the
|
|
pure-helper test still passes — the behavioral tests are sensitive to the logic.
|
|
- **Deploy:** hub v0.12.0 rolled out; startup log: `Retired infra-backup: dropped 2 table(s) and
|
|
VACUUMed to reclaim plaintext pages`.
|
|
- **Tables gone:** `.tables` on the live `/data/hub.db` shows no `infra_backup*`. DB shrank
|
|
**65.8 MB → 52.2 MB**; no `-wal` sidecar lingers.
|
|
- **Secret purge proof (grep on the live post-VACUUM DB file):** the infra-backup-exclusive markers
|
|
are gone — `encryption_key_b64` 17→**0**, `controller_config_b64` 11→**0**, and the actual
|
|
AES-key value from spike record 109 present→**0** (physically reclaimed, not just delinked). The
|
|
residual `cf_api_token`/`cf_tunnel_token` hits are in `customer_configs.config_json` (the hub's
|
|
legitimate config store), not the infra-backup blobs (which held CF tokens base64-encoded inside the
|
|
now-zero `controller_config_b64`).
|
|
- **Endpoints retired:** `POST /api/v1/infra-backup` and `GET …/versions` return **404**; control
|
|
`POST /api/v1/report` still returns **401** (routed, unauth) — confirming the 404s are route removal,
|
|
not a blanket failure.
|
|
- **Demo email fix:** the next 03:00 deadline run cannot be observed within this session, but the live
|
|
demo host-report (fresh verified PBS, 13.9 h) exercises the not-missed path, and demo emits
|
|
`db_dump_completed` daily (never missed) — so `expected_backup_missed` will no longer fire.
|
|
|
|
## Flagged for the operator (out of scope here)
|
|
|
|
- **Rotate the exposed credentials** that were in the dropped blobs (Cloudflare API + tunnel tokens
|
|
for `demo-felhom.eu` first; hub/session secrets if shared). They remain valid until rotated; the
|
|
purge removes the at-rest copy but not their validity.
|
|
- **Separate historical leak:** the legacy `reports` table holds thousands of rows with a plaintext
|
|
`restic_password` value from old controller versions. The **live** controller no longer sends it
|
|
(removed in controller v0.69.0), but the historical rows persist — a distinct purge/rotation
|
|
decision, deliberately not done here.
|
|
- **`peti-felhom` is a defunct customer marked `active`** (no host-report or controller report since
|
|
Feb 2026). Its daily `expected_backup_missed` stops with this change (no host-report → no backup
|
|
alarm), but it should be marked inactive to silence all residual noise.
|
|
|
|
## Out of scope (untouched)
|
|
|
|
Credential rotation; the Komga healthcheck; the secret-free DR "recipe" (the later DR slice).
|