slice 9: GET /host/metrics + CPU/chassis-temp collector (v0.14.0)
Add a host-wide, token-authed GET /host/metrics local-API endpoint that re-serves the slice-4 collector's host + per-storage view to the customer (the de-privileged controller can't read the host itself). Add the one new collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful- null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c too. Cross-repo host-report golden updated byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,53 +1,53 @@
|
||||
# REPORT — slice 8B.2 (agent half): quiesce downtime optimization — `snapshotted` phase (v0.13.0) (2026-06-10)
|
||||
# REPORT — slice 9 (agent half): host metrics to the controller (v0.14.0) (2026-06-10)
|
||||
|
||||
> Overwrite-latest report. Cumulative history: [CHANGELOG.md](CHANGELOG.md). Implements the agent
|
||||
> half of `TASK — Slice 8B.2`. The controller early-resume is `felhom-controller` v0.38.0. No hub change.
|
||||
> Overwrite-latest report. Cumulative history: [CHANGELOG.md](CHANGELOG.md).
|
||||
|
||||
## Outcome
|
||||
## What was implemented
|
||||
|
||||
In snapshot mode an LXC vzdump only needs the app-stopped state captured at the **storage-snapshot
|
||||
moment**; after that it reads from the snapshot and the app can safely resume. The agent now emits a
|
||||
**`snapshotted`** phase on `GET /backup/status` when the snapshot is taken, so the controller resumes
|
||||
its app early — app downtime drops from *whole-backup* to *until-snapshot* with no loss of
|
||||
app-consistency.
|
||||
The agent half of **slice 9** — re-serving the host's health to the customer's controller, plus the
|
||||
one new collector (CPU/chassis temperature). The de-privileged controller (slice 8C) sees only its
|
||||
own cgroup, so it cannot read host metrics itself; the agent already collects host CPU/mem/loadavg/
|
||||
uptime + per-storage targets for the hub, and slice 9 exposes that to the customer over the local API.
|
||||
|
||||
## Phase 0 (validated first, on the demo — PVE 9.2.2)
|
||||
### `internal/hub/cputemp.go` — CPU/chassis-temp collector (new)
|
||||
- `TempReader` seam + `SysfsTempReader`: reads the CPU package temperature from **sysfs** — hwmon
|
||||
(`coretemp`/`k10temp`/`zenpower`/`cpu_thermal`, preferring the `Package id 0` input) then thermal
|
||||
zones (preferring `x86_pkg_temp`/`coretemp`/`cpu-thermal`, falling back to `acpitz`).
|
||||
- **No external binary, no privilege** (sysfs is world-readable) → the root-CLI fence is untouched.
|
||||
- **Graceful-null**: a missing sensor, an unsupported board, an implausible reading (outside
|
||||
5–150 °C), or any read error all degrade to `nil` ("n/a") — never fails the report. Same nullable
|
||||
contract as the per-disk `SmartSummary.TemperatureC`.
|
||||
|
||||
- **Snapshot mode is in effect** on local-lvm (lvmthin) — `vzdump --mode snapshot` logs `backup mode:
|
||||
snapshot`, not downgraded to stop.
|
||||
- **Marker:** `INFO: create storage snapshot 'vzdump'` — the vzdump-level step after which the backup
|
||||
reads from the snapshot (generic across snapshot-capable storage; only appears in snapshot mode).
|
||||
- **Downtime delta:** snapshot created in the first ~1s; archive runs ~23s after. So 8B (resume at
|
||||
`done`) = ~24s app-down; 8B.2 (resume at `snapshotted`) = ~1s — **~95% cut** for a 934 MB guest;
|
||||
the delta grows with guest size.
|
||||
### `internal/hub` — shared wire field + collector reuse
|
||||
- `HostMetrics` gains **`CPUTempC *int` (`cpu_temp_c`)** — nullable, on the **shared** struct, so the
|
||||
**hub report carries it too** (operator freebie). Cross-repo host-report golden updated
|
||||
**byte-identical** with the hub's copy.
|
||||
- `Collector` gains a nil-safe `temp TempReader` (defaults to the real `SysfsTempReader`;
|
||||
`SetTempReader` injects a fake in tests). `Collect()` now sets `cpu_temp_c` on the report.
|
||||
- **`Collector.HostMetricsNow(ctx)`** — a fresh `NodeStatus` + CPU-temp read returning just the host
|
||||
block; the source for the local API (current cpu%/temp, not the 15-min hub snapshot).
|
||||
|
||||
## What landed
|
||||
### `internal/localapi` — `GET /host/metrics` (new endpoint)
|
||||
- `host_metrics.go`: host-wide health (cpu%/mem/load/uptime/`cpu_temp_c`) + per-storage capacity
|
||||
(total/used/fraction, thin-pool, SMART temp+wear). Token-authed via `withGuest` (host-wide data; a
|
||||
cross-guest `?vmid=` still 403). Best-effort on storage (a view error still returns the host
|
||||
block). Served only when the `HostMetrics` provider (the shared collector) is wired in
|
||||
`buildLocalAPIServer` — else 503 "not configured".
|
||||
|
||||
- **`BackupRunner.BackupWithSnapshotHook(ctx, vmid, onSnapshot)`** — while the vzdump runs, a watcher
|
||||
tails the task log (`TaskLogTail`) for the `create storage snapshot` marker and fires `onSnapshot`
|
||||
**once**. It bails on `backup mode: stop` and the marker never appears in stop mode, so it **never
|
||||
fires in stop/downgraded mode**. `Backup` keeps its signature (scheduler/selftest); both share one
|
||||
body. `snapshotWatchInterval` is a package var (prod 1s; tests shrink it).
|
||||
- **`/backup/status` phase `snapshotted`** (between `running` and `done`): `handleBackup` passes the
|
||||
hook → `markSnapshotted` flips the running job. `done`/`failed` unchanged; `snapshotted` is additive.
|
||||
## Tests (all green)
|
||||
- `cputemp_test.go`, `hostmetrics_test.go`, `host_metrics_test.go`: hwmon/thermal-zone selection +
|
||||
**graceful-null**, `HostMetricsNow` populate/null/hard-error, endpoint populated + `cpu_temp_c:null`
|
||||
serialization + **401 without a token** + 403 cross-guest + 503 not-configured.
|
||||
- `go test ./...` green; `go vet ./internal/hub ./internal/localapi` clean.
|
||||
|
||||
## Tests
|
||||
## Versioning / docs
|
||||
- Version `0.13.0 → 0.14.0`; `CHANGELOG.md` updated. Doc 03 §6 (local-API surface) + §9 (roadmap +
|
||||
changelog) updated.
|
||||
|
||||
`go test ./...` green; `-race` green (build server). localapi: snapshot mode → `snapshotted` before
|
||||
`done`; stop mode → `snapshotted` never emitted. runner: the watcher fires on the marker; stop-mode
|
||||
log never fires.
|
||||
## Assumption (noted, not built)
|
||||
- **One customer per host** (home-server model): `/host/metrics` is host-wide. A multi-customer host
|
||||
would leak cross-customer CPU/mem → revisit then.
|
||||
|
||||
## Live validation (demo-felhom)
|
||||
|
||||
End-to-end on a provisioned guest (controller v0.38.0 + a postgres stack): the agent logged `backup
|
||||
reached snapshotted (app may resume)` mid-backup, the controller resumed the app at that point, and
|
||||
the backup proceeded to `done`. **App downtime ~3s** (quiesce→snapshotted) vs **~23s** if it had
|
||||
waited for `done` (~87% cut). The snapshot backup restored **clean** (`database system was shut down`,
|
||||
no WAL replay) — the early resume did not compromise app-consistency. Deployed to the demo service as
|
||||
v0.13.0.
|
||||
|
||||
## Deferred / dependency
|
||||
|
||||
Snapshot-capable storage (lvm-thin/ZFS) is required for the win; on stop/downgraded storage the
|
||||
controller falls back to resume-at-`done` (8B). No consistency-contract or crash-safety change. No
|
||||
secrets committed.
|
||||
## Pending
|
||||
- **Live validation** on the demo (build + deploy agent v0.14.0; controller monitoring page → real
|
||||
N100 CPU%/temp + per-storage, cross-checked vs `pvesh`/`free`/`df`).
|
||||
|
||||
Reference in New Issue
Block a user