2b (unreleased): the agent moves from a pinned Proxmox certificate to the box's root CA copy — copy, proof, then the config; the cause of a trust failure is named (R-1380)
gates / gates (push) Successful in 38s

This commit is contained in:
admin committed 2026-11-14 16:35:22 +01:00
1 parent a9ab47ec71
commit 386a0c4cf4
13 files changed
+1191 -6

No files matched your search

+21
View File
@@ -9,6 +9,27 @@ session's. `configs/` changes, so the release needs BOTH signed jobs.
name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust
file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real
runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`).
- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go`
at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every
call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy
ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets
`proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy
`agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust
(`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays.
**`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An
operator's own CA file, a box with checks switched off and a config that is not a file are never moved.
- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to
the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running
client.
- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the
served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no
copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report
(`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet**
(felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as
„host stale" on the hub.
- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST
wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`;
one mutation is not a rule and says so).
## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET
+1
View File
@@ -100,6 +100,7 @@
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+37
View File
@@ -54,6 +54,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
@@ -988,6 +989,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
osLeg := newOSLeg(cfg, client, px, logger)
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
desiredSyncer.AddConsumer(osLeg)
// R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root
// step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE
// write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and
// runs the same order again (a pinned box whose certificate changed moves right then, with no restart).
pveTrust := newPVETrust(cfg, px, osLeg, logger)
collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() })
loop.SetCollectErrorHook(pveTrust.OnCallError)
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -1018,6 +1026,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so
// a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer.
pveTrust.Ensure(ctx, "start")
go pveTrust.Run(ctx, time.Hour, 24*time.Hour)
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
@@ -1301,6 +1313,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
// it may wait for the heavy-op gate, and the signed job must not wait with it).
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
// R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again.
go pveTrust.Ensure(daemonCtx, "config bundle installed")
}}
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
@@ -3948,6 +3962,29 @@ func (f *selftestFlag) Set(v string) error {
return nil
}
// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client
// with EXACTLY the trust under test, and the running client's SetTLS.
func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager {
m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger,
EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) {
r, err := leg.PVECACopy(ctx)
return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err
},
Probe: func(ctx context.Context, t proxmox.TLSConfig) error {
p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t})
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: px.SetTLS,
}
m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint,
InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify})
return m
}
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
// the once-per-night marker live in the agent's own os/ dir.
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
// reported. The same for the profile refresh (R-1388), which had no such test.
func TestMainWiresPVETrust(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
seen := map[string]int{}
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
sel, ok := node.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
x, _ := sel.X.(*ast.Ident)
switch {
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
seen["ensure"]++
case sel.Sel.Name == "SetProxmoxTrustSource":
seen["stanza"]++
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
seen["hook"]++
}
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
seen["profile-refresh"]++
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
seen["timer"]++
}
}
case *ast.KeyValueExpr:
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
seen["apply=SetTLS"]++
}
}
}
return true
})
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2}
for k, n := range want {
if seen[k] < n {
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
}
}
}
+15
View File
@@ -124,6 +124,7 @@ type Collector struct {
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
@@ -209,6 +210,12 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
return c
}
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
c.pveTrust = f
return c
}
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
// disabled. Returns the collector for chaining.
@@ -391,6 +398,13 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
// error (no useful report — the cycle skips the POST); a failed per-guest
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
// a cloudflared probe failure yields status="unknown" and is never fatal.
func (c *Collector) proxmoxTrust() any {
if c.pveTrust == nil {
return nil
}
return c.pveTrust()
}
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
ns, err := c.px.NodeStatus(ctx)
if err != nil {
@@ -420,6 +434,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
ProxmoxTrust: c.proxmoxTrust(),
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
}
+9
View File
@@ -70,8 +70,14 @@ type Loop struct {
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
// collectErr (R-1380) sees every failed collect.
collectErr func(ctx context.Context, err error)
}
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
@@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
report, err := l.collector.Collect(ctx)
if err != nil {
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
if l.collectErr != nil {
l.collectErr(ctx, err)
}
return current
}
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
+5
View File
@@ -53,6 +53,11 @@ type HostReport struct {
// alert. Not a secret (the fp is public; the token is never reported).
LeafFingerprint string `json:"leaf_fingerprint"`
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
+3 -1
View File
@@ -139,7 +139,9 @@ type WrapperReport struct {
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
ProfileRefresh json.RawMessage `json:"profile_refresh"`
OutcomeHint string `json:"outcome_hint"`
// PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}.
PVECACopy json.RawMessage `json:"pve_ca_copy"`
OutcomeHint string `json:"outcome_hint"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
+45
View File
@@ -0,0 +1,45 @@
package osupdate
// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a).
// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan
// names nothing. internal/pvetrust decides when to ask and what to do with the answer.
import (
"context"
"encoding/json"
"fmt"
)
// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`.
const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem"
// PVECACopyResult is the wrapper's pve_ca_copy object.
type PVECACopyResult struct {
Outcome string `json:"outcome"` // written | same
Path string `json:"path"`
SHA256 string `json:"sha256"`
NotAfter string `json:"not_after"`
DaysLeft int `json:"days_left"`
Replaced bool `json:"replaced"`
}
// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that
// does not know the mode yet) or a wrapper that gave no answer.
func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) {
wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405"), map[string]any{"release_id": "pve-ca-copy",
"layer": LayerHost, "mode": "pve-ca-copy"})
if err != nil {
return PVECACopyResult{}, err
}
if wr.refused() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused)
}
if wr.failed() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed)
}
var res PVECACopyResult
if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr)
}
return res, nil
}
+40 -5
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"net/url"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
@@ -45,6 +46,39 @@ type Client struct {
node string
token string
http doer
// tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the
// box's own root CA, and a refreshed CA copy, take effect with no agent restart).
tlsMu sync.RWMutex
timeout time.Duration
}
func (c *Client) httpDoer() doer {
c.tlsMu.RLock()
defer c.tlsMu.RUnlock()
return c.http
}
// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an
// error (an unreadable CA file, a bad fingerprint) nothing changes.
func (c *Client) SetTLS(t TLSConfig) error {
tlsCfg, err := t.build()
if err != nil {
return err
}
timeout := c.timeout
if timeout == 0 {
timeout = 30 * time.Second
}
hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)}
c.tlsMu.Lock()
old := c.http
c.http = hc
c.tlsMu.Unlock()
if o, ok := old.(*http.Client); ok && o != nil {
o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call
}
return nil
}
// NewClient builds an API client. It validates required config and constructs the
@@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) {
Transport: httpx.NewTransport(tlsCfg, 0),
}
return &Client{
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
timeout: timeout,
}, nil
}
@@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader
req.Header.Set("Content-Type", contentType)
}
resp, err := c.http.Do(req)
resp, err := c.httpDoer().Do(req)
if err != nil {
return fmt.Errorf("proxmox: %s %s: %w", method, path, err)
}
+43
View File
@@ -0,0 +1,43 @@
package proxmox
import (
"crypto/x509"
"errors"
"strings"
)
// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is
// down") exists; these name the cause.
const (
TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more
TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate
TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended
TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate
TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls
)
// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout).
func TrustProblem(err error) string {
if err == nil {
return ""
}
var ua x509.UnknownAuthorityError
var inv x509.CertificateInvalidError
var hn x509.HostnameError
switch {
case strings.Contains(err.Error(), "TLS pin mismatch"):
return TrustPinnedCertChanged
case errors.As(err, &ua):
return TrustCAMismatch
case errors.As(err, &inv):
if inv.Reason == x509.Expired {
return TrustCertExpired
}
return TrustCAMismatch
case errors.As(err, &hn):
return TrustNameMismatch
case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"):
return TrustCAFileUnreadable
}
return ""
}
+421
View File
@@ -0,0 +1,421 @@
// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names
// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a).
//
// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then
// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own
// root CA verifies every certificate that CA signs.
//
// THE ORDER, and it is the whole point (a box must never be left with neither way working):
//
// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate);
// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust;
// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint;
// 4. the running client takes the new trust (no restart).
//
// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in
// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the
// fingerprint is cleared LAST, in the same write that sets `ca_file`.
//
// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is
// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move
// and the PBS seed write it. No root step touches it.
package pvetrust
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its
// root CA when less than two years are left, so a healthy box never comes near it.
const EndsSoonDays = 30
// Problems this package names beyond proxmox.Trust*.
const (
ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer
ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays
ProblemCopyEndsSoon = "ca_copy_ends_soon"
ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays
)
// Copy is what the root step answered.
type Copy struct {
Path string
Outcome string // written | same
NotAfter string
DaysLeft int
}
// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date.
type Status struct {
Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system
Problem string `json:"problem,omitempty"` // "" = none
Detail string `json:"detail,omitempty"`
CANotAfter string `json:"ca_not_after,omitempty"`
CADaysLeft *int `json:"ca_days_left,omitempty"`
MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy
CheckedAt string `json:"checked_at,omitempty"`
}
// Manager holds the trust state of one agent process.
type Manager struct {
// EnsureCopy runs the root step (osupdate.Leg.PVECACopy).
EnsureCopy func(ctx context.Context) (Copy, error)
// Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client).
Probe func(ctx context.Context, t proxmox.TLSConfig) error
// Apply gives the running client the new trust (proxmox.Client.SetTLS).
Apply func(t proxmox.TLSConfig) error
// ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved).
ConfigPath string
StateDir string
Logger *slog.Logger
Now func() time.Time
// MinGap between two runs that a failed call asked for (default 10 minutes).
MinGap time.Duration
mu sync.Mutex
cur proxmox.TLSConfig
status Status
lastRun time.Time
}
func (m *Manager) now() time.Time {
if m.Now != nil {
return m.Now()
}
return time.Now()
}
func (m *Manager) log() *slog.Logger {
if m.Logger != nil {
return m.Logger
}
return slog.Default()
}
// Start records the trust the process started with. Call once, before Ensure.
func (m *Manager) Start(t proxmox.TLSConfig) {
m.mu.Lock()
defer m.mu.Unlock()
m.cur = t
m.status = Status{Mode: modeOf(t)}
}
func modeOf(t proxmox.TLSConfig) string {
switch {
case t.InsecureSkipVerify:
return "insecure"
case t.Fingerprint != "":
return "fingerprint"
case t.CAFile != "":
return "ca"
}
return "system"
}
// Status is the current stanza (nil-safe).
func (m *Manager) Status() *Status {
if m == nil {
return nil
}
m.mu.Lock()
defer m.mu.Unlock()
s := m.status
return &s
}
// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move).
func (m *Manager) Moved() bool {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur.Fingerprint == "" && m.cur.CAFile != ""
}
// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not
// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy).
func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) {
for {
d := settled
if !m.Moved() {
d = pending
}
select {
case <-ctx.Done():
return
case <-time.After(d):
m.Ensure(ctx, "timer")
}
}
}
// Current is the trust the running client uses.
func (m *Manager) Current() proxmox.TLSConfig {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur
}
// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs
// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box
// whose CA copy was refreshed reloads it. A failure of another kind does nothing.
func (m *Manager) OnCallError(ctx context.Context, err error) {
cause := proxmox.TrustProblem(err)
if cause == "" {
return
}
m.mu.Lock()
gap := m.MinGap
if gap <= 0 {
gap = 10 * time.Minute
}
tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap
if m.status.Problem == "" {
m.status.Problem, m.status.Detail = cause, causeText(cause)
}
m.mu.Unlock()
m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err)
if tooSoon {
return
}
m.Ensure(ctx, cause)
}
func causeText(cause string) string {
switch cause {
case proxmox.TrustPinnedCertChanged:
return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)"
case proxmox.TrustCAMismatch:
return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)"
case proxmox.TrustCertExpired:
return "the certificate Proxmox serves, or its CA, has ended"
case proxmox.TrustCAFileUnreadable:
return "the CA copy is missing or holds no certificate"
case proxmox.TrustNameMismatch:
return "the certificate Proxmox serves does not name the address the agent calls"
}
return cause
}
// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns
// true when the running trust changed.
func (m *Manager) Ensure(ctx context.Context, why string) bool {
m.mu.Lock()
cur := m.cur
m.lastRun = m.now()
m.mu.Unlock()
lg := m.log().With("why", why, "mode", modeOf(cur))
set := func(mode, problem, detail string, c *Copy) {
m.mu.Lock()
defer m.mu.Unlock()
moved := m.status.MovedAt
m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)}
if c != nil {
d := c.DaysLeft
m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d
}
}
if cur.InsecureSkipVerify {
lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved")
set("insecure", "", "", nil)
return false
}
if m.EnsureCopy == nil || m.Probe == nil {
return false
}
// 1. the copy
c, err := m.EnsureCopy(ctx)
if err != nil {
lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err)
set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil)
return false
}
ca := proxmox.TLSConfig{CAFile: c.Path}
onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path
if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path {
// an operator's own CA file: never replaced
lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile)
set("ca_file_own", "", "", &c)
return false
}
if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired {
// the box already trusts by the copy and the copy did not change: nothing to prove again
problem, detail := "", ""
if c.DaysLeft < EndsSoonDays {
problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter)
lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft)
}
set("ca", problem, detail, &c)
return false
}
// 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust
pctx, cancel := context.WithTimeout(ctx, 30*time.Second)
perr := m.Probe(pctx, ca)
cancel()
if perr != nil {
cause := proxmox.TrustProblem(perr)
detail := "a call to Proxmox with the CA copy alone failed"
if cause != "" {
detail += ": " + causeText(cause)
} else {
detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")"
}
if onCopy {
lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr)
set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c)
return false
}
if cause == "" {
// Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no
// cause is named; the next run asks again.
lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr)
set(modeOf(cur), "", "", &c)
return false
}
lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr)
set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c)
return false
}
if onCopy {
// the copy was refreshed (or a trust failure asked): the running client reloads it
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c)
return false
}
}
lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome)
set("ca", "", "", &c)
return true
}
// 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof
if m.ConfigPath == "" {
lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is")
set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c)
return false
}
if werr := m.writeConfig(c.Path); werr != nil {
lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr)
set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c)
return false
}
// 4. the running client
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr)
}
}
m.mu.Lock()
m.cur = ca
m.mu.Unlock()
lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy",
"ca_file", c.Path, "ca_not_after", c.NotAfter)
set("ca", "", "", &c)
m.mu.Lock()
m.status.MovedAt = m.now().UTC().Format(time.RFC3339)
m.mu.Unlock()
return true
}
func firstNonEmpty(a, b string) string {
if a != "" {
return a
}
return b
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is
// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked.
func (m *Manager) writeConfig(caPath string) error {
raw, err := os.ReadFile(m.ConfigPath)
if err != nil {
return err
}
var doc map[string]json.RawMessage
if err := json.Unmarshal(raw, &doc); err != nil {
return fmt.Errorf("parse %s: %w", m.ConfigPath, err)
}
px := map[string]json.RawMessage{}
if cur, ok := doc["proxmox"]; ok {
if err := json.Unmarshal(cur, &px); err != nil {
return fmt.Errorf("parse the proxmox section: %w", err)
}
}
tlsSec := map[string]json.RawMessage{}
if cur, ok := px["tls"]; ok {
if err := json.Unmarshal(cur, &tlsSec); err != nil {
return fmt.Errorf("parse the proxmox.tls section: %w", err)
}
}
caJSON, _ := json.Marshal(caPath)
tlsSec["ca_file"] = caJSON
tlsSec["fingerprint"] = json.RawMessage(`""`)
b, _ := json.Marshal(tlsSec)
px["tls"] = b
b, _ = json.Marshal(px)
doc["proxmox"] = b
out, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return err
}
out = append(out, '\n')
st, err := os.Stat(m.ConfigPath)
if err != nil {
return err
}
if m.StateDir != "" {
if err := os.MkdirAll(m.StateDir, 0o700); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil {
return fmt.Errorf("parking the recovery copy: %w", err)
}
}
// In place (O_TRUNC), not tmp+rename: the folder is root-owned and the file is the agent's (the same reason as the
// backup-target move and the PBS seed). One small write, then fsync.
f, err := os.OpenFile(m.ConfigPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
_ = os.WriteFile(m.ConfigPath, raw, st.Mode().Perm()) // put the old bytes back: the fingerprint must not be lost
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
// read back: the file must say exactly what was meant
var check struct {
Proxmox struct {
TLS struct {
CAFile string `json:"ca_file"`
Fingerprint string `json:"fingerprint"`
} `json:"tls"`
} `json:"proxmox"`
}
back, err := os.ReadFile(m.ConfigPath)
if err != nil || json.Unmarshal(back, &check) != nil || check.Proxmox.TLS.CAFile != caPath || check.Proxmox.TLS.Fingerprint != "" {
_ = os.WriteFile(m.ConfigPath, raw, st.Mode().Perm())
return fmt.Errorf("the config did not read back as written (%v) — the old config was put back", err)
}
return nil
}
+489
View File
@@ -0,0 +1,489 @@
package pvetrust
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ----
type pki struct {
t *testing.T
caKey *ecdsa.PrivateKey
caCert *x509.Certificate
caPEM []byte
}
func newPKI(t *testing.T, cn string) *pki {
t.Helper()
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
return (&pki{t: t, caKey: k}).renewCA(cn)
}
// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed).
func (p *pki) renewCA(cn string) *pki {
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign}
der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
p.caCert, _ = x509.ParseCertificate(der)
p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
return p
}
func (p *pki) leaf() tls.Certificate {
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k}
}
func fp(c tls.Certificate) string {
s := sha256.Sum256(c.Certificate[0])
return hex.EncodeToString(s[:])
}
type pve struct {
srv *httptest.Server
mu sync.Mutex
cert tls.Certificate
}
func newPVE(t *testing.T, c tls.Certificate) *pve {
p := &pve{cert: c}
p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`)
}))
// GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no
// server name) that one would win over GetCertificate.
p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
p.mu.Lock()
defer p.mu.Unlock()
return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil
}}
p.srv.StartTLS()
t.Cleanup(p.srv.Close)
return p
}
// newCert is `pveupdate` making a new node certificate: served from the next connection on.
func (p *pve) newCert(c tls.Certificate) {
p.mu.Lock()
p.cert = c
p.mu.Unlock()
p.srv.CloseClientConnections()
}
// ---- the agent's side ----
type box struct {
t *testing.T
m *Manager
px *proxmox.Client
cfgPath string
copyPath string
source []byte // what the root step would copy (nil = the root step refuses)
events []string
applyErr error
}
func cfgJSON(fingerprint, caFile string) string {
return fmt.Sprintf(`{
"hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"},
"proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]},
"unknown_key": {"stays": true}
}`, fingerprint, caFile)
}
func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box {
t.Helper()
dir := t.TempDir()
b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source}
if start.CAFile == "COPY" {
start.CAFile = b.copyPath
if err := os.WriteFile(b.copyPath, source, 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil {
t.Fatal(err)
}
pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second}
px, err := proxmox.NewClient(pcfg)
if err != nil {
t.Fatal(err)
}
b.px = px
b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"),
EnsureCopy: func(context.Context) (Copy, error) {
b.events = append(b.events, "copy")
if b.source == nil {
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`)
}
outcome := "same"
if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) {
outcome = "written"
if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil {
t.Fatal(err)
}
}
return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil
},
Probe: func(ctx context.Context, tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint))
c := pcfg
c.TLS = tl
p, err := proxmox.NewClient(c)
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: func(tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != ""))
if b.applyErr != nil {
return b.applyErr
}
return b.px.SetTLS(tl)
},
}
b.m.Start(start)
return b
}
type fileTLS struct {
Fingerprint string `json:"fingerprint"`
CAFile string `json:"ca_file"`
}
func (b *box) fileTLS() fileTLS {
var d struct {
Proxmox struct {
TLS fileTLS `json:"tls"`
} `json:"proxmox"`
}
raw, _ := os.ReadFile(b.cfgPath)
if err := json.Unmarshal(raw, &d); err != nil {
b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw)
}
return d.Proxmox.TLS
}
func (b *box) call() error {
_, err := b.px.Version(context.Background())
return err
}
func (b *box) has(prefix string) bool {
for _, e := range b.events {
if strings.HasPrefix(e, prefix) {
return true
}
}
return false
}
// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written.
func TestNewBox_CAOnly_NothingToMove(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("a box that already trusts by the copy changed its trust")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) || b.has("probe") || b.has("apply") {
t.Fatalf("the config was touched or a proof ran: %v", b.events)
}
if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the call fails: %v", err)
}
}
// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only
// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config
// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails.
func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
old, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "start") {
t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events)
}
want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"}
if strings.Join(b.events, " | ") != strings.Join(want, " | ") {
t.Fatalf("order = %v\nwant %v", b.events, want)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
raw, _ := os.ReadFile(b.cfgPath)
for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} {
if !strings.Contains(string(raw), keep) {
t.Fatalf("the write lost %s:\n%s", keep, raw)
}
}
if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) {
t.Fatal("no recovery copy of the old config was parked")
}
if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" {
t.Fatalf("status = %+v", st)
}
// and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client
srv.newCert(p.leaf())
if err := b.call(); err != nil {
t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err)
}
}
// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in
// the running client — and the cause is named.
func TestProofFails_TheFingerprintStays(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("the trust changed although the proof failed")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) {
t.Fatal("the config was written although the proof failed")
}
if b.has("apply") {
t.Fatalf("the running client was changed: %v", b.events)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the fingerprint no longer works: %v", err)
}
}
// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the
// box moves to the CA copy right then, and the next call works — no restart.
func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
srv.newCert(p.leaf())
err := b.call()
if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged {
t.Fatalf("fixture: want a pin mismatch, got %v", err)
}
b.m.OnCallError(context.Background(), err)
if err := b.call(); err != nil {
t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
// a failure that is NOT one of trust does nothing
n := len(b.events)
b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused"))
if len(b.events) != n {
t.Fatalf("a refused connection started a trust run: %v", b.events[n:])
}
}
// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client
// reloads it; the config is not written again.
func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
oldCA := string(p.caPEM)
p.renewCA("PVE CA")
srv.newCert(p.leaf())
b.source = p.caPEM
before, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "daily") {
t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events)
}
if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) {
t.Fatal("the copy was not refreshed")
}
if !b.has("probe") || !b.has("apply") {
t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written again")
}
if err := b.call(); err != nil {
t.Fatalf("the call fails after the CA's renewal: %v", err)
}
}
// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named.
func TestRootStepRefuses_NothingChanges(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") {
t.Fatalf("something ran without a copy: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") {
t.Fatalf("status = %+v", st)
}
}
// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client.
func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root writes a read-only file")
}
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
if err := os.Chmod(b.cfgPath, 0o400); err != nil {
t.Fatal(err)
}
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("the trust changed without a saved config: %v", b.events)
}
if got := b.fileTLS(); got.Fingerprint != fp(leaf) {
t.Fatalf("the fingerprint is gone from the file: %+v", got)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved {
t.Fatalf("status = %+v", st)
}
}
// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file.
func TestNeverMoved(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
own := filepath.Join(t.TempDir(), "own-ca.pem")
os.WriteFile(own, p.caPEM, 0o644)
for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} {
b := newBox(t, srv, start, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("%s: moved (%v)", name, b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatalf("%s: the config was written", name)
}
}
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.ConfigPath = ""
if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved {
t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status())
}
}
// The copy ends soon: the cause is named (the box still works).
func TestCopyEndsSoon(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
inner := b.m.EnsureCopy
b.m.EnsureCopy = func(ctx context.Context) (Copy, error) {
c, err := inner(ctx)
c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z"
return c, err
}
b.m.Ensure(context.Background(), "daily")
if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") {
t.Fatalf("status = %+v", st)
}
}
// A trust failure asks at most once per MinGap.
func TestOnCallError_Throttled(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
srv.newCert(p.leaf())
err := b.call()
now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC)
b.m.Now = func() time.Time { return now }
b.m.OnCallError(context.Background(), err)
n := len(b.events)
b.m.OnCallError(context.Background(), err)
if len(b.events) != n {
t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:])
}
now = now.Add(11 * time.Minute)
b.m.OnCallError(context.Background(), err)
if len(b.events) == n {
t.Fatal("no new run after the gap")
}
if st := b.m.Status(); st.Problem == "" {
t.Fatalf("a blind box names no cause: %+v", st)
}
}
// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written,
// and the box counts as not moved, so the timer asks again soon.
func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.Probe = func(context.Context, proxmox.TLSConfig) error {
return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")
}
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("moved without a proof")
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() {
t.Fatalf("status = %+v moved=%v", st, b.m.Moved())
}
}